The Emotion Gap Behind 90.7% Voice Accuracy

TL;DR for operators

A customer says everything is fine, but sounds frustrated. Can a voice AI reliably identify the emotion simply because it receives both the words and the audio? On eNTERFACE, GPT-audio reaches 90.7% overall accuracy with text-plus-audio input, yet scores 94.4% when emotional cues agree versus 85.1% when they conflict—a 9.3-percentage-point gap (Section V-A, Table II).

Fengji Jing and Guy Laban¹ find that greater disagreement between spoken words and vocal delivery is associated with higher GPT-audio classification-error odds across three speech corpora. After statistical controls, each standard-deviation increase in measured disagreement corresponds to 15%–25% higher error odds under joint text-and-audio inference. These are associations, not proof that multimodal integration causes the errors; text-only results, model differences, and evaluation sensitivities complicate that explanation.

For teams evaluating voice agents or customer-support emotion analytics, high aggregate accuracy is not evidence of reliability when emotional signals conflict. Test performance across disagreement levels, compare modalities within the same model, report invalid outputs, and check measurement robustness. The study supplies a reproducible evaluation method, not evidence that deployment-time conflict flagging improves outcomes.

The accuracy gap appears inside a strong result

A customer tells a support assistant that everything is fine, but the delivery sounds frustrated. The words indicate satisfaction; the voice suggests otherwise. A system receiving both the transcript and audio has access to more information, but that does not guarantee a more reliable emotional interpretation.

The problem becomes visible in a seemingly strong benchmark result.

On eNTERFACE'05, GPT-audio using a rule-guided prompt and text-plus-audio input achieves 90.7% overall emotion-classification accuracy. Its accuracy is 94.4% on emotionally congruent utterances and 85.1% on incongruent ones—a 9.3-percentage-point difference (Section V-A, Table II).

That difference is descriptive. The two groups also differ in their distribution of emotion labels, so the gap alone cannot establish that conflicting signals caused the additional errors.

Nevertheless, it exposes a procurement and quality-assurance risk. A team selecting an emotion-recognition component based on aggregate accuracy could overlook the performance of a subset relevant to its intended use: customers whose words and vocal delivery disagree.

The study investigates whether that gap persists after accounting for observable differences between utterances.

Measuring disagreement before measuring model failure

The researchers represent lexical and acoustic emotion in a shared three-dimensional space: valence (pleasantness), arousal (activation), and dominance (perceived control).

Text-derived VAD values come from the NRC VAD Lexicon, averaging affective ratings for matched words. Audio-derived VAD values come from a wav2vec2-large regression model trained for acoustic emotion estimation.

For each utterance, the distance between these vectors measures the extent to which the two representations disagree.

A two-component Gaussian mixture model is fitted to training-split distances. The posterior responsibility of the higher-distance component becomes the continuous incongruence score, $p_{\mathrm{incon}}$, ranging from zero to one.

The score does not require a gold emotion label for each utterance, although it depends on pretrained affect extractors and lexical resources. A threshold of 0.5 supports descriptive grouping, while the main statistical tests retain the continuous measure.

Importantly, $p_{\mathrm{incon}}$ is not a calibrated probability that a human would perceive emotional conflict. It describes membership in a fitted high-distance component and preserves the ordering produced by the underlying Euclidean distance.

The measure has independent perceptual support. Across eNTERFACE, MELD, and CMU-MOSEI, correlations with human incongruence judgments are 0.65, 0.58, and 0.62, respectively (Section VI). Inter-rater agreement is only moderate, however, indicating that human identification of emotional disagreement is itself imperfect.

The controlled association survives across three corpora

The primary evidence comes from GPT-audio evaluations on three different forms of speech: acted emotional utterances in eNTERFACE, conversational dialogue in MELD, and spontaneous opinion monologues in MOSEI.

The researchers fit pooled logistic regressions controlling for emotion class, prompt condition, transcript length, audio duration, and distances to text and audio emotion prototypes. They account for repeated observations through speaker- or utterance-clustered standard errors and adjust significance levels for multiple comparisons.

The resulting estimates are:

Source: Section V-G, Table VII. OR denotes odds ratio; SD denotes one standard deviation of the standardized incongruence predictor. Confidence intervals are 95%.

For eNTERFACE, an odds ratio of 1.25 means that a one-standard-deviation increase in incongruence corresponds to 25% higher adjusted odds of classification error. It does not mean a 25-percentage-point increase in error probability.

The positive text-plus-audio associations across all three corpora provide evidence that measured emotional disagreement identifies a reliability problem in GPT-audio.

But the text-only results complicate the explanation.

On MOSEI, the text-only association is approximately as large as the text-plus-audio association. A model receiving only transcripts cannot experience conflicting acoustic evidence during inference. Its association with the incongruence score must therefore reflect something else, potentially including characteristics of the transcripts themselves.

The relevant distinction is between disagreement present in the stimulus, difficulty arising from the transcript, and a failure to integrate multimodal evidence. The study establishes an association with errors; it does not isolate the third mechanism as their cause.

The disagreement is not inherently unusable information

A comparison with conventional supervised classifiers complicates the integration-failure interpretation further.

On eNTERFACE, logistic regression, linear SVM, and multilayer perceptron classifiers trained on the same six text-and-audio VAD features achieve accuracies of 85%, 89%, and 96%, respectively. These supervised classifiers perform better on incongruent utterances, reversing the pattern observed for several prompted audio-language-model conditions (Section V-C, Figure 3).

This is a comparison between different learning and inference approaches, not a controlled test of one model architecture. The classifiers receive structured affective features and task-specific supervision, while the audio LLMs interpret raw multimodal inputs through prompts.

The result establishes that emotional disagreement need not make the underlying features uninformative. It remains consistent with an integration difficulty in some prompted models, without directly revealing their internal fusion mechanisms.

A separate text-reader ablation demonstrates another evaluation hazard. When the same eNTERFACE transcripts are processed by gpt-5.2, the adjusted incongruence-error odds ratio is 1.12 [0.91, 1.37]. Using gpt-audio-1.5 instead yields 0.78 [0.63, 0.97], despite similar overall accuracies of 83.2% and 81.8% (Section V-H, Table V).

This ablation isolates a reader-model difference; its text-only regression is not identical to the primary pooled specification. It shows that changing the model used for a transcript baseline can materially alter the apparent relationship between incongruence and errors.

Cross-model comparisons also limit generalization. On MELD, Qwen2.5-Omni and GLM-4-Voice have positive text-plus-audio estimates, but their confidence intervals include one. On MOSEI, neither model reproduces the significant positive association found for GPT-audio. These extensions use matched stratified samples rather than the full-corpus GPT-audio evaluations (Section V-I, Table VI).

The evidence supports a GPT-audio finding across three corpora, not a universal weakness of audio-capable LLMs.

A conflict-aware evaluation protocol for voice systems

For teams considering emotion predictions in customer-support routing, escalation, or service analytics, the research supports a more discriminating evaluation process.

1. Stratify performance by emotional disagreement. Retain continuous incongruence scores and report accuracy across conflict levels, not just aggregate accuracy. Inspect results within emotion classes to avoid mistaking changes in class composition for a uniform reliability effect.

2. Hold the model constant in modality comparisons. Use the same audio-capable model, matched utterances, and consistent prompting when comparing transcript-only with text-plus-audio inference. Report whether adding audio changes accuracy, not simply whether it changes predictions.

3. Treat invalid outputs as part of system reliability. Report valid-response-only performance alongside results that count invalid outputs as failures. Recheck conflict associations using alternative affect extractors, particularly before selecting a fixed operational threshold.

4. Validate interventions independently. High-incongruence speech could trigger an uncertainty flag, alternative processing, or a more conservative downstream decision. Each policy should be tested against classification quality, intervention frequency, and operating cost on representative traffic.

These are Cognaptus operational inferences from the evaluation evidence, not interventions whose benefits were demonstrated in the paper.

The researchers also release an incongruence-aware evaluation repository containing VAD features, scores, predictions, validity flags, prompts, and analysis code. That artifact can support replication and comparative testing, although the original speech datasets are distributed separately.

Where the evidence remains fragile

Three boundaries are especially relevant before converting this diagnostic into a deployment policy.

First, measurement depends on the affect extractors. An alternative acoustic VAD model produces weaker or reversed associations in some settings. The extractor sensitivity analysis uses a reduced regression specification (Section V-K, Table X), so its coefficients should not be compared directly with the main full-control estimates. The GMM score is the only evaluated label-free measure with significant positive text-plus-audio associations across all three corpora, but it is not the strongest predictor in every corpus.

Second, dataset coverage and response validity are uneven. About 20.5% of MELD test utterances lack defined lexical VAD, disproportionately excluding short conversational backchannels. MELD labels also depend on dialogue context that the main independent-utterance evaluations do not provide. MOSEI simplifies multilabel emotion intensities into single-label targets, and roughly 14.6% of its text-plus-audio GPT-audio responses are invalid. Counting invalid responses as errors removes the significance of its reported incongruence association in the sensitivity specification.

Third, the relationship varies by emotion and model family. Within-emotion comparisons identify harmful associations for some labels and protective associations for others. The evaluated datasets are English-language corpora, and no randomized manipulation of emotional incongruence identifies a causal effect.

These boundaries prevent a single numerical incongruence threshold from being treated as a validated indicator of unsafe or incorrect emotional predictions.

The next benchmark needs to test disagreement explicitly

The central contribution is an evaluation capability: a continuous measure of text-audio emotional conflict, linked to model performance and available for conflict-stratified testing.

For GPT-audio, higher incongruence predicts greater adjusted error odds under joint text-and-audio inference across three distinct speech corpora. The result is meaningful even without proving the underlying integration mechanism, because it identifies a measurable condition associated with less reliable classification.

For an organization deploying voice emotion recognition, the appropriate next step is to establish whether this relationship appears in its own model, speech domain, and decision workflow. Only then can a conflict-aware intervention be tested for measurable benefit.

A system’s ability to process both words and voice establishes access to two signals. Reliable emotional interpretation requires evidence about what happens when those signals disagree.

Cognaptus: Automate the Present, Incubate the Future.