Reading Between the Generated Lines: What AI Emotion Scores Really Measure

TL;DR for operators

A conversational AI assistant may correctly identify a character’s emotions in a benchmark while remaining unreliable when a customer expresses frustration or concern indirectly.

In ToMStories, the strongest open-weight model achieves 88.4% multiple-choice accuracy, above a small human baseline of 80.3%. Yet the highest model score on open-ended questions is 58.6%, below the human baseline of 64.2%.

A more revealing result concerns the source of benchmark success. Models whose behavioral descriptions resemble those used by the synthetic-data generator tend to achieve substantially higher accuracy. This association persists after accounting for model size and preference for descriptive evidence, but it does not establish a causal shortcut.

For organizations selecting customer-facing AI, the appropriate response is to supplement synthetic benchmark rankings with independently authored narratives, open-ended interpretations, and cases where a person’s words contradict their behavior. The study supports better evaluation design, not claims that models can reliably infer real people’s psychological states.

A customer may say one thing and signal another

Suppose a customer repeatedly asks about cancellation terms while assuring a support assistant that everything is fine. The words provide one signal; the surrounding interaction suggests another. An effective assistant needs to recognize the uncertainty without assuming it knows the customer’s feelings.

For a company choosing a conversational AI model, this creates a difficult evaluation question. Does a high score on psychological-understanding tests indicate that the model can interpret such interactions, or merely that it recognizes familiar descriptions of emotions and motivations?

Melanie Subbiah, Chia-En Jacklyn Tsai, and Kathleen McKeown investigate this question in ToMStories: Interpreting Theory-of-Mind in Synthetic Narratives.¹ Their results show why selecting an answer from four alternatives and interpreting psychological meaning without supplied answers can produce different assessments of model capability.

Controlled narratives make psychological inference measurable

ToMStories contains 5,932 synthetic narrative vignettes, generated using Claude Sonnet 4.6. Each scenario combines a setting, a relationship between characters, and a target psychological state drawn from a writer-oriented thesaurus.

The benchmark covers four categories: emotions, fears, motivations, and emotional wounds. It includes three narrative variants:

• Implicit (1,988): The target state is suggested through actions and contextual details rather than directly stated.
• Explicit (1,988): The state is communicated more directly, without simply reproducing the target answer.
• Conflicted (1,956): Dialogue and descriptive behavior suggest different psychological states.

Matched implicit and explicit scenarios allow the researchers to examine how presentation affects interpretation while holding the underlying target state and situation constant.

The construction provides known intended answers, which is valuable for controlled measurement. It also introduces a dependency: the benchmark measures how models interpret psychological conventions expressed by one language-model generator.

Choosing the right answer is easier than producing one

The headline performance comparison comes from implicit narratives (Section 5.1, Table 4).

The leading models differ across the two formats. The human baseline is also limited: four computer science doctoral students answered a subset of multiple-choice questions, while two supplied 78 open-ended responses. These figures should not be interpreted as a definitive human-versus-model ranking.

There is another measurement detail. Although open-ended responses are generated without visible answer options, the benchmark converts them into accuracy scores by embedding each response with all-MiniLM-L6-v2 and selecting whichever of four predefined answer alternatives is semantically closest.

Consequently, open-ended accuracy remains partly dependent on the benchmark’s answer taxonomy and embedding model. It does not measure unrestricted human judgment of whether an interpretation is thoughtful, appropriately uncertain, or justified by the narrative.

Still, the difference between formats matters. A model may recognize a psychological category when alternatives are provided without producing an equally reliable interpretation on its own. The relatively high explicit-variant accuracy—at least 93% for every tested model with 8B or more parameters—further indicates that directly expressed states are comparatively straightforward.

Models perform better when they share the generator’s behavioral vocabulary

The researchers investigate whether different language models associate similar observable behaviors with the same psychological states.

They ask thirteen open-weight models and Claude Sonnet 4.6 to generate behavioral or physical indicators for 100 sampled states. After normalizing and semantically clustering 13,798 phrases, they measure how often each evaluated model shares indicator clusters with Claude.

The relationship with benchmark performance is striking (Section 5.3, Table 5):

• Multiple-choice accuracy: Pearson correlation of 0.89 with indicator overlap.
• Open-ended accuracy: Pearson correlation of 0.92 with indicator overlap.

Both correlations have reported p-values below 0.01.

Across twelve models, standardized regressions incorporating model size, preference for descriptive evidence, and generator-indicator overlap find that overlap is the only statistically significant predictor in the primary specification. Its standardized coefficients are 0.60 for multiple choice and 1.03 for open-ended accuracy.

General dominance analysis attributes 45% and 66% of the respective regressions’ explained variance to overlap. These are shares of model-explained variation, not percentages of accuracy caused by shared phrases.

A robustness analysis in Appendix G.2 varies the clustering threshold from 0.40 to 0.60. The open-ended association remains significant in all joint specifications, while the multiple-choice association remains significant in most. This tests sensitivity to phrase-clustering decisions rather than independently validating the proposed mechanism.

The interpretation is consequential: when a generator and an evaluated model associate similar behaviors with similar emotions, the evaluated model may find the intended answer easier to recognize.

However, the analysis is observational. Only twelve model-level observations enter the main regressions, and the predictors are correlated. Shared indicators could reflect broader model capabilities rather than a specific shortcut. The evidence makes generator alignment a plausible explanation for benchmark performance, not a proven causal mechanism.

When words and behavior disagree, models favor words

A separate experiment examines which evidence models prioritize when dialogue and behavioral description imply competing psychological states.

Unlike the standard implicit questions, these conflicted narratives have no uniquely correct interpretation. Their purpose is to reveal relative reliance on different information sources (Section 5.1, Figure 2).

Models generally select dialogue-aligned interpretations more frequently than human annotators. Meanwhile, models with greater preference for descriptive evidence tend to score better on implicit multiple-choice questions, with a reported correlation of 0.80. The corresponding relationship for open-ended accuracy is not statistically significant.

For customer-facing applications, this suggests a concrete evaluation dimension: whether an assistant can consider behavioral and contextual evidence when verbal statements are incomplete or contradictory.

It does not establish that dialogue should always receive less weight. In real interactions, context can be misleading, and a customer’s explicit statement may deserve priority. The operational capability to test is evidence integration under uncertainty, not automatic distrust of spoken words.

Internal representations suggest a narrower concept of implicitness

The authors also investigate whether implicit and explicit narratives produce distinguishable internal representations (Section 5.4).

Linear classifiers trained on transformer hidden states recover answer-related information at approximately similar layers for both variants, although explicit narratives yield higher probe accuracy. This shows that answer information is more readily decodable from explicit examples; it does not establish which information the model actually uses when answering.

The authors then construct an activation direction from the difference between average implicit and explicit hidden-state representations. Adding this direction during generation—known as activation steering—produces measurable changes in writing.

Steering toward implicitness generally produces longer, more ornate text while reducing emotion-associated word rates in seven of eight measured emotion categories. Sadness is the exception. Generated text also exhibits lower lexical valence and arousal.

These intervention results suggest that models partly encode implicit psychological expression as less emotionally explicit language. But psychological subtext can involve intense feelings conveyed indirectly. Reduced emotional vocabulary is therefore an incomplete representation of the underlying phenomenon.

The experiment demonstrates changes in linguistic output, not improved interpretation of human emotions.

What a conversational AI selection process should test

For procurement and product teams, Cognaptus draws an evaluation-design implication rather than a model-ranking recommendation.

A stronger assessment would combine independently written customer scenarios with open-ended questions requiring the assistant to explain its interpretation and identify supporting evidence. It would also introduce conflicting verbal and contextual signals, ambiguous cases where several interpretations remain plausible, and culturally varied communication patterns.

Evaluation should separately measure whether the assistant recognizes possible concerns, expresses uncertainty appropriately, and chooses a suitable operational response. Identifying a possible emotion and deciding how to handle a customer request are related but distinct tasks.

Generator-indicator overlap could help flag potential benchmark-specific alignment. It should not be used as an automatic rejection criterion without independent performance evidence.

The boundary of the findings

ToMStories provides controlled evidence about synthetic psychological narratives, not direct evidence about real-world customer interactions.

All stories originate from one generator, and the underlying psychological-state resources reflect Western English-language writing conventions. Human comparisons are small, while regression results involve few models. The interpretability experiments characterize internal information availability and changes in generated language without demonstrating reliable psychological inference outside the benchmark.

The most defensible conclusion is therefore narrower than human-level theory of mind: language models can perform strongly on structured psychological-state recognition, but their scores are associated with the conventions of the system that generated the test.

For organizations deploying conversational AI, trustworthy evaluation requires evidence that extends beyond those conventions. A model’s ability to select the intended emotional label is only one component of its ability to respond appropriately when human intentions remain uncertain.

Cognaptus: Automate the Present, Incubate the Future.