TL;DR for operators

Five patients receive drug A. Three improve, and two do not. Should an AI verifier label both “The patients improved” and “The patients didn’t improve” as False? Under the group-statement semantics tested in HomogeneityNLI, both are Neither—neither true nor false. Yet “Not all the patients improved” is True. The evidence is complete; the difference lies in what each statement means when outcomes are mixed.

Across seven instruction-tuned open-weight models, the mean zero-shot rate of correctly assigning Neither to both opposing claims was just 0.05 (5%), despite high accuracy on explicit quantifiers and uniform-group controls. Negated group claims were labeled False in 84% of pooled zero-shot judgments. Supplying semantic definitions or additional reasoning instructions did not reliably correct the problem, and sometimes worsened performance.

For operators evaluating AI-assisted claim verification or group summaries, the finding motivates testing matched positive–negative claims under mixed evidence. Track correct Neither–Neither judgments, incorrect False–False judgments, and ordinary control accuracy separately. These are diagnostic recommendations supported by a controlled English benchmark, not established production remedies or estimates of real-world error rates.

Five patients, two claims, and a surprising answer

Consider an AI assistant reviewing treatment outcomes. Five patients received drug A. Three improved, and two did not. The evidence is complete: every patient’s outcome is known.

Now consider two statements:

  • “The patients improved after taking drug A.”
  • “The patients didn’t improve after taking drug A.”

A conventional verification system might label both statements False because neither accurately describes every patient. That seems cautious, but under the linguistic interpretation examined in this research, it is incorrect.

Both statements receive Neither: neither true nor false. Meanwhile, “All the patients improved” is False, and “Not all the patients improved” is True.

The distinction comes from homogeneity, a semantic account of statements about plural groups. Under this account, an affirmative plural-definite statement is true when every member satisfies its predicate, false when none do, and neither true nor false when the group is mixed. Its negation also receives Neither in the mixed case.

Importantly, Neither does not mean that the evidence is missing. It represents a truth-value gap despite complete information.

For a verification system, this creates a specific classification problem. Rejecting both opposing group claims as False may look conservative, while actually overlooking a distinction built into their semantics.

A benchmark isolates the missing truth value

In Neither True nor False: Do LLMs Represent Truth-Value Gaps in Plural Predication?, Nakshbandi and colleagues introduce HomogeneityNLI to measure this behavior directly.1

The benchmark contains 1,000 labeled items generated from 100 English scenarios. Each scenario contributes ten logically related conditions covering plural definites, their negations, explicit quantifiers, and uniform-group controls. Predicates span actions, properties, locations, and states.

Seven instruction-tuned open-weight models, ranging from approximately 7 billion to 9 billion parameters, are tested under four prompting strategies.

The central measurement evaluates positive and negated plural-definite statements as a pair. Two metrics distinguish the outcomes:

Homogeneity Gap Rate (HGR) measures how often a model correctly assigns (Neither, Neither) to the pair.

Bivalent Collapse Rate (BCR) measures how often it instead assigns (False, False), incorrectly replacing the truth-value gap with binary falsity.

This distinction matters because evaluating individual statements separately can conceal whether a model consistently represents the underlying semantic relationship.

The main zero-shot results are striking.

Diagnostic measure Finding
Mean HGR across seven models 0.05
Highest model-specific HGR 0.15, gemma-2-9b-it
Neither responses for positive definites 40%
Neither responses for negated definites 8%
False responses for negated definites 84%

Source: Section 5.1, Table 3 and polarity-asymmetry analysis. Rates for individual polarities are pooled across the seven models.

In approximately 95% of model–scenario evaluations, the model fails to assign Neither to both statements.

Yet the neighboring logical controls perform substantially better. Zero-shot accuracy is at least 0.96 for explicit universal claims, at least 0.99 for negated universal claims, and at least 0.95 on average for uniform-group definite controls.

This comparison is central to the paper’s argument. The models can usually handle statements equivalent to “all” and “not all” when those quantifiers are explicit. Their difficulty emerges when plural-definite wording requires a third semantic value.

The authors describe the tendency to force these judgments into classical True or False categories as Bivalence Bias.

Negation makes the problem especially visible. A model may correctly identify that not every patient improved while still assigning False to “The patients didn’t improve,” where Neither is required under the benchmark’s semantics.

More reasoning does not reliably correct the error

A natural response is to explain the semantic rule to the model or require it to reason through each group member.

The paper tests three such interventions against strict-logic zero-shot prompting. These are intervention comparisons, not independent demonstrations of a production solution.

Prompting strategy Mean HGR
Zero-shot 0.05
Definition-aware prompting 0.11
Chain-of-thought 0.01
Member-by-member enumeration 0.14

Source: Section 5.2, Table 3. The degenerate DeepSeek enumeration run is excluded from pooled enumeration results.

Definition-aware prompting produces the clearest individual improvement: Yi-1.5’s HGR rises from 0.01 to 0.56. But Llama-3.1’s BCR increases from 0.54 to 0.99 under the same general strategy. For that model, supplying the semantic definition makes incorrect False–False pairs substantially more frequent.

Chain-of-thought prompting performs worse on average than zero-shot. All seven models have HGR at or below 0.03, and five exhibit increased bivalent collapse.

Enumeration offers partial improvements. Llama-3.1’s HGR rises from 0.02 to 0.29, but its mean accuracy on uniform-group controls falls from 1.00 to 0.79.

These results show why a prompting intervention cannot be judged solely by improvements on the targeted examples. A model may recover some missing truth-value distinctions while losing accuracy on statements it previously classified correctly.

The model may find the right condition but assign the wrong label

The authors also examine chain-of-thought rationales to investigate a possible mechanism.

Among rationales ending in False for affirmative plural definites, 92% (457 of 497) reformulate the statement using universal language such as “all” or “every.” For negated definites classified False, 48% (271 of 566) contain universal-negative reformulations.

These findings are consistent with an important distinction: identifying the conditions required for a statement to be true is not the same as assigning its truth value correctly.

A model may recognize that a homogeneous affirmative claim requires every member to satisfy a predicate. Upon finding an exception, it may jump directly to False rather than recognizing that the mixed group creates a truth-value gap.

This is interpretive evidence, not proof of the model’s internal computation. Generated rationales may not faithfully describe underlying reasoning, and the chain-of-thought prompt itself mentions quantifiers that could encourage these reformulations.

A separate sensitivity analysis adds nuance. Models accept affirmative plural definites as True in 25% of near-uniform mixed groups, compared with 9% of approximately balanced groups and 6% when only one member qualifies. Acceptance also differs by predicate class.

Such non-maximal interpretations indicate sensitivity to group composition and wording. They do not demonstrate reliable recognition of the required third truth value.

Evaluate paired claims alongside ordinary accuracy

For a product team selecting a model for AI-assisted claim verification, the findings suggest a more targeted acceptance test.

Suppose the application summarizes customer outcomes, employee participation, or case-resolution records. Where individual outcomes are available, evaluators can construct matched affirmative and negated group statements across all-member, no-member, and mixed-member cases.

The evaluation should track three distinct properties.

First, HGR establishes whether the model recognizes both members of a mixed-group pair as neither true nor false under the specified semantics.

Second, BCR reveals whether apparently cautious rejection is actually incorrect binary classification.

Third, ordinary quantified and uniform-group control accuracy checks whether an intervention preserves basic judgments rather than trading one error class for another.

These measures address different failure modes. An aggregate accuracy score can obscure their distribution, and an improved HGR alone does not guarantee that overall semantic reliability has improved.

For evidence-summarization products, Cognaptus further infers that a structured representation distinguishing all, none, and mixed outcomes may reduce dependence on ambiguous categorical judgments. Where incomplete evidence is possible, missing information should be represented separately from a semantic Neither label.

That distinction is a proposed design principle. The benchmark does not test whether this architecture improves real-world summaries.

The benchmark diagnoses a weakness, not deployment risk

The evidence is strongest for the controlled behavior it measures: seven specified 7–9B open-weight models evaluating template-generated English statements with distributive predicates.

It does not establish the prevalence of the same failures in larger or untested models, other languages, collective predicates, or naturalistic documents. Contextual relevance and conversational non-maximality are not directly manipulated.

Nor does the study measure downstream clinical, legal, or business harm. Its patient example illustrates a semantic distinction; it is not an evaluation of medical decision-making.

The prompt comparisons also do not rule out other interventions, including different training procedures or application-level verification methods.

These boundaries affect what organizations can claim from the evidence. HomogeneityNLI provides a reproducible way to diagnose a specific semantic failure. Deployment reliability requires additional evaluation using the actual documents, judgments, and consequences of the intended workflow.

When rejecting both claims is still an error

The broader lesson is about what an evaluation measures.

High accuracy on explicit quantifiers does not necessarily mean a model can represent the semantics of closely related plural statements. More reasoning text does not necessarily correct a truth-value assignment error. And rejecting two opposing claims does not automatically mean a model has handled ambiguity appropriately.

HomogeneityNLI makes these distinctions measurable through paired judgments and targeted controls.

For organizations assessing AI-generated group summaries, that is its immediate contribution: a more precise way to test whether the system preserves what mixed evidence actually permits it to say.


Cognaptus: Automate the Present, Incubate the Future.


  1. Duaa Nakshbandi and Zhiyu An and Yiping Li and Sachin Giri and Wan Du (2026). Neither True nor False: Do LLMs Represent Truth-Value Gaps in Plural Predication?. arXiv:2610.14438. https://arxiv.org/abs/2610.14438 ↩︎