TL;DR for operators
A model can receive a camera feed, spoken report, reference image, and text record without meaningfully reasoning across all four. In C$^3$PO1, 86–95% of observed failures across ten models in the paper’s failure analysis were classified as dominance-driven: one modality or prior drove the answer while other evidence was effectively ignored.
That changes what multimodal evaluation should measure. The benchmark separates tasks that require combining complementary evidence from tasks that require resolving deliberate contradictions. Difficulty does not rise cleanly with modality count. Some three-modality conflict settings outperform two-modality ones because the added evidence can help adjudicate the dispute.
The paper’s internal probes provide a diagnostic hypothesis for why. Correct answers are generally associated with broader mid-layer attention across modalities, while several “audio-bypass” models devote about 87% of measured attention to text and chat-template tokens. But the paper does not show that forcing higher attention entropy would improve accuracy, and its main attention analysis excludes video positions.
For product selection, test what a model does when evidence disagrees. For safety- or compliance-sensitive workflows, detect those disagreements and seek independent corroboration before acting. Treat attention diagnostics as investigative tools, not production confidence scores.
Four inputs do not guarantee four-way reasoning
Consider a system processing a live camera feed while also receiving a spoken report, a still image, and a text record. When all four agree, even weak integration can look competent. Reliability becomes visible when they disagree.
The central question is then which evidence survives the model’s reasoning process.
C$^3$PO is designed around that distinction. Its 25 templates divide evaluation into Information Composition, where the answer requires combining evidence dispersed across modalities, and Counterfactual Conflict, where modalities intentionally contradict one another and the model must decide which evidence should govern the answer. The benchmark primarily reports 3,404 samples built from authentic media drawn from eight public datasets plus generated complementary modalities.
The headline performance gap is substantial. Humans reached 88.64% accuracy on a stratified 250-sample evaluation. Gemini-3.1-Pro, the strongest evaluated model by overall accuracy, reached 73.17%. The strongest open-source model reported in the main results table, Qwen-3-Omni-Instruct, reached 43.98%.
Those figures establish that cross-modal reasoning remains difficult. The more informative result is how the systems fail.
Most failures look like commitment, not partial fusion
The paper classifies 86–95% of observed failures across ten models as modality-dominance failures. Roughly 5–14% are categorized as incomplete fusion.
That distinction changes diagnosis.
Incomplete fusion suggests that a model is trying to combine evidence but fails to integrate enough of it. Dominance suggests something earlier: the system commits to one modality or prior and behaves as though competing evidence carries little weight.
The dominant channel is not fixed. In the paper’s template-level analysis, AcousticSceneMismatch shows substantial audio-dominance failures, while CausalMisattribution and ProcedureVsEvidence contain large text-or-prior-dominance components. The issue is consequently more specific than a universal “text bias” or “audio bias.” Task structure changes which learned shortcut becomes attractive.
For a product owner evaluating a multimodal assistant, this means separate modality competence from evidence arbitration. A model can recognize speech, images, and video individually yet still become unreliable when those channels disagree.
More modalities can make a conflict easier
Information Composition behaves roughly as expected: accuracy generally falls as tiers become more demanding.
Counterfactual Conflict does not.
Many evaluated models perform particularly poorly at Tier 2, improve sharply at Tier 3, and then deteriorate again at Tier 4. Qwen-3-Omni-Instruct, for example, scores 36.74% on Tier 2 conflict tasks and 77.76% on Tier 3 before falling to 15.64% at Tier 4.
The paper traces part of this irregular pattern to evidence structure rather than modality count. Two templates involving contradictory audio, CausalMisattribution and DisclaimedEvidence, differ by roughly 56 percentage points in cohort-level performance. In DisclaimedEvidence, an explicit conflict-resolution cue and an image can provide an independent reference for deciding between competing signals. CausalMisattribution requires more direct arbitration between audio and video.
A third source can therefore function like an independent witness: adding information increases input complexity while reducing decision ambiguity.
This is one of the benchmark’s most consequential design findings. Counting modalities is a poor proxy for reasoning difficulty when those modalities play different logical roles.
The probes point toward premature concentration
The paper then asks whether dominance is visible inside the models rather than only in their final answers.
Several diagnostic tests address different stages of the process:
| Diagnostic | Likely purpose | What it supports | Boundary |
|---|---|---|---|
| Layer-wise attention | Examine where measured attention is allocated | Some models heavily concentrate on text/template tokens | Video positions are excluded from this probe |
| Attention entropy | Measure whether attention remains spread across modalities | Correct answers generally retain broader mid-layer attention | Association does not establish causation |
| Logit-lens recovery | Test whether modality-relevant information is latent before decoding | Helps distinguish missing representation from later reasoning failure | Available only for probe-compatible architectures |
| Caption fidelity | Test how faithfully audio and image content can be rendered | Helps separate perception/fidelity problems from arbitration failures | Captioning performance is not equivalent to task reasoning |
| Template failure decomposition | Compare behavior across evidence structures | Dominant failure modality depends on template design | Comparisons are descriptive, not causal identification |
The attention result is particularly suggestive. The authors measure between-modality Shannon entropy,
where higher entropy means measured attention remains more evenly distributed across available modalities.
Across the statistically tested model-category comparisons, five of six bootstrap 95% confidence intervals show higher mid-layer entropy for correct answers. After within-model normalization, the pooled correctness-associated gap is +0.43 standard deviations for Counterfactual Conflict and +0.21 for Information Composition.
Meanwhile, several audio-bypass models allocate about 87% of measured per-layer attention mass to text and chat-template tokens, leaving only around 11–13% for audio and image.
The plausible interpretation is premature commitment: retaining several evidence channels through the middle layers may give later computation more opportunity to arbitrate among them.
It remains a hypothesis. Attention mass is not a complete measure of semantic use, the probe excludes video, and the analysis is observational.
Procurement tests should introduce disagreement deliberately
Cognaptus infers a practical evaluation change from these results: multimodal procurement should include controlled contradiction tests, not only separate vision, audio, and video accuracy scores.
A procurement team comparing candidate models could vary three properties independently: which modalities disagree, whether an independent corroborating source exists, and whether the task explicitly identifies which evidence should be authoritative. The output to inspect is not only answer accuracy but also the pattern of failures—audio dominance, visual dominance, text/prior dominance, or incomplete fusion.
For system architects, the same logic supports an explicit verification layer in workflows where an incorrect action carries compliance or safety consequences. When sensor evidence and linguistic reports conflict, the workflow can flag the contradiction, seek another source, or escalate rather than allowing one channel to win silently.
For model-development teams, the paper’s probes provide a way to narrow the intervention. Poor modality fidelity points toward perception or encoding. Strong unimodal representations combined with dominance failures point further downstream toward routing, fusion, or arbitration.
The benchmark does not establish which architectural change will fix those failures. It does make “use a larger multimodal model” an incomplete diagnosis.
What remains uncertain
Three boundaries matter for practical interpretation.
First, Gemini-3-Pro participates in benchmark generation, so Gemini-family performance is not fully independent of the construction pipeline. Second, semantic scoring ultimately relies on an LLM judge, although the authors report 93% average agreement and Cohen’s $\kappa$ of 0.88 against a second judge across four response sets. Third, the mechanistic probes cover only architectures that expose the required internals, and the attention analysis cannot include video because of memory constraints.
There is also a manuscript-level counting inconsistency: the abstract, Section 3, and Appendix Table 13 report 3,404 samples, while earlier Tables 1 and 2 report 3,342. The paper does not reconcile the difference.
These issues weaken attempts to treat individual numbers as universal model rankings or the attention results as a validated monitoring mechanism. They do not erase the benchmark’s central diagnostic result: multimodal input capability and cross-modal evidence arbitration are separate capabilities.
For deployed systems, the more revealing question is no longer how many modalities a model accepts. It is what happens when those modalities disagree—and whether the surrounding workflow notices before the model acts.
Cognaptus: Automate the Present, Incubate the Future.
-
Swapnanil Mukherjee and Agyeya Negi and Tanuja Ganu and Ponnurangam Kumaraguru (2026). C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models. arXiv:2608.05381. https://arxiv.org/abs/2608.05381 ↩︎