Correct Answer, Weak Evidence: Measuring Multimodal Reasoning at the Fact Level

TL;DR for operators

A correct multimodal answer with citations can still have a weak evidence trail. In human evaluation on Video-MMMU, Gemini-3-Pro reached 86% question-answering accuracy but only 16.3 on MuRGAt-Score, meaning answer performance and fact-level grounding ranked very differently.

MuRGAt evaluates two requirements separately. Coverage asks whether verifiable claims receive citations. Attribution asks whether the cited frames, audio, figures, or temporal segments are actually relevant to and sufficient for the factual claims they accompany. The combined score rewards systems only when both conditions hold.

This distinction changes several operational decisions. Procurement teams should not infer auditability from benchmark accuracy. Product teams should test citation architectures on both answer quality and grounding. Human-review systems can focus on unsupported factual units rather than rechecking an entire response. And inference-budget policies should measure attribution directly rather than assume that more reasoning automatically produces a better evidence trail.

The paper’s program-aided experiments also expose a real design trade-off: stronger grounding improved MuRGAt-Score on WorldSense while reducing answer accuracy on average. That result supports measurement of the trade-off, not a general claim that additional grounding machinery is always preferable.

A correct answer does not verify the reasoning behind it

Suppose a multimodal system watches a video, listens to its audio, identifies several events, and answers a question correctly. It also provides timestamps beside its explanation. From an operational perspective, one more question remains: do those timestamps actually support the factual steps the explanation makes?

That question matters because two reassuring signals can separate. The final answer may be right even when intermediate claims rely on unsupported inference, internal knowledge, or weakly related source material. A response may also cite almost every claim while attaching evidence that does not establish what the text says.

Wan and colleagues address this problem with MuRGAt, or Multimodal Reasoning with Grounded Attribution, an evaluation framework for fact-level multimodal grounding.¹ The paper’s human evaluation makes the separation unusually clear: on Video-MMMU, Gemini-3-Pro achieved 86% answer accuracy but a MuRGAt-Score of only 16.3.

For a team deciding whether a model is auditable, those are different properties and should be treated as different release criteria.

MuRGAt checks evidence against smaller factual units

A sentence can contain several independently checkable claims. Evaluating its citation as one block can therefore hide partial support: a cited segment may establish one part of the sentence while leaving another unsupported.

MuRGAt first identifies sentences containing externally verifiable claims. It then decomposes those sentences into atomic facts and propagates the associated citations to those smaller factual units. Finally, an evaluator tests whether each cited evidence segment is relevant and whether the cited evidence collectively provides sufficient support.

The resulting measurement has three layers:

Formally, the combined score is:

Attribution itself combines precision—whether individual citations are relevant—with recall—whether the available citations jointly provide enough evidence to establish each atomic fact.

This is why high citation coverage is not enough. A system can attach citations almost everywhere while still producing weak attribution.

Fact-level decomposition improves the evaluator, not just the definition

The paper also tests whether this evaluation procedure can scale beyond human annotation.

Against human end-to-end judgments, the decomposed evaluator reached a Pearson correlation of 0.86 for MuRGAt-Score. The strongest reported disentangled sentence-level baseline reached 0.58, while holistic evaluation reached 0.35.

That comparison matters because the contribution is not merely a more demanding definition of grounding. The paper provides evidence that decomposing the task into verifiability, atomic facts, and evidence entailment produces automatic scores that track human judgments substantially better in the studied setting.

The supporting checks are best interpreted as robustness tests, not separate headline findings. The authors use 10,000 bootstrap resamples and subsampling analyses to test whether the correlation depends on a small set of examples. A manual review of 50 automatic entailment judgments agreed with the evaluator in 47 cases. These checks strengthen confidence in the measurement pipeline without establishing that it will transfer unchanged to every multimodal domain.

Citation strategies solve different problems

The model comparisons show why grounding architecture should be evaluated rather than assumed.

With Gemini-3-Flash on WorldSense, explicit citation generation produced a MuRGAt-Score of 64.4, while post-hoc attribution reached 69.2. On Video-MMMU, however, explicit citation generation scored 56.9, versus 44.1 for post-hoc attribution.

The qualitative interpretation in the paper helps explain the difference. Post-hoc attribution can work when a conclusion maps closely to perceptual evidence that can be located afterward. But for deductions, the model can retroactively attach plausible frames to reasoning that those frames did not actually support. The evidence trail then looks complete without faithfully representing the basis for the claim.

Citation prompting also interacts with task accuracy rather than moving it in one direction. For Gemini-3-Pro, explicit citations changed WorldSense accuracy from 71.4% to 70.0%, while Video-MMMU accuracy moved from 85.3% to 86.0%.

The evidence therefore supports task-specific comparison among grounding strategies, not a universal preference for inline or post-hoc citation.

More reasoning does not guarantee better grounding

Reasoning effort shows the same lack of monotonicity.

For Gemini-3-Flash on WorldSense, MuRGAt-Score fell from 69.7 at Minimal effort to 64.4 at High effort. Gemini-3-Pro moved in the other direction, from 45.6 at Low effort to 51.7 at High effort.

Additional inference compute can improve some model-task combinations, but the paper does not support treating reasoning budget as a proxy for evidence faithfulness.

The program-aided experiments make the trade-off even more explicit. On WorldSense with Gemini-3-Flash, variants that made evidence retrieval and attribution checking part of the runtime process improved MuRGAt-Score by an average 9.6 points relative to Base+Citation, while answer accuracy fell by an average 7.4 points. The strongest reported logic-centric imperative variant reached 76.4 MuRGAt-Score, with 60% answer accuracy.

These experiments are exploratory design comparisons rather than causal evidence about model internals. Their value is that they expose an engineering choice: constraining reasoning around explicit evidence discovery and verification can produce a more defensible evidence trail while also disrupting answer performance.

Treat accuracy and traceability as separate deployment criteria

The paper directly shows that answer accuracy, citation coverage, and fact-level attribution can produce materially different rankings across models and configurations.

Cognaptus inference: teams deploying multimodal systems should preserve those dimensions separately in evaluation dashboards rather than collapsing them into one trust score. A high-accuracy model with weak attribution may still be suitable where only the answer matters. The same system may be inappropriate where reviewers, auditors, clinicians, analysts, or other decision-makers must inspect the evidence behind individual claims.

The separation also improves diagnosis. Low Coverage means the model omitted evidence. High Coverage with low Attribution means evidence was supplied but does not adequately support the claims. Those failures call for different interventions.

For human review, atomic fact-level scoring could also provide a more targeted queue: reviewers can examine unsupported facts and their cited timestamps rather than replaying an entire multimodal input for every response.

The evidence supports measurement more strongly than generalization

The direct human validation remains bounded. The end-to-end reference set covers 20 distinct multimodal inputs, 80 model responses, and 600 annotated sentences, although the paper supplements this with larger component-level annotation sets, bootstrap analysis, evaluator substitutions, and manual checking.

The empirical evidence also comes from WorldSense and Video-MMMU and from the evaluated Gemini, Qwen, and Molmo configurations. The scalable evaluator itself relies heavily on model-based judgments, including Gemini-family components.

So the strongest conclusion is not that one citation strategy, reasoning budget, or grounding architecture should become a default everywhere. It is narrower and more actionable: grounding quality can be measured independently of answer accuracy, and systems should be compared on that axis whenever evidence traceability matters.

A citation makes a reasoning trace inspectable only when the cited evidence supports what the trace claims. MuRGAt gives teams a way to test that distinction instead of assuming it.

Cognaptus: Automate the Present, Incubate the Future.