The Right Number Can Still Answer the Wrong Question

TL;DR for operators

A financial agent can retrieve the correct filing, extract a defensible figure, ask a relevant follow-up question—and still return the wrong answer for what the user meant.

The clearest evidence is the gap between knowing the interpretation and having to recover it. When evaluated models are supplied with the disambiguating interpretation, their accuracy rises to roughly 90-95%. Under full free interaction, where the agent must decide what to clarify and use the answer correctly, the best result is 28.9%.

For operators, this separates two controls that are often bundled together: evidence retrieval and intent resolution. A production financial assistant should not treat a successful search or even a sensible clarification as sufficient evidence that a number is ready for downstream use. The resolved interpretation—entity, metric definition, period, filing version, or accounting treatment—needs to persist as explicit state through final answer generation.

The paper’s benchmark diagnosis is considerably stronger than its proposed remedies. Category-aware prompting and category-guided training improve some results, but those experiments are small and do not yet isolate which intervention components are responsible.

A retrieved number is not yet a resolved answer

Consider a routine question to a financial assistant: “What was the company’s revenue?”

The filing may contain several defensible answers. The user may mean consolidated revenue rather than a segment figure, GAAP rather than a non-GAAP measure, a fiscal period rather than a calendar period, or one filing version rather than another. Retrieval can be flawless while interpretation remains unresolved.

This is the problem measured by FinInteract, a bilingual benchmark from Wang, Kwok, Tai, and Cheng.¹ It contains 173 ambiguous financial QA instances across 61 companies, with 53 English and 120 Chinese cases. Each item pairs a default interpretation with a separate intended interpretation, along with evidence for both.

That pairing matters because conventional financial QA evaluation usually has one reference answer. If a model gives a plausible number that matches that reference, the benchmark records success even if the question itself supported another reasonable reading.

FinInteract makes that mismatch measurable.

Single-gold scoring can reward the wrong interpretation

The paper demonstrates this directly by grading the same model outputs against two different readings.

For GPT-4o in the retrieval-augmented setting, accuracy is 12.1% when answers are judged against the intended interpretation. Re-grade those same outputs against the default interpretation and accuracy rises to 37.0%.

Nothing about the model output improved. Only the scoring target changed.

The authors call this the single-gold illusion: one accepted answer can make an agent look substantially better at ambiguity resolution than it actually is. The issue is especially relevant in finance because several numbers can be simultaneously well grounded in the same filing while answering different versions of the question.

For model-governance teams, this suggests that retrieval accuracy and answer accuracy should not be treated as sufficient proxies for intent fidelity. An agent can be grounded and still be grounded to the wrong reading.

Supplying intent nearly solves what interaction does not

The strongest diagnostic result comes from the oracle comparison.

When the intended interpretation is supplied to the model, evaluated systems achieve roughly 90-95% accuracy. Qwen3-30B reaches 90.2%; GPT-4o reaches 93.1%; GPT-5 and GPT-5-mini reach 95.4%.

By contrast, the best full-scale free-interaction result is 28.9%.

This is a substantial change in where the failure should be located. The benchmark does not primarily describe agents that lack the financial information needed to answer. Once the ambiguity is resolved for them, they usually can.

The difficult part is the sequence before that answer: detect that the question is underspecified, identify which dimension is underspecified, ask the right question, absorb the user’s reply, and then answer under the resolved interpretation.

FinInteract therefore decomposes “clarification ability” into separate capabilities rather than treating interaction as one behavior.

Asking the right question does not guarantee the right answer

A natural assumption is that once an agent recognizes the ambiguity and asks about the relevant dimension, most of the problem has been solved.

The error analysis shows otherwise.

Across interactive failures, the paper’s E5 category—clarification misuse, where the agent asks appropriately but still produces the wrong answer—accounts for 67% of GPT-5 failures, 89% for GPT-4o, 90% for GPT-5-mini, and 75% for Qwen3-30B.

That is the operationally important distinction. Clarification is not merely a conversational action. It creates new state that must constrain subsequent retrieval, reasoning, and answer generation.

The paper also shows why interaction rate alone is a poor success metric. GPT-5 improves from 6.4% with search to 20.2% with interaction, while GPT-4o falls from 12.1% to 4.6%. Forcing four clarification questions pushes GPT-5 down to 1.5% and GPT-4o to 0%.

More questioning is therefore not a monotonic path to better answers. The system has to identify the uncertainty worth resolving and preserve the resulting interpretation.

Ambiguity should become explicit system state

FinInteract divides financial ambiguity into five categories: temporal scope, metric definition, entity scope, filing vintage, and recognition policy.

This makes clarification quality more inspectable. Its AC@1 metric asks whether the agent’s first clarification targets the relevant ambiguity category rather than merely producing a generic follow-up.

The benchmark suggests that models can often touch the correct category while still failing later. A separate hidden-state analysis also finds that ambiguity category can be linearly decoded from larger open models: entity versus metric type reaches 0.80 decoding accuracy against a 0.60 majority baseline.

That result should be interpreted narrowly. Linear decodability shows that information is available in the representation; it does not show that the representation causally drives the agent’s decision to clarify.

The more defensible system-design inference is simpler: intent should be represented explicitly rather than assumed to survive implicitly across turns.

For a financial research assistant, that could mean maintaining a structured interpretation record alongside retrieval state:

Before a number is committed to a research note, report, compliance workflow, spreadsheet, or downstream agent, the system can check whether the relevant fields are resolved rather than relying on conversational context alone.

That design implication is a Cognaptus inference, not a directly tested architecture in the paper.

The proposed interventions are promising, not settled

The paper tests two directions for turning the taxonomy into action.

In a stratified 50-item GPT-5 pilot, Category-Aware Clarification ReAct raises accuracy from 34.0% to 46.0% and improves expected calibration error from 0.308 to 0.241. But the policy bundles ambiguity pre-checking, category-conditioned clarification, and gated answering, so the experiment does not identify which component produces the gain.

Training experiments show a similar pattern. On a held-out set of 51 cases, category-guided supervised fine-tuning raises AC@1 from 25.0 to 68.6 and interaction from 54.9 to 100%. Subsequent KTO and GRPO stages add little beyond SFT within the reported run-to-run variation.

These results support ambiguity categories as usable supervision signals. They do not yet establish a preferred production recipe.

The benchmark measures resolution, not when to ask

FinInteract is deliberately constructed from ambiguous questions. That makes it useful for testing what an agent asks and what it does with the answer, but not whether an agent is well calibrated about when clarification is necessary in the first place.

The dataset is also small and unevenly distributed. Entity scope accounts for 94 instances and metric definition for 63, while temporal scope has seven, recognition policy nine, and filing vintage none. The recognition-policy results are therefore exploratory rather than a stable estimate of model capability.

The questions are reverse-constructed from filings rather than sampled from natural analyst conversations, and GPT-5-family models participate in construction, verification, and user simulation. The inference and training interventions are preliminary, while the representation probes are correlational.

Those limitations do not materially weaken the central benchmark result: giving the model the resolved interpretation produces near-ceiling performance, while requiring the agent to recover and preserve that interpretation produces much lower accuracy.

Financial agents need an interpretation checkpoint

The practical lesson is not that financial agents should ask more questions. FinInteract provides evidence against that policy.

The stronger requirement is that a system should know which interpretation it has committed to, how that interpretation was resolved, and whether the final number remains consistent with it.

That changes the control point. Search success is not enough. A relevant follow-up is not enough. Even a grounded final answer is not enough if the system cannot establish that the evidence, interpretation, and committed number refer to the same financial object.

For high-stakes financial QA, intent resolution should be treated as a separately testable system capability rather than an assumed by-product of retrieval and conversation.

Cognaptus: Automate the Present, Incubate the Future.