Before You Retry the Agent, Check the Evidence

TL;DR for operators

An agent can already have the information needed to answer a question and still produce the wrong answer. Diehl and Lim’s Evidence-Inference Reconstruction (EIR)¹ addresses that failure mode by storing retrieved source sentences separately from the agent’s evolving interpretations, then generating the scored answer in a fresh call that sees only the original question and accumulated evidence.

Across six combinations of three multi-hop QA benchmarks and two language models, EIR improves Answer F1 over direct interleaved reasoning by 8.27 to 32.79 points. Averaged across those settings, it reaches 66.89 Answer F1 with 4.85 model calls per question, versus 47.52 with 35.29 calls for Agentic SSR and 58.56 with 12.41 calls for oracle-stopped Reflexion.

The design does not make retrieval-time reasoning dispensable. That reasoning still determines which documents are read. If evidence is never acquired, the clean final-answer stage cannot reconstruct it. The operational lesson is therefore narrower: preserve evidence independently from agent beliefs, diagnose whether a bad output reflects missing evidence or bad inference over available evidence, and spend retry budget accordingly.

A wrong answer does not always require another trajectory

Suppose an agent has searched several documents, found the facts needed to answer a question, and then reaches the wrong conclusion. A common response is to verify its reasoning, refine the intermediate steps, or run another retrieval-and-reasoning attempt.

That repair can be expensive because it assumes the trajectory itself must be repeated. Yet the failure may occur later: the retrieved material is adequate, while the agent’s provisional interpretation of that material is not.

This distinction matters whenever latency and inference calls are part of the system budget. If evidence acquisition succeeded, rerunning retrieval may add cost without addressing the actual failure. EIR tests whether the system can instead retain what was found while discarding what the agent thought about it.

EIR keeps source evidence outside the agent’s mutable beliefs

During retrieval, EIR maintains two kinds of state.

The first is an exact evidence store. Every sentence read from a selected paragraph is retained as source material rather than rewritten into the model’s own representation.

The second is mutable retrieval state: a requirement list tracks information still needed, while a conclusion list records claims currently considered supported. A model call can update these lists and choose the next retrieval action jointly, so maintaining them does not require a separate model call.

The decisive separation comes after retrieval stops. EIR makes one fresh call to the same underlying language model using only the original question and accumulated source sentences. The requirement list, conclusion list, intermediate reasoning and provisional answer are withheld.

This creates an evidence-inference boundary. Retrieval-time reasoning still influences what enters the evidence store, but it cannot directly enter the final answer prompt. A bad provisional conclusion therefore survives only indirectly, if it caused the agent to retrieve the wrong evidence or stop before acquiring enough of it.

That architecture is more consequential than adding another verification prompt. It divides two failure modes that are otherwise entangled: failure to acquire the necessary evidence and failure to infer correctly from evidence already acquired.

The accuracy gain comes with much less corrective compute

The main evaluation uses matched sets of 1,000 questions from HotpotQA, 2WikiMultiHopQA and answerable MuSiQue with Haiku 4.5 and GPT-4.1 Mini.

EIR beats Direct in all six benchmark-model combinations. The gains range from 8.27 Answer F1 points on Haiku 4.5 HotpotQA to 32.79 points on GPT-4.1 Mini 2WikiMultiHopQA, and all six differences are reported as statistically significant.

The aggregate comparison makes the compute trade easier to see:

This does not mean EIR is the cheapest method. Direct uses fewer calls. The comparison that matters is different: when a system is already considering corrective reasoning or another trajectory, EIR achieves higher average accuracy than the tested correction methods while using substantially fewer calls.

The Reflexion comparison also needs a qualification. Its reported oracle stopping procedure uses the benchmark reference answer to decide whether another trial is necessary. That makes it useful for evaluating the value of additional trials, not a deployable stopping mechanism.

Corpus search suggests the separation survives a harder retrieval environment

The main benchmarks provide small candidate paragraph pools. The paper therefore adds an extended HotpotQA evaluation in which agents search 5,233,329 Wikipedia introductory paragraphs using BM25.

This is a transfer test of the retrieval environment rather than a new question sample: it reuses the same 1,000 HotpotQA questions per model.

With GPT-4.1 Mini, EIR reaches 61.67 Answer F1 versus 56.99 for oracle-stopped Reflexion and uses 7.61 calls per question versus 27.92. With Haiku 4.5, EIR scores 61.35 compared with Reflexion’s 62.21; the 0.86-point difference is statistically inconclusive.

A particularly useful saved-run analysis replaces the answer submitted during retrieval with EIR’s fresh evidence-only answer. Answer F1 rises by 21.20 points for the relevant GPT-4.1 Mini runs and 6.88 points for the relevant Haiku 4.5 runs. That directly supports the paper’s central mechanism: in some failures, the evidence was already present and the inference needed to be reconstructed rather than the whole trajectory replayed.

Remaining errors turn evidence sufficiency into the harder systems problem

EIR does not remove the need for good retrieval.

Across the 6,000 main evaluation runs, 2,758 EIR answers fail Exact Match. Of those, 1,248 had missed at least one annotated supporting paragraph. The remaining 1,510 were wrong despite having read all annotated supporting paragraphs.

The paper’s missing-evidence intervention helps separate these cases. When omitted supporting paragraphs are supplied to failures that had missed annotated support, 26.84% become exact matches. Simply repeating the final-answer call over unchanged evidence recovers only 1.36%.

That result shifts attention from generic “reasoning retries” toward diagnosis. Some failures need a better inference pass. Others need additional evidence.

But indiscriminate retrieval is not free either. Allowing an additional retrieval decision for every question raises aggregate Answer F1 from 66.89 to 69.27, while average calls rise from 4.847 to 6.654. The intervention also turns some previously exact answers into non-exact ones.

More retrieval is therefore not a monotonic repair strategy. The unresolved engineering problem is deciding when the evidence store is sufficient.

Cognaptus inference: evidence and belief should be different system objects

For enterprise RAG and agentic research systems, the paper supports a useful architectural distinction even though it does not directly evaluate enterprise deployments.

The source record can be treated as durable state: exact retrieved content, provenance and citations remain available for inspection. The agent’s requirements, hypotheses and provisional conclusions remain mutable state that may change as the task progresses.

That separation changes the recovery decision. When an answer is challenged, the system can first ask whether the evidence store is deficient. If the evidence appears sufficient, a fresh inference pass may be preferable to replaying search and reasoning. If critical evidence is absent, spending compute on retrieval is more defensible than repeatedly asking the model to reason over the same incomplete record.

It also creates a clearer audit surface. An operator can inspect what the system actually retrieved separately from what the model inferred from it. The paper does not test governance outcomes, but the architecture makes those two records technically distinguishable.

The evidence supports a compute architecture result, not general agent reliability

The empirical evidence is strong within the tested settings: matched questions, fixed model snapshots, paired statistical comparisons, multiple ablations and a large-corpus retrieval extension.

Its scope is still narrow. Only two model backbones and three primary multi-hop QA benchmarks are tested. Each model-benchmark configuration is evaluated once rather than through repeated experimental replications. The full-corpus experiment changes the retrieval environment but not the HotpotQA question sample. Complete-method comparisons also differ in interaction procedures, so they do not isolate every architectural component independently.

Most importantly, successful retrieval does not guarantee correct inference, and EIR cannot recover information that was never retrieved. The authors explicitly do not establish reliability for high-stakes decision-making.

The useful conclusion is therefore specific: before paying for another agent trajectory, determine whether the system needs new evidence or merely a clean inference over evidence it already has. EIR shows that those are different problems, and treating them as different problems can materially change both accuracy and test-time cost.

Cognaptus: Automate the Present, Incubate the Future.