The First Document Should Change the Next Search

TL;DR for operators

A retrieval system can find a useful first document and still be searching incorrectly afterward. That document may reveal a person, relationship, date, product, or policy that makes the missing evidence much easier to specify than it was from the original request.

EviReform¹ makes that change explicit: it reads initially retrieved evidence, generates new queries describing what remains unresolved, retrieves against those queries, preserves part of the original-question signal, and only then uses graph connections to consolidate evidence. The paper’s component tests indicate that this query reformulation accounts for most of the improvement; graph propagation adds a smaller, consistent increment.

For enterprise RAG, the practical design choice is consequently more specific than “add a graph.” When tasks require several documents to support one conclusion, invest separately in changing the search target after evidence arrives and connecting the resulting evidence afterward. The paper provides strong benchmark support for that sequence under its tested retrieval budgets, but not a general guarantee for every production corpus or retrieval stack.

Finding one document can change what should be searched next

Consider a knowledge assistant investigating a contractual obligation. The first retrieved document identifies a subsidiary, an amendment date, or a policy exception that the user’s original wording never mentioned. At that point, continuing to rank the corpus against the original query alone leaves information on the table: the system now knows how to describe the missing evidence more precisely.

That is the retrieval problem EviReform formalizes. In the paper’s terms, the relevance of a passage after observing evidence need not equal its relevance under the original question alone:

The distinction matters because graph retrieval and query revision solve different problems. A graph can help move retrieval signal among connected propositions. It does not by itself guarantee that the system has reformulated the semantic target after learning something new.

EviReform first retrieves propositions for the original question and lets an LLM select evidence whose source passages are then observed. From those passages, another LLM generates up to three residual queries: search requests describing information still required but not yet established.

The residual signals are separately normalized and mixed with the original-question signal. In the main configuration, half the retrieval mass remains tied to the original query:

That design prevents the new, narrower query from simply replacing the user’s original constraints.

Only after this semantic revision does the method spread the combined signal once across propositions sharing entities. The main configuration again uses an even weighting between direct and propagated evidence. Proposition scores are finally aggregated back into one passage ranking.

The ablation says reformulation is doing most of the work

A reader could reasonably see a GraphRAG method and attribute its advantage to better graph traversal. The paper’s 2×2 component study points elsewhere.

With graph propagation already enabled, adding evidence-guided reformulation changes complete-chain recovery at five passages as follows:

Once reformulation is already present, adding graph propagation increases Chain@5 by only 1.2, 1.2, and 2.1 points on the same datasets.

This does not make the graph component irrelevant. It changes its role. Under this architecture and retrieval budget, reformulation appears to acquire much of the missing evidence; shared-entity propagation then helps consolidate evidence reached through the original and revised semantic channels.

Two controls strengthen that interpretation. Repeating the original question for another retrieval round performs materially worse than generating residual queries, so extra retrieval calls alone do not explain the gain. And a control that merely reorders the original candidate pool cannot reproduce the result because it cannot introduce supporting passages that were absent from that pool.

The candidate-pool analysis makes the acquisition effect concrete. Among initially incomplete evidence pools, residual retrieval turns 416 of 459 into complete pools on 2Wiki, 62 of 93 on HotpotQA, and 150 of 533 on MuSiQue.

Complete evidence chains expose failures that Hit@5 hides

Finding any relevant passage is a weak success criterion for tasks whose answer depends on several documents.

On 2WikiMultiHopQA, CatRAG already reaches 99.9% Hit@5, meaning it almost always retrieves at least one supporting passage. Yet its Chain@5—the fraction of questions for which all required supporting passages appear in the first five results—is 72.4%. EviReform also records 99.9% Hit@5, but Chain@5 rises to 94.9%.

HotpotQA shows the same pattern: CatRAG reaches 99.2% Hit@5 and 81.7% Chain@5, while EviReform reaches 99.6% and 93.8%.

That distinction is operationally relevant wherever a conclusion must be supported by a sequence of records rather than one plausible source. Due diligence, compliance investigation, scientific evidence synthesis, and technical support can all fail after retrieving a document that is relevant but insufficient.

The broader benchmark results are consistent with the chain-recovery result. EviReform improves Recall@5 over the strongest metric-wise comparator by 5.00 percentage points on 2Wiki, 2.65 on HotpotQA, and 5.59 on MuSiQue. Using a shared QA reader, F1 improves over GeAR by 3.91, 2.28, and 4.50 points respectively. Paired question-level bootstrap intervals exclude zero for those reported F1 differences.

The Medical evaluation provides additional transfer evidence at the answer level: EviReform reports mean accuracy of 71.75 versus 69.86 for HippoRAG 2, 69.25 for GeAR, and 67.48 for S2G-RAG. Because that benchmark has no gold supporting passages, however, it does not independently reproduce the paper’s chain-retrieval analysis.

The architectural implication is to separate search revision from evidence consolidation

What the paper directly shows: within the evaluated multi-hop benchmarks, evidence-conditioned reformulation contributes most of EviReform’s incremental retrieval gain, while graph propagation adds a smaller positive increment.

Cognaptus inference: enterprise teams building multi-document RAG systems should consider those as separate system capabilities and therefore separate engineering investments.

The first capability asks: given what we have just learned, what exactly is still missing? That component can be inspected through generated residual queries, candidate-pool coverage, and retrieval cost.

The second asks: given evidence reached from several semantic directions, which pieces reinforce or connect to one another? That is where graph structure becomes useful as an aggregation layer.

Separating the two also makes diagnosis easier. If the required document never enters the candidate pool, improving graph ranking may address the wrong failure mode. If the expanded candidate pool contains the complete chain but the final context omits part of it, the remaining problem is selection and consolidation.

MuSiQue illustrates the latter case. After residual retrieval, complete-chain coverage in the expanded pool exceeds final Chain@5 by 14.8 points. The method has found additional evidence that its compact final ranking does not always keep together.

More reformulation is not automatically better

The main EviReform design uses one reformulation round, which is relevant for production systems sensitive to serial latency and LLM-token consumption.

Additional rounds provide little benefit on 2Wiki and HotpotQA. MuSiQue does benefit: continuing beyond the first round raises Recall@5 by 1.74 points and Chain@5 by 2.50 points. The improvement comes with substantially higher feedback-token usage and additional serial requests.

That result argues for a bounded control policy rather than an open-ended retrieval loop. Workloads with longer or more weakly specified evidence chains may justify another round; simpler workloads may not.

The cost table also needs careful interpretation. EviReform averages 2.454 seconds and about 2,974 retrieval-time LLM tokens per query across the three multi-hop datasets. The authors explicitly note that wall-clock comparisons are descriptive because concurrency differs among methods, making token use the cleaner comparison.

The evidence is strong within a defined experimental boundary

The paper combines heterogeneous baselines, shared evaluation conventions, component ablations, matched-query and reranking controls, candidate-pool analysis, paired bootstrap intervals, embedding-model tests, parameter checks, and cost analysis. Replacing BGE-M3 with NV-Embed-v2 leaves EviReform strongest at Recall@5 among the tested graph retrievers on all three datasets, supporting robustness to that embedding substitution.

The uncertainty estimates are narrower than “system-level reproducibility,” however. The 10,000 bootstrap resamples vary evaluated questions. They do not rebuild the LLM-generated index or repeat retrieval-time LLM inference, and each method–dataset combination is evaluated once.

The principal retrieval evidence also comes from three English multi-hop QA benchmarks, using one main propagation rule and one passage readout. The experiments do not establish that reformulation will dominate under arbitrary graph operators, larger feedback budgets, non-English corpora, or production workloads with different document structure.

Those boundaries do not weaken the paper’s architectural result. They define where it has actually been tested.

Retrieval should learn from what retrieval has already found

The operational lesson from EviReform is not that every RAG system needs a more elaborate graph. It is that multi-hop retrieval can change state after the first useful evidence arrives.

If that evidence reveals a bridge that was absent from the original request, the next retrieval instruction should be allowed to change. The paper’s experiments show that making this semantic revision explicit recovers substantially more complete evidence chains, while graph propagation provides an additional consolidation step.

For systems whose job ends after locating one relevant passage, that architecture may be unnecessary. For systems expected to assemble a defensible chain across several documents, it identifies a more precise place to spend retrieval complexity: first describe what remains unknown, then connect what has been found.

Cognaptus: Automate the Present, Incubate the Future.