Search Wider or Read Deeper: Where Long-Document Agents Should Spend the Next Token

TL;DR for operators

A document assistant has already found several relevant pages but still cannot support an answer. The next action should depend on why the evidence is inadequate: perhaps another page is missing, or perhaps the answer is already present but buried in a table, region, or cross-page relationship that needs closer inspection.

D2-ScaleAgent turns that distinction into a routing mechanism. Instead of applying a fixed retrieval-and-reasoning workflow to every query, it maintains a shared evidence state, diagnoses the remaining gap, and directs additional computation either outward toward more retrieval or inward toward finer analysis. The strongest architectural signal is an ablation: removing the Verifier-driven loop reduces MMLongBench-Doc accuracy from 52.0 to 44.1, a 7.9-point decline and the largest reported drop among its single-component removals.

For enterprise document QA, the practical idea is not “use more agents.” It is to make the next unit of inference budget conditional on the evidence state. That can support higher accuracy where incomplete evidence justifies extra work without assuming that every query deserves the same expensive workflow.

The cost is explicit. With GPT-4o, D2-ScaleAgent averages 21.4K tokens and 16.22 seconds per MMLongBench-Doc query, while the authors identify additional latency and computational overhead as deployment limitations. Adaptive scaling is therefore a resource-allocation mechanism, not an efficiency guarantee.

A document can be incomplete in two different ways

Suppose an analyst asks a system to reconcile a figure in one section of a report with a qualification buried elsewhere. The system retrieves three plausible pages but still lacks enough support to answer.

One response is to search outward: perhaps another relevant page was never retrieved. Another is to inspect inward: perhaps the needed evidence is already present, but the system has not localized the right region, extracted the relevant detail, or connected it to the other evidence.

These are different failures and call for different computational responses.

Hao Zhang and colleagues formalize that distinction in D2-ScaleAgent: Dual-Dimensional Scaling for Long Document Understanding.¹ They describe missing relevant evidence as a problem of retrieval breadth and insufficient processing of located evidence as a problem of reasoning depth.

The distinction matters because many document pipelines effectively decide their retrieval depth, number of agent calls, or reasoning workflow in advance. Increasing every budget can compensate for some failures, but it spends computation without first asking which resource the current query actually lacks.

The Verifier decides where the next unit of work goes

D2-ScaleAgent organizes its current state in an Evidence Bank containing page-level, region-level, and atomic evidence, together with an assessed completeness score and an unresolved evidence gap:

The architecture then uses a Verifier to inspect that state.

If the problem is insufficient breadth, the system expands retrieval. Queries are decomposed into weighted perspectives, candidate pages are accumulated and rank-fused, and retrieval can continue until the evidence set stabilizes.

If the problem is insufficient depth, computation is routed among three reasoning roles: a Global Surveyor, Region Locator, and Fine-grained Extractor. Their purpose is progressively more detailed inspection of evidence already considered relevant.

The loop ends only when the Verifier considers the evidence sufficiently complete and reports no remaining gap:

“Closure” here should not be read as a proof that the answer is globally correct. It is a system-defined stopping condition based on the Verifier’s assessment.

Nor is the computation unbounded. The experimental configuration retains explicit limits, including up to six decomposed queries, five retrieval rounds, 20 reasoning rounds, three sub-agent retries, and an 8,192-token limit per LLM call. Top-k retrieval is adaptive, but the surrounding system still operates inside fixed ceilings.

The routing loop contributes more than any one specialist

The main benchmark comparison spans six multimodal long-document datasets and four foundation-model backbones. D2-ScaleAgent records the highest average score within every tested backbone block:

This is the paper’s principal comparative evidence. It does not establish universal dominance. With Gemini-3-flash-preview, for example, whole-document VQA scores higher on MMLongBench-Doc, 65.0 versus 63.0, and LongDocURL, 65.4 versus 64.4. The result is best average performance within each backbone block, not a win in every benchmark cell.

The more diagnostic result comes from the component ablation on MMLongBench-Doc. The complete GPT-4o system scores 52.0. Removing retrieval scaling lowers the result to 46.8; removing the Surveyor, Locator, or Extractor produces scores of 46.5, 47.1, and 47.5. Removing the Verifier-driven loop takes performance down to 44.1.

That experiment is an ablation, so its purpose is narrower than the main benchmark table: it helps identify which architectural components account for performance within the tested configuration. It suggests that the mechanism deciding what kind of additional work is needed is especially consequential. It does not independently prove that the same routing policy will be optimal for other document domains or production workloads.

The paper’s extended ablations on PaperTab and UniDoc-Bench play a robustness role. They test whether the component pattern survives beyond the primary MMLongBench-Doc ablation rather than establishing a separate performance thesis.

Better evidence coverage has a measurable bill

The retrieval experiments report improvements in Recall, Precision, NDCG, and MRR on MMLongBench-Doc and LongDocURL relative to the evaluated retrieval baselines. These results support the narrower mechanism claim that attribute-guided, multi-perspective retrieval can improve the evidence pool compared with a single static retrieval step.

But adaptive evidence acquisition requires additional calls.

With GPT-4o, the compute profile looks like this:

The execution-trace case study helps explain where this work goes: the Verifier detects missing cross-page evidence, triggers additional retrieval, and later requests finer-grained extraction before answering. That trace is useful as a mechanism illustration, not as quantitative evidence of general performance.

For enterprise QA, the design variable becomes the evidence gap

Cognaptus inference: the paper suggests a useful control architecture for teams building assistants over reports, contracts, research archives, filings, manuals, or other long multimodal documents.

The affected user is the product or platform team deciding how much inference budget to allocate per request. The decision is whether a query should trigger more retrieval, deeper inspection of existing evidence, or termination. The relevant condition is an evidence state that can be assessed well enough to distinguish missing coverage from insufficient analysis.

Under those conditions, a system can make cost escalation conditional rather than uniform. Easy queries may close quickly. Queries with dispersed evidence can search outward. Queries with already-located but difficult evidence can spend additional compute on localization or extraction.

The shared Evidence Bank also points toward more traceable document workflows because support is represented at page, region, and atomic levels. That structure may be useful where operators need to inspect how an answer was assembled, although the benchmark results do not establish auditability or compliance performance in production.

The deployment boundary is latency, plus trust in the Verifier

Two uncertainties matter before translating the architecture directly into a product.

First, the system buys some of its accuracy with additional inference. Multi-round retrieval and on-demand sub-agents may be acceptable for research analysis or asynchronous document review but poorly matched to interactions with tight response-time requirements. Whether the additional accuracy is worth 16-second-class latency and higher token consumption depends on the user’s task and cost of error.

Second, routing quality depends on the Verifier correctly assessing completeness and identifying the remaining gap. The stopping rule is only as reliable as that assessment. The paper provides strong comparative benchmark and ablation evidence within its tested settings, but it does not establish how reliably those judgments transfer to production document distributions, provider-specific latency regimes, or domains where missing evidence is harder to recognize.

Several benchmark scoring procedures also depend on GPT-4o for answer extraction, correctness scoring, or judging. That does not nullify the reported comparisons, but it is part of the evaluation boundary.

Spend computation according to the failure

D2-ScaleAgent’s most transferable idea is smaller than its full architecture.

When a long-document system lacks enough evidence, “do more work” is not yet an actionable policy. The system needs to identify whether the next unit of compute should search for evidence that is absent or examine evidence that is already present.

The paper turns that diagnosis into a closed-loop routing decision and provides benchmark evidence that the distinction can matter: average performance improves across four tested backbones, while removing the routing loop produces the sharpest component-level degradation in the primary ablation.

For operators, the opportunity is a more explicit inference policy: spend additional retrieval and reasoning where the evidence state justifies it. The remaining engineering question is whether the resulting accuracy gain clears the latency and cost threshold of the actual workflow.

Cognaptus: Automate the Present, Incubate the Future.