The Search Stack Changes the Answer

TL;DR for operators

An AI research assistant’s accuracy cannot be reliably predicted by selecting the strongest language model, connecting a powerful search service, and adding specialized agents. Those components must work together.

In HyperBrowseComp¹, replacing native search with Exa improved GPT-5.6 Sol accuracy from 19.15% to 26.71%, while reducing Gemini 3.7 Flash accuracy from 31.68% to 22.22%. A separate multi-agent configuration completed only 330 of 423 attempts and answered 17.73% correctly overall.

For organizations deploying research agents, the operational unit of evaluation should be the complete model–retrieval–execution system. Test answer correctness, evidence accessibility, completion reliability, latency, and resource consumption on representative business tasks. The benchmark’s deliberately difficult questions expose system weaknesses; their low scores should not be interpreted as ordinary web-search failure rates.

The same search change can improve one model and weaken another

Consider an organization selecting an AI assistant to investigate information scattered across foreign-language websites, scanned reports, videos, and maps. Several configurations seem plausible: a model with built-in search, that model connected to an external retrieval service, or a coordinated group of specialized agents.

The natural expectation is that broader retrieval access should make difficult information easier to find.

HyperBrowseComp complicates that expectation. On the same 423 questions, GPT-5.6 Sol answered 81 correctly using its native search interface and 113 using Exa. Gemini 3.7 Flash moved in the opposite direction, from 134 correct answers to 94.

The difference is substantial enough to change which system appears preferable.

The surrounding search tools, interfaces, and execution procedures constitute a retrieval harness. A harness determines how an agent searches, accesses documents, and incorporates external evidence into its answer.

These results show that the evaluated system is the model operating through a particular harness. They do not establish that Exa intrinsically helps one model or harms another: changing retrieval interfaces can also change tool interactions, accessible sources, and research behavior.

That distinction matters when procurement teams compare standalone model rankings with products assembled from independently selected components.

HyperBrowseComp makes evidence discovery deliberately difficult

The benchmark’s central contribution is its construction of questions that require sustained open-web investigation rather than straightforward factual recall.

The researchers began with 500 manually authored questions and retained 423 across 13 languages. Native or highly proficient speakers created questions directly within their respective linguistic and information environments.

This differs from translating a shared collection of English questions. A translated benchmark can preserve comparable underlying tasks across languages; independently authored questions can instead expose locally specific sources, relationships, and cultural contexts.

Each question requires a concise, publicly verifiable answer. Annotators document acceptable answers, supporting URLs, exact evidence locations, and solution traces. Independent reviewers check interpretation, uniqueness, source accessibility, and temporal specificity.

The evidence requirements are unusually heterogeneous:

These categories overlap. Arithmetic describes a task requirement rather than an evidence modality. Approximately 64.3% of questions are classified as requiring interpretation of at least one non-text modality.

The researchers also screened all 500 candidates using seven language models without internet access. They excluded 54 questions solved by at least two models and another 23 with highly specific answers produced by exactly one model.

This screening limits questions that might be answered without searching, although it cannot distinguish memorization from successful guessing.

Across five models using native search, 244 of 423 questions (57.68%) received no recorded correct answer. Another 49 were solved by exactly one model.

The shared failures indicate substantial remaining difficulty for the evaluated systems. They do not establish which underlying operation failed: source discovery, access, visual interpretation, evidence combination, or answer verification.

Search integration changes accuracy and resource consumption

The benchmark compares five frontier models using provider-native search. Three are also evaluated through Exa-backed search and document-fetching tools.

The matched-question comparisons expose striking reversals.

Source: Section 4.2, Table 2. All configurations were evaluated on 423 questions.

Gemini 3.7 Flash achieves the highest native-search accuracy, while GPT-5.6 Sol gains the most from the external retrieval configuration.

Resource consumption also varies sharply. GPT-5.6 Sol consumes 43.6 million tokens with native search versus 928.8 million with Exa. For Gemini 3.7 Flash, the corresponding totals are 14.1 million and 663.9 million.

Greater token expenditure therefore accompanies both improved and degraded accuracy, depending on the configuration.

These are resource measurements, not directly comparable monetary costs. They also cannot establish whether the additional expenditure reflects better evidence exploration, inefficient interaction, or other implementation differences.

For a team optimizing a research agent, the implication is practical: retrieval policies should be evaluated jointly for correctness and resource use, rather than optimized around retrieval volume or model capability alone.

More specialized agents introduce more execution paths

The researchers additionally evaluate Gemini 3.7 Flash within OWL, a multi-agent browsing framework with management, coordination, web-research, multimodal-research, and synthesis roles.

OWL has access to broader browsing and multimodal analysis capabilities. It also introduces additional interactions, budgets, and failure points.

A terminal execution failure occurs when an attempt ends without a usable completed answer because of a timeout, processing problem, or exhausted resource budget.

OWL encounters 93 such failures among 423 attempts. Of those, 51 are worker-step timeouts and 28 involve model-budget exhaustion.

Its accuracy differs depending on the denominator:

• 17.73% overall: 75 correct answers across all 423 attempts.
• 22.73% conditional: 75 correct answers among 330 completed attempts.

The system makes 11,486 external tool calls and 56,342 model calls across the evaluation.

Its overall accuracy remains below Gemini 3.7 Flash with native search. Nevertheless, the comparison does not isolate whether multi-agent coordination caused the difference. OWL changes architecture, available tools, execution limits, and operating procedures simultaneously.

The operational lesson concerns evaluation design: counting only completed tasks would conceal failures that directly affect users. Accuracy should therefore be reported alongside completion rates and budget exhaustion.

Human researchers and agents solve different questions

A smaller evaluation examines 30 randomly selected Indonesian, Thai, and Vietnamese questions.

Human evaluators, browsing without LLM assistance, solve 15. Gemini 3.7 Flash solves 13 on the same subset. Both succeed on only eight questions.

This incomplete overlap suggests that human review could recover some agent failures, while automated research could potentially succeed where individual human evaluators do not.

But the study does not test a combined human–agent workflow, and its 30-question sample is insufficient to establish general human superiority or the value of a particular escalation policy.

Human attempts also require a mean of 89.50 minutes, including give-ups. Comparing correctness without the associated time and operating conditions would miss a significant deployment constraint.

The supplementary failure analyses identify higher shared-failure frequencies in some visual categories, domains, and languages. These are descriptive patterns. Because language-specific questions were independently authored, differences in scores cannot be attributed solely to language competence.

Enterprise evaluation must follow the intended research workflow

For organizations deploying multilingual research assistants, Cognaptus draws four implementation implications from the evidence.

First, construct tests around actual information needs. International research teams should include questions originating in their target languages, rather than relying exclusively on translated English prompts. Reference answers should have documented source locations and acceptable aliases.

Second, evaluate the assembled system. Compare the intended model, retrieval service, document-access tools, and execution protocol on matched questions. Record whether the workflow reaches the necessary evidence, not just whether it produces a plausible answer.

Third, measure operational completion separately from correctness. Timeouts, tool errors, exhausted budgets, and unresolved questions should remain visible in reported results. Resource use and latency belong beside answer accuracy.

Fourth, define evidence-based escalation. For consequential research decisions, unresolved or insufficiently supported answers can be routed for further search or human verification. HyperBrowseComp motivates testing such policies but does not demonstrate that any particular escalation design is effective.

These recommendations concern evaluation and control procedures. They are not production-performance predictions for the models or frameworks tested.

What the benchmark cannot determine

HyperBrowseComp intentionally concentrates unusually difficult questions. Its scores cannot estimate how frequently an enterprise assistant will fail on everyday searches.

Live-web changes also affect reproducibility. Search rankings, source availability, geographic restrictions, and tool behavior may differ across evaluation periods.

The results are descriptive comparisons of complete systems rather than causal estimates of individual component effects.

Automated judging introduces another uncertainty. A supplementary robustness check regrades 3,384 saved ReAct responses with three independent judges. Agreement exceeds 98.7%, and aggregate accuracy changes by no more than 0.80 percentage points. This supports aggregate scoring stability without validating every individual judgment.

The benchmark consequently provides stronger evidence about observed system-level differences than about the specific mechanisms responsible for each failure.

The research agent is the whole system

HyperBrowseComp shows why selecting a research assistant from isolated model, search-tool, or architecture rankings can be misleading.

Its distinctive contribution is a difficult, auditable collection of multilingual and multimodal research tasks. Its evaluations then demonstrate how substantially performance can change when retrieval integration and execution architecture change.

For enterprise adoption, the most defensible comparison is between complete workflows performing the questions an organization actually needs answered, under explicit reliability and resource constraints.

The model matters. So do the sources it can reach, the tools through which it reaches them, and whether the research process completes.

Cognaptus: Automate the Present, Incubate the Future.