When Retrieval Isn’t Enough: The DEEPSYNTH Wake‑Up Call

Search is easy to admire because it looks busy.

The agent opens pages. It follows links. It finds PDFs. It writes Python. It returns a neat JSON object, ideally with the confidence of someone who has just discovered government statistics. This is the part of AI demos that makes executives lean forward: the machine appears to have become an analyst.

DEEPSYNTH is useful because it ruins that pleasant little scene.

The paper, “A Benchmark for Deep Information Synthesis,” introduces a benchmark designed not for shallow fact lookup, but for realistic multi-source information synthesis: tasks where an agent must locate data, understand source structure, perform calculations or comparisons, and return an exact structured answer.¹ The benchmark contains 120 tasks spanning countries, domains, document types, and synthesis operations. The answers are not essays. They are JSON-style outputs with verifiable keys and values. No charming prose can save the model when the number is wrong. A cruel but healthy arrangement.

The headline result is blunt: even strong LLMs and deep-research-style agents perform poorly. The best full-benchmark F1 score reported in the main table is 8.97 for o3-deep-research, while GPT-5.2-Pro reaches 8.70. Exact match remains extremely low: GPT-5.2-Pro reaches 6.25%, o3-deep-research 2.50%, and several systems score zero. The softer LLM-judge metric is more forgiving, but still not flattering.

The business lesson is not “AI agents are useless.” That is too easy, and also wrong. The sharper lesson is that retrieval is not the same as research. A useful enterprise agent must manage a chain: plan the search, reach the right sources, operate hostile websites, parse tables and files, calculate correctly, preserve intermediate results, and output the answer in the requested structure. One weak link is enough. DEEPSYNTH is mostly a study of weak links.

The real task is a chain, not a search query

DEEPSYNTH tasks are designed around a simple but important distinction: finding information is not the same as synthesizing it.

A typical benchmark question might ask for one obscure fact. A DEEPSYNTH question asks for an answer that must be constructed. For example, the paper includes tasks such as ranking Canadian polling companies by mean-squared error against election results, calculating average annual change rates for Swiss railway transport categories, identifying ASEAN countries with negative correlations between exports of certain services and ASEAN GDP, or computing airport PageRank values from OpenSky flight data.

These are not “what is the capital of X?” tasks wearing a lab coat. They require several actions:

This is why DEEPSYNTH’s design matters. The benchmark’s tasks require an average of 7.54 intermediate steps, navigation through 4.2 web pages per task, and took annotators about 5.5 hours per task to construct. The benchmark covers operations such as counting and comparing, ranking, trend detection, averaging, correlation, anomaly detection, and filtering. Counting and comparing account for the largest share of tasks at 33.72%, followed by trend detection at 20.93% and ranking at 19.77%.

Those numbers are not decoration. They tell us what kind of work is being tested. DEEPSYNTH is closer to “junior analyst asked to produce a clean spreadsheet-backed answer” than to “chatbot asked to summarize a Wikipedia paragraph.”

That matters for business adoption because many enterprise AI proposals quietly assume the second while selling the first.

The benchmark is strict because business answers are strict

The authors make the answers verifiable by requiring structured JSON-like outputs. That design choice narrows the benchmark, but it also gives the evaluation teeth.

In a memo, a vague paragraph can hide a lot. In a structured answer, every key and value must survive inspection. If a model returns the wrong country name, omits a category, rounds incorrectly, or calculates a number from the wrong base year, the score falls.

DEEPSYNTH uses exact match as the strictest metric: every required key and value must be correct. It also reports precision, recall, F1 over key-value pairs, and an LLM-judge score that allows semantic equivalence and small numerical deviations. The appendix explains why exact match is appropriate here: over 95% of answers are numeric, and the keys correspond to unambiguous factual fields.

For business readers, this is the right kind of harshness. A compliance report, pricing comparison, policy briefing, investment screen, or supplier risk table does not become correct because the model “basically understood the idea.” Approximate correctness is useful for brainstorming. It is not enough when the output feeds a decision.

The gap between exact and soft evaluation is therefore revealing. On the DEEPSYNTH-Dev subset, GPT-5.2-Pro achieves the highest F1 among standalone LLMs at 15.64, while Gemini-Pro-3 leads among standalone models on the LLM-judge metric at 15.0. o3-deep-research reaches the highest LLM-judge score among agents at 20.0. The soft metric sees partial semantic progress. The strict metric still sees a machine that often cannot deliver the exact answer.

That is not pedantry. That is the difference between a useful assistant and a confident intern who copied the wrong cell.

Tool use helps, but it does not magically create an analyst

The obvious response to poor standalone LLM performance is: give the model tools. Web search. Browser simulation. Document processing. Code execution. Agent loops. A clipboard full of little digital prosthetics.

DEEPSYNTH tests this intuition. Tool-augmented systems generally improve, but not enough.

On the full benchmark, o3-deep-research performs best overall in F1 at 8.97, higher than the base o3 result of 3.29. Smolagent with GPT-5 reaches 6.42, and OWL with GPT-4.1 reaches 5.41. These are improvements over some base systems, but the absolute scores remain low. The conclusion is not that tools do nothing. The conclusion is that tools expose the next layer of difficulty.

The ablation study makes this clearer. Using OWL as the full system, the authors remove tool capabilities one at a time. Removing the search toolkit reduces F1 from 5.41 to 3.60, the largest drop among the tool ablations. Removing document processing lowers F1 to 4.90, removing code execution to 4.82, and removing the web browsing toolkit to 4.80.

This test is best read as an ablation, not as a second thesis about which tool is universally most important. Since the baseline is already weak, the measured differences are modest. Still, the direction is useful: search, browsing, document handling, and computation each contribute. The work is multifaceted. A model with search but no reliable document processing will fail on files. A model with code but weak navigation will never reach the right data. A model with the right source but weak synthesis may still produce nonsense with immaculate formatting.

For enterprise AI, this means “we added browsing” is not a research workflow. It is one component in a workflow. A procurement team would not hire an analyst merely because he owns a web browser. The same standard should apply to agents, despite their charming habit of printing logs.

Planning is not optional; it is the missing operating system

One of the strongest signals in the paper comes from the intermediate-step experiments.

The authors test what happens when models receive the ground-truth intermediate reasoning steps, without receiving the intermediate answers. In other words, the model is given a better plan but still has to execute it. Performance rises substantially.

For GPT-4.1, F1 rises from 3.46 to 9.36, and exact match from 0.0 to 5.0. For Smolagent with GPT-4.1, F1 rises from 3.75 to 10.50, and exact match from 2.50 to 10.0. Additional appendix results show the same pattern for o3 and Gemini-Pro-2.5: o3 rises from 3.29 to 12.87, and Gemini-Pro-2.5 from 6.25 to 10.40 when intermediate steps are provided.

This is not merely “chain-of-thought is useful.” It is more operational than that. The benchmark suggests that agents often fail before they have a stable decomposition of the task. They may not know which data source must be found first, which variables must be aligned, which years must be filtered, or which transformation must be applied before ranking.

A business translation:

The tempting enterprise shortcut is to ask one large model to do everything in one pass: “Research this market and return the answer in JSON.” DEEPSYNTH says that shortcut is expensive in a boring way. The agent may look productive while quietly corrupting its own pipeline.

A better workflow separates planning from execution. First, the system should generate a plan with explicit evidence requirements. Then it should validate each source and intermediate result. Only then should it synthesize. This is slower than a demo. It is also less theatrical. Unfortunately, reality has a persistent bias toward the less theatrical answer.

The failure modes are mechanical, not mystical

DEEPSYNTH’s manual error analysis is especially useful because it turns “the model failed” into a map of where it failed.

The authors manually analyze a subset of tasks where OWL with GPT-4.1 made errors. They identify four categories:

The categories are multi-label, so one run can contain more than one error. The pattern is still clear: navigation and synthesis dominate.

That division is important. Navigation errors are not solved by better arithmetic. Synthesis errors are not solved by a shinier search API. Technical issues are not solved by telling the model to “think harder.” The failure modes live in different parts of the system.

The appendix adds a more detailed process evaluation. On 40 DEEPSYNTH tasks, GPT-4.1, DeepSeek-R1, and GPT-5.2 show steep intermediate-step accuracy decay. Step 1 F1 ranges from 4.1 to 11.2. Step 2 ranges from 2.6 to 12.4. By step 5, performance is essentially gone. Error propagation is severe: when a step fails, the subsequent step also fails 91–100% of the time, with recovery rates below 10%.

This is one of the paper’s most business-relevant findings. The agent does not usually recover gracefully. If it chooses the wrong source early, the rest of the pipeline becomes a beautifully reasoned walk in the wrong direction.

The GPT-5.2 process-evaluation detail is also instructive. With web search enabled, it sometimes identifies correct databases, API endpoints, indicator codes, and download URLs, but reports structured failures when the execution environment cannot process JavaScript-rendered tables, binary Excel files, or bulk downloads requiring interactive queries. This is not pure hallucination. It is infrastructure friction. The model can see the door, but the handle is weird.

For enterprises, that distinction matters. Some failures require better models. Others require better connectors, file parsers, API integrations, browser automation, authentication handling, and deterministic data pipelines. Calling all of that “reasoning” is how organizations buy a chatbot and expect a data engineering department.

Regional coverage exposes a second reliability problem

DEEPSYNTH is not only multi-source; it is also multi-regional. The benchmark spans 67 unique countries in the datacard and evaluates performance across regions.

The regional breakdown is uncomfortable. On Africa-related tasks, all reported models in the regional analysis score 0.0 F1. Performance is higher in some other regions: for example, o3-deep-research reaches 12.70 on Asia-related tasks, 10.83 on Europe, 14.43 on Oceanic tasks, and 25.00 on South America, though South America accounts for only 5% of tasks. Smolagents performs strongly on Oceanic tasks at 24.00, but scores 0.0 on South America and Africa in that table.

The paper interprets this as evidence of geographical bias and uneven performance across information ecosystems. That interpretation is plausible, though the sample sizes vary and some regional categories are small. The business implication is still serious: agent reliability is not globally uniform.

This matters for companies operating across emerging markets, cross-border trade, public-sector research, development finance, supply-chain monitoring, and regional risk analysis. A model that performs acceptably on U.S. or European sources may struggle with government portals, local statistical agencies, multilingual PDFs, inconsistent table formats, or lower-resource web ecosystems elsewhere.

The danger is not that the model will always fail. The danger is that it will fail unevenly while presenting the same tone of confidence everywhere.

That is a governance problem. AI systems should not be evaluated only on the countries, languages, and document ecosystems where demos are easiest. If the business process depends on Southeast Asian customs data, African public-health reports, Latin American procurement portals, or local-language filings, the evaluation set should contain those sources. Otherwise, the benchmark measures comfort, not capability.

Best-of-N finds occasional success, not dependable work

The paper also tests whether multiple attempts help. On DEEPSYNTH-Dev, Best@5 improves results: Smolagents with GPT-4.1 reaches 25.0% LLM-judge accuracy compared with 17.5% for GPT-4.1. Tool use appears to introduce beneficial variation across runs.

But self-consistency performs badly. Majority voting at five attempts yields only 5% accuracy for both systems, with low average consistency scores of 0.27. The paper’s interpretation is straightforward: correct answers sometimes appear, but they are rarely the majority.

This is a subtle point with large practical consequences. In many LLM workflows, repeated sampling is treated as a cheap reliability trick. Ask several times, vote, and move on. DEEPSYNTH suggests that for deep information synthesis, the correct answer may be a rare event rather than the center of the distribution.

That makes majority voting a poor substitute for verification. If five analysts produce five different spreadsheet outputs, and four are wrong for different reasons, voting does not magically reveal the truth. It merely gives bureaucracy to the error.

For business systems, Best-of-N is useful only if paired with an external verifier: source checks, deterministic recomputation, schema validation, numeric tolerance rules, and human review for ambiguous cases. Without verification, multiple attempts may increase the chance that the system prints something plausible. That is not the same as increasing reliability.

What DEEPSYNTH directly shows, and what businesses should infer

The paper directly shows that current LLMs and agent frameworks struggle on a benchmark where answers require multi-source, multi-step, verifiable synthesis. It also shows that tool access helps but does not solve the problem; planning support improves results; navigation and synthesis errors are common; early errors propagate; and performance varies by region.

Cognaptus’ business inference is narrower but useful: companies should stop treating “deep research agent” as a finished job role. It is better understood as a component in a controlled research workflow.

A practical enterprise design would look less like one autonomous agent and more like a pipeline:

1. Task formalization Convert the business question into expected output fields, evidence requirements, time periods, formulas, and source constraints.

2. Planning gate Generate and review the intermediate steps before search begins. For high-stakes workflows, require human approval or a deterministic checklist.

3. Source acquisition layer Use search, but also maintain trusted source registries, APIs, downloaded reference files, and domain-specific connectors.

4. Extraction and computation layer Handle PDFs, spreadsheets, web tables, APIs, and code execution through tools that can be tested independently of the LLM.

5. Intermediate validation Check each retrieved value, date range, unit, and transformation before downstream synthesis.

6. Structured synthesis Let the model assemble the final answer, but only from validated intermediate objects.

7. Final verification Enforce schema, key completeness, numeric recomputation, rounding, citation integrity, and exception reporting.

The ROI is not “replace analysts.” That phrase is usually where serious process design goes to die. The ROI is cheaper first-pass evidence collection, better task decomposition, faster cross-checking, and a clearer audit trail for human analysts. In boring departments, this is called productivity. In AI slides, it is often called autonomy, because apparently the word “workflow” does not raise enough money.

Where the result should not be overextended

DEEPSYNTH is strong because it is strict, but that strictness also defines its boundary.

First, the benchmark emphasizes official, data-rich, numeric, and verifiable tasks. That makes it highly relevant to compliance research, policy analysis, market intelligence, procurement comparison, economic monitoring, and due diligence. It is less directly informative for creative strategy writing, open-ended synthesis, qualitative interviews, or exploratory brainstorming where there may be no single exact JSON answer.

Second, exact match is appropriate for DEEPSYNTH’s task design, but not every business question should be evaluated that way. An executive memo can be useful without every sentence mapping to a key-value pair. The important lesson is not that all work must become JSON. The lesson is that when a business process does require exact structured output, approximate narrative competence is not enough.

Third, some comparisons involve systems with different tool capabilities and operating constraints. o3-deep-research, Smolagents, and OWL do not have identical access to code interpreters, browser simulation, document processing, web search, or APIs. The benchmark is best read as a stress test of current agent configurations, not as a permanent ranking of vendors.

Fourth, the regional analysis is important but should be interpreted with care. Some regional categories have limited task counts, and a question can span multiple regions. The direction of the concern is still valid: reliability differs across information ecosystems. The exact magnitude should not be treated as a universal law.

Finally, DEEPSYNTH is a benchmark, not a full operating manual. It diagnoses failure patterns. Building production-grade research systems still requires domain-specific workflow design, source governance, integration engineering, and user-interface decisions. Yes, the boring parts again. They keep winning.

The wake-up call is about coordination

The easiest misconception is that retrieval unlocks research. It does not. Retrieval unlocks the possibility of research.

DEEPSYNTH shows the gap between “the model found something” and “the system produced the correct structured answer.” In that gap live planning, navigation, parsing, computation, synthesis, validation, regional source familiarity, and error recovery. Current agents can perform pieces of this chain. They do not yet coordinate the chain reliably enough to be treated as autonomous analysts for exact, data-heavy work.

For businesses, the useful response is not panic. It is architecture.

Use agents where they reduce search friction. Use models where they help decompose tasks and explain results. Use deterministic tools where arithmetic, parsing, and validation must be correct. Use human reviewers where judgment, accountability, and ambiguous interpretation matter. Above all, inspect the intermediate steps, because DEEPSYNTH’s most practical warning is simple: when the first wrong number enters the pipeline, the final answer rarely heals itself.

Retrieval is a door. Synthesis is the building. Most current agents are still bumping into the furniture.

Cognaptus: Automate the Present, Incubate the Future.