From Search to Synthesis: Why AI’s Next Leap Requires Structured Thinking

Spreadsheet.

That is where many impressive AI research reports quietly go to die.

A model can browse twenty web pages, produce a polished executive memo, cite three market reports, and still fail at the boring part: comparing numbers, checking whether a table supports a claim, generating the right chart, and then explaining what the chart actually means. The output looks like research. The mechanism underneath is closer to literary confidence with a browser tab.

This is the useful discomfort in Towards Knowledgeable Deep Research: Framework and Benchmark.¹ The paper argues that the next leap for AI research agents will not come from search alone. It will come from systems that can combine unstructured web content with structured data, execute computations, validate figures, and synthesize the results into reports that are not merely fluent, but evidentially grounded.

The distinction matters because “Deep Research” has become one of those phrases that product pages love and users vaguely trust. Many systems now browse, summarize, and produce long-form reports. That is progress. It is also a ceiling. Once the question requires quantitative comparison, table-grounded reasoning, or domain-specific evidence selection, web summarization starts to look like a consultant who read everything except the appendix.

The paper gives this harder task a name: Knowledgeable Deep Research, or KDR. Its proposed system, Hybrid Knowledge Analysis (HKA), is less interesting as another multi-agent framework and more interesting as a statement about where AI research products are heading: from text retrieval to structured reasoning.

The real bottleneck is not finding information, but using the right kind of evidence

Most current research agents are built around a simple operating assumption: if the model can search enough sources and summarize them well enough, it can produce a useful report.

That assumption works for many descriptive questions. It fails when the question asks for analysis over evidence that does not naturally live in paragraphs.

Consider questions like:

• What explains regional differences in ESG investment?
• Which transport markets show the strongest structural shift over the last decade?
• How have policy changes affected a specific sector across different countries?
• Which indicators support a forecast rather than merely decorate it?

These are not just “find the answer” questions. They require several different operations: locate relevant data, choose the right table, compute comparisons, generate supporting figures, interpret the output, reconcile it with textual evidence, and then write a coherent report.

The paper’s central correction is therefore simple: deep research is not one knowledge problem. It is at least two.

This framing is more useful than another generic “AI agents are improving” narrative. The authors are not saying that search is useless. They are saying search is incomplete. A system that reads the web but cannot reason over tables will tend to produce conclusions that sound grounded while avoiding the disciplined work of grounding.

That is a product problem, not just a research problem. Many enterprise AI tools already sit on top of document stores, CRM exports, BI dashboards, KPI tables, regulatory databases, and operational logs. The business question is rarely “can the model summarize this text?” It is “can the model combine the text with the numbers without embarrassing us in a meeting?”

HKA separates the job because mixed evidence creates mixed failure modes

The HKA framework is built around four sub-agents: a Planner, an Unstructured Knowledge Analyzer, a Structured Knowledge Analyzer, and a Writer.

At first glance, that sounds like another agentic workflow diagram. Fine. The world has enough boxes connected by arrows to power a small consulting industry.

The important point is not that HKA uses multiple agents. The important point is what the separation protects.

This separation reflects a practical truth: text and data do not fail in the same way.

Text retrieval fails when the search intent is vague, when pages are noisy, or when summaries compress away important context. Table reasoning fails differently. The system may select the wrong table, misunderstand the schema, write faulty computation code, generate an empty chart, or produce a figure that is visually correct but irrelevant to the question.

A single “search and summarize” loop cannot handle those failures well because it treats every source as another chunk of text. HKA instead assigns structured data its own pathway. That is the architectural lesson.

The Structured Knowledge Analyzer is the paper’s actual business-relevant mechanism

The Structured Knowledge Analyzer, or S.K.A., is the most important part of HKA. It has four steps:

1. retrieve structured knowledge relevant to the subtask;
2. generate computation code based on the table schema;
3. execute the code and retry when execution fails;
4. use a vision-language model to inspect generated figures or tables and produce corresponding insights.

The paper’s most practical design choice is that HKA does not simply paste full tables into a prompt and hope the model behaves. Instead, it represents table data as executable Python objects while giving the code model a compact schema-like description. The model writes computation code against the schema; the actual objects are injected during execution.

That design has two implications.

First, it lowers context pressure. Large tables do not have to be fully stuffed into the prompt, which is a cheerful way to spend tokens and a miserable way to reason.

Second, it turns “data reasoning” into an execution problem. The system can run code, observe errors, retry, and validate outputs. This matters because table-based reasoning is not just language. It is procedural.

The reported failure reductions are worth pausing on. In the code execution step, retry mechanisms reduce execution failure from 31.7% to 0.51%. In the result analysis step, validation and retry reduce failed visual/table outputs from 55.5% to 1.7%. These are not minor polish numbers. They indicate that the naive version of the pipeline is operationally fragile, and that the reliability layer is not optional.

For builders, the lesson is blunt: if an AI product claims to analyze data but has no execution trace, no schema discipline, and no validation loop, it is probably doing interpretive theater.

KDR-Bench evaluates whether a report uses knowledge, not just whether it sounds good

The paper’s benchmark, KDR-Bench, is designed around the same argument. It contains 41 expert-level questions across 9 domains, supported by 1,252 tables and annotated with 41 main conclusions and 261 key points.

The domains include agriculture, politics and economics, energy and environment, finance and insurance, metals and electronics, society, art, technology, and transportation. This range is not just decorative. The task is meant to force systems to combine domain context with structured evidence rather than overfitting to one familiar benchmark style.

The dataset construction process has three layers:

The most important move is in the evaluation design. Traditional report evaluation often asks whether an answer is coherent, comprehensive, deep, and readable. These metrics are useful. They are also easy to game with confident prose.

KDR-Bench adds two more categories: knowledge-centric metrics and vision-enhanced metrics.

Knowledge-centric metrics ask whether the report aligns with the main conclusion, covers key points, and supports those key points using the appropriate tables. Vision-enhanced metrics compile reports into PDFs and use a multimodal judge to compare reports while considering layout, figures, and visual content.

That last part matters because a text-only judge cannot really evaluate a chart. It can see an insertion marker, perhaps a caption, and then politely pretend that the visual evidence exists. Very academic. Very dangerous.

The main result is not “HKA beats everyone”; it is more specific than that

The experimental results are more interesting when read carefully.

On general-purpose metrics, HKA scores 48.4 on average. Gemini scores 50.2, the best among the compared systems. That means HKA does not simply dominate all visible dimensions. It trails Gemini on general report quality as judged by text-oriented criteria.

But on knowledge-centric metrics, the picture shifts. HKA reaches 82.1 on Main Conclusion Alignment and 61.7 on Key Point Coverage. Gemini still has a slightly higher Main score at 82.6, but HKA leads on Key Point Coverage over Gemini’s 58.3. HKA also reports 27.8 on Key Point Supportiveness, while shallow table or hybrid variants of LangChain and ThinkDepth remain around 18.3–21.2.

That distinction is the article’s center of gravity.

HKA is not necessarily the most fluent general report writer in the benchmark. It is stronger where the task requires using the correct structured evidence. This is exactly the difference businesses should care about when moving from “nice report” to “decision support.”

The paper also reports that HKA generates, on average, 5.75 figures from tables and cites 0.98 figures from web pages per report. In transportation, HKA generates 9.50 figures and tables per report, compared with Gemini’s 2.00. In finance and insurance, the gap is smaller because web pages already contain richer structured materials; HKA generates 9.67, while Gemini generates 5.17.

That domain difference is important. It prevents the conclusion from becoming simplistic. Structured reasoning helps most when the relevant evidence is not already conveniently packaged in the surrounding web text. When the web already contains many tables, strong web-based systems can partly close the visual-evidence gap.

The ablation study explains why shallow hybrid search is not enough

The benchmark comparison shows that HKA performs well. The ablation study explains why.

The authors test four variants: removing the Structured Knowledge Analyzer, removing the Unstructured Knowledge Analyzer, removing the reranking step in table retrieval, and replacing compact schema descriptions with table data in the code-generation prompt.

The most interesting ablation is not the one with the largest drop. It is the contrast between shallow hybrid access and HKA’s specialized integration.

The paper modifies LangChain and ThinkDepth to support web-only, table-only, and hybrid search settings. The hybrid versions do not gain much over the table versions, and in ThinkDepth’s case hybrid search can even hinder structured knowledge utilization. That result is easy to misunderstand. It does not mean combining web and tables is bad. It means combining them by dumping both into the same general reasoning flow is weak.

This is a familiar enterprise pattern. Adding a database connector to a chatbot does not create an analyst. Adding a spreadsheet upload button does not create quantitative reasoning. Adding a chart renderer does not create evidence discipline. The architecture has to know when each kind of evidence is needed, how to operate on it, and how to reconcile it with the final narrative.

Otherwise, “hybrid” becomes a nice word for “now the model has two ways to be confused.”

The evaluation reliability tests are guardrails, not a second thesis

The paper includes reliability checks for the evaluation framework. These should not be overread, but they are useful.

First, the authors compare the default judge model with Claude-haiku-4.5-thinking on representative metrics. Rankings are relatively consistent on general-purpose average and main conclusion alignment. For key point coverage, some ordering changes occur among lower-scoring systems, but the absolute scores remain similar.

Second, they run generation three times for selected systems and report standard deviations. HKA’s standard deviation is 0.2 on general-purpose average, 0.2 on main conclusion alignment, and 0.5 on key point coverage. Key point coverage is naturally more unstable because it depends on fine-grained agent choices: which table is retrieved, which computation is executed, and which claim survives into the final report.

Third, they compare multimodal judge preferences with human preferences across 80 report pairs, reporting 86.3% pairwise agreement.

These tests support the benchmark’s internal consistency. They do not eliminate all evaluation risk. LLM-as-judge and MLLM-as-judge methods are still judge-model-dependent. Human preference was checked on a limited sample. The structured resources are curated. But the tests are useful because they show the authors are not simply throwing a judge model at generated reports and calling it science. A low bar, perhaps, but one the field does not always clear with dignity.

The case study shows the workflow, not just the scoreboard

The paper’s case study is useful because it makes the mechanism visible.

For a question in the Art domain, the Planner decomposes the task and invokes both structured and unstructured analyzers. The Structured Knowledge Analyzer retrieves a table about how Brexit-related regulatory changes affected artist mobility in the UK art market, generates a figure, and derives insights. The Unstructured Knowledge Analyzer searches for contextual material about Brexit, artist mobility, EU exhibitions, and international fairs. The Writer then inserts the figure and integrates the insights into the report. Finally, the judge recognizes that the resulting text matches an annotated key point.

This matters because it shows what “grounded synthesis” looks like at runtime:

Question
  → subtask planning
  → table retrieval
  → code execution
  → figure generation
  → VLM validation
  → web-context retrieval
  → written synthesis
  → key-point support

Notice the sequence. The figure is not decoration added after the report. It is part of the reasoning path that produces the report. That is the shift many AI products still underprice.

What this means for AI research products

The paper directly shows that, on KDR-Bench, HKA improves structured knowledge use and multimodal report generation compared with a range of LLMs, closed-source deep research agents, and open-source agent frameworks.

Cognaptus’ business inference is broader but still bounded: the next serious research-agent products will compete less on “how many pages can you browse?” and more on how reliably they move from evidence to computation to narrative.

For enterprise users, this suggests five design priorities.

1. Treat structured data as a first-class source, not an attachment

A report generator that can read documents but cannot operate on tables will be limited to narrative synthesis. That is fine for briefings. It is not enough for market analysis, policy evaluation, financial research, supply-chain monitoring, or operational diagnostics.

Structured sources need metadata, schemas, access controls, table descriptions, retrieval indexes, and execution environments. This is less glamorous than prompt design. It is also where much of the value lives. Annoying, how reality keeps doing that.

2. Make code execution part of reasoning governance

If a system uses generated code to analyze data, the organization needs logs: what code was generated, what data objects were used, what outputs were produced, which retries occurred, and which final claim depended on which computation.

This is not just for debugging. It is for accountability. A business user should be able to ask, “Why does the report say this?” and receive more than a paragraph of retrospective explanation. The system should expose the evidence path.

3. Separate evidence selection from final writing

A strong writer model can hide weak evidence selection. That is the problem.

HKA’s architecture is valuable because it separates planning, retrieval, computation, validation, and writing. In business workflows, this separation makes review easier. Analysts can inspect the table chosen by the system, the computation performed, and the figure generated before trusting the final prose.

4. Evaluate research agents on evidence use, not only output quality

Many organizations test AI tools by reading the final answer and asking whether it sounds reasonable. That is not evaluation. That is vibe auditing in a suit.

A better evaluation set should include known source tables, expected key points, required computations, and checks for whether claims are supported by the correct evidence. KDR-Bench provides a template, even if firms need to build their own domain-specific versions.

5. Treat visual output as analytical content

Charts and tables are often treated as presentation artifacts. In KDR-style systems, they become intermediate reasoning objects. A figure can reveal a pattern, validate a trend, or expose a mismatch between narrative and evidence.

That means visual evaluation cannot be an afterthought. If the report contains figures, the evaluator must understand figures. Text-only scoring will miss part of the product.

Where the paper’s evidence stops

The paper is useful, but its practical boundary is clear.

KDR-Bench contains 41 expert-level questions. That is meaningful for a research benchmark, but small compared with the range of enterprise research tasks. The structured knowledge base is carefully built, with curated table descriptions, human-reviewed questions, and annotated key points. Real corporate data is usually messier: inconsistent fields, missing values, version drift, access restrictions, duplicated dashboards, and business definitions that changed because someone renamed a KPI in 2022 and then left.

The evaluation also relies heavily on LLM and MLLM judges. The authors run reliability checks, including judge comparison, generation consistency, and human preference alignment. Those checks help. They do not make judge-based evaluation equivalent to audited ground truth.

Finally, the paper does not prove enterprise ROI. It does not measure deployment cost, latency, analyst adoption, governance overhead, or failure rates on proprietary databases. HKA is an architectural and benchmark contribution, not a completed business case.

So the correct business reading is not “install HKA and win.” It is this: if your AI research workflow does not have structured-data retrieval, executable computation, validation, multimodal evidence handling, and evidence-aware evaluation, it is probably not ready for high-stakes analytical work.

From search product to analysis system

The industry has spent enormous energy making AI systems better at reading and writing. That was necessary. It was not sufficient.

The next phase is less poetic: schemas, tables, code execution, retries, visual validation, evidence mapping, and benchmark design. This is where research agents become less like verbose search engines and more like junior analysts with supervision, tools, and a paper trail.

The paper’s best contribution is not a single score on a table. It is the architectural reminder that intelligence in business research is not the ability to produce a long answer. It is the ability to decide which evidence matters, operate on it correctly, and explain the result without losing the thread.

Search finds material. Synthesis organizes it. Structured thinking makes it worth trusting.

Cognaptus: Automate the Present, Incubate the Future.