A Novel Hypothesis May Still Follow an Old Direction

TL;DR for operators

An AI research assistant may produce proposals that look substantially different from existing papers without identifying fundamentally new research directions.

A study comparing 524,308 LLM-generated hypotheses with published scientific research finds that generated ideas have 15–23% lower median directional novelty, even though most evaluated models produce hypotheses farther from their nearest published neighbors.

For R&D leaders, this changes how automated ideation should be evaluated. Semantic distance can help identify unusual candidates within an established research program. It should not, by itself, justify claims that a system has discovered a new research direction.

The tested remedies—more literature context, peripheral-paper retrieval, repeated agent critique, and unconventional prompting—do not consistently close the gap. Research teams should measure conceptual departure separately and require scientific validation before treating a generated proposal as a substantive discovery.

The proposal list grows, but does the research agenda change?

Consider an R&D team using an AI assistant to identify opportunities beyond its current portfolio. The assistant reviews recent literature and generates dozens of hypotheses. Many differ substantially from the papers supplied as context.

The team now faces a selection problem. Should it interpret those differences as evidence of new research directions, or as variations within familiar approaches?

In Large Language Models Extend Known Research But Do Not Create New Research Directions, Zhihua Liu and colleagues examine precisely this distinction.¹

Their central finding is that generated scientific proposals can appear novel when compared with individual papers while remaining aligned with established conceptual directions.

The difference matters whenever a research organization expects an AI system to do more than expand its existing list of experiments.

Distance from existing papers is not the same as directional novelty

The authors evaluate novelty using three measurements derived from scientific-document embeddings.

Nearest-paper distance measures how far a hypothesis lies from its closest published research neighbor. A high value indicates that the proposal does not closely resemble any individual reference paper.

Topic-centroid distance measures how far a hypothesis lies from the center of its research topic. It identifies proposals positioned away from the topic’s typical research content.

Neither measure determines whether a hypothesis introduces a substantially different conceptual direction.

The paper therefore introduces orthogonal novelty score (ONS). Scientific titles and abstracts are represented as vectors, and existing literature defines a conceptual subspace within that representation. ONS measures the proportion of a hypothesis’s embedding variation that falls outside this literature-spanned subspace.

A proposal can move a considerable distance along directions already represented in the literature. Its nearest-paper distance may increase substantially while its orthogonal component remains relatively small.

This is the measurement problem at the center of the study: distance captures displacement, while ONS attempts to capture departure from established directions.

ONS remains an embedding-based proxy. It does not determine whether the underlying scientific hypothesis is correct, feasible, or genuinely original.

The directional novelty gap appears across models and disciplines

The primary comparison covers approximately 26.9 million published papers from 2005–2026, organized into 1,325 research topics across six natural-science fields.

The authors use SPECTER2 embeddings and compare published papers with 524,308 hypothetical titles and abstracts generated by eight LLMs from the Gemini, GPT, Qwen, and DeepSeek families.

Figure 2 shows a consistent asymmetry.

• Generated hypotheses have 15–23% lower median ONS than published papers from the same topics.
• Their within-topic ONS win rates are 18–28%. This means a randomly selected generated proposal has only an 18–28% chance of scoring higher on ONS than a randomly selected published paper from the same topic.
• The directional novelty deficit appears in 86–97% of the 1,325 topics for each model and in at least 84 of 86 subfields.

Yet most models generate hypotheses farther from their nearest published neighbors than real papers are from theirs. On distance from the topic centroid, generated proposals generally underperform, with median win rates of 35–48%.

These measurements explain why an evaluation based only on similarity to the nearest paper can give a misleading impression of conceptual exploration.

The cross-disciplinary results also matter. Medicine comes closest to published-paper performance on ONS, but its reported median win rates remain below parity at approximately 30–41%.

Larger models do not consistently improve ONS within the evaluated families. Gemini’s larger-model comparison declines, GPT and Qwen show no statistically meaningful improvement, and DeepSeek improves without reaching the published-paper baseline.

The principal finding is therefore not confined to one model, one discipline, or one notion of distance.

More context and stronger critique do not reliably change the direction

The authors conduct follow-up experiments using Gemini-3.1-Flash-Lite and GPT-5.4-nano across 86 selected research topics.

These tests examine whether changing the generation process improves directional novelty. They are interventions on system configuration and sensitivity checks, not independent demonstrations of scientific discovery.

The peripheral-retrieval experiment is particularly informative. Selecting papers farthest from the topic centroid produces the strongest directional novelty among the tested selection strategies. Nevertheless, ONS win rates remain 39% for GPT-5.4-nano and 45% for Gemini-3.1-Flash-Lite, without establishing a general parity result.

The critique experiment shows a similar distinction. Relative to first-turn outputs, ONS improves by 5–8% for Gemini-3.1-Flash-Lite and 2–4% for GPT-5.4-nano. These are improvements within the generated-output distribution, not evidence that the system has caught up with published research.

Persona results are also instructive. Wild ideator prompts produce lower ONS than conservative prompts in direct same-turn comparisons for both models. Wild judge prompts increase nearest-paper distance more strongly, but that additional displacement does not produce a corresponding ONS improvement.

The pattern suggests that agents can respond to instructions encouraging greater novelty while optimizing a different aspect of novelty from the one the research organization actually wants.

R&D systems need different standards for exploration and discovery

For an R&D leader, the appropriate response depends on the purpose of the research system.

When extending an established program, an LLM can generate candidate variations and help researchers explore alternatives within known conceptual directions. Nearest-paper distance and candidate diversity may be relevant screening signals. The team must still assess feasibility and scientific value.

When identifying new research directions, the evaluation should become more demanding. Directional novelty should be measured separately from semantic distance, and promising proposals should undergo expert literature assessment and empirical validation.

For autonomous-research platform developers, Cognaptus infers that these objectives should translate into distinct evaluation criteria rather than one generic novelty score.

A practical evaluation can track three dimensions: how different the proposal is from existing papers, whether it departs from established conceptual directions, and whether its scientific claims survive substantive validation.

The first two can support automated candidate screening. The third cannot be established by embedding geometry alone.

This also changes investment decisions. Before allocating additional compute to larger context windows or longer critique loops, platform teams should test whether those changes improve the specific novelty dimension their customers require. Peripheral literature retrieval may be worthwhile when greater displacement is the objective, but the paper does not establish that it creates fundamentally new research directions.

For research portfolio managers, high-volume ideation should not substitute for explicit ownership of research priorities. AI-generated candidates can inform those decisions without independently validating a strategic shift.

What the novelty measurements cannot establish

The study’s strongest conclusion concerns a measurable gap in scientific-text embedding space.

Generated hypotheses are titles and abstracts, not experimentally executed studies. Published research is also a selected and developed population. Comparing those populations does not isolate human and machine creativity under equivalent conditions.

ONS depends on SPECTER2 and on how adequately the available literature represents established conceptual directions. A low score does not mean a hypothesis is useless; a high score does not guarantee a discovery.

The authors suggest that LLM training may favor recombination and extrapolation because existing scientific knowledge is more strongly represented than underexplored conceptual directions. That explanation is plausible but not causally demonstrated by these experiments.

The intervention results also cover two models and 86 selected topics, rather than every possible research-agent architecture. Specialist scientific AI systems and experimental-feedback workflows remain outside the tested scope.

Consequently, the paper does not prove that LLMs are incapable of making scientific discoveries. It shows that the evaluated systems underperform published research on a particular proxy for opening new conceptual directions.

A broader proposal space is not automatically a broader research agenda

AI research assistants can increase the number and apparent variety of hypotheses available to an R&D organization. That capability remains relevant to scientific work.

But organizations seeking genuinely new research directions need more than a high-throughput generator and a semantic-distance ranking.

The study provides a reason to distinguish exploration within known directions from movement beyond them—and to evaluate each objective on its own terms.

For autonomous scientific research, the critical question is not simply how many ideas a system can generate or how unusual they appear. It is whether those ideas lead to scientifically defensible directions that existing research has not already substantially explored.

Cognaptus: Automate the Present, Incubate the Future.