Predicting the Experiment Is Easier Than Knowing When to Trust the Prediction

TL;DR for operators

SciPredict finds that frontier LLMs predict outcomes of recent natural-science experiments with roughly 14-26% accuracy, compared with about 20% for domain experts. That headline can make model performance look surprisingly competitive. It should not be read as evidence that these systems are ready to decide which experiments can safely be skipped.

The operational difference is that human experts have a useful signal for when they are likely to be right. Their accuracy rises from roughly 5% on tasks they judge infeasible to predict without experimentation to roughly 80% on tasks they judge feasible. Model accuracy, by contrast, remains weakly related, non-monotonic, or essentially flat across the models’ own confidence, difficulty, and feasibility ratings.

Context also requires control. Expert-curated background knowledge improves model accuracy by about three percentage points on average. Letting the model generate its own background knowledge often provides no benefit and can reduce accuracy.

For R&D organizations, this supports a bounded role: use model predictions as another input to experiment prioritization, with an independent reliability check and human review. Do not use model confidence as the approval threshold, and do not evaluate scientific systems only on multiple-choice questions when the deployed task requires open-ended or numerical predictions.

A research team needs to know which prediction deserves money

Experimental portfolios are allocation problems. A research group may have ten plausible hypotheses, limited instrument time, finite reagent budgets, and only enough staff capacity to pursue a subset. A prediction system becomes valuable when it helps distinguish the experiments that need physical validation from those whose outcomes can already be inferred with sufficient reliability.

Being right occasionally does not solve that problem. A predictor that is correct one time in four but cannot tell decision-makers which quarter contains the correct answers gives a program manager little basis for deferring experiments or concentrating resources elsewhere.

The stakes therefore extend beyond prediction accuracy. Before a forecast can influence laboratory spending, the user needs some reliable indication of when that forecast is likely to hold.

SciPredict directly tests that distinction.¹

SciPredict tests empirical prediction rather than scientific recall

The benchmark contains 405 expert-curated tasks drawn from empirical studies that first appeared after March 31, 2025. The tasks cover 33 specialized subfields across physics, biology, and chemistry and ask for multiple-choice, free-form, or numerical predictions about actual experimental outcomes.

Fifteen LLMs are evaluated zero-shot with web search disabled. A separate group of domain-matched human experts answers under comparable conditions. The study also changes which scientific background information is provided and asks both models and humans to rate confidence, difficulty, and whether the outcome appears predictable without physically running the experiment.

That setup matters because SciPredict is not principally asking whether a model remembers scientific facts. It asks whether those facts, the described experimental conditions, and the model’s reasoning are sufficient to anticipate an unseen empirical result.

Across models, accuracy lands between approximately 14% and 26%. Human experts average about 20%. Some frontier models therefore exceed the aggregate human baseline.

That comparison is real, but it is not the result an R&D manager should use to authorize autonomous experiment selection.

Human-level average accuracy does not produce a human-like reliability signal

Human experts are not especially accurate overall. Their advantage appears when the benchmark asks them whether a result can reasonably be predicted without running the experiment.

For tasks humans judge infeasible, their accuracy is roughly 5%. For tasks they judge feasible, accuracy rises to roughly 80%. Their assessment separates cases with radically different probabilities of being correct.

The models do not show an equivalent pattern. Accuracy does not reliably increase as self-reported confidence rises or as the model declares a task easier or more feasible. In operational terms, their own ratings provide a poor ranking of which forecasts should be trusted.

This is the relevant form of calibration for experiment triage: not merely producing a confidence number, but producing a reliability signal that corresponds to observed correctness.

For a scientist deciding what to test next, a 25% predictor with reliable case-level uncertainty could potentially be more actionable than a somewhat more accurate predictor whose errors cannot be identified in advance. SciPredict does not establish that current models possess that first capability.

More context helps only when someone controls what enters the context

The study’s background-knowledge experiments probe another common assumption: if a model struggles with scientific prediction, provide more scientific context.

When experts select the relevant background knowledge, accuracy improves by about three percentage points on average, with reported model-level gains of roughly 1.2 to 5.8 percentage points. But when models generate their own background material, performance is often unchanged or worse. Combining self-generated material with expert context generally fails to improve on expert context alone.

The result becomes more informative when compared with Appendix C, Table 7. On separate checks of the required background knowledge, evaluated models answer between about 72% and 95% correctly, with many above 90%. The models therefore already possess much of the relevant factual knowledge.

Yet explicitly placing appropriate facts in context still improves prediction.

The likely mechanism is not simply knowledge acquisition. Expert context helps determine which existing facts and assumptions deserve attention for this experiment. Self-generated context can instead introduce irrelevant information, unsupported assumptions, or misleading premises.

For organizations building retrieval systems around scientific models, Cognaptus would therefore treat context provenance and selection as controlled parts of the decision process. Increasing context volume is not equivalent to improving the evidence available to the predictor.

Multiple choice can make scientific forecasting look easier than deployment

SciPredict also finds a consistent format gradient: multiple-choice tasks are generally easiest, free-form tasks harder, and numerical prediction hardest.

The matched MCQ-to-free-form experiment is especially useful because it functions as a sensitivity test rather than a second benchmark thesis. The researchers rewrite multiple-choice questions so that the underlying scientific problem remains comparable while the answer options disappear. Accuracy systematically falls.

That supports a recognition-versus-generation interpretation. Answer choices constrain the possible outcome space. An open prediction requires the system to construct and commit to the outcome itself, while numerical tasks require further precision.

The paper’s alternative free-form judging and decoding experiments serve as robustness checks: they test whether evaluation details are driving the reported findings. They do not establish that the systems would remain reliable in live laboratories with retrieval, tools, iterative experimentation, or human-model interaction.

Error analysis points in the same operational direction. Incorrect responses are dominated by factual or extraction problems and logical or reasoning flaws, rather than simple misunderstanding of the requested output. False certainty also appears among the scientific-rigor failures. The primary problem is therefore not merely interface compliance.

What changes in an R&D prioritization workflow

The paper directly supports a relatively conservative deployment architecture.

A research team can use an LLM prediction as one feature in deciding which hypotheses merit further investigation. Expert-curated retrieval can improve that signal modestly. But the benchmark does not support using the model’s own confidence or feasibility rating to decide that physical validation is unnecessary.

Cognaptus would go one step further in interpretation. If the decision being automated is which experiments receive scarce laboratory resources, reliability estimation should be separated from prediction generation. That reliability estimate could come from human feasibility review, independently validated calibration methods, empirical similarity to previously tested cases, or another mechanism whose relationship to correctness has actually been measured. SciPredict does not determine which alternative is best; it shows why the model’s self-report is currently insufficient.

Procurement tests should also resemble the intended deployment decision. A vendor score dominated by multiple-choice scientific questions provides weak evidence for a workflow that requires scientists to forecast quantities, describe expected outcomes freely, or determine whether an experiment can safely be deferred.

The benchmark supports triage research, not skipping experimental science

The evidence is substantial within its stated setting: 405 expert-curated tasks, 15 models, domain-matched human comparisons, five context conditions, three evaluation trials, format tests, alternative judging, and decoding robustness checks.

Its external boundary is narrower. The benchmark covers physics, biology, and chemistry, not engineering or computational fields. Chemistry is particularly difficult, including for the human baseline. The task set is static, web search is disabled, and expert curation determines which background information is treated as relevant. A live research agent with retrieval, simulation, instruments, iterative tool use, and opportunities to revise predictions is a different system.

Those differences could improve performance. They could also introduce additional failure modes. SciPredict does not resolve that question.

What it does establish is a more demanding criterion for scientific prediction systems. Increasing average accuracy is only one part of progress. A system that influences experimental budgets must also identify which predictions warrant action, operate on controlled evidence, and remain reliable when the answer cannot be selected from a prepared list.

Until those capabilities are demonstrated together, the defensible role for LLM experimental predictions is reviewed prioritization rather than autonomous substitution for physical validation.

Cognaptus: Automate the Present, Incubate the Future.