The 53 Regressions Behind a Better Scientific Model

TL;DR for operators

A scientific assistant can improve its overall score while becoming worse at problems that matter to its users. In a 250-question inertial confinement fusion (ICF) study, the specialized model Prospero gained 10.7 percentage points over its base model but regressed on 53 questions, or 21.2% of the benchmark.¹

The evaluation also finds substantial critical errors in numerical physics and model-dependent benefits from retrieval. Gemma 4 improves markedly when supplied with external scientific evidence; GPT-5.4 does not show a statistically significant improvement.

For teams choosing between stronger base models, retrieval infrastructure, and additional training, the practical sequence is to evaluate existing capabilities, test retrieval, inspect individual regressions, and independently validate calculations and physical assumptions. A higher benchmark score does not establish dependable scientific performance.

A better score can conceal worse scientific decisions

A research organization wants an AI assistant that can interpret technical literature, derive equations, and support calculations. Its existing model handles some problems well but lacks specialist knowledge. Further training appears attractive because a higher domain benchmark score would seemingly justify the investment.

The difficulty is that scientific tasks do not carry equal consequences. An improvement in explaining plasma terminology cannot necessarily compensate for a regression in calculating instability growth or selecting an ignition model. A single aggregate score can conceal these changes.

Radha and colleagues examine this problem through ICF and high-energy-density physics, where plasma dynamics, hydrodynamics, radiation transport, thermodynamics, and nuclear reactions interact. Their study provides a domain-specific evaluation and tests whether specialization or additional retrieved evidence produces meaningful improvements.

The benchmark checks physical correctness, not just plausible explanations

The researchers construct 250 question-answer pairs curated by ICF scientists, with each pair reviewed by two independent researchers. Questions span ten overlapping scientific categories and several response types.

The benchmark evaluates 22 model endpoints, with three planned independently generated answers per question. This corresponds to 16,500 answer-generation calls.

Its evaluation protocol recognizes that a mathematically plausible response may still be physically incorrect. Numerical checks examine tolerances, units, and dimensional consistency. Computer algebra checks symbolic equivalence. Three blinded language-model judges apply question-type-specific rubrics covering correctness, methods, assumptions, and completeness.

Responses containing critical physics errors are capped at 40%, while correct numerical values accompanied by invalid or contradictory derivations are capped at 70%. Invalid responses receive zero. Scores are aggregated across attempts and questions, with question-level bootstrap resampling used to estimate uncertainty.

This design is the paper’s principal measurement contribution: it evaluates whether a model can produce a scientifically defensible answer, not simply recognizable technical language.

The aggregate rankings still provide a useful starting point:

Yet numerical calculation is the hardest question type. Claude Sonnet 5 leads numerical performance with 58.9, while critical-error rates across reported numerical responses range from approximately 58% to 71%, depending on the model.

These scores describe performance under the study’s evaluation protocol. They are not calibrated probabilities of correctness or estimates of reliability in operational scientific workflows.

Prospero gains 10.7 points but loses ground on 53 questions

The researchers develop Prospero from Llama-3.3-Nemotron-49B-Instruct-128K through two training stages.

First, domain-adaptive pretraining (DAPT) continues training on approximately 1.3 million deduplicated scientific documents, totaling about 17 billion tokens. Supervised fine-tuning (SFT) then uses 1,048,592 examples, combining broader mathematical and STEM reasoning with ICF-specific material.

The resulting model improves its aggregate ICF score by 10.7 percentage points, approximately 36% relative to its base model.

The improvement is uneven across tasks:

• Qualitative reasoning: +18.4 points
• Equation formulation: +11.3 points
• Symbolic derivation and reasoning: +6.4 points
• Numerical calculation: +4.4 points

The smaller gain in numerical calculation is consequential because quantitative physics is already the weakest capability identified by the benchmark.

Question-level comparisons provide an even sharper picture.

The specialized model is better on most questions, but its improvement is not uniform. For a scientific team, the deployment decision therefore depends on which questions regressed, their severity, and their relevance to actual work.

There is also an attribution boundary. Prospero combines DAPT and SFT without separate experimental ablations isolating either stage. Intermediate pretraining reportedly degraded broader benchmark performance and increased unanswered responses before later SFT recovered performance.

The reported 10.7-point gain establishes the result of the combined training process. It does not demonstrate that DAPT alone was necessary or beneficial.

Retrieval helps Gemma 4 substantially, but not GPT-5.4 significantly

Training model weights is not the only way to improve scientific answers. Models can also receive relevant scientific material during inference.

The study compares closed-book answering with evidence supplied by URSA and Open Deep Research. These are additional evaluation conditions, not part of the main closed-book benchmark ranking.

For Gemma 4, the improvement is substantial: 13.8 points with URSA and 14.7 with Open Deep Research.

The approximately 0.94-point difference between the two retrieval methods is not statistically established. Its reported paired 95% confidence interval runs from −3.70 to +5.80 points.

GPT-5.4 experiences much smaller observed gains, neither statistically significant.

These comparisons suggest that access to additional evidence can compensate for some knowledge limitations, particularly in a smaller model. They do not establish that retrieval improves every scientific reasoning task or that more evidence resolves errors in calculation and model selection.

Retrieval also has infrastructure trade-offs. In the reported comparison, Open Deep Research is approximately twice as fast per response as URSA while consuming about 3.4 times as many tokens. Organizations evaluating retrieval should measure answer quality, latency, and token consumption together.

Knowing an equation is not knowing when it stops applying

The paper’s Rayleigh-Taylor instability stress test exposes a failure not captured by general scientific fluency.

In the linear regime, perturbation growth follows a relation involving acceleration, density contrast, and wavelength. For the specified initially stationary interface, the linear solution is proportional to $\cosh(\gamma t)$.

But sufficiently large perturbations enter nonlinear behavior. Continuing to apply the linear solution at late times can produce an inappropriate amplitude estimate.

None of the four leading benchmark models correctly calculates the nonlinear late-time amplitude using the provided reference models. Claude Sonnet 5 and Inkling Small recognize nonlinear behavior but do not complete the appropriate calculation. GPT-5.4 and Nemotron Ultra fail to account for the nonlinear transition in their reported solutions.

This appendix stress test examines physical-regime selection, a distinct requirement from recalling the governing equation.

An interactive follow-up with targeted prompting and additional reasoning tools allows GPT-5.6Sol to reconstruct a piecewise nonlinear solution. That result illustrates the potential of assisted reasoning, not independently reliable autonomous model selection.

A separate proton-boron-11 ignition study provides a complementary exploratory demonstration. GPT-5.6Sol combines supplied references, analytic derivations, Python calculations, optimization, and sensitivity analysis to produce conditional power-balance scenarios.

However, the calculations do not self-consistently resolve all relevant physics, including electron-ion collisions. The resulting ignition thresholds are conditional modeling outputs, not experimentally validated ignition predictions.

Both stress tests locate a practical boundary: scientific agents must validate the applicability of their assumptions as well as the arithmetic performed under those assumptions.

A deployment decision should begin with the failure profile

For organizations building scientific AI systems, Cognaptus draws four operational implications from the benchmark and its supplementary tests.

First, evaluate the workflow before selecting a training strategy. A team supporting simulation interpretation needs different competency thresholds from one answering literature questions. Its evaluation set should represent actual calculations, model choices, and high-consequence failure cases.

Second, preserve question-level regression tests. When adapting an existing model, compare individual answers against the base model. Aggregate gains should not override deterioration on tasks essential to the intended scientific application.

Third, measure retrieval before committing to expensive specialization. The Gemma 4 result establishes a plausible alternative route to higher domain performance. Whether retrieval is sufficient depends on the chosen model, knowledge requirements, and demonstrated behavior on the target workflow.

Fourth, separate language generation from computational verification. Scientific assistants can use deterministic numerical checks, unit validation, symbolic equivalence tests, and explicit physical-assumption checks. An apparently correct output should not bypass scrutiny merely because its explanation is coherent.

These are workflow-design inferences, not deployment interventions experimentally validated by the paper.

What the study cannot establish

The benchmark covers 250 expert-curated questions, a small portion of the possible physical regimes and research workflows in ICF. The questions are obtainable from the corresponding author upon reasonable request rather than supplied as a fully public evaluation artifact.

Automated scoring also has validation limits. Three judges and physics-aware checks strengthen consistency, but the reported direct expert comparison covers only 30 selected assessments. Three attempts per question cannot fully characterize response variability.

The specialization experiment does not separate DAPT from SFT. Retrieval tests cover selected models and two evidence systems. The appendix investigations are scientific stress tests rather than systematic demonstrations of autonomous experimental reliability.

These boundaries limit conclusions about universal scientific capability, production dependability, and experimental prediction.

The practical result is a better evaluation sequence

The study demonstrates that domain specialization can deliver meaningful aggregate gains while degrading performance on a substantial minority of individual problems. Retrieval can offer significant improvements without modifying model weights, but its value depends on the model. Even leading systems remain vulnerable to critical quantitative errors and inappropriate physical assumptions.

For organizations developing scientific assistants, the next investment should follow the observed failure profile: establish a domain-specific baseline, evaluate retrieval against it, examine regressions from specialization, and separately verify calculations and physical regimes.

The objective is not simply a higher score. It is a scientifically defensible answer on the problems the organization actually needs to solve.

Cognaptus: Automate the Present, Incubate the Future.