A Successful Run Is Not a Reproduced Result

TL;DR for operators

An agent can finish a scientific simulation, generate plausible figures, and still leave the published claim unverified. PaperBenchX¹ tests this distinction across 93 reproduction tasks. Its leading configuration earns 59.08% average rubric credit but fully reproduces only 13 tasks (13.98%) under the benchmark’s strict scoring rule. Validation is the weakest assessed stage. For R&D leaders, the decision is not simply which agent can operate scientific software; it is which agent can regenerate the specified evidence, demonstrate agreement with the scientific target, and report what remains unresolved. These results describe scoped computational tasks, not the probability of success in scientific research generally.

A completed simulation can leave the central claim unanswered

A research team asks an AI agent to reproduce a published simulation. The agent builds the model, launches the native solver, and returns logs, plots, and numerical outputs. Before those results can inform a design decision or another experiment, the team needs to know whether the geometry, physical assumptions, numerical settings, and measured outputs actually support the paper’s claim.

PaperBenchX: Can AI Reproduce Science Across Disciplines? makes that acceptance question measurable. It assembles 93 expert-defined tasks from 93 published papers across 12 research areas, including photonics, electromagnetics, computational chemistry, materials science, bioinformatics, and embodied AI. The work runs in ten domain-native environments, including HFSS, Meep, Lumerical, Quantum ESPRESSO, and PySCF. These are reproductions of scoped key experiments, not entire publications or physical laboratory experiments.

Partial progress rarely becomes full reproduction

The benchmark evaluates ten model–scaffold configurations on the same 93 tasks. Each submission earns a weighted score across scientific requirements: reconstructing the model, executing the relevant software, and validating regenerated observables. The reported Partial metric averages those rubric scores across tasks; it does not mean that a corresponding percentage of papers was reproduced. Full is an indicator that the task’s weighted score is strictly greater than 90%.

GPT-6-Astra with Codex leads with 59.08% Partial and 13.98% Full—13 of 93 tasks. Across all ten configurations, the mean Partial score is 49.26%, while the mean Full rate is 8.60%. Only 80 of the 930 configuration–task outcomes pass the Full threshold, covering 22 distinct tasks. On 71 of 93 tasks, none of the tested configurations reaches it (Section 4.2, Table 2; Section 5).

This gap matters because weighted progress can be genuine without being sufficient. An agent may satisfy substantial modeling and execution requirements while missing a scientific observable, a necessary variant, or a comparison that establishes the target result. A high Partial score measures coverage of specified criteria, not the strength of the final conclusion taken in isolation.

Validation is weaker than model construction and execution

The stage-level averages locate much of the difficulty:

Among 304 outcomes where both Modeling and Execution score at least 80%, just 53 (17.4%) achieve Full reproduction. Their average Validation score is 58.37%. This conditional analysis reinforces the completion gap: apparently strong upstream work often does not yield a sufficiently verified result. It does not establish validation as the sole causal source of every failure.

Three diagnostic traces make the problem concrete. In smartphone MIMO antenna matching, F-stub antenna S-parameters, and lithium-niobate-on-insulator edge-coupler checks, substantial native execution evidence still falls short of demonstrating the principal scientific targets (Section 5; Appendix C.2). These are illustrative failure analyses, not a representative estimate of how frequently those particular errors arise.

Clean replay tests evidence rather than the agent’s account of its work

A key contribution is the verification contract. The agent submits a runnable package with a reproduce.sh entry point. The evaluator deletes prior generated outputs and mutable state, then reruns the submitted workflow offline within task-specific resource limits. Only newly regenerated logs, numerical arrays, trajectories, plots, and measurements can support rubric credit (Section 3.4; Appendix B.3).

That procedure separates a workflow capable of regenerating results from a folder containing attractive artifacts. The hidden rubric evaluates requirements independently, using deterministic checks where possible and a common GPT-5.6-sol judge for criteria requiring scientific interpretation. Expert-reviewed reference workflows and rubric construction help anchor the evaluation, but do not eliminate judge dependence.

The appendix’s regression and adversarial checks test the verification protocol: cached outputs, corrupt numerical values, missing provenance, unmatched variants, and prompt-injection attempts should not earn inappropriate credit. They are not a separate measurement of the judge’s scientific accuracy. Nor does clean replay prove a physical model is correct; a faithfully rerun computation can reproduce the same mistaken assumptions.

What R&D buyers should ask an agent to demonstrate

For an R&D automation lead deciding whether to accept agent-produced simulations, the appropriate procurement exercise is a small set of representative internal reproduction tasks in the actual scientific stack. Define the target observables, experimental variants, numerical tolerances, and required outputs before evaluating the agent. Then score the complete chain—model specification, native execution, and independent claim validation—rather than ranking agents by how often they finish a run.

The deliverable should include an executable replay entry point, pinned environment and dependencies, inputs and parameter settings, solver and convergence records, raw observables, and reproducible post-processing. Where a resonance, fit, or scientific target cannot be identified, the result should remain explicitly unresolved. Expert review is particularly valuable before expensive sweeps when an early parameter or solver-setting error could propagate into later figures. This is a Cognaptus operational inference from the benchmark design and case evidence; the paper does not measure the cost savings or return on investment of these controls.

A secondary finding also cautions against rewarding activity. Across the ten configurations, trimmed mean turns correlate only weakly with Partial score ($r=-0.14$). In 200 sampled trajectories, native execution occupies 51.8–63.3% of elapsed tool-mediated time. Neither descriptive result shows that more turns cause worse outcomes or that runtime reduction would improve validation. Activity is an input cost; scientifically adequate regenerated evidence is the acceptance criterion.

The rankings are conditional, not model-only verdicts

The experiment compares model–scaffold configurations, not foundation models under a fully crossed design. Two GPT configurations use Codex, while the other eight use Claude Code. Reported Fable 5 and Fable 5.1 results also incorporate Opus 5 fallback submissions when those models refuse tasks. Interpretive criteria share one LLM evaluator, and only 12 of the 93 tasks have public demonstrations; three of these involve commercially licensed software not redistributed with the release.

Different disciplines and resource budgets further limit extrapolation. The results are useful for evaluating agents on similarly scoped computational reproductions, but cannot certify performance on an organization’s proprietary workflows, experimental laboratories, or novel discoveries. Procurement teams should reproduce the evaluation process in matched environments, and audit ambiguous grading decisions, rather than treating the published ranking as a deployment guarantee.

The central acceptance standard is narrower and more demanding than “the solver completed”: Can the submitted workflow independently regenerate the evidence that substantiates the specific scientific claim? If the answer is not yet demonstrated, a completed simulation remains an intermediate result.

Cognaptus: Automate the Present, Incubate the Future.