A Successful Run Is Not a Reproduced Result
TL;DR for operators An agent can finish a scientific simulation, generate plausible figures, and still leave the published claim unverified. PaperBenchX1 tests this distinction across 93 reproduction tasks. Its leading configuration earns 59.08% average rubric credit but fully reproduces only 13 tasks (13.98%) under the benchmark’s strict scoring rule. Validation is the weakest assessed stage. For R&D leaders, the decision is not simply which agent can operate scientific software; it is which agent can regenerate the specified evidence, demonstrate agreement with the scientific target, and report what remains unresolved. These results describe scoped computational tasks, not the probability of success in scientific research generally. ...