Running Is Not Correct: Why Scientific Code Needs Graded Verification

TL;DR for operators

For generated engineering code, successful execution should be treated as a feasibility check, not a correctness certificate. In the paper’s 7B ablation, reinforcement learning based only on program validity reaches pass@1 of 0.58 and pass@8 of 0.76. Adding continuous trajectory accuracy raises those figures to 0.71 and 0.84.

The mechanism is straightforward. Generated solver code must first run, return the required output shape, and avoid non-finite values. Programs that clear those checks are then graded by how closely their numerical trajectories match hidden references and, where available, how consistent their outputs are with the governing PDE.

That richer verifier becomes a training signal. Across the eight PDE families used for training, RLVP produces large gains for 3B, 7B, and 14B code models. Under the same direct-generation protocol, the 7B RLVP model also exceeds Claude Sonnet 4.5 on pass@1, pass@8, and numerical error.

For scientific-software and engineering teams, the investment implication is conditional but concrete: where outputs can be executed and continuously graded against trustworthy domain criteria, building the verifier may create more task-specific value than simply moving to a larger general model. The paper does not establish universal PDE-solving ability, however. Transfer to unseen equations remains uneven, and the comparison does not cover retrieval, debugging agents, verifier-guided search, or other inference-time scaffolding.

Code can run and still solve the wrong problem

Suppose an AI system generates a numerical solver. The program executes without an exception. It returns an array of the expected dimensions. Every value is finite.

That is useful evidence, but it says little about whether the computed trajectory is accurate enough to trust.

This gap is particularly severe in scientific computing. A solver can satisfy the software interface while using an unstable timestep, mishandling a boundary condition, or accumulating enough discretization error to produce a materially incorrect result. Binary tests catch some failures, but scientific correctness often has magnitude: two executable solvers can differ substantially in how close they are to an acceptable physical solution.

Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewards introduces RLVP to turn that graded difference into a post-training signal.¹ The paper studies LLMs that generate conventional numerical PDE solver code; it is not training a neural surrogate to directly predict solution fields.

The central design separates two questions that should remain separate in deployment: is this program valid enough to evaluate, and how good is its scientific output once evaluated?

The verifier checks feasibility before measuring quality

RLVP first applies a hard validity gate:

A generated program receives zero physical reward if it fails to execute, returns the wrong tensor shape, or produces non-finite values.

Programs that pass are evaluated again. One diagnostic measures normalized function-space error against a hidden numerical reference over the relevant spatial domain and, for time-dependent problems, over time. In practical terms, the verifier asks how far the generated solver’s computed solution is from a trusted reference rather than stopping at “the code ran.”

The paper converts diagnostic losses into continuous rewards by treating acceptance tolerance as distributed rather than fixed. The resulting score can be interpreted as an expected probability of passing across possible tolerances. A residual-based term can additionally compare the PDE residual of the generated result with that of the numerical reference.

The complete reward is:

This architecture creates a layered verifier. Invalid programs are blocked. Valid programs are ranked by numerical and physical quality.

The distinction is empirically consequential rather than cosmetic. In the paper’s 7B ablation, adding trajectory accuracy to validity-only reinforcement learning increases seen-task pass@1 from 0.58 to 0.71 and pass@8 from 0.76 to 0.84. The authors also report 38% lower numerical error. This experiment is an ablation: its purpose is to isolate the contribution of graded trajectory feedback beyond the binary validity signal.

Continuous scores become parameter updates

The training pipeline begins with supervised fine-tuning on 2,048 examples drawn from a solver bank covering eight PDE families and 71 numerical schemes. That stage helps place executable numerical-program structures within the model’s generation distribution.

RLVP then samples multiple candidate programs and executes them on hidden PDE instances. Their verifier scores allow better and worse outputs within a group to be compared. Using GRPO, those relative scores are converted into parameter updates that shift generation probability toward programs producing better verified solutions.

The relevant change is where verification enters the system. Numerical feedback is not confined to evaluation or an inference-time debugging loop. It changes the model itself.

The training cases vary grids, parameters, output times, initial conditions, and boundaries. The study evaluates three Qwen2.5-Coder-Instruct sizes—3B, 7B, and 14B—on fixed case sets.

Targeted post-training can outweigh raw scale on the bounded task

On the eight trained PDE families, the RLVP models reach:

The base checkpoints are far weaker under the same evaluation. More strikingly, the 7B RLVP model exceeds Claude Sonnet 4.5 under the paper’s common direct-generation protocol: pass@1 is 0.68 versus 0.24, pass@8 is 0.83 versus 0.50, and median best-of-8 numerical error is $6.13\times10^{-4}$ versus $1.15\times10^{-2}$.

The appropriate interpretation is narrow but commercially relevant. The experiment does not show that a 7B specialized model is generally superior to a frontier model. It shows that, for this bounded solver-generation workflow, task-specific post-training backed by a strong executable verifier can outweigh large differences in general model scale.

For an engineering organization, that changes the model-selection calculation. If a workflow supplies reliable numerical references, simulation checks, residual operators, or similarly dense domain feedback, verifier engineering becomes part of the model strategy rather than only part of quality assurance.

Held-out equations expose the generalization boundary

The paper also evaluates ten PDE prompts excluded from both supervised and reinforcement-learning training. These tests are best read as held-out transfer evidence, not as proof of general scientific-code autonomy.

The 7B model shows the strongest aggregate transfer. Its pass@8 rises from 0.28 after supervised fine-tuning to 0.57 after RLVP, while median best-of-8 normalized error falls from 0.145 to 0.00401.

Inspection of generated code is consistent with reuse of numerical motifs learned across the trained tasks: finite-difference stencils, CFL-based timestep control, Runge-Kutta integration, periodic-boundary logic, and related components appear in combinations suited to some unseen equations.

But the transfer is selective. Heat, 2D advection, advection-reaction-diffusion, and Cahn-Hilliard improve strongly, while advection-diffusion, Darcy-reaction, and KdV remain difficult. At 14B, held-out pass@8 actually falls from 0.60 after SFT to 0.50 after RLVP, even though pass@1 improves.

The evidence therefore supports recombination of useful numerical structure on related tasks, not an assumption that reinforcement learning across several PDE families automatically yields a general PDE solver generator.

The business case starts with verifier economics

What the paper directly shows: On a structured PDE benchmark, hybrid validity-plus-continuous verification can provide a substantially stronger post-training signal than binary validity alone. It can also make relatively compact code models highly competitive under a controlled direct-generation evaluation.

Cognaptus inference: Teams building AI for simulation, quantitative engineering, numerical analysis, or other executable technical workflows should evaluate verifier construction as a first-class capability investment. Where correctness can be graded continuously, a verifier can support training, model selection, regression testing, and deployment controls from the same domain-grounded evidence.

This can alter deployment economics. A smaller specialized model may be attractive when local execution, predictable cost, privacy, or throughput matter—provided its outputs can be evaluated with criteria strong enough to justify specialization.

What remains uncertain: RLVP depends on trustworthy numerical references, and residual-based rewards require usable residual operators. The training scope covers eight structured PDE families rather than complex geometries, adaptive meshes, stochastic systems, or broad multiphysics. Solver-bank validation uses convergence tests, manufactured solutions where available, and published-reference cross-validation rather than formal proof of every implementation. The paper also reports fixed seeds without confidence intervals from repeated independent training runs.

Finally, the frontier comparison excludes retrieval, automated debugging, verifier-guided search, test-time scaling, and richer agent architectures. Organizations choosing between specialized local models and frontier APIs should therefore compare complete production systems, not transfer the paper’s direct-generation ranking to a different inference stack.

Better scientific-code models begin with better judgments

RLVP’s strongest contribution is not that reinforcement learning can make an LLM write more PDE code. It is the demonstration that the quality of the training signal changes materially when scientific correctness is represented as a graded executable quantity.

For technical AI systems, compilation and interface compliance answer only the first verification question. If the domain can also quantify how wrong a valid output is, that information can influence both post-training and deployment.

The opportunity is largest in workflows where such judgments can be made reliably and repeatedly. The unresolved work lies outside that boundary: equations, geometries, physical regimes, and production inference systems for which the verifier itself becomes uncertain.

Cognaptus: Automate the Present, Incubate the Future.