TL;DR for operators
A simulator can clear an agent while the physical deployment simultaneously fails one operational claim and satisfies another. In the paper’s 240-trial calibration block, every source-cleared execution was a false clearance for immediate completion, while only 20 of 240 failed on reported state, 42 of 240 failed on observable physical effect, and none failed on the eventual settled postcondition.
SimVerity1 turns that ambiguity into an assurance process. A team declares the property it intends to certify and the read boundary at which it should hold, aligns simulator and deployment traces, and requires an independent physical observation channel that meets predeclared qualification rules. Insufficient evidence remains an abstention rather than being silently treated as success, and a frozen calibration profile can estimate the risk that a source pass becomes a deployment failure.
For release governance, the relevant question is therefore not “Did the simulator pass?” It is “Which property did it pass, when was that property checked, and what independent deployment evidence supports the same verdict?” The reported risk estimates remain local to the evaluated deployment conditions, and under stationary conditions a more granular deployment-specific lookup performed better than SimVerity’s structured predictor.
One simulated task can produce four deployment answers
A familiar release workflow starts with a simulation. The agent completes a task, the benchmark reports success, and the team must decide whether the same workflow can be authorized on physical devices.
The SimVerity evidence shows why treating that pass as one coherent deployment claim can be misleading. In the P1 calibration block, 240 executions that had cleared the source simulator produced four radically different physical results:
| Property being claimed | False clearances | What the result means |
|---|---|---|
| Completion | 240/240 | The deployment had not yet satisfied the declared completion condition at the read boundary |
| Reported state | 20/240 | Software state usually, but not always, agreed with the cleared source verdict |
| Observable effect | 42/240 | The intended physical effect was absent at the declared observation point in a substantial minority |
| Settled postcondition | 0/240 | Every eligible case eventually reached the declared settled state |
The operational problem is temporal. An agent can return before a software state update arrives. Software can update before a device visibly responds. The physical device can lag both yet eventually settle correctly. Collapsing those stages into “task success” removes the information a release team needs to decide whether immediate completion, reported state, physical effect, or eventual outcome is the actual requirement.
Only after that distinction is explicit do the paper’s two central measures become useful. Verdict Fidelity measures how often source and deployment assign the same verdict to the same declared property. False Clearance Risk measures the probability that the deployment fails that property given that the source passed it.
The latter is the more directly actionable quantity for a release gate:
It is not a universal simulator error rate. The property, read boundary, path, source rung, deployment rung, and evidence conditions are part of the claim.
SimVerity makes the assurance contract explicit
SimVerity is an assurance layer rather than an agent policy. It does not select actions or improve the agent. Its job is to determine whether a source clearance survives deployment under a declared contract.
The system aligns source and target traces across semantic stages such as request, agent read, dispatch, software feedback, and observable effect. Property monitors then assign pass, fail, or abstain verdicts for claims including reported completion, read-after-write state, observable effect, settled postcondition, effect order, fanout, and sensor-to-effect behavior.
Physical evidence also has an admissibility rule. Camera-based optical witnesses, for example, must satisfy frozen qualification criteria, and sessions require at least 95% trace completeness plus settled-control agreement. Missing stages or evidence that cannot meet the qualification rule produce abstention.
For QA teams, that design changes the meaning of missing evidence. An unavailable physical observation is not converted into agreement merely because the simulator passed. The release record retains the coverage gap.
Structured calibration predicts some failures before the held-out deployment test
The paper’s main predictive experiment asks whether calibration data can estimate false-clearance risk on a physically unmeasured path-rung pairing. This is main evidence, not an ablation: predictors and backoff rules are frozen before held-out evaluation, and a second confirmatory cohort is evaluated without refitting.
SimVerity conditions risk on property, stage or direction, and path. On the eight-session confirmatory cohort, its Brier score was 0.0878, compared with 0.1995 for the registered path-only baseline. Lower is better. SimVerity won all eight session-level comparisons, with sign and exact Wilcoxon tests reporting p=0.0039.
The first three-session cohort pointed in the same direction: Brier 0.1113 for SimVerity versus 0.1616 for path-only. Across the two cohorts it beat path-only in all eleven sessions, although the paper correctly treats that combined tally as descriptive because the confirmatory protocol was registered after the first cohort’s results were known.
This supports a bounded operational inference: when a team cannot directly calibrate every deployment cell, preserving structure about which property, which stage, and which path failed can provide more useful transfer information than pooling risk at the path level alone.
Local memorization can still beat structured transfer
The strongest comparator prevents a broader claim.
In the confirmatory cohort, a deployment-local strong lookup achieved a Brier score of 0.0341, substantially below SimVerity’s 0.0878, and it outperformed SimVerity in seven of eight sessions. Its advantage arose under controlled, stationary conditions where granular local history remained highly informative.
For operations teams, this creates a model-selection question rather than a universal ranking. If the environment is stable and the same device-level cells repeat, local lookup may be the better risk estimator. Structured conditioning becomes more relevant when the organization expects unsupported cells or changing combinations for which exact local history is unavailable.
The paper tests the latter proposition only partially. Its principal predictor evaluation covers one held-out physical path-rung pairing, so broader transfer across heterogeneous deployments remains unresolved.
A second simulator is not automatically an independent witness
The cross-simulator experiment is a comparison designed to test whether simulator consensus can substitute for deployment-grounded evidence.
It cannot in this study. SimuHome and S5-HES produced zero eligible disagreements across 160 frozen physical anchors. Requiring strict agreement reduced coverage by 25 percentage points while lowering conditional false-clearance risk by only 1.25 percentage points. The second simulator often contributed abstention rather than a distinct failure signal; its 100-ms clock could not represent one 0.05-second boundary.
Agreement therefore established consistency between the two source systems, not independent physical correctness. A procurement or QA team evaluating multiple simulators should test whether their errors actually separate against physical anchors before counting consensus as additional assurance.
Auditability belongs to the executable configuration
The live-agent experiments serve a different purpose: testing whether the same assurance discipline survives dynamic action selection.
One planner-executor physical configuration matched all 32 declared scenarios and still exhibited property-selective failures. Its immediate return produced 32/32 completion false clearances because the system read state before the update became visible.
More revealingly, scenario matching varied with the executable configuration. Custom agent loops achieved roughly 52–88% matching depending on configuration, while the unmodified Hermes harness remained fully matched. A registered intervention that changed the full ReAct model-client/serving configuration restored 100% matching.
That result does not identify whether model weights, provider, endpoint, client, or serving behavior caused the improvement; those components changed together. It does show why reproducibility records for agent evaluation cannot stop at labels such as “ReAct” or even at a model name. The executable stack is part of the audited object.
The deployment rule is local, explicit, and allowed to abstain
Two additional preregistered deployments—a separate office and another home on a different vendor stack—showed zero false clearances for completion, reported state, and settled postcondition in the evaluated comparisons. The paper attributes this to confirmation-gated integration semantics.
Those replications support portability of the audit procedure. They do not validate the frozen false-clearance predictor across sites, and they should not be pooled with the principal home deployment as evidence of universal risk.
The physical scope is also deliberately narrow: commodity lights, contact sensors, and isolated proxies rather than hazardous actuators. Camera cadence and ambient conditions sometimes prevent qualified short-timescale judgments, in which case the system abstains.
For deployment governance, the resulting release discipline is straightforward. Define the operational property. Fix the read boundary. Specify the source and target rungs. Qualify an independent witness. Estimate false-clearance risk from calibration where appropriate. Preserve abstention when the evidence channel cannot support a verdict.
That is a narrower claim than “simulation predicts reality,” but it is a more usable one. A simulator pass becomes deployment evidence only after the organization states exactly what that pass is supposed to certify and demonstrates that the corresponding verdict survives on the physical system.
Cognaptus: Automate the Present, Incubate the Future.
-
Zhonghao Zhan and Yefan Zhang and Krinos Li and Hamed Haddadi (2026). SimVerity: When Does Simulated Agent Success Survive Physical Deployment?. arXiv:2608.25067. https://arxiv.org/abs/2608.25067 ↩︎