TL;DR for operators

A simulator can rank candidate policies accurately without making the final deployment decision substantially cheaper to validate. In two reinforcement-learning control tasks, world-model rankings achieved Kendall tau values of 0.962 and 0.800, yet real-environment episode-saving ratios reached only 1.00 and 1.33, below the researchers’ predefined twofold target.

The paper explains why relative rankings can remain accurate despite biased predictions, then tests whether those shared errors also reduce validation uncertainty. They largely do not: matched audit-error correlations are close to zero, and statistical certification frequently remains inconclusive.

For operators selecting automated strategies, the actionable distinction is between using simulation to prioritize candidates and using real outcomes to justify a selection. Measure ranking accuracy, selection regret, audit variance, confidence coverage, and certification rates independently. A strong simulated ranking is evidence of potential screening value, not proof of cheaper or sufficient validation.

The simulator can rank the options without reducing the testing bill

An automation team has several candidate control policies and limited opportunities to test them in the actual operating environment. Running each policy through a learned simulator is relatively inexpensive. Testing them against real outcomes consumes the scarce evaluation budget.

If simulated performance consistently orders the candidates correctly, management might reasonably expect to reduce the number of real tests needed to select the best one.

The difficulty is that choosing the correct policy and demonstrating that the choice is reliable are different statistical problems.

Ahanaf Hasan Ariq and Hasaan Ahmad examine this separation in World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits.1 Their experiments report strong simulated ranking agreement at an evaluation horizon of 500 steps, but the anticipated twofold reduction in real-environment episodes does not materialize.

The result raises a narrower question than whether world models are generally accurate: which parts of policy selection can a biased simulator perform reliably, and which still require expensive real evidence?

Shared prediction errors can cancel in policy comparisons

Suppose a world model overestimates the expected return of two policies by similar amounts. Both absolute predictions are wrong, but their predicted difference can remain accurate.

The paper formalizes this possibility through an exact pairwise simulation identity:

$$ \Delta(\pi,\mu)-\widehat{\Delta}(\pi,\mu) = \frac{ \mathbb{E}_{d^\pi}\widehat{b}^{\pi} - \mathbb{E}_{d^\mu}\widehat{b}^{\mu} }{1-\gamma}. $$

Here, $\Delta$ denotes the true return difference between policies, $\widehat{\Delta}$ the predicted difference, $\widehat{b}$ the policy-specific Bellman model residual, and $d$ the normalized state-action occupancy distribution under the true environment.

The identity says that the error in a policy comparison depends on the difference between occupancy-weighted model residuals, rather than directly on the sum of the policies’ absolute prediction errors.

The authors then introduce a shared reference residual, $c$, to separate ranking error into two components:

  1. How much each policy’s residual deviates from the common residual.
  2. How differently the two policies encounter that common residual through their state-action visitation distributions.

If both components are sufficiently small, substantial absolute prediction bias can disappear from the relative comparison. The resulting bound can certify a policy ordering when the predicted return gap exceeds the valid bound on pairwise error.

This is a conditional mathematical result, not a guarantee that all imperfect simulators rank correctly. The underlying assumptions include discounted Markov dynamics, compatible state representations, and bounded relevant quantities. Moreover, the residual-occupancy bound uses true-environment quantities; it is not automatically a computable deployment certificate.

Its value is conceptual and methodological: policy ranking should be evaluated through errors in differences, not only through errors in individual predicted returns.

Why shared bias does not guarantee cheaper real audits

A second question follows. If prediction errors cancel when comparing expected returns, can paired real-environment tests exploit that cancellation to reduce sampling uncertainty?

Not necessarily.

Expected prediction bias concerns average errors across evaluation conditions. Statistical efficiency in repeated paired audits depends on whether the errors of two policies actually move together across those conditions.

Ariq and Ahmad introduce RAVEL—Relative Auditing of Value Errors in Learned models—to separate these questions operationally.

RAVEL uses inexpensive world-model evaluations to prioritize candidate comparisons. It then estimates the true pairwise return gap by adding a correction based on held-out real-minus-model return differences. The correction uses coefficient one and is unbiased under the stated sampling assumptions.

For elimination decisions, the procedure constructs simultaneous confidence intervals across policy pairs and possible audit sample sizes. These intervals use bounded differential errors and a Hoeffding union-bound adjustment. Finite uncertainty in imagined model rollouts must also be accounted for.

An elimination is justified by the real-outcome confidence comparison, not merely because the simulator ranks a candidate lower.

This separation is essential to the guarantee. If candidates are excluded by model score before valid covered comparisons, the resulting shortlist does not inherit a certificate protecting the globally best candidate.

The authors also test whether starting policies from matched conditions makes the audit corrections less variable. In their horizon-500 experiments, the advantage is negligible.

Mean matched audit-error correlations are 0.007 for cheetah-run and 0.090 for walker-walk. Relative to independent-error evaluation, matching changes differential-error standard deviation by -2.5% and +1.9%, respectively.

For walker-walk, binned correlation declines from 0.164 to 0.024 as occupancy maximum mean discrepancy increases. This is consistent with a mechanism in which closed-loop policy trajectories diverge despite matched starting conditions. The experiment supports that interpretation but does not independently isolate trajectory divergence as the cause.

A world model can therefore preserve the ordering of average policy returns while failing to provide correlated audit errors that would substantially reduce the number of observations needed for validation.

Strong rankings, modest savings, and frequent abstention

The main evaluation uses two DeepMind Control Suite tasks: cheetah-run and walker-walk.

Each task contains 45 policies from SAC, TD3, and PPO training runs, with five checkpoints from each of three seeds per algorithm. A five-member learned dynamics ensemble evaluates the policies. Each policy has 128 imagined rollouts and 50 held-out real-environment episodes for analysis.

At horizon 500, the headline measurements show how differently ranking quality and audit utility behave.

Measurement Cheetah-run Walker-walk
World-model ranking Kendall $\tau$ 0.962 0.800
Same-run differential-error ratio 0.674 0.106
Mean matched audit-error correlation 0.007 0.090
Real-episode saving ratio 1.00 1.33
Certified decisions at eight audits 13.2% 0%

Source: Sections 5.2–6.3, including the precommitted verdicts in Table 1. The episode-saving target was 2.00; neither task achieved it.

The low differential-error ratios support the proposed cancellation mechanism for same-run policy pairs. Both pass the precommitted threshold of 0.7. However, policy relatedness does not provide a universal ordering of cancellation strength across the two tasks.

The stronger operational test is whether model guidance improves the quality of the selected policy at a fixed audit budget.

For walker-walk, normalized top-1 selection regret—lower is better—shows a budget-dependent result:

Audit budget RAVEL regret Real-only successive halving
$n=2$ 0.03221 0.03624
$n=4$ 0.01890 0.02784
$n=8$ 0.01833 0.01152

Source: Section 6.3, Table 2. These audit levels correspond to 16, 32, and 64 real episodes under the paper’s budget definition.

RAVEL performs better at the first two budgets, but real-only successive halving overtakes it at the largest tested budget. The model-guided approach has measurable low-budget value without establishing consistent superiority.

The paired and unpaired RAVEL variants produce identical point selections under the tested equal-sample design. Pairing changes confidence-interval widths, but those changes are negligible.

Statistical validity introduces another constraint. The simultaneous confidence intervals achieve high empirical coverage, with reported minimum coverage of 0.9995 for cheetah-run and 0.9927 for walker-walk. Yet at eight audits, only 13.2% and 0% of decisions are certified.

The intervals can be reliable while remaining too wide to resolve most selections. High coverage is therefore not equivalent to an operationally decisive answer.

Selecting more candidates introduces another source of risk

The candidate-pool experiments assess whether choosing from more policies changes the quality of model-based selection.

For walker-walk, raw-model normalized top-1 regret rises from 0.047249 with eight candidates to 0.082782 with 32. For cheetah-run, it falls from 0.001073 to 0.000111.

Selection pressure thus behaves differently across the two tasks. A model with strong overall ranking correlation can still become less reliable for the particular decision of identifying the single best candidate.

A separate frozen-model optimization stress test examines 500 additional policy-gradient steps for ten policies per task.

Mean real returns decline by 22.43 in cheetah-run and 28.50 in walker-walk. Crucially, predicted returns also decline, by 4.56 and 17.45. The test demonstrates deterioration in realized performance and ranking reliability, but it does not establish systematic exploitation that raises predicted rewards.

These experiments extend the main findings rather than independently demonstrating failures across other environments. Candidate pools are repeatedly sampled from the same fixed policy suites.

What an operational selection process should measure

For an organization evaluating automated control policies or simulation-optimized AI agents, the paper suggests separating three questions before replacing real tests with model-based evaluations.

Evaluation question Required measurement Decision it informs
Are predicted outcomes reliable in level? Absolute calibration error Whether simulated return estimates can support outcome forecasts
Are alternatives ordered correctly? Pairwise inversions, ranking agreement, top-1 regret Whether the simulator can prioritize candidates
Does simulation reduce validation costs? Audit-error variance, episode savings, confidence coverage, abstention Whether real-test budgets can safely be reduced

This is a Cognaptus operational inference from the research, not a demonstrated result in enterprise deployments.

An engineering manager choosing between candidate control policies could initially use simulation to allocate scarce trial capacity toward promising alternatives. However, consequential elimination or deployment decisions should depend on real-outcome comparisons that meet the intended statistical requirements.

Management should also distinguish the scope of the certificate. A validated winner within a prefiltered shortlist is not necessarily validated against every originally available policy.

The final cost calculation must include more than environment episodes. The paper does not measure wall-clock time, energy, or end-to-end computation. An episode-count improvement alone would not establish lower total operating costs.

Where the evidence stops

The formal relative-error identities and conditional certificates apply under their mathematical assumptions. The empirical findings are narrower: two state-based simulated-control tasks, a principal 500-step horizon, fixed trained policy suites, and reference returns estimated from finite held-out episodes.

Overlapping candidate pools and policy pairs are dependent observations, so the results cannot be treated as independent policy-training replications. The experiments also do not establish performance for learned latent-state models, substantially different dynamics-training budgets, physical robots, or shifted deployment distributions.

The absence of meaningful matched-audit covariance at horizon 500 does not rule out efficiency gains under shorter horizons or different pairing mechanisms.

Most importantly, the study does not prove that model-guided auditing is generally ineffective. It demonstrates that accurate ranking alone is insufficient evidence for the specific savings and certification claims an operator might make.

The deployment decision needs more than a ranking

The paper’s theoretical contribution explains how an inaccurate world model can still compare policies correctly. Its experimental contribution shows why that capability cannot be assumed to make reliable policy selection substantially cheaper.

For operational deployment, the relevant standard is not simply whether simulated rankings agree with observed performance. It is whether those rankings improve selection outcomes, reduce the required real evidence, and produce sufficiently decisive conclusions under the actual evaluation budget.

In the tested setting, those outcomes remain distinct.


Cognaptus: Automate the Present, Incubate the Future.


  1. Ahanaf Hasan Ariq and Hasaan Ahmad (2026). World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits. arXiv:2611.01139. https://arxiv.org/abs/2611.01139 ↩︎