Cover image

A Strong Ranking Is Not a Cheap Audit

TL;DR for operators A simulator can rank candidate policies accurately without making the final deployment decision substantially cheaper to validate. In two reinforcement-learning control tasks, world-model rankings achieved Kendall tau values of 0.962 and 0.800, yet real-environment episode-saving ratios reached only 1.00 and 1.33, below the researchers’ predefined twofold target. The paper explains why relative rankings can remain accurate despite biased predictions, then tests whether those shared errors also reduce validation uncertainty. They largely do not: matched audit-error correlations are close to zero, and statistical certification frequently remains inconclusive. ...

November 15, 2026 · 9 min · Zelina