TL;DR for operators A mathematical verifier is not an invariant scoring utility. Esther Xin’s Where the Verifier Fails1 audits four implementations and configurations using 307,420 verifier verdicts and finds acceptance of mathematically equivalent answers ranging from 53.8% to 95.2%. Two extraction configurations of the same math-verify library disagree on 49.9% of equivalent cases where both return a verdict. ...
TL;DR for operators Before choosing the highest-scoring model, ask whether the evaluator still preserves distinctions that competent behavior requires. In one audited NAVSIM configuration, a route-blind policy called Ignore-All, which does not observe surrounding actors, scored 79.6 EPDMS, above human replay at 74.0 and PDM-Closed at 67.1. The paper does not interpret this as evidence that ignoring traffic works. Wei et al. instead trace the ranking inversion to a dependency-sensitive numerical failure inside the score calculation.1 Both candidate and reference trajectories pass through the same unstable velocity-refit and rollout transformation. When the transformed human reference fails structural criteria, a forgiveness rule can turn corresponding candidate failures into passes. ...