When the Scorecard Forgets the Traffic
TL;DR for operators Before choosing the highest-scoring model, ask whether the evaluator still preserves distinctions that competent behavior requires. In one audited NAVSIM configuration, a route-blind policy called Ignore-All, which does not observe surrounding actors, scored 79.6 EPDMS, above human replay at 74.0 and PDM-Closed at 67.1. The paper does not interpret this as evidence that ignoring traffic works. Wei et al. instead trace the ranking inversion to a dependency-sensitive numerical failure inside the score calculation.1 Both candidate and reference trajectories pass through the same unstable velocity-refit and rollout transformation. When the transformed human reference fails structural criteria, a forgiveness rule can turn corresponding candidate failures into passes. ...