When the Scorecard Forgets the Traffic

TL;DR for operators

Before choosing the highest-scoring model, ask whether the evaluator still preserves distinctions that competent behavior requires. In one audited NAVSIM configuration, a route-blind policy called Ignore-All, which does not observe surrounding actors, scored 79.6 EPDMS, above human replay at 74.0 and PDM-Closed at 67.1.

The paper does not interpret this as evidence that ignoring traffic works. Wei et al. instead trace the ranking inversion to a dependency-sensitive numerical failure inside the score calculation.¹ Both candidate and reference trajectories pass through the same unstable velocity-refit and rollout transformation. When the transformed human reference fails structural criteria, a forgiveness rule can turn corresponding candidate failures into passes.

The strongest evidence is not the strange leaderboard result itself. It is that changing the solver, changing the dependency stack while keeping scorer source fixed, or replacing and guarding the unstable transformation eliminates rollout divergence and restores actor-aware-over-blind ordering.

For teams using simulation scores in model selection, release gates, vendor comparison, or safety justification, the implication is concrete: treat behaviorally incompetent probes and numerical-stability tests as evaluator validation, not as optional benchmark diagnostics.

A leaderboard should survive an intentionally bad policy

Any model ranking used for an operational decision carries an implicit prerequisite: the metric must preserve distinctions relevant to that decision.

For defensive driving, one such distinction is actor sensitivity. A system that sees surrounding vehicles has information needed to respond to them. A deliberately actor-blind system does not. Actor awareness alone does not prove safe driving, but an evaluation intended to support actor-sensitive claims should at least avoid systematically rewarding blindness.

That sanity condition fails under the affected stock scoring configuration examined in the paper.

Across the complete 12,146-token NAVSIM navtest split, Ignore-All scores 79.6. A second probe that follows route information while remaining blind to surrounding actors scores 79.2. Both exceed human replay at 74.0; the route-aware blind probe beats human replay by 5.3 points, with a 95% confidence interval of 4.4 to 6.2.

That is a diagnostic result, not a driving result. The relevant question becomes: which part of the evaluator allows the distinction to disappear?

The scorer sees actors, so the failure is deeper in the pipeline

One tempting explanation is that the evaluator itself simply fails to incorporate surrounding traffic.

The paper tests that possibility. Collision-risk scoring still responds to the actual actors in the scene. The Ignore-All advantage also survives when reactive traffic is replaced by non-reactive log replay: it remains 5.9 points above human replay.

Those checks matter because they narrow the diagnosis. The anomaly is not adequately explained by traffic yielding to the blind agent or by the scorer being globally insensitive to actors.

Attention instead shifts to how trajectories are numerically transformed before individual score channels are calculated.

Both the evaluated proposal and the logged human reference traverse a shared velocity-refit and rollout transformation. Under the affected dependency stack, that transformation can become numerically unstable. The consequences are not subtle: the paper reports kilometer-scale path deviations, and an exact-input diagnostic produces an implausible fitted initial velocity of −491.34 m/s and maximum acceleration of $3.06 \times 10^5$ m/s² under the affected default pseudoinverse. Hermitian and control-stack calculations instead agree with a high-precision solution near 12.53 m/s and 1.04 m/s².

This is why numerical robustness belongs inside benchmark validity. The solver is not merely producing slightly different decimals. It can change the behavioral meaning of the score.

Shared reference failure turns penalties into passes

The second ingredient is a scoring rule that excuses the evaluated agent on a criterion when the logged human reference also fails that criterion.

Formally, the paper defines a reference-failure mask for score channel $j$:

When that mask is active, filter-on scoring assigns the evaluated agent a score of one on that channel:

Such forgiveness can be reasonable when the reference itself legitimately violates a criterion. The failure arises when proposal and reference share a broken transformation.

Under the unstable rollout, the human reference itself fails structural compliance channels broadly. For Ignore-All, raw failures are subsequently converted to passes on 99.98% of DAC, 99.98% of DDC, and 99.92% of lane-keeping evaluations. Overwrite rates for TTC and no-at-fault-collision are much lower, at 2.63% and 2.39%.

The authors call the resulting mechanism shared-failure forgiveness collapse: a common numerical failure makes the reference fail, and reference-conditioned forgiveness then removes corresponding constraints from the evaluated policy.

Crucially, the paper does not argue that forgiveness caused the numerical instability. The unstable velocity refit is the trigger; forgiveness propagates its effects into score credit.

The controlled reversals identify the mechanism

A surprising ranking can reveal a problem, but it cannot identify its cause. The paper’s more consequential contribution is its sequence of controls.

The full-navtest Ignore-All versus human-replay margin moves from +5.6 points under stock filter-on scoring to −75.6 points under the two-sided diagnostic. The fallback produces a −75.1-point margin.

On the 450-token solver-control pool, changing only the velocity-fit solver moves Ignore-All from 81.8 with the affected stock pseudoinverse to 43.9 with direct solve and 45.7 with the Hermitian pseudoinverse. Human replay scores 62.8 and 66.1 under those respective alternatives.

These interventions serve different evidentiary roles. The alternative score bases are diagnostics, not proposed replacement leaderboards. The solver and dependency controls are more important for localization: they show that stabilizing the shared transformation restores the necessary ranking distinction while forgiveness remains enabled.

Benchmark validation should become part of the release gate

What the paper directly shows: under the audited NAVSIM configuration, a dependency-sensitive numerical instability can interact with reference-conditioned scoring to erase structural failures and invert rankings.

Cognaptus inference: teams should validate the evaluator before allowing aggregate benchmark scores to allocate consequential decisions.

For a model-selection or release workflow, that means keeping deliberately deficient probes alongside strong models. An actor-blind policy is useful precisely because nobody expects it to win. If it does, the ranking should be suspended until the score basis is explained.

Benchmark owners can also expose measurements that are usually hidden behind the aggregate: reference-failure masks, fail-to-pass overwrite rates, fitted-state diagnostics, rollout deviation, solver choice, dependency versions, and numerical backend. A frozen exact-input test can help distinguish a model regression from an evaluator regression when environments change.

Vendor comparison has the same dependency. A two-point advantage between systems has little decision value if a solver or library stack can reverse much larger behavioral distinctions.

The audit validates a score basis, not driving safety

The evidence is strong for the configuration the authors audit, but its scope is narrow.

The demonstrated failure concerns NAVSIM v2.2 original-scene single-stage pseudo-simulation under an affected documented dependency stack on the authors’ audited x86-64 OpenBLAS backend. The study does not establish how often the same condition occurred in published evaluations, whether it appears on other software branches or numerical backends, or whether similar failures exist in unrelated benchmarks.

The paper also does not evaluate real-world driving safety. Restoring actor-aware-over-blind ordering establishes a necessary validity condition, not anticipation, comfort, collision avoidance, or deployment readiness.

That boundary should shape how the result is used. The right response is not to discard simulation benchmarks. It is to stop treating their aggregate rankings as self-validating.

A benchmark used to choose systems should first demonstrate that its own numerical machinery preserves the behavioral distinctions those choices depend on.

Cognaptus: Automate the Present, Incubate the Future.