When Worse Inputs Score Better: Audit the Credibility Behind the Benchmark

TL;DR for operators

A benchmark score can be high without being equally trustworthy as a measure of generalization. In this study, the researchers deliberately degraded benchmark questions before they reached the answering model. At a noisy-router count of eight, 10 of the 12 evaluated models nevertheless scored above their own clean baseline. At nine routers, eight models still did so, and the mean positive excess among above-baseline cases reached 0.086.

The operational lesson is not that these models were proven to have seen the benchmark during training. They were not. The result is an audit signal: some models respond unusually strongly when deleted, rewritten, perturbed, and aggregated fragments reconstruct possible benchmark-related semantic cues. For release teams, procurement functions, and model-risk owners, that creates a second selection dimension alongside score level: test whether the score remains credible when the information path is deliberately altered.

When degraded information raises the score

Suppose two models are being compared before procurement or deployment. The normal workflow is familiar: run a benchmark, compare accuracy, inspect cost and latency, then decide whether the higher-scoring system is worth the trade-off.

That workflow assumes a basic property of the benchmark result. Giving a model a faithful version of the question should normally be at least as helpful as deleting useful information, rewriting the wording, adding irrelevant material, and forcing several imperfect reconstructions to be merged before answering.

The experiment reported by Song and colleagues challenges that expectation.¹ With eight noisy routing paths, 10 of 12 models scored above their own clean-condition accuracy. If degraded transmission repeatedly improves performance, the benchmark number needs an explanation beyond its level.

The audit changes the information path, not the questions

The paper tests this with what it calls a router-worker audit.

The researchers fixed a sample of 100 multiple-choice questions and reused it across 12 models. In the clean condition, one router was instructed to preserve the problem as fully as possible before passing it to a worker model. In noisy conditions, between one and nine routers independently deleted useful information, rewrote the problem, introduced irrelevant noise, and produced fragments that were aggregated before the worker answered.

The worker never saw the original question and had to return a single option letter. The questions, scoring rule, sample, and answering constraint were held constant. What changed was the transmission regime.

For each noisy setting, the main statistic is simply the noisy-minus-clean accuracy difference:

A positive value means the degraded route beat the clean reference. The paper calls the positive portion of that gain the positive excess.

The theoretical argument matters because it defines what counts as surprising. If the clean router already transmits nearly all task-relevant information, additional deletion and noise should not systematically improve correctness except for a limited benefit from combining independent router outputs. Persistent gains beyond that expectation become the behavior to investigate.

The anomalies are widespread, but the models do not behave alike

The aggregate result is not a monotonic story in which every additional router improves every model. The stronger finding is heterogeneity.

Some models crossed their clean baseline repeatedly. Qwen3-Next-80B did so in all nine noisy settings, while Seed-2.0-Lite did so in eight. DeepSeek-Chat crossed it only once, and Qwen3.5-122B twice.

Frequency and magnitude also separate. Qwen3.5-35B exceeded the clean baseline in only five of nine settings, yet its maximum positive excess was 0.260, the largest model-level maximum reported. Qwen3-Next-80B violated the baseline more consistently, but its maximum excess was only 0.070.

That distinction matters for evaluation design. A model can show broad but modest responsiveness to the perturbation, or rare but large jumps. Collapsing both patterns into a single yes-or-no contamination flag would discard information the experiment is specifically designed to expose.

The question-level reversals make a generic aggregation story less convincing

A noisy condition outperforming a clean one could still arise from ordinary aggregation effects. Multiple imperfect rewrites might accidentally surface useful information that one clean router failed to emphasize. Random variation over 100 questions can also produce score differences.

The question-level transitions add another piece of evidence.

At one router, the models collectively changed 112 previously wrong answers to correct ones, while 150 correct answers became wrong. By eight routers, that direction reversed: 150 wrong-to-correct transitions versus 110 correct-to-wrong. At nine routers, the difference widened to 180 versus 116, producing a net improvement of 64 transitions.

The pattern does not prove a contamination mechanism. It does make the anomaly more specific. Higher noisy-router settings are associated with increasingly directional recovery rather than merely symmetric answer instability.

The authors’ proposed mechanism is that several partial rewrites can collectively recreate semantic neighbors of benchmark-related material. If a model has latent memory responsive to those cues, overlap between the aggregated text and those latent cues can contribute to answer probability even though the transmitted problem contains less faithful task information.

That is the paper’s interpretation of contamination sensitivity: responsiveness to potential benchmark-related cues, observed behaviorally rather than established from training-corpus inspection.

Score confidence asks a different question from score level

The paper’s more consequential move is conceptual.

Benchmark accuracy answers, “What score did the model achieve?” Score confidence asks whether that score is a credible indicator of contamination-reduced generalization.

Those dimensions need not move together. A model can retain genuine benchmark competence while also exhibiting stronger sensitivity to reconstructed benchmark cues. The authors therefore do not propose discarding benchmarks. They propose auditing the credibility attached to their scores.

For an organization choosing between models, that reframes the decision. A higher benchmark number may still be relevant, but it no longer exhausts the evidence. The organization can also ask whether that number behaves as expected when the informational form of the test is disrupted.

What this changes for release and model selection

What the paper directly shows: models differ substantially in both the frequency and size of noisy-over-clean anomalies under this 100-question audit. It also shows that higher router counts can produce more wrong-to-correct than correct-to-wrong transitions.

Cognaptus inference for model developers: when a benchmark improvement is central to a release claim, the release owner can add a perturbation audit before treating the gain as evidence of broader capability. A model with frequent or large above-baseline anomalies deserves additional validation on fresh or privately held evaluation material before the organization attributes the improvement to generalization.

Cognaptus inference for procurement teams: when candidate systems have similar benchmark performance, contamination sensitivity can act as a secondary comparison criterion. Violation breadth and anomaly magnitude should remain separate fields because this paper shows that they capture different behaviors.

Cognaptus inference for governance teams: the audit is better used as a trigger for deeper review than as an automatic rejection rule. Its value is in identifying benchmark scores whose credibility warrants additional evidence.

The audit diagnoses sensitivity, not training-data provenance

The strongest boundary is causal. The experiment does not inspect model training corpora and cannot establish that a particular benchmark question appeared in training. An above-baseline anomaly is evidence consistent with sensitivity to benchmark-related semantic cues, not proof of leakage.

The empirical base is also narrow. The study uses one fixed sample of 100 questions from a public multiple-choice benchmark that is not named in the paper’s HTML record. The main results are descriptive; no confidence intervals, p-values, or repeated sampling seeds are reported.

Finally, the theoretical expectation that noisy routing should not outperform the clean condition depends on the clean router being close to full-information transmission and on benefits from cross-router complementarity remaining limited. If either assumption fails materially, some positive gains could have explanations unrelated to contamination-sensitive memory.

Those constraints do not erase the anomaly. They define its proper use: as a behavioral audit signal that identifies where a benchmark score needs stronger corroboration.

A benchmark score now has two questions attached

Leaderboards compress evaluation into an attractive single number. This paper shows why organizations making release, procurement, or deployment decisions may need one additional question before treating that number as capability evidence.

The first is still: How high did the model score?

The second is: How credible is that score when the benchmark’s wording and information path are deliberately disturbed?

The router-worker audit does not answer whether contamination occurred. It does something narrower and operationally tractable: it tests whether the score behaves in ways that make contamination sensitivity plausible. For benchmark-dependent decisions, that is enough to justify treating score credibility as a separate field rather than an assumption embedded inside accuracy.

Cognaptus: Automate the Present, Incubate the Future.