Decontamination Is a Dial, Not a Delete Key

TL;DR for operators

A benchmark score can become unreliable when test items, close paraphrases, or related material have entered training. Removing suspicious questions sounds straightforward, but any detector that misses contaminated items leaves score inflation behind, while filtering also changes the evaluation set.

Chai, Zhe, and Sakuma propose DeconIEP,¹ a white-box inference-time intervention that keeps the benchmark text and model weights fixed. Instead, it learns small, input-specific changes to the model’s embeddings so contaminated behavior moves closer to a comparatively less-contaminated reference model.

The main operational lesson is not that stronger decontamination is always preferable. In one LLaMA-3 experiment, increasing intervention strength reduced residual contamination on TruthfulQA from 0.512 to 0.022 and on MMLU from 0.344 to 0.032. Over the same range, benign utility loss rose from zero to 0.062 and 0.102 respectively. The evaluation control therefore has to be calibrated against both contamination suppression and damage to uncontaminated performance.

For teams using open-weight model benchmarks in selection, deployment approval, procurement, or governance, DeconIEP suggests a more disciplined workflow: preserve the test, apply a controlled mitigation, and judge the result on contamination reduction and clean-task harm together. The evidence is strong inside the paper’s simulated-contamination setting, but it does not establish equivalent behavior under unknown real-world leakage.

The benchmark can be preserved while inference changes

Suppose two internally fine-tuned models compete for deployment and one posts a surprisingly high benchmark score. The team suspects leakage, but does not know exactly which items were exposed. Deleting every suspicious question changes the test. Leaving the benchmark untouched risks rewarding memorization.

The paper’s alternative is to intervene after training but before the model processes each evaluation item. DeconIEP leaves the written prompt unchanged and modifies its embedding representation:

Here, $\delta(x)$ is a learned perturbation specific to the input, and $\zeta$ caps its magnitude. The generator producing those perturbations is trained so the contaminated model’s output distribution approaches that of a less-contaminated reference model.

That design matters because it separates two decisions that filtering conflates. The benchmark curator can keep the evaluation set intact, while the model evaluator changes how strongly suspected memorization pathways influence inference.

The paper’s detect-then-filter analysis is supporting motivation rather than the central experimental result. It shows formally and through simulation that false negatives can leave substantial score inflation as contamination prevalence rises. That is enough to justify testing inference-time mitigation; it does not show that every real benchmark detector will fail in the same way.

The control is a trade-off, not a cleanup switch

The most informative experiment varies the perturbation budget rather than reporting one preferred setting.

On contaminated LLaMA-3 with leakage occurrence $o=3$, the authors increase $\zeta$ from $10^{-5}$ to $10^{-2}$. Residual Contamination, or RC, measures how far the mitigated model remains from the matched clean model on contaminated evaluation data. Benign Utility Drop, or BUD, measures how much the intervention changes performance on uncontaminated data.

The pattern is monotonic enough to be operationally useful: stronger perturbation suppresses more contamination signal, but eventually costs more clean utility.

This also corrects a tempting interpretation of RC. A low RC score does not establish that an intervention has recovered clean reasoning. A method could approach clean-model benchmark performance simply by degrading the contaminated model. RC therefore has to be read jointly with BUD.

That distinction turns $\zeta$ into a policy parameter. An evaluator choosing it is deciding how much clean-task change is acceptable in exchange for reduced contamination-driven inflation.

The main experiments test whether the trade-off generalizes

The paper’s central evidence spans Mistral, Qwen2.5, and LLaMA-3 on MMLU and TruthfulQA, with exact leakage, meaning-preserving paraphrases, and domain-level contamination. Leakage strength is varied by repeating leaked items one, three, or five times during fine-tuning.

Across those conditions, DeconIEP usually produces the lowest or near-lowest RC. At $o=3$, its cross-benchmark BUD ranges from 0.009 to 0.041 across the reported model-benchmark pairs.

That clean-utility result matters when comparing white-box methods. Shortcut Neuron and Short Circuit can also suppress contamination, but their clean-performance losses are often substantially larger. For example, on Qwen2.5 the reported BUD for DeconIEP is 0.009 on TruthfulQA and 0.029 on MMLU, compared with 0.207 and 0.256 for Shortcut Neuron.

Black-box methods occupy another part of the trade-off. ITD often produces almost no BUD, but in the reported contamination experiments it frequently leaves much more residual inflation. The comparison therefore is not simply “white-box beats black-box.” Different methods occupy different points on the contamination-versus-utility frontier.

The ablations test whether the effect is more than injected noise

A small perturbation could appear to work merely because noise disrupts memorized responses. The random-noise ablation addresses that possibility.

Matched random perturbations become less stable as intervention strength increases, while learned, instance-adaptive perturbations retain better decontamination and utility behavior. This supports the paper’s proposed mechanism: direction matters, not only magnitude.

Other ablations serve narrower robustness purposes. Performance remains largely stable when the reference model’s contamination rises from 0% to 30%, suggesting that the reference need not be perfectly clean within that tested range. Increasing the auxiliary training set generally reduces RC, with the largest relative improvement arriving when moving from no auxiliary data to a small nonzero sample and diminishing returns later.

The Qwen2.5 experiments from 1.5B through 14B test model-scale sensitivity rather than establishing a separate contribution. The HumanEval extension performs a similar role across task type: on LLaMA-3-8B-Instruct at $o=5$, RC falls from 0.36 without mitigation to 0.08 with DeconIEP, while the reported BUD remains below TED’s. Because this extension uses one backbone, it is supporting evidence rather than broad code-generation validation.

Evaluation teams can govern the RC-BUD operating point

For organizations comparing open-weight models, Cognaptus sees the business relevance in the control structure rather than the specific algorithm alone.

An evaluation team with uncertain model provenance can keep the benchmark fixed, select an intervention strength, measure movement toward clean-like behavior, and simultaneously test for damage on unrelated clean tasks. That produces a reviewable operating decision instead of a single adjusted score.

A reasonable deployment workflow would require three checks before benchmark results influence model selection: contamination-strength stress tests, cross-benchmark clean-utility tests, and sensitivity to the chosen reference checkpoint. A base or earlier checkpoint of the same architecture may be a workable reference when no perfectly clean model exists, but the paper only demonstrates tolerance to moderate contamination, not arbitrary mismatch.

The affected decision is narrow but consequential: whether a benchmark score is credible enough to support comparative model governance. DeconIEP does not remove memorized training data, certify provenance, or prove that a model has recovered the reasoning process it would have learned without leakage.

Where the evidence stops

The strongest limitation is external validity. The experiments create contamination deliberately using matched clean and contaminated fine-tuning conditions. That is excellent for isolating the effect of leakage and mitigation, but production models rarely arrive with a known clean twin or known contamination history.

The method is also white-box. It requires access to input embeddings and, in the reported training setup, gradients. Standard closed-model APIs therefore fall outside the demonstrated deployment setting.

Finally, high cosine similarity between original and perturbed embeddings supports limited representation drift but does not guarantee preservation of meaning or task difficulty for every item. Reference quality also remains an open dependency once contamination exceeds the tested 30% range or alignment behavior differs substantially.

The paper’s strongest contribution is thus a change in evaluation design: contaminated benchmarks need not be treated only as sets to filter or abandon. Under controlled open-model conditions, contamination can instead be managed as a measurable inference-time intervention with an explicit cost in benign utility.

That framing is more useful than any single decontaminated score. It forces the evaluator to specify what contamination reduction is worth, what clean-performance loss is acceptable, and which assumptions make the adjusted benchmark credible enough to govern a real decision.

Cognaptus: Automate the Present, Incubate the Future.