When the Model Knows but Doesn't Remember: The Hidden Blind Spot in LLM Contamination Detection

Audit.

That is the word companies like to use when they want uncertainty to sound disciplined. Model audit. Benchmark audit. Contamination audit. The phrase suggests a clean checklist: run the detector, read the score, decide whether the benchmark is safe.

The paper behind today’s article makes that picture less comfortable. It studies Contamination Detection via output Distribution, or CDD, on small language models and finds a simple but awkward failure mode: a model can be trained on contaminated benchmark examples, learn from them, and still avoid the kind of verbatim memorization that CDD is designed to catch.¹

That distinction matters because CDD is attractive for exactly the reason business users like simple audits: it can work from sampled text alone. No training corpus. No internal logits. No privileged model access. Ask the model the same prompt many times, check whether the outputs collapse into the same answer, and infer whether the example was memorized.

Elegant. Also fragile. Naturally, reality has chosen the fragile part.

The mechanism is the article: contamination does not always become memorization. Sometimes it only shifts the model’s probabilities. When that happens, a detector watching generated text can see nothing, while a detector watching probabilities can already see the leak.

CDD detects collapse, not contamination itself

CDD starts from a sensible intuition. If a model has memorized a specific answer, random sampling should not change much. The greedy output and temperature-sampled outputs should look unusually similar. The output distribution becomes “peaked.”

The paper operationalizes this through a peakedness score. For each prompt, CDD generates one deterministic greedy output and multiple stochastic outputs. It then computes token-level edit distance between the greedy output and each sampled output. If enough sampled outputs are close to the greedy output, the prompt is classified as contaminated.

In plain language:

The key phrase is “output collapse.” CDD does not directly detect whether an example appeared in training. It detects a behavioral symptom that often follows memorization.

That symptom is useful when it appears. But the paper asks a sharper question: what if contamination changes the model internally without making the generated outputs collapse?

The answer is not flattering to CDD.

The experiment isolates learning from memorization

The study uses controlled contamination experiments on Pythia models with 70M, 160M, and 410M parameters. The author fine-tunes them on three benchmark domains:

The contamination setup is deliberately controlled. Examples are split into training, contamination, and evaluation subsets. Contaminated examples are inserted into fine-tuning data at different repetition levels: 0, 1, 5, or 10 times.

The fine-tuning regime is where the mechanism becomes visible. The paper compares:

Training duration is also varied: 3 epochs and 20 epochs. The exact trainable parameter count ranges from 98K in the smallest low-rank LoRA setup to 405.3M in full fine-tuning of Pythia-410M. This gives the experiment its real value: it does not merely ask whether CDD works. It asks what kind of training dynamics make CDD work.

The answer is a threshold. Not a vague “performance improves gradually with scale” threshold. A fairly sharp one.

Below the threshold, the model learns from contaminated data but keeps producing varied outputs. Above it, the model begins reproducing contaminated answers consistently enough for CDD to notice.

The uncomfortable result: contaminated, detectable by probabilities, invisible to CDD

The core result appears in the low-capacity LoRA regime.

On GSM8K, with LoRA rank 8 and 3 training epochs, CDD stays around chance-level accuracy across model sizes and contamination levels. Even with Pythia-410M and contamination repeated 10 times, CDD does not meaningfully separate contaminated from clean examples.

That is the “no memorization, no detection” result.

But the contamination is not imaginary. The paper compares CDD with three baselines:

The n-gram method confirms the contamination injection. More importantly, perplexity and Min-k% Prob detect contamination in many conditions where CDD remains at chance.

This is the paper’s most practical distinction:

That is not a small technicality. It means a black-box detector can provide false comfort precisely when the model has absorbed leaked evaluation material without becoming a parrot.

The audit says “clean.” The model quietly says, “I have seen this neighborhood before.”

The threshold is driven by trainable capacity, duration, and scale

The paper’s second contribution is the memorization threshold.

On Pythia-410M with GSM8K at contamination level 10, CDD accuracy moves from chance under LoRA rank 8 to about 0.915 under LoRA rank 256 after 3 epochs. Full fine-tuning reaches about 0.955 after 3 epochs and 0.985 after 20 epochs.

Extended training can compensate for low adapter rank. LoRA rank 8 at 20 epochs reaches about 0.920 in the same Pythia-410M GSM8K high-contamination setting, similar to LoRA rank 256 at 3 epochs.

Model size also matters, but only after capacity is sufficient. With LoRA rank 256 at 20 epochs and contamination level 10 on GSM8K, CDD accuracy rises from 0.640 for 70M to 0.765 for 160M and 0.925 for 410M. But with LoRA rank 8 at 3 epochs, all three model sizes remain at chance.

This is the important correction to lazy scale talk. Bigger helps when the training regime can already push the model toward memorization. Bigger does not magically rescue a detector whose signal has not appeared.

This also explains why the result does not simply contradict earlier CDD findings on larger 7B models. A LoRA rank that sounds “small” on a 7B model can still mean millions of trainable parameters. On a 70M or 160M model, the same nominal rank leaves far less trainable capacity. The label is the same. The memorization budget is not.

Training loss is not the audit signal people wish it were

One tempting response is to look at training loss. If the model learned the contaminated data, surely the loss curve should tell us whether detection should work.

The paper blocks that escape route.

Training loss decreases even in conditions where CDD fails. For Pythia-410M with LoRA rank 8 after 3 epochs, the loss falls from 2.35 to 1.26, while CDD accuracy is still about 0.505. The model learned. It did not memorize enough for CDD.

The paper describes two regimes:

The transition is abrupt. There is not much of a useful middle zone where CDD partially works. It either sees collapse or it does not.

This is exactly why the mechanism-first reading matters. If we only summarize the tables, CDD looks like a weak method in small models. If we understand the mechanism, the result becomes more general: any detector that equates contamination with near-verbatim output collapse is exposed to the same conceptual weakness.

Not all contamination has the decency to look like plagiarism.

The qualitative examples show the difference better than the metrics

The paper includes a GSM8K example involving a crayon arithmetic problem. Both models are trained on the contaminated example 10 times.

Under LoRA rank 8, the model produces answers in the GSM8K style but gets different wrong results across greedy and sampled generations. The format has been learned: step-by-step arithmetic, annotations, final answer markers. But the exact solution has not been reproduced. The greedy output gives 42. A sampled output gives 38. The ground truth is 29.

Under full fine-tuning, both greedy and sampled outputs reproduce the ground-truth solution essentially verbatim. The output distribution has collapsed. CDD can now detect it.

That contrast is the whole story in miniature:

This is also where the business interpretation becomes less abstract. Many organizations fine-tune small or mid-sized models on domain-specific data and then evaluate them on curated task sets. If evaluation items leak into fine-tuning, the model may benefit from exposure without reproducing answers verbatim. A black-box text-only audit may miss that.

The benchmark score can improve. The detector can stay quiet. Everyone gets a neat dashboard. Beautiful, in the way a locked door is beautiful when the window is open.

Probability-based methods win because they look earlier in the causal chain

The paper compares CDD against perplexity and Min-k% Prob across Pythia-410M conditions for GSM8K, HumanEval, and MATH. Using the paper’s above-chance threshold of accuracy greater than 0.55, CDD exceeds that threshold in only 7 of 27 reported conditions. Perplexity does so in 26, and Min-k% Prob in 25.

The pattern is especially clear at lower contamination levels. On GSM8K with LoRA rank 256 and 3 epochs, CDD is 0.50 at contamination level 1, 0.59 at level 5, and 0.92 at level 10. Perplexity reaches 0.75 already at level 1 and 1.00 by level 5.

The same logic appears across domains. At single-repetition contamination, CDD is at or near chance in all nine combinations of fine-tuning method and dataset reported for Pythia-410M. Probability-based methods show meaningful signal in most of them.

Here is the practical reading:

Probability-based methods win because they inspect the model’s token probabilities. Fine-tuning on contaminated data can lower perplexity or raise the probability of otherwise difficult tokens before the model can reproduce the full answer consistently.

CDD waits for the final symptom. Perplexity and Min-k% Prob listen to the earlier cough.

The appendix is mostly robustness, not a second thesis

The appendix matters because it tests whether the central result is an artifact.

The threshold-free metrics in Appendix A report AUROC and AUPRC. These avoid dependence on a single classification threshold. The result remains the same: probability-based methods generally dominate CDD. This supports the interpretation that the CDD failure is not merely caused by an unlucky threshold choice.

Appendix B extends the GSM8K-style analysis to HumanEval and MATH. The same two-regime pattern appears: low-capacity fine-tuning produces little or no CDD separation, while higher-capacity settings can create peakedness separation.

Appendix C adds qualitative examples. In each domain, the lower-capacity LoRA model produces diverse, often incorrect outputs, while the higher-capacity LoRA model collapses toward the memorized training sequence.

The appendix is therefore not trying to introduce a separate grand claim. Its role is more disciplined: confirm that the mechanism survives alternative metrics, additional datasets, and qualitative inspection.

That is useful because the main claim is subtle. The paper is not saying “CDD always fails.” It is saying “CDD works only when contamination produces the behavioral symptom CDD measures.”

That is a much sharper and more operationally relevant statement.

What this means for business audits

For business users, the paper does not say that contamination detection is hopeless. It says the audit stack needs to match the mechanism of contamination.

A useful operational framework is:

The business value is not “use perplexity and declare victory.” Perplexity and Min-k% Prob require access to model probabilities, and not every vendor exposes them. N-gram overlap requires training data access, which is often unavailable. CDD remains attractive precisely because it works with generated text only.

But the paper changes how CDD should be positioned. It is not a general contamination detector. It is a detector for contamination that has become output collapse.

That distinction should appear in audit reports. A clean CDD result should not be written as “no contamination detected.” It should be written as something closer to: “no evidence of near-verbatim output collapse under this sampling protocol.”

Less comforting, yes. Also less misleading. A rare bargain.

What Cognaptus infers, and what remains uncertain

The paper directly shows that, in controlled fine-tuning experiments on Pythia 70M–410M, CDD can fail at chance level even when contamination is injected and probability-based methods detect it. It also directly shows that CDD improves sharply when trainable capacity, training duration, and model size push the model toward memorization and output collapse.

Cognaptus infers three practical lessons.

First, contamination audits should be multi-signal. If log probabilities are available, probability-based methods should be part of the audit. Sampled-text methods are useful, but they should not be treated as complete.

Second, PEFT-heavy workflows need special care. LoRA can reduce memorization risk, which is good, but that same reduction weakens detectors that depend on memorization. A cleaner-looking output trace does not necessarily mean a cleaner evaluation.

Third, audit language should separate “learning,” “memorization,” and “detectability.” These are not the same layer of the system. A benchmark example can influence the model without appearing as repeated text in generations.

The uncertain part is also important. The paper’s evidence is strongest for small Pythia models, controlled repeated fine-tuning contamination, and the three studied domains. It should not be automatically extrapolated to 7B+ models, pretraining contamination, retrieval-augmented systems, or alternative black-box two-sample tests.

The authors themselves note that other sample-based distribution tests might behave differently. CDD uses a specific peakedness statistic based on edit distance to a greedy output. Other methods could compare broader distributions using embeddings or classifiers, though they would require nontrivial design choices and reference distributions.

So the right conclusion is not “black-box contamination detection is dead.” It is narrower and more useful: black-box detection based on output collapse can miss contamination that changes probabilities without producing stable repeated answers.

The audit should measure the failure mode it claims to detect

The paper’s title says “No Memorization, No Detection.” That is accurate, but the business lesson is slightly more pointed: no mechanism, no audit.

If the detector is built to catch memorization, then its silence tells us about memorization. It does not necessarily tell us about contamination. That is the small linguistic mistake that becomes a large governance mistake.

This matters because the industry is moving toward smaller, specialized, fine-tuned models. They are cheaper to run, easier to adapt, and increasingly attractive for internal workflows. They are also exactly the kind of systems where parameter-efficient fine-tuning can create learning without obvious verbatim recall.

A good audit should not ask one detector to answer three different questions:

1. Did the model see the evaluation example?
2. Did the model’s probabilities shift because of that exposure?
3. Did the model memorize the answer enough to reproduce it?

CDD mostly answers the third. Perplexity and Min-k% Prob are closer to the second. N-gram overlap addresses the first, but only when training data is available.

The practical audit stack should reflect that division.

Otherwise, we end up with the worst kind of model governance: technically sophisticated, operationally neat, and pointed at the wrong signal.

The model may know. It may not remember. And if the audit only checks memory, it may confidently miss the knowledge.

Cognaptus: Automate the Present, Incubate the Future.