Retrieval Changes the Exposure List, Not Just the Score

TL;DR for operators

Adding retrieval to an LLM does not necessarily make previously extractable training text inaccessible. A controlled study finds that some continuations remain exposed across different retrieved contexts, others become less extractable, and some become newly exposed.¹ Longer prefixes make exposure particularly stable.

For teams evaluating retrieval-augmented generation (RAG), an improved aggregate extraction score is therefore incomplete evidence of privacy protection. Test the same target continuations with and without retrieved documents, and track persistent, suppressed, and newly enabled exposures. Include longer prefixes, multiple retrieval contexts, and realistic prompt formatting.

These findings support a stronger evaluation procedure. They do not establish actual leakage rates in deployed applications.

A lower extraction count can conceal newly exposed text

Suppose an enterprise AI team adds document retrieval to an assistant and observes fewer training-text continuations passing its memorization test. The result appears favorable: retrieved information may be reducing the model’s reliance on its internal training data.

But the aggregate count leaves an unresolved question. Are the previously exposed continuations now inaccessible, or has the collection of extractable text changed?

The difference becomes visible in research by Ali Satvaty and colleagues. For Llama-3-8B-Instruct, using 32-token prefixes, the number of exposed continuations falls from 116 without retrieved context to 95 with relevant retrieved context. Yet the two exposure sets overlap imperfectly.

The reported counts and overlap imply approximately 88 continuations remain exposed under both conditions, 28 lose exposure, and seven become newly exposed. The net reduction of 21 therefore conceals two opposing changes.

For a team concerned about a particular sensitive passage, the seven newly exposed continuations matter even though the overall count improves.

This is the paper’s central contribution: retrieval changes not only how much text qualifies as extractable, but also which text qualifies.

A stable core persists as prefixes get longer

The researchers evaluated 50,000 randomly sampled prefixes from Pile-CC against three instruction-tuned models: Llama-3-8B-Instruct, OLMo-2-1124-7B-Instruct, and Falcon-7B-Instruct.

Each example had a fixed 32-token target suffix. Prefixes contained 32, 64, or 128 tokens. The same suffix was scored under three conditions: no added context, a semantically relevant retrieved document, and an irrelevant retrieved document.

Rather than requiring a model to reproduce the suffix in one generated response, the researchers measured the log-probability assigned to the complete target continuation. An example was classified as exposed if its suffix log-probability exceeded -16. This is a probability-based classification, not evidence of an observed disclosure.

The researchers then compared exposed sets using Jaccard overlap: the number of examples exposed under both conditions divided by the number exposed under either condition.

The main overlap results are striking.

Jaccard overlap between empty and relevant retrieved context; Appendix C, Table 2.

For all three models, exposure stability rises substantially as prefixes grow longer. At 128 tokens, more than 91% of the union of exposed examples is shared between the two conditions.

These percentages do not describe the fraction of all 50,000 samples that were exposed. They describe agreement among examples exposed in at least one condition. The absolute exposed counts remain relatively small.

The interpretation is nevertheless consequential. A longer prefix supplies more information about its original continuation, making that continuation’s probability less sensitive to preceding retrieved text. The study observes this pattern consistently, although it does not establish an internal neural mechanism.

The resulting picture has two components: a context-robust core whose exposure persists, and a context-sensitive boundary whose membership changes.

Exposure changes concentrate near the probability threshold

The boundary is not distributed uniformly across examples.

Section 5.2 and Appendix B examine how far each sample’s empty-context extraction score lies from the exposure threshold. Examples near that threshold change classification more frequently when context is introduced. Those substantially above or below it rarely switch.

This helps explain why retrieval can produce both suppression and newly enabled exposure.

A relatively strong continuation may remain above the threshold despite contextual changes. Another example may lie just below the threshold, so a modest increase in suffix likelihood is enough to classify it as newly exposed.

The measurement threshold also matters. The main cutoff of -16 corresponds to a probability of approximately $1.1 \times 10^{-7}$ for the complete 32-token suffix. It does not imply that the model will ordinarily generate that suffix in a single response.

The authors repeat their exposure analysis across thresholds from -22 to -10. The qualitative pattern persists: robust exposures remain prominent, while suppressed and context-enabled cases continue to appear.

This sensitivity analysis strengthens the finding that exposure is not invariant to retrieval. It does not transform a probabilistic threshold into an observed attack-success rate.

Relevant retrieval and additional context have different effects

A plausible explanation for lower extraction scores is that a relevant document directs the model toward retrieved information instead of its memorized continuation.

The experiments show why this explanation is incomplete.

The researchers separate two measured contrasts:

• Generic context effect: the change in suffix log-probability when an irrelevant retrieved document is added to an otherwise isolated prefix.
• Semantic context effect: the additional difference when the irrelevant document is replaced by a relevant one.

The generic effect is model-dependent. For 32-token prefixes, Table 1 reports mean generic effects of +0.52 for Llama, -0.42 for OLMo, and -0.94 for Falcon among the selected exposed or boundary examples. Simply adding text does not consistently reduce extraction likelihood.

Relevant context often produces an additional suppressive effect, particularly at shorter prefixes. For Llama’s selected 32-token examples, the mean semantic effect is -3.35 log-probability units, and 70.1% have a negative semantic effect. At 128 tokens, the corresponding mean is only -0.32.

This pattern is not universal. Some model-prefix conditions show weaker effects, and individual examples can move in either direction.

There is also an identification boundary. Relevant and irrelevant documents are selected as nearest and farthest matches by an E5 retriever from 67,655 StackExchange, PubMed, and Wikipedia documents. Those documents can differ in properties beyond semantic relevance.

Consequently, the contrast describes differences associated with retrieval relevance, not a fully isolated causal effect of meaning alone.

For RAG evaluation, retrieval configuration deserves measurement alongside model identity and prompt length.

Robustness checks do not establish deployment safety

The supplementary experiments address two concerns about the main findings.

First, the threshold-sensitivity tests show that the robust-core and context-sensitive-boundary pattern is not confined to the principal cutoff.

Second, an instruction-framed evaluation preserves the increasing stability with prefix length. Relevant-versus-irrelevant Jaccard overlap rises from 0.694 to 0.927 for Llama, 0.693 to 0.920 for OLMo, and 0.740 to 0.958 for Falcon between 32-token and 128-token prefixes (Appendix D, Table 3).

A separate chat-template comparison substantially lowers measured memorization. But the template introduces an assistant-response boundary between the original prefix and its target suffix, changing the continuation task. This result cannot establish that chat formatting removes the underlying memorization.

The broader boundaries remain significant. The main experiments cover three approximately 7–8-billion-parameter open-weight models, one principal retrieval configuration, CommonCrawl-derived targets, and fixed 32-token suffixes. The study does not independently verify training-data membership for every evaluated model-sequence pair.

Most importantly, it scores target-suffix probabilities offline. It does not measure unauthorized disclosure in a production RAG system.

What enterprise RAG evaluations should measure

For an enterprise security team deciding whether to approve a RAG architecture, the practical extension is to preserve sample identity throughout memorization testing.

An evaluation should report the exposed set under each retrieval condition, rather than only its total size or mean extraction score. Samples can then be classified into persistent, suppressed, and context-enabled exposures.

The study also suggests three specific extensions to conventional testing. Include longer target prefixes because these produced the most stable exposures. Repeat comparisons across probability thresholds because boundary cases can change classification. Finally, test deployment-relevant prompt templates and retrieval configurations because contextual formatting can materially alter the measured continuation task.

These are Cognaptus evaluation recommendations derived from the experiments, not demonstrated production controls.

Organizations should separately assess two risk channels: potential extraction of text represented in the model’s parameters, and disclosure of information contained in the retrieved document database. Improving one channel does not establish protection against the other.

Where actual disclosure risk determines deployment approval, offline probability measurements should be complemented by tests of realistic generation and attack conditions. The current study offers no quantified conversion from exposure counts to production leakage rates.

The deployment decision depends on the exposed examples

The paper challenges an overly broad interpretation of retrieval-based privacy improvements.

A lower aggregate extraction count may reflect genuine suppression of some continuations. It can simultaneously coexist with a persistent exposed core and newly enabled exposures. Longer prefixes make the persistent component especially apparent.

For operators, the appropriate conclusion is a change in evaluation design. A model or retrieval configuration should not be judged solely by whether its aggregate memorization score improves. The identity and stability of exposed continuations must also be examined.

Whether those probabilistic exposures lead to real-world disclosure remains an empirical question for the specific model, application, documents, prompts, and attack conditions.

Cognaptus: Automate the Present, Incubate the Future.