The Sink That Remembers: Solving LLM Memorization Without Forgetting Everything Else

TL;DR for operators

Deletion is simple in a database. It is not simple in a neural network that has already used the deleted record to improve its internal machinery. That is the unpleasant little invoice this paper presents.

Gaurav R. Ghosal, Pratyush Maini, and Aditi Raghunathan study why repeated natural text is hard to remove from language models after training, then propose MemSinks, a training-time mechanism designed to make memorization easier to isolate later.¹ The important shift is not “better pruning.” It is architectural accounting. Instead of hoping that memorized text happens to live in a few removable neurons, MemSinks gives repeated sequences a controlled place to accumulate memorization during training.

The paper’s main operational message is this: if an organisation wants models that can later forget sensitive or copyrighted sequences without losing broad capability, it may need to plan for removability before training, not improvise it after deployment. Post-hoc unlearning remains useful, but the paper argues that it faces a structural handicap when memorization shares the same mechanisms as general language ability.

The evidence is staged carefully. In TinyStories experiments, post-hoc localization methods struggle more with natural repeated sequences than with artificial canaries. A naive attempt to force localization through gradient masking hurts generalization and still creates co-adaptation. MemSinks then preserves validation performance close to standard repeated-data training while making repeated sequences less memorized after sink neurons are dropped. At larger scale, the authors test 360M and 1.7B SmolLM-style models trained on SlimPajama mixtures with repeated TinyStories data; MemSinks keeps the generalization benefit of repetition while closing at least half of the memorization-validation loss gap.

For business readers, the result is promising but not yet a compliance appliance. It is strongest as a design pattern for training-time memory governance: track repeated or sensitive data, route its verbatim residue into removable capacity, and evaluate models with those components removed. Its boundaries are equally practical. The method depends on consistent sequence IDs or grouping metadata, focuses mainly on verbatim memorization, and still needs stronger evidence against adversarial extraction and frontier-scale deployment. So, no, this is not a magic “right-to-be-forgotten” button. The industry loves buttons. Reality keeps filing objections.

The real problem is not remembering; it is remembering with the wrong machinery

A language model can memorize a repeated sequence in two very different ways.

The convenient version is tidy. The model stores the sequence in some special-purpose internal location. Find that location, disable it, and the model forgets the unwanted text while keeping its useful language ability. This is the mental model behind many neuron-localization approaches to unlearning. It is also the mental model that makes policy people sleep better than they probably should.

The inconvenient version is messier. The repeated sequence is natural text, so it shares syntax, style, concepts, and statistical structure with the rest of the training distribution. The model does not treat it like an alien string taped onto the dataset. It uses the same representational machinery that helps it predict ordinary text. Memorization then becomes entangled with generalization. Remove the memorized material, and you may also damage the very components that made the model competent.

That distinction drives the entire paper. The authors are not merely asking whether a model memorizes. They are asking where the training process puts memorization, and whether that location remains removable without breaking useful behaviour.

This is why the paper’s mechanism-first framing matters. If memorization were a detachable blob, post-hoc neuron surgery would be an engineering nuisance. But if natural-text memorization is absorbed into general capability, then post-hoc removal becomes closer to extracting dye from water. You can still try. You may even remove some. But pretending the dye was sitting politely in one corner is how one ends up with optimistic dashboards and disappointing audits.

Natural repeated text is harder to remove than artificial canaries

The paper begins with a controlled comparison. The authors train models on TinyStories under two kinds of induced memorization.

The first setting, TS-Repetition, repeats 100 natural TinyStories sequences 128 times, alongside 20,000 unrepeated TinyStories sequences. These repeated examples look like ordinary language from the same distribution as the rest of the data.

The second setting, TS-Canary, appends random token canaries to stories and repeats those. These are atypical strings, closer to the artificial memorization probes often used in prior work.

That contrast is not decorative. It is the first hinge of the paper.

Artificial canaries are useful because they are easy to identify and measure. But they are not the same as a repeated contract clause, patient note, forum post, code snippet, or licensed paragraph that resembles the rest of the corpus. A canary is a neon sign. Natural text is a building in the same city.

The authors test post-hoc localization approaches, including pruning-style methods and integrated gradients. The evaluation asks two questions at once:

The results are deliberately sobering. Localization methods achieve partial success, but they struggle especially on natural repeated sequences. Integrated gradients is much more effective on canaries than on TS-Repetition. Removing neurons can raise loss on memorized examples, but it also risks degrading validation performance, especially when the memorized text resembles normal training data.

The learning curve adds the key interpretation. In TS-Repetition, the repeated-sequence loss and validation loss fall together. The model is not entering a clean “first learn general language, then memorize the repeated stories” phase separation. Memorization and generalization improve simultaneously.

That matters because it challenges a common operational assumption: “We can train normally now and clean up memorization later.” Maybe. But the paper’s evidence says that for natural text, later cleanup may be fighting the training dynamics themselves.

Gradient descent prefers the neat solution only when reality is unusually kind

The authors support the empirical result with a theoretical analysis in a simplified setting. The simplified model is not a production transformer, and it should not be read as one. Its job is to isolate the mechanism.

The setup separates possible features into a generalizing subspace and a memorization subspace. In principle, a model could memorize a repeated natural sequence in two ways.

The theoretical point is that gradient flow can be biased toward the entangled solution. In the paper’s simplified analysis, the minimum-norm bias of gradient descent favours reusing existing generalizing features rather than creating a clean, orthogonal storage location for the repeated natural sequence.

Translated into operator language: standard training does not automatically organise memory for later deletion. It organises memory for predictive efficiency. Those are not the same requirement. In fact, they may conflict.

This is the quiet sting of the paper. The failure of post-hoc unlearning is not framed as “existing localization tools are bad.” The deeper claim is that the standard training process may put memorization in places that make clean localization unlikely. Blaming the pruning method alone is a bit like blaming the locksmith after the architect poured concrete over the door.

Forced localization is too rigid and still not clean

The obvious next idea is to force separation during training. If repeated sequences are likely to create memorization risk, route their gradients into designated memorization neurons. Route everything else into generalization neurons. Then, after training, delete the memorization neurons. Simple. Elegant. Suspiciously elegant.

The paper tests this through a gradient-masking scheme. In each transformer MLP layer, neurons are partitioned into memorization and generalization groups. Gradients from repeated sequences update only the memorization block. Gradients from unrepeated examples update the generalization block.

This experiment has a clear purpose: it is not the proposed solution. It is an ablation of a tempting design philosophy. The authors are testing whether direct hard routing can create the desired separation.

It fails in two ways.

First, forced localization impairs generalization. The model trained with gradient masking has worse validation loss than standard training even before the memorization neurons are removed. The reason is intuitive once stated: repeated data can contain useful general information. If the shared/general neurons are denied that signal, they learn less. The policy may be clean, but the model becomes worse. Governance achieved by starving the model is not especially impressive governance.

Second, even when memorization is routed into designated neurons, removing those neurons still hurts validation performance as training progresses. The paper attributes this to co-adaptation. The memorization neurons may be segregated by gradient updates, but their activations still participate in the forward pass. Generalization neurons can adapt around signals produced by memorization neurons. Later, when those memorization neurons are dropped, the rest of the model no longer behaves like a model that never saw the repeated sequence.

This distinction is important. Localization of updates is not the same as independence of function. A component can be “where memorization is stored” and still become load-bearing for other behaviour. Enterprise AI teams may recognise the pattern from software systems: just because a module owns a feature does not mean the rest of the stack has not quietly built dependencies on it. Neural networks, being professionally inconsiderate, do the same.

MemSinks gives memorization a controlled place to accumulate

MemSinks is the paper’s proposed escape hatch. It tries to avoid both failures: the entanglement of standard training and the rigidity/co-adaptation of gradient masking.

The design uses sequence-dependent dropout over a pool of memorization sink neurons inside transformer MLP layers.

The mechanics are straightforward enough to be useful:

Repeated sequence
        ↓
Stable sequence ID
        ↓
Deterministic mask over memorization sink neurons
        ↓
Same subset of sink neurons activated across repetitions
        ↓
Unrelated sequences mostly do not update those same sink neurons
        ↓
Memorization accumulates in removable sink capacity
        ↓
At evaluation or removal time, sink neurons are dropped

There are three details worth not mangling.

First, MemSinks does not allocate a private neuron set to every document. In the TinyStories setup, the authors use a GPT-Medium-style model where 70% of MLP neurons are shared and 30% are allocated to the memorization pool. They explicitly note that there are far fewer memorization neurons than total sequences. The method uses deterministic masks from sequence IDs, not one bespoke filing cabinet per text.

Second, the shared neurons still learn from repeated sequences. This is the big difference from gradient masking. MemSinks does not throw away the generalization benefit of repeated data. It lets the model learn broadly useful structure while encouraging the verbatim residue to collect in a known place.

Third, the sink neurons are activated infrequently and selectively. That matters because memorization and generalization follow different training dynamics. Generalizable signals are reinforced across many examples. Sequence-specific memorization is vulnerable to interference from unrelated examples; a repeated sequence is learned when seen, then partially forgotten as other data updates the same parameters. In standard training, that learning-forgetting cycle happens throughout the model. In MemSinks, the chosen sink neurons are shielded from much of that interference, so sequence-specific residue can accumulate there.

This is the central mechanism: not “detect memorization after training,” but “shape the training dynamics so memorization prefers removable capacity.”

The TinyStories results test whether the mechanism works before scale complicates the picture

The first MemSinks validation happens in the controlled TinyStories setting. This is the right place to test the mechanism because the authors can manipulate repetition, sequence identity, and removal more cleanly than in open-ended web-scale pretraining.

The paper reports that standard training with repeated sequences outperforms training without repeated sequences on validation loss. That is important: repetition is not merely a liability. In this setup, repeated documents provide useful training signal.

MemSinks, evaluated with memorization neurons removed, achieves validation loss comparable to standard training with repetitions and better than the no-repetition baseline. This is the generalization side of the bargain. The model keeps the benefit of repeated data even after the sink neurons are removed.

On memorization, the contrast goes the other way. A standard model trained on repeated TinyStories drives loss on those repeated sequences close to zero. That is the “yes, it memorized” condition. With MemSinks, dropping the memorization neurons significantly increases loss on the repeated sequences, reaching roughly 66% of the loss of a standard model that did not memorize them.

That number is not “complete deletion.” It is meaningful mitigation. The practical interpretation is: MemSinks does not prove that every trace of a sequence is gone, but it shows that a substantial share of the verbatim advantage can be made removable while preserving validation performance.

There is also a subtle dynamic result. Later in training, loss on memorized sequences with sequence-tied dropout begins to increase. The authors interpret this as evidence that shared neurons may initially implement some memorization, but further training shifts more of the sequence-specific burden into the sinks. That is precisely the sort of behaviour the mechanism predicts. The sinks are not merely passive storage; their activation pattern changes the learning-forgetting trajectory.

The larger-scale experiments ask a more business-relevant question: can repetition help without becoming verbatim residue?

Small controlled tasks can be illuminating and deeply misleading. The paper therefore moves to larger pretraining experiments using SmolLM-style models of 360M and 1.7B parameters trained on 1B and 2B tokens, respectively. The data mixture is mostly SlimPajama, with 5,000 TinyStories examples included as an under-sampled domain.

The business-relevant scenario is this: an organisation may want to upsample a valuable domain because it improves performance on that domain, but not want the model to reproduce the exact repeated examples. Think technical manuals, support tickets, legal templates, code repositories, medical notes, or licensed educational content. The useful distribution is valuable. The verbatim residue is the problem. Annoyingly, the model is not born knowing your procurement constraints.

The experiments compare three broad regimes:

The reported result is the paper’s most practically interesting claim. MemSinks preserves the generalization benefit of repetition: validation loss is comparable to the repeated baseline and better than the deduplicated baseline. At the same time, after dropping sink components, MemSinks substantially reduces memorization, closing at least 50% of the gap between validation loss and training loss on repeated examples. In the more heavily repeated setting, the mitigation is even more pronounced.

Again, this is not a guarantee of perfect unlearning. But it is a credible proof of concept for a more useful objective: do not choose between domain adaptation and removability quite so crudely.

The ablations are mostly about practicality, not a second thesis

The paper’s supporting tests should be read according to their purpose. They are not all equal pieces of evidence, and treating them as a pile of “more experiments” would blur the argument.

The activation-ratio result is especially operational. If too many sink neurons are activated for too many sequences, the sinks are no longer well shielded from interference. The method degrades. That means MemSinks is not merely “add sparse capacity.” Its value depends on matching the activation pattern to the learning dynamics.

The sequence-ID noise result is also practical. The authors perturb repeated-sequence IDs and find robustness to small noise levels, up to 10%, but failure at high inconsistency such as 50%. That is exactly the kind of boundary that matters outside the lab. If your data pipeline cannot reliably recognise repetitions or group related sequences, the sink has nothing stable to remember into. The method does not abolish data governance; it makes data governance load-bearing.

What Cognaptus infers for enterprise AI governance

The paper directly shows that MemSinks can isolate a substantial part of repeated-sequence memorization under the tested conditions while preserving the generalization benefit of repeated data. It does not directly show a complete enterprise compliance workflow. That part requires inference.

The inference is still useful.

Modern AI governance often treats training data risk as an external process: filter the corpus, document provenance, deduplicate, redact, then train. After training, if something goes wrong, attempt unlearning, patching, refusal tuning, or retrieval-side controls. These steps remain necessary. MemSinks suggests an additional layer: design the model’s training dynamics so risky memorization is easier to locate later.

That shifts the control model from “prevent or repair” to “prevent, structure, and repair.”

The most valuable use case is not random internet-scale memory cleanup. It is controlled-domain training where the operator knows which repeated or sensitive groups matter. Examples include proprietary support logs, paid publisher corpora, internal knowledge bases, regulated customer data, or partner-provided datasets with contractual deletion terms.

For those cases, a MemSinks-like system could make “train on useful repeated data” less incompatible with “avoid verbatim retention.” The business value is not merely privacy theatre. It is the possibility of retaining performance gains from upsampling a scarce domain while reducing the downstream cost of removal.

That said, this only works if the organisation can maintain stable identifiers through tokenization, packing, streaming, and training. The appendix implementation makes this concrete: in the larger-scale setup, sequence IDs are hashes of document tokens, interleaved with token streams during preprocessing, and used to generate masks online. In other words, removability becomes a data engineering property, not just a model property. The governance team may now need to care about token-stream metadata. Glamorous? No. Important? Unfortunately, yes.

What remains uncertain before this becomes a product pattern

The paper is careful about its boundaries, and they matter.

First, the work is primarily about verbatim memorization of repeated sequences. That is already important, but it is narrower than “the model knows something it should not know.” A model can encode facts, associations, style, or derived information without reproducing exact text. MemSinks may inspire broader localization schemes using domain or topic annotations, but this paper does not prove that.

Second, the approach depends on consistent sequence metadata. The authors show robustness to modest noise, but high inconsistency damages isolation. In messy enterprise corpora, near-duplicates, partial overlaps, templated documents, copied clauses, and transformed records make sequence identity less obvious. The method’s real-world performance will depend heavily on how well these groups are defined.

Third, adversarial extraction remains an open test. A higher loss on repeated sequences after sink dropout is encouraging, but privacy risk is not measured only by average loss. Attackers search, prompt, paraphrase, and exploit rare completions. The paper explicitly names robustness to adversarial extraction as future work. Sensible. Necessary. Slightly inconvenient for anyone hoping to invoice this as a solved compliance product by Friday.

Fourth, the large-scale experiments are larger than toy settings but not frontier-scale. The 360M and 1.7B SmolLM-style runs are meaningful proof-of-concept evidence. They do not establish behaviour in models with much larger capacity, more diverse data mixtures, instruction tuning, RLHF, tool use, retrieval augmentation, or deployment-time guardrails. The method may scale well; the paper gives reasons for optimism. It does not remove the need to test.

Finally, dropping sink neurons is a coarse operation. The paper evaluates models with memorization neurons removed, showing preserved validation performance under its settings. But an operational system would need decisions about when to drop all sinks, source-specific sinks, sequence-specific masks, or more granular subsets. That design space is where research becomes infrastructure and infrastructure becomes meetings. Many meetings.

The strategic lesson: make memory auditable while the model is still being built

The strongest idea in the paper is not the specific implementation detail of masking MLP neurons. It is the strategic reversal.

Most unlearning workflows ask: “Given a trained model, can we find and remove the unwanted memory?” MemSinks asks: “Can we train the model so unwanted memory has a better chance of ending up somewhere removable?”

That is a more mature question. It treats memorization as an architectural and data-pipeline design concern rather than a post-deployment surprise. It also recognises that repeated data is not always bad. Sometimes repetition is how a model learns an underrepresented domain. Deduplication can reduce memorization risk, but it can also throw away useful signal. MemSinks tries to keep the signal while isolating the residue.

For operators, the paper suggests a practical evaluation template:

The paper does not answer all of these. It gives a technically coherent reason to ask them earlier.

Conclusion: the sink is less glamorous than the slogan, which is why it is interesting

MemSinks is not a universal cure for LLM memorization. It does not prove perfect forgetting. It does not dissolve copyright risk. It does not make data provenance optional. It does not save anyone from the eternal spreadsheet of compliance evidence. A tragedy, truly.

What it does offer is more useful: a mechanism for making memorization less accidentally entangled with general capability. The paper shows why post-hoc localization struggles when natural text is memorized through the same features that support language modelling. It shows why naive forced localization can damage generalization and still fail through co-adaptation. Then it proposes a training-time structure that lets repeated data remain useful while giving verbatim residue a place to collect.

That is the right direction for serious AI governance. Not magical erasure after the fact, but models designed with memory boundaries from the beginning.

The sink remembers so the rest of the model does not have to. For once, the plumbing metaphor is doing actual work.

Cognaptus: Automate the Present, Incubate the Future.