Forgotten Until Asked Differently: Unlearning Needs an Adversarial Sign-Off

TL;DR for operators

A model can pass an ordinary unlearning evaluation while still yielding supposedly forgotten information when the request is reformulated strategically. Gupta and colleagues demonstrate this gap in a controlled benchmark of LLM unlearning: on the 1% forget split, four fine-tuning-based methods produced average adversarial recovery rates between 72.8% and 84.3%, compared with 87.5% for the unprotected model.¹

For teams approving deletion requests, privacy controls, model releases, or unlearning vendors, the relevant distinction is between passing the expected test and remaining resistant when someone actively tries to recover the information. The paper supports adding adversarial recovery testing before sign-off.

It does not show that all LLM unlearning fails. The experiments use one Llama-3.2-3B-Instruct victim-model family, synthetic data, and adversarial testing limited to the smallest forget split. Its adversarial metric also depends on an LLM judge whose binary decisions show errors in a small human audit.

A deletion check can pass before the information becomes hard to recover

Suppose a model has been instructed or fine-tuned to forget selected information. The model is tested with ordinary questions about that information, performs as expected, and clears the evaluation. For an operational approval process, that can look like evidence that the deletion worked.

The problem is that the evaluation may only establish how the model behaves under the queries it was tested on. A user who actively wants the information is not required to ask those questions in the same way.

That distinction is the focus of Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation. The authors first compare six unlearning methods under one controlled setup and then challenge four fine-tuning-based methods with strategically constructed prompts.

The controlled dataset is TOFU, a synthetic benchmark containing 200 fictitious authors and 4,000 question-answer pairs. Because every method starts from the same TOFU-fine-tuned Llama-3.2-3B-Instruct checkpoint and uses the same forget splits and evaluation implementation, differences between methods are less entangled with incompatible experimental setups.

Strong ordinary-query scores coexist with high recovery rates

The standard evaluation includes Forget Quality, a metric intended to assess whether post-unlearning behavior resembles successful forgetting under ordinary benchmark queries.

Several fine-tuning-based methods score strongly by that measure. On the 1% forget split, Forget Quality is 0.919 for NPO, 0.999 for SimNPO, and 0.919 for AltPO. RMU is lower at 0.765.

The paper then asks a different question: what happens when prompts are deliberately constructed to recover the targeted information?

The authors evaluate eight attack suites, including roleplay, authority framing, system injection, direct and hypothetical rephrasing, multiple choice, structured chain-of-thought, and completion prompts. A separate hacker model helps generate and refine the attacks. An LLM judge scores the resulting answers for leakage, and responses exceeding a fixed leakage threshold count toward Attack Success Rate, or ASR.

The important result is not that the protected models leak at exactly these percentages in every setting. It is the coexistence of two observations within the same controlled experiment: some methods look highly successful under ordinary forgetting evaluation, yet most tested targets remain recoverable under many adversarial formulations.

That means a high Forget Quality score should not be interpreted as proof that the knowledge has been erased from model parameters. The experiment measures behavioral accessibility. Its evidence is consistent with a weaker form of success: the model may suppress targeted answers under the benchmark’s query distribution without making those answers robustly inaccessible.

The attack-level results reinforce that interpretation. Authority and system-injection prompts reach ASRs as high as 90% for AltPO, while multiple-choice prompts are less effective, ranging from 45% to 70% across the four methods. Vulnerability also correlates only weakly across unlearning methods. A prompt strategy that exposes one method is therefore not necessarily an equally good probe of another.

For red teams, a single attack template is consequently a poor acceptance test.

The unified benchmark matters before the attacks do

The adversarial result gets the attention, but the paper’s comparison design is also consequential.

Six methods are evaluated against the same starting checkpoint, forget sets, and OpenUnlearning metrics. The set includes two prompt-based approaches, SPUL and ICUL, alongside NPO, SimNPO, RMU, and AltPO.

Under clean-query evaluation, NPO, SimNPO, and AltPO maintain high Forget Quality across the 1%, 5%, and 10% forget splits. By contrast, SPUL and ICUL fall toward zero as the forget set grows: at 10%, their Forget Quality scores are 0.001 and 0.006 respectively.

The authors therefore restrict the subsequent adversarial study to the four fine-tuning-based approaches. That is an experimental scope decision, not evidence that prompt-based unlearning has been shown adversarially safer or less safe.

For vendor evaluation, the broader lesson is procedural. Comparing unlearning products from separate papers can mix differences in models, forget sets, implementations, and metrics with differences in the methods themselves. A common evaluation substrate removes some of that ambiguity before robustness testing even begins.

Attack Success Rate is a stress test, not ground truth

The adversarial metric creates its own measurement problem.

Across 1,280 adversarial responses, the paper’s LLM-judge leakage scores correlate strongly with ROUGE-L similarity to the target answers: Pearson $r=0.924$. That provides evidence that the judge is tracking answer recovery rather than producing arbitrary scores.

But the authors also perform a blinded manual audit of ten cases. Human judgment agrees with the resulting binary ASR classification in seven.

Three disagreements in ten examples are too few observations to estimate a stable error rate, but they are enough to reject an interpretation of ASR as unquestionable ground truth. Judge behavior and the chosen threshold can generate both false positives and false negatives.

For an operational acceptance gate, Cognaptus therefore infers a two-layer measurement design: use automated adversarial recovery testing for coverage, but calibrate the decision threshold against a substantially larger human-labeled sample before treating the resulting percentage as a compliance boundary.

Multilingual probing tests a different question

The paper also translates the forget-set questions into German, French, Spanish, Hindi, Chinese, Arabic, and Japanese. After back-translation checks, 279 of 280 translated questions are retained.

Across 1,116 model-probe evaluations, only 33 responses cross the paper’s ROUGE-L leakage threshold, an overall rate of 2.95%.

That number is much lower than the English adversarial ASRs, but it is not evidence that changing languages protects an unlearned model. The multilingual experiment uses clean translated questions rather than adversarially optimized multilingual prompts, and leakage is measured differently.

Its likely role is therefore a sensitivity probe: ordinary cross-lingual reformulation alone produces relatively little measured leakage in this setup. Whether strategic multilingual attacks would produce the same result remains unanswered.

Unlearning sign-off needs a recovery test

For privacy, deletion, governance, and vendor-acceptance workflows, the paper changes what constitutes sufficient evidence.

A clean-query forgetting score can remain useful. It tells an evaluator how a model behaves under a standardized set of expected requests and enables controlled comparison between methods.

What it does not establish is resistance to a user deliberately searching for another route to the same information.

Cognaptus therefore infers that an operational unlearning sign-off should separate at least two questions:

1. Does the model satisfy the ordinary forgetting benchmark while preserving acceptable utility?
2. Can targeted information still be recovered under a diverse adversarial search over prompt formulations?

Passing the first should not automatically answer the second.

The recovery suite should also vary by method rather than assuming one universal attack template, and its automated leakage threshold should be validated against enough human judgments to support the decision being made.

The evidence supports a stronger test, not a universal verdict

The study provides moderate evidence within a controlled setting. Standardized checkpoints and metrics make the method comparison relatively clean, while eight attack suites produce a substantial adversarial evaluation over the 1% forget split.

External validity is narrower. All victim-model experiments use Llama-3.2-3B-Instruct. TOFU contains fictitious authors rather than real personal, duplicated, or context-dependent information. Adversarial testing does not extend to the 5% and 10% forget splits, and the attack suite excludes adaptive multi-turn recovery, gradient-based attacks, broader automated prompt search, and indirect multi-hop extraction.

The human validation of the leakage judge is also only ten examples from one annotator.

Those boundaries do not erase the observed gap. They define what can be concluded from it: within this benchmark, ordinary evidence of forgetting is insufficient evidence of robust inaccessibility.

For an organization deciding whether an unlearning request is complete, that is enough to change the acceptance process. The next question after “does the model appear to have forgotten?” is whether someone trying to prove otherwise can still recover the target.

Cognaptus: Automate the Present, Incubate the Future.