Forgetting by Remembering: A Smarter Path to Machine Unlearning

TL;DR for operators

Deletion sounds simple until the deleted record has already shaped millions of model parameters. The clean answer is to retrain the model without that record. The operational answer is usually less glamorous: nobody wants to burn a full training cycle every time a user, regulator, data-quality team, or security analyst says, “Remove this.”

The paper behind Influence Approximation Unlearning, or IAU, proposes a useful reframing: approximate forgetting as a form of remembering in reverse.¹ Instead of using influence functions in the expensive classical way—compute a Hessian, invert it, and estimate how parameters should move after deleting a sample—the method asks whether deletion can be approximated through gradient-based incremental updates. In plain terms: if adding a point changes the model one way, then adding an “opposite influence” should imitate removing the original point.

That is the clever part. The careful part is that the authors do not stop at naive gradient ascent on the forgotten data. They add gradient correction to preserve behaviour on retained data, and gradient restriction during training to prevent individual samples, especially outliers, from having destabilising gradients when later deletion requests arrive.

The business takeaway is narrow but valuable. IAU points toward cheaper, faster model-maintenance infrastructure for repeated deletion requests, poisoned-data cleanup, and compliance-adjacent data hygiene. It is not a certificate of legal deletion, not a foundation-model eraser, and not a reason to throw away audit logs while humming about “privacy by design.” That would be adorable. It is an approximate unlearning mechanism whose value depends on validation, risk tier, and how closely “similar to retraining” matches the organisation’s actual obligation.

The real cost of forgetting is not memory, but recomputation

A deployed model does not store each training record as a neat row with a delete button beside it. Training data changes the model indirectly: through gradients, parameter updates, learned representations, class boundaries, and all the little statistical compromises that training quietly makes. So when a user requests deletion, the hard question is not “Can we remove the row from the database?” That part is clerical. The hard question is whether the trained model still behaves as if that row had participated in training.

The gold standard is retraining from scratch on the retained dataset $D_r$, excluding the forgotten subset $D_f$. This gives the cleanest target: the unlearned model should resemble the model that would have existed had the deleted data never been used.

The paper formalises this as an approximate goal:

where $h(D)$ is the original trained model and $h_u$ is the unlearning mechanism. That approximation hides the whole problem. Similar in what sense? Test accuracy? Membership-inference resistance? Internal activations? Parameter distance? Regulatory defensibility? The paper focuses on three practical properties: removal guarantee, unlearning efficiency, and comparable model utility.

The tension is familiar. Exact unlearning methods try to give stronger removal behaviour, often by partitioning training data and retraining affected shards. Approximate methods try to adjust model parameters directly. Influence-based unlearning belongs to the second family: estimate how much a point affected the model, then move the parameters as if that point had not been there.

Elegant, but expensive. Influence functions usually require Hessian or inverse-Hessian calculations. In a small convex model, this is tolerable. In a large neural network, the Hessian is where optimism goes to become a budget request.

Influence functions know where to move, but charge heavily for directions

The paper’s starting point is the standard influence-function approximation. If removing a sample $z^{-}$ changes the optimum parameters from $\theta^\ast$ to $\theta^\ast_{{z^{-}}}$, then the change can be approximated as:

This is attractive because it points to the direction in parameter space that deletion should induce. It is unattractive because $H^{-1}_{\theta^\ast}$ is the inverse Hessian over model parameters. For modern neural networks, “just compute the inverse Hessian” is less an algorithmic instruction than a small act of violence against the infrastructure team.

The authors’ key move is to compare deletion with addition. They show that adding a point $z^{+}$ has the opposite form:

So, if one could add a point whose gradient is approximately the negative of the point to be forgotten,

then adding that point would imitate deleting the original point.

That is the paper’s conceptual bridge: forgetting can be treated as counterfactual incremental learning. The model does not need to “forget” by solving the full second-order problem. It can approximate the same effect by updating as if it were learning an opposing influence.

The practical shortcut is even simpler. Instead of constructing the actual opposite sample $z^{+}$, IAU applies gradient ascent on the forgotten point:

This is not merely “do gradient ascent and hope.” The paper positions the update as a first-order approximation to the influence-function deletion direction. The Hessian is not solved; it is bypassed.

That distinction matters. A lot of approximate unlearning can look like parameter vandalism with a research citation attached. IAU at least gives the vandalism a map.

IAU is three mechanisms, not one trick with a nicer name

The full method has three parts. Treating IAU as only “gradient ascent on deleted samples” misses why the method works as well as it does in the experiments.

The combined unlearning update is:

The first term keeps the retained dataset anchored. The second pushes away from the forgotten dataset. The effect is not just deletion pressure; it is deletion pressure plus behavioural stabilisation.

The gradient-restriction part happens earlier, during model training:

This matters because unlearning quality is partly determined before any deletion request arrives. If a few samples have unusually large gradients, deleting those samples later can cause disproportionate parameter movement. The paper’s GR loss tries to train a model whose per-sample gradients are smaller and more uniform, so future unlearning updates are less explosive.

For operators, this is the most interesting design lesson in the paper: unlearning should not be bolted on only after deployment. If deletion requests are expected, the model should be trained to be deletable.

The missing Hessian is the business case

The complexity analysis gives the business case its skeleton. The paper describes IAU’s unlearning cost as first-order gradient computation, with time complexity $O(nt_1 + nk_1p)$ under its notation. Hessian-based influence approaches require substantially heavier second-order computation: direct Hessian handling involves terms such as $O(nk_2p^2)$ and inversion can add $O(p^3)$. Even approximate inverse-Hessian methods remain much more expensive in the authors’ analysis.

The exact symbols matter less than the shape of the argument: IAU scales with gradients; influence-function unlearning pays for curvature. In large models, curvature is the expensive part.

This is where the paper becomes operationally relevant. If deletion requests are rare, a firm can often queue retraining, absorb delay, or manually handle high-risk cases. If deletion requests are frequent, or if contaminated data must be cleaned repeatedly, retraining becomes a maintenance tax. IAU offers a way to turn deletion into a lighter parameter-update routine.

But “lighter” should not be confused with “free.” IAU still computes gradients over retained and forgotten data. It aggregates deletion into one update rather than retraining from scratch or repeating an expensive second-order process per deletion event. That is a meaningful infrastructure improvement, not a miracle. Miracles rarely come with PyTorch version numbers.

The experiments mainly show speed with competitive approximation

The experiments compare IAU against retraining, USGD, Bad Teaching, Amnesiac Unlearning, and Fisher-style influence scrubbing. The main datasets are CIFAR10 and SVHN with LeNet5 and ResNet18. Complementary experiments use Purchase100 with an MLP, CIFAR100 with VGG19, and an outlier-removal task on SVHN.

The metrics are important:

• Model Utility (MU) is the test-accuracy gap between the unlearned model and the retrained model. Lower is better.
• Unlearning Time is runtime in seconds. Lower is better.
• Unlearning Efficacy (UE) is the gap in membership-inference attack success between the unlearned model and the retrained model on the erased data. Lower is better.
• Average Rank summarises the trade-off across the three metrics. Lower is better.

A condensed reading of the results looks like this:

The strongest consistent result is runtime. IAU is the fastest method across the reported settings. That is the cleanest empirical claim.

The more nuanced claim is quality. IAU often achieves the best or near-best average rank, but it does not dominate every metric in every setting. On CIFAR100 with VGG19, for example, IAU has the best UE and runtime but a worse utility gap than the alternatives. On LeNet5/CIFAR10, it is very fast but not the top average-rank method. This is not a flaw in the paper; it is the trade-off the paper is actually about.

The article version of the result is therefore not “IAU wins everything.” The useful claim is: IAU buys major speed improvements while remaining competitive on utility and membership-inference-based unlearning efficacy. That is a stronger and more honest sentence. It also has the rare advantage of being true.

The visual and ablation tests explain why the method is not just fast

The activation-map experiment is qualitative evidence. The authors unlearn one CIFAR10 image in a ResNet18 model and compare activation maps across methods. Retraining produces a modest shift, which makes sense: removing one sample should not annihilate everything related to its class. USGD appears to shift attention strongly, suggesting excessive removal of shared information. Bad Teaching and Amnesiac Unlearning diverge more visibly from retraining. Fisher and IAU look closer to the retrained pattern, with IAU also preserving activation structure on the retained image.

This is not the main quantitative proof. It is a diagnostic illustration of the failure mode: bad unlearning can make the model behave too differently, which may itself leak that something was removed. The paper even frames this as a kind of “Streisand effect.” A deletion mechanism that makes deleted samples theatrically strange is not exactly subtle.

The ablation study is more important. It tests Incremental Approximation (IA), IA plus Gradient Correction (IA+GC), and the full IA+GC+GR method. The pattern is intuitive:

The ablation tells a useful story. IA alone forgets aggressively but damages utility. Adding GC improves utility but weakens efficacy. Adding GR improves the balance by making gradients less pathological. In other words, the method’s performance is not coming from a single clever sign flip. It comes from shaping both the unlearning update and the original training process.

The GR experiments add another layer. The paper reports fewer epochs under early stopping when using GR loss across LeNet5 and ResNet18 on CIFAR10 and SVHN. It also compares the gradient distribution after convergence against original loss, gradient clipping, and SignSGD-style restriction. The authors argue that ordinary clipping and SignSGD do not guarantee small real sample gradients in the trained model, whereas GR directly penalises gradient magnitude in the loss.

For business readers, the implementation lesson is subtle: the system must decide at training time whether future unlearning matters. Waiting until the deletion ticket arrives may be too late to get the cleanest behaviour.

Where this fits in an enterprise model stack

IAU is best read as a model-maintenance primitive. It is relevant when the organisation needs faster approximations to retraining in settings where exact retraining is too slow or too expensive for every request.

The most plausible near-term use cases are:

The important distinction is between what the paper shows and what Cognaptus infers.

The paper directly shows that IAU can be much faster than retraining and several baselines while often achieving competitive utility and membership-inference-based efficacy on benchmark classification tasks. It also shows that the three components address different failure modes.

The business inference is that approximate unlearning can become part of a deletion operations pipeline: fast unlearning first, validation second, periodic retraining third, high-risk cases escalated. That pipeline is plausible because IAU reduces the cost of the first step.

What remains uncertain is whether this approach is sufficient for high-stakes regulated deletion, large generative models, retrieval-augmented systems, recommender systems with continual updates, or models whose risks are measured by memorised text rather than classification behaviour. Those are not small details. They are where the expensive lawyers and infrastructure teams live.

The misconception: approximate unlearning is not legal erasure in a lab coat

The obvious over-reading is to say that IAU “solves the right to be forgotten for AI models.” It does not.

The paper evaluates approximate unlearning by similarity to retraining, model utility, runtime, and membership-inference-based efficacy. That is useful. It is not the same as proving that no information remains, that all downstream artefacts are clean, or that a regulator will accept the process as sufficient.

There is also a theory-to-practice boundary. The proof of the incremental approximation relies on assumptions such as strict twice differentiability and convexity. The experiments apply the idea to neural networks, where those assumptions do not neatly hold. This is normal in machine learning papers, but normal does not mean irrelevant. The theory motivates the update; the experiments support that it works in the tested settings. They do not turn non-convex deep learning into a clean courtroom theorem.

The evaluation boundary matters too. Membership inference is one useful privacy-oriented test, but it is not a full audit of information removal. The target model may leak through other channels. A production system may include embeddings, logs, feature stores, caches, fine-tuning datasets, prompts, vector indexes, analytics copies, and monitoring artefacts. Deleting influence from one trained classifier is not the same as deleting a user from the organisation’s entire AI estate.

This is the sober interpretation: IAU improves the engineering economics of approximate unlearning. It does not eliminate the need for data lineage, model inventory, validation, periodic retraining, and risk-based governance. Sorry. The spreadsheet survives.

The deeper idea is training models to be easier to clean

The most durable contribution may not be the exact update rule. It may be the design philosophy.

Most unlearning discussions start after the mistake has already happened: the data was used, the model was trained, and now someone wants removal. IAU points to a different posture. If future deletion is expected, train the model in a way that makes deletion less destructive. GR loss is an example of that posture: constrain sample gradients now so that future unlearning updates are less unstable.

This shifts unlearning from a post-hoc patch to a lifecycle design problem. A serious AI governance stack would not treat deletion as a one-off emergency. It would classify models by deletion exposure, choose training regimes accordingly, log the relationship between training data and model versions, define when approximate unlearning is acceptable, and specify when retraining is mandatory.

IAU does not provide all of that. It gives one technical piece that makes the operating model more plausible.

The slogan version is tempting: forgetting by remembering. The operational version is better: reduce deletion cost by converting second-order influence removal into first-order gradient maintenance, then train the model so those gradients behave. Less poetic, more useful. Conveniently, businesses tend to pay for the second one.

Conclusion: cheaper forgetting changes the maintenance equation

Machine unlearning is usually framed as a privacy problem. It is also a maintenance problem. Models ingest flawed, sensitive, stale, poisoned, ambiguous, and later-withdrawn data. Organisations need ways to correct that history without rebuilding the entire system every Tuesday.

IAU’s contribution is to recast forgetting as incremental learning against the influence of deleted data. The mechanism is attractive because it avoids Hessian inversion, corrects for retained-data utility, and uses gradient-restricted training to make later unlearning less brittle. The experiments support a clear practical claim: IAU is consistently fast and often competitive across the tested datasets, models, and deletion scenarios.

The boundary is equally clear. This is approximate unlearning, not certified oblivion. It belongs in a governed workflow, not in a slide deck labelled “compliance solved.”

Still, the direction is important. If deletion becomes a frequent operational requirement, the winners will not be the systems that forget perfectly once. They will be the systems designed to forget cheaply, repeatedly, and measurably.

Cognaptus: Automate the Present, Incubate the Future.