Catastrophic-Forgetting

Fine-tuning looks deceptively democratic. Every response token gets its little vote in the gradient. The commas, the boilerplate, the obvious connective tissue, the wrong kind of certainty, the genuinely task-bearing step in the middle of the answer: all are invited to update the model. A charmingly egalitarian arrangement. Also a rather efficient way to teach a model to forget things it used to know. ...

Forgetting That Never Happened: The Shallow Alignment Trap Forgetfulness is an expensive diagnosis. When an internal AI system performs well on last month’s support taxonomy, then underperforms after being fine-tuned on this month’s compliance cases, the obvious story is simple: the model forgot. That story usually triggers an equally obvious response: replay old data, retrain more broadly, freeze more parameters, or panic politely in a meeting while calling it “model lifecycle management.” ...