Continual Learning

Opening — Why this matters now Continual learning is supposed to be the adult version of fine-tuning: learn new things, keep the old ones, don’t embarrass yourself. Yet large language models still forget with the enthusiasm of a goldfish. Recent work complicated this picture by arguing that much of what we call forgetting isn’t real memory loss at all. It’s misalignment. This paper pushes that idea further — and sharper. It shows that most modern task alignment is shallow, fragile, and only a few tokens deep. And once you see it, a lot of puzzling behaviors suddenly stop being mysterious. ...

In the era of foundation models, one challenge looms increasingly large: how to safely, scalably, and reversibly compose AI systems from multiple task-specific fine-tunings. Traditional solutions — from naïve weight averaging to adapter stacking — often create interference, forgetfulness, and compliance nightmares. But a recent paper introduces a promising new direction: Modular Delta Merging with Orthogonal Constraints (MDM-OC). Rather than combining entire model weights, MDM-OC treats each task-specific fine-tuned model as a delta from a shared base. Think of these deltas as compact, focused perturbations that encode only what changed to solve a given task. The twist? Before merging, each delta is orthogonalized — projected into a subspace that doesn’t overlap with others. This creates a modular, mathematically principled structure for interference-free integration. ...