Model Merging

Software agents fail in a familiar way. They do not always fail because they are stupid. Sometimes they fail because they are busy. They search too widely, inspect too much, edit too early, revise the wrong file, run out of context, and then collapse under the weight of their own half-formed investigation. In enterprise language: they generate activity before they stabilize a diagnosis. We have seen humans do this too, usually in Slack threads with too many tabs open. The machines are catching up nicely. ...

A coding agent can fail in two very different ways. One failure is obvious: it does not think enough. It sees an error report, guesses the wrong file, edits too early, and then spends the rest of the trajectory debugging its own mistake. Anyone who has watched an autonomous coding agent wander through a repository has seen this little tragedy. The machine is busy, but not necessarily useful. ...

Think Longer, Act Worse? What M2A Teaches About Reasoning Agents A coding agent does not fail only because it cannot think. Sometimes it fails because it keeps thinking after it should inspect the repository. Sometimes it writes a plausible explanation before checking the relevant file. Sometimes it burns the context window by wandering through hypotheses, each one almost reasonable, none of them decisive. The result is not stupidity in the familiar sense. It is a coordination failure: the model does not know when to reason, when to call a tool, when to absorb feedback, and when to edit. ...

In the era of foundation models, one challenge looms increasingly large: how to safely, scalably, and reversibly compose AI systems from multiple task-specific fine-tunings. Traditional solutions — from naïve weight averaging to adapter stacking — often create interference, forgetfulness, and compliance nightmares. But a recent paper introduces a promising new direction: Modular Delta Merging with Orthogonal Constraints (MDM-OC). Rather than combining entire model weights, MDM-OC treats each task-specific fine-tuned model as a delta from a shared base. Think of these deltas as compact, focused perturbations that encode only what changed to solve a given task. The twist? Before merging, each delta is orthogonalized — projected into a subspace that doesn’t overlap with others. This creates a modular, mathematically principled structure for interference-free integration. ...