Merge Without Mayhem: How Orthogonal Deltas Could Revolutionize Model Composition

TL;DR for operators

Model composition usually sounds harmless until someone asks the obvious production question: “Can we remove that client-specific update without retraining the whole thing?” At that point, many elegant AI stacks quietly become sedimentary rock.

The MDM-OC paper proposes a cleaner model lifecycle: keep a shared base model, express every fine-tuned specialist as a task delta, orthogonalize those deltas so they interfere less, merge them with tuned coefficients, and subtract a selected delta later when a capability, customer, or data source needs to be removed.¹ The important claim is not “we found another averaging recipe.” The claim is that model updates can be treated as separable components in parameter space.

The evidence is directionally strong but not production-final. The paper reports 78.4% accuracy on CIFAR-100, 6.3 percentage points above the best baseline cited there, and far smaller unmerge accuracy drops than Task Arithmetic, TIES-Merging, and AdapterFusion. It also reports memory growth from 2.1GB to 8.7GB when scaling from 5 to 50 merged models, compared with GEM’s 47GB. The ablation table matters most: removing orthogonal projection drops CIFAR-100 performance from 78.4% to 72.2%, making the geometry the central mechanism rather than decorative mathematical furniture.

For business readers, the practical interpretation is straightforward. MDM-OC points toward AI systems where fine-tunes become auditable inventory: added, weighted, inspected, rolled back, and selectively removed. That has obvious relevance for enterprise AI, federated learning, compliance-sensitive deployments, and model operations teams trying not to rebuild everything whenever one component changes.

The boundary is equally straightforward. The framework assumes a shared base model and compatible update format. It is tested with ResNet-50 and BERT-large using LoRA rank 8, plus reported larger-scale checks on BERT-large and GPT-2 medium. It is not proof that arbitrary models from arbitrary vendors can be snapped together like office furniture. Sadly, IKEA has not yet entered neural parameter space.

The real problem is not training models. It is taking them apart.

Most organizations now understand fine-tuning. A support model gets tuned for one product line. A compliance assistant gets tuned on internal policy. A sales model gets tuned on customer objections. A regional model gets tuned on local language patterns. Eventually the team has a cupboard full of specialized models, each useful, each expensive, and each slightly annoying.

The naïve operational answer is to merge them. But merging models is not like merging spreadsheets. Fine-tuned weights are not passive files. They encode task-specific changes inside a high-dimensional parameter space, and those changes can collide. One update may improve a capability while degrading another. A second update may undo part of the first. A third may work only because the first two happen to be present in the right proportions. Congratulations: your model portfolio has become soup, but without the warmth.

MDM-OC, short for Modular Delta Merging with Orthogonal Constraints, starts from a different intuition. Instead of treating model composition as a bulk averaging problem, it treats each fine-tune as a displacement from a common origin:

That is the first important move. A task model is not stored conceptually as a whole new model; it is stored as “what changed from the base.” This is already how many adapter and LoRA workflows behave operationally. The paper then makes the second, more distinctive move: before merging those deltas, it projects them into orthogonal subspaces.

The merged model is then written as:

The $\alpha_i$ terms are merge coefficients. The $\Delta\theta_i^\perp$ terms are the orthogonalized task deltas. The superscript is doing a lot of work.

The paper’s central bet is that if task deltas are geometrically separated, they are easier to combine and easier to remove. This is the mechanism that makes the article worth reading. The benchmarks are useful. The reversibility numbers are useful. But the actual strategic idea is that a model update can be made closer to an independent module.

Orthogonal deltas are not just cleaner averaging

A common misread of this paper would be to file it under “another weight-averaging method.” That would miss the interesting bit.

Weight averaging asks: how do we combine several trained models into one parameter set? MDM-OC asks a more operationally useful question: how do we combine task-specific changes so that each contribution remains identifiable enough to later subtract?

The difference matters because model operations are not static. A deployed system may need to add a task, retire a task, remove a customer-specific update, roll back a bad adaptation, or comply with a deletion request. If the merged model is only a blended artifact, removal becomes difficult. If the merged model is a base plus separable deltas, removal becomes at least algebraically plausible.

The orthogonality condition is simple in form:

In plain English: after projection, the update direction for one task should not overlap with the update direction for another. The paper enforces this using Gram-Schmidt projection. For a new task delta, it subtracts the projection of that delta onto prior orthogonalized deltas:

The intuition is not mystical. If two updates push the model in overlapping directions, they can interfere. If each update is moved into a direction independent of the previous ones, the merge becomes more stable and the removal operation becomes cleaner.

The paper’s workflow follows five steps:

This is why mechanism-first is the right way to read the paper. The practical value does not come from “MDM-OC beats baseline X.” It comes from the fact that the method changes what a fine-tuned model is inside the operating system of AI: not a monolith, but a detachable contribution.

The workflow turns model updates into inventory

The business version of MDM-OC is not “better benchmarks.” It is inventory control.

In normal software, components can be installed, versioned, disabled, and rolled back. Machine learning systems have historically been worse at this. Once training data, task behavior, and gradient updates are absorbed into a model, they are hard to isolate. This is one reason AI governance often feels like documenting fog.

MDM-OC offers a cleaner mental model:

Shared base model
        ↓
Task-specific fine-tunes
        ↓
Delta extraction
        ↓
Orthogonal projection
        ↓
Weighted merge
        ↓
Composable model
        ↓
Selective unmerge when needed

The paper’s Figure 1 is best read as an implementation diagram, not just a conceptual sketch. It shows the base model, fine-tuned models, extracted deltas, orthogonal projection, evolutionary merging, continual addition, and reversible unmerging. Its purpose is to communicate the pipeline: the system does not simply average trained models and hope the result behaves.

The paper’s Figure 2, a PCA projection of task deltas before and after orthogonalization, is a diagnostic visualization. It is not the main evidence that the model works. It supports the mechanism by showing reduced overlap in projected delta space. In other words, it helps the reader see the geometry, but the performance claims still need the benchmark and unmerging results.

Figure 3, showing optimization over merge coefficient space, is also an implementation-oriented figure. It supports the idea that coefficient selection is a real part of the method rather than a footnote. One small textual wrinkle: the method section says the merge weights are optimized via Adam, while the implementation details, figure caption, and ablation table discuss CMA-ES. That inconsistency does not destroy the paper’s argument, but it does mean readers should treat “coefficient optimization matters” as the stable claim, rather than becoming too attached to one named optimizer.

Figure 4, on structured unmerging, illustrates the removal logic. It is the visual version of:

That formula is the reason the paper is interesting for compliance and lifecycle management. If a task contribution is stored as an orthogonalized delta with a known coefficient, then removal is not a vague aspiration. It is a defined operation.

Of course, defined does not mean legally sufficient. Removing a delta from a model is not automatically equivalent to satisfying every privacy, contractual, or provenance obligation. Lawyers remain tragically unorthogonal. But as an engineering primitive, reversible unmerging is a meaningful step beyond “we will retrain later and send a PDF.”

What the experiments actually support

The paper evaluates MDM-OC across vision and NLP settings. For vision, it uses CIFAR-100 split into 20 tasks of 5 classes each and ImageNet-100 split into 10 tasks. For language, it uses AG News, DBpedia, and Yahoo Answers as a multi-domain benchmark. ResNet-50 and BERT-large are used as base architectures, and all methods use LoRA with rank $r=8$ for consistent parameterization.

That last detail matters. The experiments are not a free-for-all comparison among incompatible systems. They are testing composition under a controlled adapter-style setup. This makes the evidence cleaner, but it also narrows the deployment interpretation.

The paper compares MDM-OC against continual learning and model-merging baselines including Sequential Fine-Tuning, EWC, SI, GEM, A-GEM, Fisher averaging, Task Arithmetic, TIES-Merging, AdapterFusion, LoRA merging, Model Soups, and DiWA. That is a broad enough comparison set to make the empirical results worth attention.

The evidence can be grouped by purpose:

The strongest evidence is not the broad claim that MDM-OC “wins.” The strongest evidence is the alignment between mechanism and ablation. When orthogonal projection is removed, CIFAR-100 performance falls from 78.4% to 72.2%. That 6.2-point degradation is larger than the reported degradation from removing CMA-ES coefficient optimization, removing EWC plus replay, or removing search-space reduction.

That matters because it tells us the core idea is not merely the surrounding machinery. The geometry is doing the work.

The unmerging result is the business result

The headline accuracy number is useful. The unmerging result is more strategically interesting.

In Table I, MDM-OC reports much smaller unmerge accuracy drops than the compared methods:

UAD, or Unmerge Accuracy Drop, measures how much performance deteriorates after a selected component is removed. Lower is better. The paper reports 1.8% for vision and 2.3% for language under MDM-OC, compared with much larger drops for the baseline merging methods.

This is not merely a model-quality metric. It is an operational control metric. A system that performs well only before removal is less useful in regulated or multi-tenant settings. Enterprises do not only ask, “Can we add this capability?” They also ask:

• Can we revoke it?
• Can we isolate it?
• Can we explain where it came from?
• Can we roll it back without harming everything else?
• Can we remove a client-specific contribution if the client leaves?

MDM-OC gives a technical vocabulary for those questions. A task contribution becomes a stored delta with a coefficient and an orthogonalized direction. The paper does not solve the whole governance stack, but it gives governance something more concrete to govern.

That is an underrated distinction. Many AI compliance discussions operate at the level of policy prose. MDM-OC moves part of the problem into model architecture and parameter management. The result is not compliance by magic subtraction. It is more like compliance by better bookkeeping, which is less glamorous and more useful.

Accuracy gains matter, but the ablation tells the real story

The paper reports that MDM-OC reaches 78.4% on CIFAR-100 and outperforms the best cited baseline, TIES-Merging, by 6.3 percentage points. That is a meaningful result, but the ablation study is more informative for a serious reader.

The ablation is doing its job: it asks which part of the system actually matters. The answer is fairly clear. Orthogonal projection contributes the largest improvement. Coefficient optimization matters, but less. Stability modules help, but less. Search-space reduction barely changes CIFAR-100 accuracy, though it may still matter for efficiency.

For operators, this suggests a hierarchy of adoption priorities.

First, the shared-base delta representation must be clean. If the deltas are poorly tracked, inconsistent, or generated from incompatible bases, the rest of the system collapses.

Second, orthogonal projection is the core differentiator. If an implementation skips or weakens that step, it is no longer really testing the paper’s main idea.

Third, coefficient optimization should not be dismissed. Equal weighting is attractive because it is simple, and simple things are beloved by teams that have not yet debugged them. But the paper’s results suggest that learned or searched coefficients help balance the merged model.

Fourth, EWC and synthetic replay are stability supports. They are not the conceptual core, but they help preserve robustness across repeated merges and unmerges.

This is a useful lesson for business readers. The paper is not saying, “buy a bigger GPU and all your fine-tunes will get along.” It is saying that model composition needs structure. Without structure, every merge is a negotiation among invisible parameter updates, and invisible negotiations rarely produce good governance.

Scaling is promising, but not permission to hoard deltas forever

The paper reports that MDM-OC scales linearly from 5 to 50 merged models, with peak memory growing from 2.1GB to 8.7GB. It contrasts this with GEM’s 47GB. The method uses delta compression and PCA-based subspace reduction, with the paper describing a reduction from more expensive orthogonalization to a lower-dimensional approximation.

This is encouraging for model operations because memory footprint is not a cosmetic metric. If every new task requires storing, replaying, or retraining large amounts of data or parameters, the system becomes expensive and slow. A compact delta inventory is easier to move, version, inspect, and deploy.

But “scales to 50” should not be read as “scales indefinitely.” The paper itself notes that projection complexity still restricts very large model pools. Orthogonalization has an $O(N^2)$ element in the number of components, even if dimensionality reduction mitigates the practical cost. At some point, a system may need hierarchy, clustering, approximate orthogonality, or a routing architecture.

There is also a conceptual scaling issue. If many tasks are closely related, strict orthogonality may prevent useful sharing. Suppose two customer-support fine-tunes both improve refund-policy reasoning, or two medical-domain adapters both encode overlapping terminology. Forcing their updates into separate directions may reduce interference, but it may also reduce beneficial transfer. The paper acknowledges this limitation. Operators should, too.

The practical takeaway is not “orthogonalize everything.” It is “use orthogonal deltas when reversibility, task isolation, and low interference are worth more than unconstrained sharing.” That is a narrower claim. It is also a better one.

What this changes for enterprise AI architecture

MDM-OC fits naturally into a more modular AI operating model.

The old model lifecycle looks like this:

Fine-tune → deploy → discover issue → fine-tune again → redeploy → hope nothing important broke

The MDM-OC lifecycle looks more like this:

Base model → task deltas → orthogonal registry → weighted merge → monitored deployment → selective add/remove

That changes the architecture discussion in four ways.

First, it encourages a shared-base strategy. If an enterprise wants composable models, it needs discipline around base model selection. Random fine-tunes from different foundations are not convenient raw material for MDM-OC. The shared base is the anchor.

Second, it makes delta provenance important. Each task delta should have metadata: source task, training data scope, validation performance, owner, creation date, allowed deployment contexts, and revocation conditions. Without that, reversible unmerging becomes technically possible but operationally confused.

Third, it supports customer-specific or department-specific specialization. A bank could imagine separate deltas for retail compliance, wealth-advisory language, fraud operations, and internal policy reasoning, all derived from a common base. A SaaS vendor could maintain client-specific behavior updates while retaining a unified deployment model. Again, this is an inference from the mechanism, not a direct production result from the paper.

Fourth, it creates a cleaner rollback path. If a fine-tune introduces bias, hallucination, data leakage risk, or degraded performance, subtracting its orthogonalized delta may be cheaper than rebuilding the model. That does not replace testing. It makes testing less existential.

The table below separates what the paper shows from what Cognaptus infers.

This is where the paper becomes more than a machine-learning contribution. It points toward a governance architecture in which model components are not merely trained, but catalogued.

The compliance angle is useful, but easy to overstate

The paper repeatedly connects reversible unmerging to compliance, including GDPR-style removal. The connection is reasonable. It is also not the same as proof of regulatory satisfaction.

There are at least three levels of removal:

1. Engineering removal: subtracting a task delta from a merged model.
2. Functional removal: showing that the removed contribution no longer affects relevant outputs.
3. Legal removal: satisfying the applicable regulatory, contractual, and audit standard.

MDM-OC mainly addresses the first and provides evidence relevant to the second. It does not fully settle the third. That is not a criticism of the paper; it is a boundary condition. Papers do not usually come with compliance officers, although that would make reviewer comments more entertaining.

The important point is that MDM-OC gives organizations a better primitive for compliance engineering. A system that can identify and subtract a component is easier to audit than one where all updates are blended into a single opaque parameter mass. But responsible use would still require logs, validation tests, provenance records, deletion policies, and post-unmerge evaluation.

In regulated industries, that distinction matters. A technical mechanism can support compliance without replacing the compliance process. The subtraction operation is a tool, not absolution.

The method’s boundaries are not incidental

The paper’s limitations are not boilerplate. They materially affect adoption.

The first boundary is the shared base model. MDM-OC assumes that task-specific models are fine-tuned from the same base. This is natural in disciplined internal AI platforms, but less natural when organizations inherit models from vendors, teams, acquisitions, or open-source experiments. If two models do not share a base, their deltas are not directly comparable in the way the framework requires.

The second boundary is strict orthogonality. Orthogonal separation reduces interference, but related tasks sometimes benefit from shared representations. If tasks are genuinely complementary, forcing separation may leave performance on the table. The paper’s task similarity analysis is encouraging, reporting lower performance variance for MDM-OC than for baselines, but the general question remains open.

The third boundary is scale. The paper reports tractable scaling to 50 merged models, aided by compression and PCA-style reduction. But enterprises with many product lines, jurisdictions, client variants, and domain adapters may exceed that number quickly. A practical deployment may need hierarchical delta organization, grouping related deltas before orthogonalizing, or approximate methods that trade perfect separation for tractability.

The fourth boundary is optimizer clarity. The paper’s text contains a mismatch between Adam and CMA-ES descriptions around coefficient optimization. This does not undermine the broad mechanism, but it should make implementers cautious. When reproducing the work, the exact coefficient-search procedure should be treated as an implementation detail requiring validation, not as a sentence to copy blindly into production.

The fifth boundary is evaluation realism. CIFAR-100, ImageNet-100, AG News, DBpedia, and Yahoo Answers are useful benchmarks. They are not the same as messy enterprise workflows with tool use, retrieval, policy constraints, multilingual customer data, and long-tail failure modes. The paper is a promising mechanism with supportive evidence, not a procurement-ready guarantee.

The strategic shift: from model deployment to model accounting

The deeper implication of MDM-OC is that AI teams may need to think less like model trainers and more like accountants of parameter change.

A fine-tune should not merely answer: “Did accuracy improve?” It should also answer:

• What base model does this update depend on?
• Which task or customer does this delta represent?
• What validation data justified its coefficient?
• Which other deltas does it interact with?
• Can it be removed cleanly?
• What happens to adjacent capabilities after removal?
• Who is authorized to merge or unmerge it?

That is model accounting. It sounds dull because mature infrastructure often does. Version control sounded dull until it became impossible to build software without it. MDM-OC belongs in that same category of ideas: not glamorous, but potentially foundational.

The paper does not fully build this infrastructure. It provides a method that makes such infrastructure more plausible. If task deltas can be stored, orthogonalized, weighted, and subtracted, then an enterprise model platform can maintain a registry of model components instead of a graveyard of full fine-tunes.

For AI vendors, this could become a product distinction. A platform that supports reversible composition has a better answer to customers asking for customization without lock-in, personalization without model sprawl, and compliance without ritualistic retraining. For internal AI teams, it could reduce the cost of experimentation: new capabilities can be evaluated as deltas before becoming permanent residents of the deployed model.

The key word is “could.” MDM-OC still needs reproduction, scale testing, and integration into real deployment systems. But the architectural direction is clear.

Conclusion: the merge is only impressive if the unmerge works

MDM-OC is interesting because it treats model composition as a lifecycle problem, not a one-time merging trick. The paper’s mechanism is clean: represent each fine-tune as a delta from a shared base, orthogonalize deltas to reduce interference, optimize merge coefficients, and support continual addition or algebraic removal.

The evidence supports the mechanism in the tested settings. The reported CIFAR-100 gain is notable. The ablation makes orthogonal projection look genuinely central. The unmerging table is the most business-relevant result, because it measures whether a system can remove a component without damaging everything around it. The scaling results are encouraging, though not unlimited.

For operators, the lesson is not to rush out and orthogonalize every model by Monday morning. The lesson is to rethink the unit of model management. A fine-tune should not be treated as an isolated artifact or a one-way modification. It can be treated as a governed, reversible, testable component.

That is the real promise of orthogonal deltas. They make model composition less like alchemy and more like inventory control. Less magical, perhaps. But in production AI, magic is usually just technical debt wearing a cape.

Notes

Cognaptus: Automate the Present, Incubate the Future.