When One Merged Model Is One Too Few

TL;DR for operators

Combining eight specialized AI adapters into one deployable adapter can reduce infrastructure complexity, but it can also discard capabilities when the adapters encode poorly aligned parameter updates.

Di Wu and Shuaidong Pan’s Group-wise Rank-Aware Model Merging (GRAM)¹ addresses this by clustering compatible task updates before merging them. It also allocates limited representational capacity according to the strength of each task’s learned directions. The merging procedure requires model weights, not task training or calibration data.

On eight GLUE tasks using Flan-T5-Base, two GRAM adapters achieved an average benchmark score of 83.8, compared with 81.7 for the strongest reported single-model baseline, TSV-Merge. Removing coalition formation also reduced GRAM’s ablation score to 81.7.

For deployment teams, the finding supports testing a small portfolio of compatible merged adapters rather than automatically requiring one universal adapter. The trade-off includes additional serving capacity, task routing, and per-task retention checks. The paper’s theoretical guarantees do not establish that every actual merge will improve downstream performance.

The performance cost of forcing eight tasks into one model

An AI team develops eight specialized adapters from the same foundation model. Each adapter has been trained for a different task and can be deployed independently. Maintaining all eight preserves specialization but introduces additional storage, version management, and serving complexity.

Merging them into one adapter appears to simplify deployment. The problem is that independently fine-tuned models do not necessarily learn compatible changes to their shared foundation model. A limited-capacity merged adapter can retain some changes while losing others.

This makes the choice of which tasks to merge consequential, even when the merging algorithm itself remains unchanged.

The GRAM paper illustrates this with a component-removal experiment on Flan-T5-Base across eight GLUE language tasks. The complete method achieved an average score of 83.8. Removing its coalition-formation component reduced the score to 81.7 (Section 5.5, Table 4).

The 2.1-point decrease is evidence that grouping contributed to performance in this configuration. It does not establish a universal gain of that size, but it challenges the assumption that maximum consolidation necessarily provides the best performance-cost balance.

The resulting deployment question has three parts: which adapters can share a representation, how much capacity each should receive, and how many merged adapters the organization can afford to maintain.

GRAM groups tasks by what their updates learned

GRAM begins with task vectors: the differences between a specialized model’s parameters and those of its shared pretrained backbone.

For task $i$, the update is:

Rather than comparing updates solely through their raw parameter values, GRAM uses singular value decomposition (SVD) to identify the directions represented in each update.

Two tasks can have overlapping learned directions, making them more compatible with a shared low-rank representation. Other tasks may require substantially different directions. Forcing those tasks into the same limited-dimensional representation can introduce subspace interference.

The authors characterize this geometric loss through principal angles. For two orthonormal subspaces of dimension $r$, the unweighted projection error is:

Here, $U_i$ represents the task’s left singular subspace, $B$ is a shared projection basis, and $\theta_j$ denotes their principal angles. Aligned directions contribute no geometric projection loss; increasingly misaligned directions contribute more.

GRAM converts this relationship into a clustering procedure. It calculates squared chordal distances between task-update subspaces, averages those distances across adapted layers, and applies Grassmannian k-means++ to form $K$ coalitions. Each coalition receives its own merged model.

The grouping is based on learned parameter geometry, not application labels.

On Flan-T5-Base/GLUE, the method grouped six tasks together—CoLA, MRPC, QNLI, QQP, RTE, and SST-2—while placing MNLI and STS-B in the second coalition.

A supplementary grouping analysis found an intra-group to inter-group subspace-similarity ratio of 4.06 for GRAM, compared with 0.87 for random grouping. The corresponding average benchmark scores were 83.8 and 82.4 (Appendix D.1, Table 7).

This robustness comparison supports the proposed mechanism: the selected partition was both more geometrically coherent and more accurate than the tested random partition. Average-linkage agglomerative clustering recovered the same grouping and score, suggesting that the subspace representation itself matters, not exclusively the choice of k-means++.

Compatible tasks still compete for limited rank

Grouping determines which task updates share a model. GRAM’s second contribution addresses how that shared model uses its limited capacity.

Within each coalition, task updates may contain different numbers of strong singular directions. Assigning identical ranks to every task can waste capacity on weak directions while excluding stronger ones elsewhere.

GRAM pools the squared singular values of member tasks and allocates the coalition’s rank budget to the largest values. Under its stated separable truncated-SVD objective, this greedily minimizes the sum of omitted singular-value energies.

The selected components are then assembled and orthogonalized to construct the coalition update.

The component ablations show why these two decisions should be distinguished.

Source: Section 5.5, Table 4. These are component ablations, not comparisons with separately optimized production systems.

Removing adaptive allocation cost 1.6 points, while removing the subspace-similarity component cost 3.0 points. The individual effects should not be added together because the components interact.

The rank-allocation optimality result also has a precise boundary: it minimizes individual truncated-SVD tail errors under a shared budget. It does not prove that the resulting merged matrix globally minimizes the complete multi-task reconstruction objective.

Two merged models perform well across three evaluated domains

The principal empirical comparisons use $K=2$ coalition models, while conventional merging baselines generally deploy one merged model.

Sources: Tables 1–2 and Appendix D.5, Table 9. GLUE averages combine task scores including accuracy and scaled correlation; vision averages are test accuracy percentages. Differences are benchmark-score points, not statistical effect sizes.

The Flan-T5-Base improvement is concentrated partly in tasks that benefit substantially from separating incompatible updates. MNLI increased from 74.7 under TSV-Merge to 82.8 under GRAM, while STS-B increased from 83.2 to 86.7.

The largest vision improvement appears on ViT-B/32. On ViT-L/14, however, the advantage is only 0.1 point, too small to support a strong reliability claim without repeated experiments.

The four-expert bilingual medical evaluation provides a separate comparison. Using Qwen2-7B-Instruct adapters, GRAM achieved 75.54% average relative improvement over the pretrained baseline, compared with 71.48% for calibration-dependent RegMean. Its average per-metric rank was 2.40 versus 3.40.

These medical figures aggregate ten heterogeneous accuracy, BLEU, and ROUGE measurements; 75.54% is not medical question-answering accuracy. RegMean also performed better on some individual multiple-choice metrics. This is benchmark evidence from public adapters, not validation of a deployed multi-institution clinical service.

The best number of adapters depends on serving costs

The sensitivity experiments show that increasing $K$ beyond two produces relatively small additional gains on GLUE.

For Flan-T5-Base, moving from two to three groups increased the average from 83.8 to 83.9. Keeping all eight task-specific adapters produced 84.6, but required substantially more deployed adapter parameters.

Relative to the one-model configuration, the paper reports approximately 1.8 million extra parameters for $K=2$ and 12.4 million for retaining all eight adapters (Appendix D, Table 6). These are parameter overheads, not measured end-to-end serving-cost savings.

For an organization already maintaining a common backbone, this suggests a concrete comparison: measure whether two or three merged adapters can meet task-level quality thresholds at a lower total operating cost than the original specialized portfolio.

Routing is part of that cost. The main evaluations assume the task identity or coalition is known when a request arrives. An optional Mahalanobis router achieved 100% group-routing accuracy for $K=2$ on the evaluated GLUE validation examples, using approximately 85 labeled routing examples.

That result demonstrates feasibility in the tested setting, not guaranteed routing across ambiguous real-world requests. It also means that data-free merging does not imply data-free task-agnostic deployment.

Geometric guarantees stop short of performance guarantees

GRAM’s theory explains why separate coalition bases can reduce interference. Under the optimal-basis projection objective, allowing each group its own rank-constrained basis cannot increase total projection error relative to one globally optimal basis.

The k-means++ approximation analysis adds a conditional result under an idealized latent-group model. Neither theorem guarantees higher downstream accuracy for arbitrary task collections.

The distinction becomes clear when complete reconstruction error is separated into two components:

The first term measures information lost outside the coalition subspace. The second measures how poorly the final merged matrix fits the original task update within that subspace. The paper’s direct bound for the implemented GRAM basis controls the first term, not the second (Appendix C.2, Equation 28).

Other boundaries affect deployment confidence. Results are reported from single seeds without confidence intervals, major evaluations cover limited model families, and the primary protocol relies on task-aware routing. Scalability experiments reaching 1,024 adapters use synthetic perturbed copies rather than equivalently large collections of independently developed real adapters.

The main vision table reports 88.7 for ViT-B/32 GRAM, while the component-ablation table reports 88.8. This small discrepancy does not change the broader pattern but warrants preserving the published distinction rather than treating the numbers as identical.

A deployment policy for specialized adapter portfolios

Cognaptus’ inference from these results is that adapter consolidation should be managed as a constrained model-portfolio decision, not simply a weight-compression operation.

For teams whose adapters share a pretrained backbone, a defensible evaluation would compare individual deployment, conventional global merging, and a small number of geometry-aware coalitions. Each configuration should be assessed using per-task quality thresholds, not only mean benchmark performance.

The same comparison should account for the number of stored adapters, routing errors, loading and memory requirements, latency, and operational maintenance. Where adapters originate from separate organizations, weight-only merging may reduce the need to centralize training data, but sharing model updates still requires privacy, authorization, and safety assessments. Geometric similarity does not establish that two capabilities are semantically compatible or safe to combine.

GRAM’s contribution is to make the allocation of shared model capacity more structured: first identify compatible learned directions, then distribute limited rank according to retained update energy. Its experiments indicate that this can preserve more task performance than forcing all adaptations into one model in several tested settings.

The broader lesson is conditional. The most economical merged system is not necessarily the one with the fewest adapters; it is the smallest portfolio that reliably meets the required capabilities and operating constraints.

Cognaptus: Automate the Present, Incubate the Future.