Add the Skill, Keep the Answers

TL;DR for operators

A model team can add coding capability to an existing assistant while still needing answers on established workloads to remain stable. ATLAS treats that as a joint release problem: reach the required coding performance while limiting how much the adapted model’s token predictions move from the frozen model on retained inputs. On Qwen3-8B with GSM8K retained, ATLAS has lower mean retained-output KL than every published baseline configuration tested at the same 13 MBPP+ score requirements, with reductions of 10.16% to 17.60%.

The largest mechanism gain comes from using the frozen model’s retained-domain hidden states as reference points for each token update: centering around those routed states lowers KL by 38.44% relative to a plain residual, while distance and directional conditioning add smaller preservation gains. For release teams, the operational implication is to require both a target-skill threshold and a bound on movement over retained workloads. The paper does not show that retained-task accuracy necessarily rises or that non-target behavior is frozen.

A new capability can quietly rewrite an old one

A model team adds coding skill to an assistant already used for mathematics or other stable workflows. The new version improves on code but also changes answers nobody intended to change.

That makes target accuracy an incomplete release criterion. If two fine-tuning methods reach the same coding level, the operationally preferable one may be the one that disturbs established behavior less, especially when changed answers trigger regression review.

The paper by Jiangtao Lin, Bangyang Wei, Yihang Ding, Siyi Liu, and Yuhan Dong introduces ATLAS for this trade-off.¹ On Qwen3-8B with GSM8K retained, ATLAS has lower mean retained-output KL than all eight published baseline configurations at the same 13 MBPP+ score requirements, covering 148 to 160 solved problems. The reduction ranges from 10.16% to 17.60%.

The comparison is conditioned on comparable target learning. ATLAS is not rewarded for preserving behavior merely by learning less.

Retained examples become part of the adapter’s geometry

The method starts from hidden states produced by the frozen model on retained-domain examples. These activations are clustered into local regions, each storing a center and dominant directions estimated with PCA. Together they form an activation atlas.

For each token, ATLAS softly routes its hidden state toward nearby regions, constructs a token-specific reference center, and attenuates directions strongly represented in that retained geometry. One shared low-rank residual then reads the centered, filtered state and emits a filtered update. Only the low-rank factors are trained; the backbone and atlas stay fixed.

The atlas does not select separate experts. It changes how the same trainable residual acts on each token.

The structural ablations show where most of the preservation comes from. Centering around routed retained states lowers KL by 38.44% relative to a plain residual. Full ATLAS lowers KL another 4.64% relative to the centered residual and 2.31% relative to calibrated distance scaling.

Directional geometry adds a smaller but measurable contribution. Under matched effective residual norm across four Qwen backbones, ATLAS has lower KL in 11 of 12 comparisons, and fitted local bases beat random local bases in all 11 matched comparisons. These are mechanism checks: they support a role for update direction beyond simply shrinking the residual.

The benchmark controls for how much coding was learned

Preservation is measured as average next-token KL divergence from the frozen model on fixed teacher-forced retained contexts. Lower KL means the adapted model’s predictive distribution moved less on those examples.

Methods are compared at shared achieved MBPP+ requirements. For each required coding score, the method receives its lowest observed retained KL among checkpoints that reach at least that score. This directly asks how much behavioral movement is needed to obtain a given level of new capability.

Beyond the main Qwen3-8B comparison, all 54 primary seed-level comparisons against Plain, Replay, and Output-KL favor ATLAS across five backbones and two retained domains on pairwise shared target-score ranges. The reported aggregate geometric KL reduction is 62.03%.

That supports the paper’s comparative benchmark claim within the tested protocol. It does not establish causal identification, and some tested backbones do not learn the target task well enough for the same trade-off to be evaluated broadly.

Lower KL has an answer-level counterpart

A lower distributional divergence would matter less operationally if answer identities changed just as often. The paper therefore examines answer churn.

On Qwen3.5-9B with GSM8K retained, rewritten answers fall from 22.43% for matched comparators to 10.18% for ATLAS, a 54.60% reduction. Harmful correct-to-wrong flips and incidental wrong-to-correct changes both fall by about 60%.

That distinction is central. Preservation is not retained-task improvement. Stabilizing behavior can prevent regressions while also suppressing accidental corrections.

On CSQA, conditional-choice churn falls by 38.42%, while conditional-choice accuracy is almost unchanged: 85.22% for ATLAS versus 85.20% for matched comparators. In eight target-matched Qwen3-8B comparisons with LoRA-Null and OPLoRA, changes among distinct incorrect answers fall from 123 to 96, a 21.95% reduction; the paired-question bootstrap interval is 3.03% to 40.68%.

The evidence therefore connects lower distributional movement to observable version-to-version continuity, not to a general accuracy gain.

For model releases, specialization becomes a two-gate decision

Cognaptus inference: teams specializing an existing model should consider a joint release gate.

One gate asks whether the new capability clears its required threshold. The second asks how much the model moved on workloads the organization intends to preserve. ATLAS suggests retained examples can help shape the adapter itself, not only score the second gate after training.

This is most relevant to platform and product teams for whom answer churn creates downstream test failures, manual review, or revalidation cost even when aggregate accuracy changes little.

There is a deployment cost. On Qwen3-8B, ATLAS slows decoding by 2.52% to 3.13% and prefill by 5.83% to 12.26% relative to the frozen model; the compressed atlas is 15.85 MiB. The decision is therefore whether lower behavioral movement justifies retained-domain calibration data and modest serving overhead.

The boundary is model- and task-dependent

The evidence is strongest for MBPP coding specialization with GSM8K or CSQA retained under single-layer adaptation. A CSQA-as-target experiment yields only a 0.104 percentage-point best gain. Mistral-7B and Llama3-8B show no positive mean MBPP+ learning under the tested protocol, and Gemma-2-9B has no sustained target-learning interval.

Atlas resolution also does not transfer uniformly. Qwen settings generally favor local charts, while GLM-4-9B strongly favors a one-chart Global-PCA construction. Atlas granularity therefore needs validation as part of the deployment configuration.

ATLAS makes a focused contribution: retained behavior can remain computationally active while a new skill is learned. For teams shipping specialized model versions, the broader lesson is to measure target performance and behavioral continuity together—and to consider retained examples as part of the control mechanism, not only the regression test.

Cognaptus: Automate the Present, Incubate the Future.