TL;DR for operators
A team that has trained separate mathematics, coding, instruction, multilingual, and safety adapters still has one deployment decision left: which saved checkpoint from each expert should enter the merged model. The checkpoint with the lowest held-out validation loss is the expert’s individual stopping point, called $T^\ast$, but that does not guarantee it will cooperate best with the other experts.
Across every tested merge method and model size, choosing the best duration from the evaluated checkpoint grid outperformed merging all experts at $T^\ast$ by default. The direction of the adjustment, however, reversed across operators: straightforward averaging generally favored earlier checkpoints, while methods that filter weak updates and resolve conflicting parameter signs—such as TIES and DARE+TIES—often benefited from additional training.
Checkpoint duration should therefore be selected jointly with the merge operator rather than fixed during expert training. Retain checkpoints on both sides of each expert’s standalone optimum, test them with the intended merge method, and evaluate safety separately from average task quality.
This choice affects training cost, checkpoint-storage policy, merged performance, safety retention, and whether consolidating several experts into one deployable model is worthwhile. The evidence supports this testing policy within the study’s Qwen 3.5 and QLoRA configuration, not a universal stopping rule.
The best standalone expert can be the wrong merge input
Early stopping answers a standalone question: at what training step does an expert reach its lowest held-out validation loss? The paper calls that expert-specific step $T^\ast$. A conventional pipeline saves each expert at $T^\ast$ and treats those checkpoints as the natural inputs to model merging.
That convention assumes checkpoint quality survives aggregation unchanged. It does not account for what happens when five independently specialized parameter updates are combined. Training that improves one expert may also make its updates larger, more idiosyncratic, or more likely to conflict with updates from other domains.
The study isolates this interaction by saving each expert at eight points: $0.25$, $0.5$, $0.75$, $1$, $1.5$, $2$, $3$, and $5$ times its own $T^\ast$. Five domain adapters are trained using rank-16 QLoRA over 4-bit Qwen 3.5 bases at 0.8B, 2B, and 4B parameters. The researchers then merge the adapters using Simple Averaging, Task Arithmetic, TIES, DARE+TIES, and Greedy Soup.
The main quality score averages mathematics, code, instruction-following, and multilingual performance. Safety is kept separate and measured through refusal behavior. This separation matters operationally: a merge can improve its average task score while weakening a capability concentrated in one expert.
Checkpoint policy reverses across merge methods
The main result is not a small correction around early stopping. Different operators favor different regions of the training trajectory.
| Merge method | Best-supported duration pattern | What the result changes |
|---|---|---|
| Simple Averaging | Usually before $T^\ast$, commonly around $0.25$–$0.75T^\ast$ | Stop experts early if every update will be averaged without filtering |
| Task Arithmetic | Broad, intermediate, and domain-dependent | Tune duration rather than applying a single early- or late-training rule |
| TIES | About $1.5$–$3T^\ast$ at smaller scales, shifting to $3$–$5T^\ast$ at 4B | Longer expert training may be valuable when the merge resolves interference |
| DARE+TIES | Generally flat or favorable beyond $T^\ast$, with scale variation | Extra training can be tolerated, but the optimum still requires validation |
| Greedy Soup | Often earlier or intermediate, with unstable safety retention | The selection objective must include capabilities that cannot be discarded |
Simple Averaging deteriorates sharply as training continues. By $5T^\ast$, its average quality is roughly 14 to 23 percentage points below its earlier peak, depending on model size. Some individual domains decline even more. Uniform averaging passes every expert update into the merged adapter, including specialized changes that do not coexist well.
TIES shows the opposite tendency. Its best durations occur after the standalone validation optimum: approximately $1.5$–$3T^\ast$ for the 0.8B model, $2$–$3T^\ast$ for 2B, and $3$–$5T^\ast$ for 4B. Selecting these later checkpoints improves TIES over its $T^\ast$ result by about 2.5 to 8.7 percentage points across scales.
DARE+TIES has a flatter response, but it also generally tolerates or prefers longer training. Task Arithmetic and Greedy Soup do not provide equally clean policies; their optima vary more across domains and scales. The practical conclusion is narrower than “overtrain the experts.” Longer training helps only when the merge operator can use the resulting specialization without transmitting too much interference.
Longer training creates signal and disagreement together
The paper’s analysis offers a plausible explanation for the reversal.
As experts continue training past $T^\ast$, their task-specific bias falls. They become more committed to their own domains. At the same time, prediction variance, error covariance, and parameter disagreement rise. Longer training therefore produces both potentially valuable specialization and more difficult merge inputs.
Simple Averaging cannot distinguish between those two effects. Every adapter update receives equal treatment, so domain signal and expert-specific noise enter the merged model together. Once disagreement becomes sufficiently large, the reduction in bias no longer compensates for the added interference.
TIES changes that trade-off. Rather than averaging all updates directly, it removes low-magnitude changes and resolves conflicting parameter signs before aggregation. DARE adds stochastic pruning and rescaling before applying TIES. Within the tested configuration, these operations appear able to discard part of the disagreement introduced by longer training while retaining stronger domain-specific updates.
This account is supported by the observed bias, variance, and covariance patterns, but it remains an explanatory interpretation rather than a causal demonstration. The experiments do not independently manipulate disagreement or prove that a particular filtered parameter caused a benchmark improvement.
The experts remain mergeable despite growing separation
One alternative explanation would be that extended training sends the experts into entirely separate regions of parameter space. If so, late checkpoints might become fundamentally unsuitable for weight-space merging.
The paper evaluates this possibility using linear mode connectivity. For two experts with parameters $\theta_A$ and $\theta_B$, it examines the straight interpolation path
The loss barrier measures how far the loss rises along that path relative to the endpoint losses. Barriers increase with training duration, indicating greater separation, but the average remains below 0.35 even at $5T^\ast$.
This result suggests that the QLoRA experts remain within a moderately connected region of the pretrained model’s parameter space. Late-training failure under Simple Averaging is therefore more consistent with poorly managed interference than with complete loss-basin separation.
That distinction affects system design. If the experts were no longer mergeable, retaining later checkpoints would provide little value. If they remain connected but increasingly disagree, then the merge operator—not training duration alone—determines whether the additional specialization is usable.
Safety requires its own checkpoint objective
The HarmBench results broadly follow the quality pattern, but they also expose a separate selection problem.
For the 4B model, TIES reaches a 68.5% refusal rate at $3T^\ast$. The best Simple Averaging result is 25.5%, reached at $T^\ast$, before falling to 6% at $5T^\ast$. DARE+TIES also strengthens with longer training at this scale, reaching 61.5% at $5T^\ast$.
These figures do not show that refusal rate alone defines a safe model. They show that the merge configuration materially affects whether the behavior learned by the safety expert survives consolidation.
Greedy Soup creates an additional risk. Its selection procedure is driven by the chosen quality objective and can omit an expert that does not improve that objective. When average accuracy guides selection, the safety adapter may be excluded even if its behavior is operationally required.
A production merge should therefore use separate acceptance thresholds for task quality and safety behavior. A single aggregate score cannot reliably protect a capability localized in one adapter.
Turn checkpoint storage into part of the merge pipeline
The paper directly supports one process change: do not discard all checkpoints except each expert’s standalone optimum.
A practical pipeline would retain a small normalized duration grid, run the intended merge operator across candidate combinations, and select against both task and safety criteria. The grid need not reproduce all eight durations used in the study, but it should cover both sides of $T^\ast$ when the merge method has no established duration profile.
Cognaptus inference: this changes the economics of checkpoint storage. Keeping several lightweight adapters is inexpensive relative to repeatedly retraining experts after discovering that the deployment merge needed earlier or later states. It also makes merge evaluation reproducible: duration becomes an explicit configuration parameter rather than an undocumented by-product of the training run.
The tested results suggest the following initial search policy:
- For Simple Averaging, search earlier checkpoints first.
- For TIES, extend the search beyond $T^\ast$, particularly as model size increases.
- For DARE+TIES, test a broad range because the profile may be flat and domain-dependent.
- For Task Arithmetic and Greedy Soup, avoid relying on a method-wide default.
- For every method, validate refusal or other protected behavior independently.
These are starting points for evaluation, not deployment constants.
Where the guidance stops
The evidence is reasonably broad inside the experiment: three model sizes, five domains, eight duration levels, five merge methods, and separate quality and safety evaluations. It is still concentrated in one technical setting.
All experts use Qwen 3.5 models, rank-16 QLoRA, 4-bit quantization, the same optimizer settings, and a constant learning rate of $2\times10^{-4}$ after warmup. A constant rate may make late-stage overtraining more pronounced than a decaying schedule would. The study does not test full fine-tuning, other parameter-efficient methods, substantially larger models, or other base-model families.
Greedy Soup and DARE’s stochastic pruning are also evaluated with one random seed. Their apparent duration profiles may therefore contain sampling variation that the design does not quantify. The paper reports raw benchmark differences rather than uncertainty intervals or standardized effect sizes.
The appropriate operational takeaway is a testing policy: select expert duration jointly with the merge operator. The evidence does not justify assuming that TIES will always reward training to $3T^\ast$, or that uniform averaging must always stop at a particular fraction of $T^\ast$.
Checkpoint selection belongs to the merged system
Early stopping remains useful for identifying when a standalone expert begins to overfit its own validation objective. It does not determine which version of that expert will cooperate best with four others.
The paper shows that aggregation changes the checkpoint optimum and that the direction of the change depends on how the merge rule handles disagreement. Simple Averaging favors restraint. Interference-resolving methods can extract value from further specialization. Safety behavior can move differently from average quality and must remain part of the selection objective.
For teams consolidating several adapters into one deployable model, the checkpoint is not merely a training artifact. It is an input to the merge architecture. Treating it that way requires more saved states and more structured evaluation, but it avoids optimizing five experts separately and discovering too late that they were the wrong models to combine.
Cognaptus: Automate the Present, Incubate the Future.