TL;DR for operators
Dividing a pretrained language model into more, smaller experts does not automatically produce a more capable model. In one Qwen3-1.7B comparison, fine-grained Drop-Upcycling averaged just 23.2% accuracy, versus 50.2% for its coarse-grained counterpart. Another fine-grained method reached 51.1%, showing that the failure is configuration-dependent.
DivMoE addresses this problem by initializing experts from separately trained domain specialists and restricting each token to at most one expert from each domain group. It achieves 55.6% average accuracy across 15 benchmarks, compared with 51.6% for Branch-Train-Mix under the same reported 540B-token training budget. Removing the routing constraint lowers DivMoE’s result to 52.5%.
For model engineering teams, this makes expert initialization and routing a joint design decision. The stronger benchmark results come with additional specialist-training costs, while inference latency, serving economics, and performance at substantially larger scales remain unverified.
Splitting a pretrained model can destroy capabilities it already has
A model engineering team has a capable pretrained language model and wants to extend it across programming, mathematics, scientific questions, and general knowledge. Reusing its existing weights is attractive: partition the network into smaller expert components, activate only a few for each token, and continue training.
The central risk is whether that conversion preserves the capabilities already embedded in the original model. More experts create additional routing choices, but those choices are valuable only if the resulting networks retain useful transformations and develop meaningful differences.
The experiments reported by Lou and colleagues in DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition1 expose how consequential the construction method can be.
On the Qwen3-1.7B evaluation, fine-grained Drop-Upcycling achieves 23.2% average accuracy across 15 benchmarks, compared with 50.2% for coarse-grained Drop-Upcycling. That is a deterioration of 27 percentage points. Yet NVIDIA Upcycling, another fine-grained method, reaches 51.1%. The results identify a severe failure in a particular initialization configuration, not a general impossibility of fine-grained upcycling (Section 4.2; Appendix C, Table 11).
The relevant question is therefore not how many experts a model contains. It is whether those experts begin with useful, differentiated knowledge and whether the routing mechanism selects complementary capabilities.
DivMoE controls expert composition before balancing their utilization
DivMoE combines two interventions.
First, it creates four specialist versions of a common dense checkpoint, trained on mathematics, code, academic/scientific material, and web/commonsense material. Each specialist receives 10B tokens of domain-adaptive continual pretraining. Its feed-forward network is then vertically partitioned into two smaller expert shards.
This produces eight fine-grained experts per mixture-of-experts layer, organized into four domain groups. Weight scaling and shared-component initialization are designed to preserve useful pretrained transformations rather than reconstruct expert functionality from random initialization.
Second, DivMoE changes how the router selects experts.
Instead of choosing the two highest-scoring experts globally, the router first identifies the highest-scoring expert within each domain group. It then selects the two groups with the highest representative scores. Each token consequently activates two experts drawn from different groups.
The structural rule is:
Here, $\mathcal{S}$ denotes the selected experts and $\mathcal{D}_i$ denotes a domain group. The constraint prevents two selected experts from belonging to the same group.
This differs from conventional load balancing. A load-balancing objective encourages broader expert utilization during training; it does not independently prohibit two experts from the same domain from handling one token. DivMoE retains an auxiliary load-balancing loss with coefficient 0.01, but uses the structural constraint to guarantee distinct group membership.
The guarantee has a narrow meaning. It does not require every domain to receive equal traffic, force equal contributions from selected experts, or establish that cross-domain composition improves every task. Learned routing scores still determine which groups participate and their relative weights (Section 3.2, Equation 7 and Algorithm 1).
The benchmark gains come from both initialization and routing
The main Qwen3-1.7B results show how the construction choices compare.
| Configuration | Average accuracy across 15 benchmarks |
|---|---|
| Fine-grained Drop-Upcycling | 23.2% |
| Coarse-grained Drop-Upcycling | 50.2% |
| NVIDIA Upcycling | 51.1% |
| Branch-Train-Mix | 51.6% |
| DivMoE without diversity constraint | 52.5% |
| DivMoE | 55.6% |
Source: Section 5.1, Table 5; Appendix C, Table 11. DivMoE without the diversity constraint is a component ablation. Branch-Train-Mix and DivMoE include an additional 40B specialist-training tokens; several other baselines do not.
The strongest full upcycling comparator, Branch-Train-Mix, also builds on domain-specialized models. Both methods use 40B tokens for specialist development and 500B for joint continual pretraining. DivMoE’s four-percentage-point advantage is therefore particularly informative: it is not explained simply by allocating more training tokens than Branch-Train-Mix.
The Llama3.2-1B experiment provides a second architectural setting. DivMoE averages 40.8%, compared with 39.1% for Branch-Train-Mix and 38.1% for the original dense base (Appendix C, Table 12).
A data-matched dense reference on Qwen3-1.7B achieves 53.4%, versus DivMoE’s 55.6%. This helps distinguish the benefits of MoE construction from additional training-data exposure, although the models do not have identical architectures or active parameter counts.
The component ablation supplies a more targeted test. Removing diversity-constrained routing while retaining domain-specialized fine-grained experts reduces the 15-benchmark mean from 55.6% to 52.5%. That 3.1-point difference supports a contribution from the routing rule beyond specialist initialization.
A separate domain-specific ablation reports improvements of 1.8 points on HumanEval, 2.3 on MBPP, 2.1 on GSM8K, and 1.4 on MATH when the constraint is enabled (Section 5.1, Table 6). Its absolute task scores differ from the full evaluation table, so these within-table differences should not be treated as interchangeable with the main benchmark scores.
Together, the comparisons support complementary benefits from the two architectural interventions. They do not constitute a complete factorial experiment establishing the independent effect of every design choice.
More training does not eliminate the measured advantage
One alternative explanation is that DivMoE merely begins joint training from a better checkpoint and that other methods eventually catch up.
The extended training comparison tests this possibility over 150B, 300B, and 500B Stage-2 tokens. On Qwen3-1.7B, DivMoE moves from 52.0% to 54.0% to 55.6%. NVIDIA Upcycling moves from 47.5% to 49.5% to 51.1%.
The 4.5-point advantage persists at both the first and last reported checkpoints (Appendix F, Table 15). This is training-dynamics evidence that the difference survives continued optimization, rather than merely reflecting an early checkpoint comparison. It does not establish whether the gap would persist under substantially different datasets or longer training.
Granularity itself also remains a tuning decision. The reported ablation favors two expert shards per domain and two active experts over the tested configurations with one or four. Increasing granularity is not monotonically beneficial under these conditions.
The additional accuracy has a measurable training cost
For teams choosing a dense-to-MoE conversion strategy, the appropriate comparison includes all stages of training.
| Training requirement | DivMoE / Branch-Train-Mix | Methods without specialist pretraining |
|---|---|---|
| Specialist-training tokens | 40B | 0 |
| Joint-training tokens | 500B | 500B |
| Total training tokens | 540B | 500B |
| Approximate H20 GPU-hours | 28,800 | 26,500 |
Source: Appendix B.4, Table 9; Qwen3-1.7B configurations using NVIDIA H20 80GB GPUs.
DivMoE requires approximately 2,300 more H20 GPU-hours than methods that omit the specialization stage. Its advantage over Branch-Train-Mix, however, is observed under equal reported token budgets and GPU-hour estimates.
A larger experiment adds another perspective. A Qwen3-4B-derived DivMoE has approximately 12B total parameters and 3.6B active parameters. After supervised fine-tuning, it achieves 64.5% across eight benchmarks, matching the reported average for the 16B-parameter Moonlight-MoE (Section 4.3; Appendix D, Table 13).
That comparison indicates competitive benchmark accuracy at fewer total parameters. It is not a controlled efficiency comparison: Moonlight-MoE has approximately 2.4B active parameters, and the checkpoints have different training histories.
Cognaptus inference: A team with a reusable dense checkpoint and sufficient domain-specialization data should evaluate initial expert diversity and routing constraints together. The relevant acceptance criteria are retained baseline capabilities, task-level accuracy under matched training budgets, active and total parameters, and measured deployment costs. A higher benchmark average alone cannot establish a lower cost per served request.
Where the evidence stops
The reported evidence covers dense bases of approximately 1B–4B parameters, four manually defined domain groups, and public academic benchmarks. The study does not establish whether a learned domain partition would perform better, whether every token benefits from cross-domain composition, or whether the same improvements persist at substantially larger scales.
It also provides no repeated-seed uncertainty estimates or statistical significance tests for the principal benchmark differences. Most importantly for deployment, production latency, throughput, memory behavior, and serving cost are not systematically measured.
These boundaries distinguish the established architectural finding from a production investment decision. The experiments show that specialist initialization and constrained routing can improve benchmark performance under the evaluated conditions. Whether that improvement pays for additional training and serving complexity remains an engineering and economic question.
The expert count is not the design objective
DivMoE’s contribution is to make expert diversity a property of both model construction and token-level selection. Its experiments show that simply creating smaller experts can fail badly, while combining domain-adapted initialization with an explicit cross-domain routing rule delivers stronger results across two tested model families.
For teams extending pretrained models, the evidence favors measuring preservation and useful specialization before pursuing additional granularity. More experts are valuable when the construction and routing design make their differences consequential.
Cognaptus: Automate the Present, Incubate the Future.
-
Yuxuan Lou and Kai Yang and Geng Zhang and Yong Liu and Yang You (2026). DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition. arXiv:2610.11317. https://arxiv.org/abs/2610.11317 ↩︎