When More Experts Preserve Less Knowledge
TL;DR for operators Dividing a pretrained language model into more, smaller experts does not automatically produce a more capable model. In one Qwen3-1.7B comparison, fine-grained Drop-Upcycling averaged just 23.2% accuracy, versus 50.2% for its coarse-grained counterpart. Another fine-grained method reached 51.1%, showing that the failure is configuration-dependent. DivMoE addresses this problem by initializing experts from separately trained domain specialists and restricting each token to at most one expert from each domain group. It achieves 55.6% average accuracy across 15 benchmarks, compared with 51.6% for Branch-Train-Mix under the same reported 540B-token training budget. Removing the routing constraint lowers DivMoE’s result to 52.5%. ...