Give the Quiet Experts More Bits

TL;DR for operators

A fixed MoE memory budget does not tell you which experts can safely absorb the lowest precision. Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees¹ argues that the experts most frequently or strongly used are not necessarily the ones that need the most bits. In its theory, experts associated with less-prevalent but still task-relevant features develop weaker activations and smaller router-norm changes, leaving less margin for quantization error.

The proposed rule gives higher precision to those smaller-change experts, then uses maximum intra-neuron variance, or MaxVar, to rescue experts whose weight distributions make them unusually noisy under very low-bit quantization. That second step matters: on Mixtral 8x7B, router ranking alone falls to 49.78 average accuracy under a three-level 2.5-bit allocation, while router norm plus MaxVar reaches 68.38 at the same average precision.

For serving teams, the attraction is not only accuracy. The assignment can be computed from router norms and weight statistics without PMQ-style GPU calibration over candidate bit maps. The paper reports PMQ allocation costs of 110 GB and 2,227 seconds for Mixtral 8x7B, and 350 GB and 6,000 seconds for Mixtral 8x22B.

The expensive mistake is assigning bits to the wrong experts

Lower-bit weights reduce MoE memory, but a deployment still has to decide which experts receive scarce higher-precision slots. A usage-based intuition would favor experts that activate often or carry large routing weights. The paper instead asks which experts have the least tolerance for quantization error.

That distinction is operationally consequential. If the wrong experts are pushed to 1 or 2 bits and quality collapses, a team may have to retreat to a higher average precision even though another assignment under the same memory budget would have worked.

Smaller router changes can signal less quantization margin

In the paper’s simplified two-layer MoE, task-relevant features occur at different prevalence levels. Experts learning more-prevalent features receive larger cumulative updates, produce stronger activations, and show larger changes in router $\ell_2$ norm. Experts learning less-prevalent but still necessary features change less and activate more weakly.

The ranking signal is

Smaller router-norm growth moves an expert toward higher precision. The claim is not that small-change experts are universally “more important.” Under the paper’s model, their weaker activations leave less margin for quantization-induced perturbation. Theorem 4.4 then gives sufficient high- and low-bit conditions under which a two-level assignment preserves the full-precision model’s generalization in that stylized setting.

The useful shift is from usage allocation to sensitivity allocation.

Router ranking still misses a separate low-bit failure mode

Router behavior does not capture every source of quantization error. An expert with unusually high within-neuron weight variance can be disproportionately damaged if it falls into the lowest precision group. MaxVar is therefore a secondary safeguard, not a competing importance metric.

The Mixtral 8x7B ablation isolates this failure:

The three-level test is an ablation, not a second headline benchmark. Its purpose is to show why router behavior alone is insufficient when assignments include 1-bit experts. At $\zeta=3$, the paper reports promoting only 11 of 256 Mixtral 8x7B experts.

The benchmark case weakens at the most aggressive precision

On Mixtral 8x7B, the method outperforms PMQ at matched average precisions above 2.0 bits. At 2.5 bits per expert, average accuracy is 68.38 versus 67.53 for PMQ, with memory at 16.1 GB versus 96.8 GB for the table’s full-precision model. At 2.0 bits, PMQ is slightly higher, 62.83 versus 62.56; at 1.75 bits, PMQ leads 60.53 to 58.95.

Mixtral 8x22B is similarly mixed. The proposed method trails PMQ at 2.5 bits, 60.10 versus 60.69, then leads at 2.25 bits, 59.17 versus 55.35; at 2.0 bits, 55.34 versus 54.80; and at 1.75 bits, 51.44 versus 47.87.

The result is therefore not “better at any low precision.” It is stronger evidence that targeted allocation can improve the accuracy-memory trade-off over a useful low-bit range, while very aggressive compression remains risky.

Better allocation does not require a more expensive search

The method’s operational appeal is that finer targeting does not require a large calibration search. Bit assignment uses router norms plus limited MaxVar reordering and is reported to require no GPU. PMQ instead evaluates expert behavior across candidate precisions using calibration data, producing the much larger allocation costs reported above.

The paper also reports lower inference time than PMQ at matched average precision because its higher-bit experts tend to be less frequently activated.

Cognaptus inference: for teams that repeatedly compress checkpoints or variants, allocation overhead belongs in compression ROI. Avoiding a GPU-heavy search can reduce preparation cost, while keeping frequently activated experts at lower precision may improve runtime economics. What remains uncertain is how much of that advantage survives different kernels, hardware, routing implementations, and latency targets.

Final router norm is a practical surrogate, not a universal equivalence

Router-norm change requires both initial and trained router weights. For pretrained models where initialization is unavailable, the paper tests final router norm as a surrogate after reinitializing and fine-tuning Switch Transformer routers with small-variance Gaussian noise.

Across the reported layers, Spearman correlations between final norm and norm change range from 0.9989 to 0.9999; Kendall correlations range from 0.9851 to 0.9980. Quantized ROUGE-2 is also nearly identical: at 2.5 bits per expert, 18.60 for norm change versus 18.59 for final norm.

This is a robustness and practicality test. It supports the surrogate under the tested initialization condition; it does not establish universal equivalence for arbitrary pretrained MoEs.

Where the evidence stops

The formal guarantee is narrower than the production systems the method targets. It assumes a shallow two-layer MoE with one MoE block, binary classification, expert-choice routing, ReLU experts, fixed second-layer weights, and quantization of only first-layer expert weights. The practical MaxVar correction also extends beyond what the simplified theorem fully covers.

Empirically, the method is tested on one fine-tuned Switch Transformer and two Mixtral scales across CNN/Daily Mail and eight zero-shot language benchmarks. That shows the signal survives beyond the stylized theory, but not that it is architecture-independent.

The source record also preserves two reporting inconsistencies: the main text and appendix disagree on whether Mixtral non-MoE parameters use 3 or 4 bits, and the published full-precision Mixtral 8x22B average appears inconsistent with its listed task scores. Neither drives the matched-budget comparisons emphasized here, but both limit confidence in absolute table aggregates.

The decision this paper changes is specific: under a fixed MoE precision budget, do not assume the most active experts deserve the most bits. Estimate which experts have the least error margin, then separately protect those whose weight distributions make extreme quantization unusually destructive. If that principle generalizes, the payoff is not merely a smaller model file; it is cheaper compression preparation and more disciplined use of scarce inference memory.

Cognaptus: Automate the Present, Incubate the Future.