Scale the Split: MoE Compute Allocation Should Move With the Budget

TL;DR for operators

When a sparse Mixture-of-Experts model receives a larger training budget, the additional computation has at least two architectural destinations: attention capacity and expert feed-forward capacity. Treating their existing balance as fixed can leave the larger model internally misallocated.

Li et al. show this experimentally in GPT-style sparse MoE models.¹ At fixed compute and sparsity, varying the expert-versus-attention allocation produces a clear loss minimum. As total compute increases, the loss-minimizing allocation shifts toward more expert computation, but the rate of that shift depends on sparsity.

The paper turns this pattern into a fitted power law. For a team planning a larger training run, that rule offers a starting point for reconsidering architecture dimensions rather than copying the split used at a smaller scale.

The boundary is substantial: this is a FLOPs optimum, not a demonstrated wall-clock, memory, infrastructure-cost, or inference optimum. Communication and hardware topology are absent from the model.

More training compute does not preserve the best architecture

Suppose a team has already trained a sparse MoE and now approves a substantially larger compute budget. Keeping the same architecture proportions may seem reasonable: increase dimensions, retain the existing balance, and scale the run.

The paper tests the assumption hidden in that choice.

A fixed training-compute budget must be divided between attention computation and expert feed-forward computation. The authors vary that split while holding per-token compute fixed. Across their tested configurations, different allocations produce measurably different losses, with a clear minimum rather than a flat region where the split barely matters.

Only after establishing that result does the paper define its central quantity:

where $C_E$ is expert computation and $C_A$ is attention computation. Importantly, $r$ is a ratio, not the fraction of total compute assigned to experts.

The loss-minimizing value is called $r^{\ast}$. The experiments indicate that $r^{\ast}$ increases as total compute grows across the tested sparsity regimes.

That changes the scale-up decision. An architecture that allocates its FLOPs well at one training scale may allocate them poorly after the budget increases.

Sparsity changes where the optimum moves

The experiments use sparse MoE decoder-only Transformers with three experts active per token: one shared expert and two routed experts. The total expert count varies across 17, 33, 65, and 129.

Because only three are active, increasing the available expert pool raises activation sparsity—the fraction of experts inactive for a token. The four reported sparsity levels are 82.35%, 90.91%, 95.38%, and 97.67%.

Across these regimes, the authors sweep $r$ from roughly 0.2 to 1.5 at multiple compute scales while independently changing attention and expert dimensions.

The main result is not merely that larger models favor more expert computation. Sparsity alters the scaling relationship itself. Lower-sparsity configurations show a faster increase in the preferred expert allocation as compute grows, while highly sparse configurations shift more slowly.

The fitted allocation law is:

with

and

Here $S$ is activation sparsity. Both the starting level of the relationship and its growth rate therefore change with the active-expert fraction.

The mechanism proposed by the paper is diminishing marginal returns. Attention computation supports token interaction, while expert computation supplies specialized feature-processing capacity. If their marginal value changes differently as compute grows—and if expert returns also depend on sparsity—the loss-minimizing balance should move rather than remain fixed.

This is a theory-guided interpretation of the observed scaling pattern, not a separately identified causal mechanism.

The held-out sparsity test makes this more than a fitted sweep

A collection of local allocation sweeps could still amount to architecture-specific tuning. The paper takes one additional step: it incorporates allocation into an extended loss scaling law and withholds the entire 97.67% sparsity regime when fitting its coefficients.

Predictions from the fitted law then align with observations in that withheld regime.

That test is the strongest evidence in the package that the relationship captures some structure beyond the individual configurations used for coefficient estimation. It is more informative than showing fit on the training observations themselves.

Its scope should remain precise. The holdout changes sparsity within the same general GPT-style autoregressive MoE experimental family. It is not a test on a new task, routing mechanism, architecture family, data regime, or hardware platform.

The paper also compares its formulation with prior MoE scaling formulas based on expert count or sparsity and reports poorer fits for those alternatives in portions of the tested regime. This comparison supports the narrower claim that explicitly modeling internal allocation adds information in these experiments; it does not establish one universal MoE scaling law.

What a model-training team can change

For architecture planners operating under a fixed training budget, Cognaptus draws a straightforward decision rule from the evidence: reopen the expert-versus-attention allocation decision whenever the target compute scale or sparsity changes.

That does not require accepting the fitted formula as universal. The paper establishes the more fundamental point that the variable is consequential and scale-dependent.

The relevant business user here is the team choosing model geometry before an expensive pretraining run. The decision is not whether experts are generally valuable; it is whether the next scale target preserves the same internal compute balance.

The optimum is not “more experts at any cost”

The paper does not support a rule that expert compute should simply keep increasing.

Its evidence shows an interior optimum: for a fixed compute budget and sparsity, both allocating too little and allocating too much compute to experts can produce higher loss. The preferred point then changes with scale.

There is also a methodological wrinkle in how some optima are recorded. The authors normally use the observed minimum-loss allocation. If an empirical optimum would make $r^{\ast}$ decrease as compute rises, however, they may choose a nearby point that preserves the expected monotonic trend when its loss is within 0.001 of the minimum.

That procedure injects a modeling prior into part of the series subsequently used to estimate the scaling relationship. Combined with the absence of confidence intervals, replicate counts, or reported uncertainty around the fitted coefficients, it argues for treating the numerical law as a provisional design prescription rather than a precision calculator.

The source record also preserves an experimental-reporting ambiguity: the text states that six benchmark model scales were used, while the reported hyperparameter table visibly contains five labels. The total number of allocation configurations is not enumerated.

FLOPs are only the first cost layer

The largest constraint on practical translation is outside the allocation equation.

Sparse expert systems incur communication, memory movement, routing, and distributed execution costs. None of those are incorporated into the paper’s optimization law. An allocation that minimizes loss for a given FLOPs budget may therefore fail to minimize elapsed training time or infrastructure spend on a particular cluster.

The empirical scope is also autoregressive language modeling with fixed sparsity and three active experts per token. Adaptive sparsity, dynamic routing regimes, multimodal tasks, and deployment optimization remain untested.

For model-training governance, the paper is still useful because it identifies a design variable that deserves explicit review. Compute budget, sparsity, and expert-attention allocation should be evaluated together before training. Systems constraints can then be layered onto that candidate architecture rather than assumed to disappear inside the FLOPs count.

The broader scaling point is modest but consequential: increasing the budget changes not only the size of the model worth training, but potentially the internal structure worth spending that budget on.

Cognaptus: Automate the Present, Incubate the Future.