MoE

TL;DR for operators When a sparse Mixture-of-Experts model receives a larger training budget, the additional computation has at least two architectural destinations: attention capacity and expert feed-forward capacity. Treating their existing balance as fixed can leave the larger model internally misallocated. Li et al. show this experimentally in GPT-style sparse MoE models.1 At fixed compute and sparsity, varying the expert-versus-attention allocation produces a clear loss minimum. As total compute increases, the loss-minimizing allocation shifts toward more expert computation, but the rate of that shift depends on sparsity. ...

TL;DR for operators Suppose a team has already bought the cluster time: 32 B200 nodes for 20 days. It might seem sensible to prefer the sparse-MoE design that manages to execute the most model FLOPs during that window. In the paper’s reported search, that rule would pick the wrong configuration. The loss-optimal candidate uses about $1.23\times10^{23}$ model FLOPs—roughly 36% fewer than the candidate that realizes about $1.94\times10^{23}$—yet reaches the lower predicted loss. ...

TL;DR for operators Sparse mixture-of-experts models can own hundreds of billions—or even a trillion—parameters while activating only a much smaller subset for each token. That makes activated parameters a more relevant starting point than total parameters when comparing computational burden. It does not settle the infrastructure question. The surveyed literature shows that tokens still have to be assigned to experts, overloaded experts need capacity controls, unused capacity can create padding, overflow may result in dropped tokens, and experts distributed across devices require substantial network traffic. Dong Pan and colleagues’ survey brings these model-level and systems-level constraints into one view.1 ...

Model architecture has a recurring habit: when something works, we freeze it into a default and move the argument elsewhere. Attention gets the drama. Routing gets the diagrams. Context windows get the product demos. Meanwhile, the feedforward network sits there, quietly holding a large share of the parameters and applying the same nonlinearity to every token, every time, as if “one curve fits all” were a law of nature rather than a convenient engineering choice. ...

Latency has a way of turning elegant model architecture into an invoice. Mixture-of-Experts models were supposed to soften that invoice. Instead of sending every token through the same dense feed-forward machinery, an MoE layer sends each token to only a few experts. In theory, this gives us scale without paying for all parameters on every token. In practice, many deployed MoE models still behave like a restaurant that insists every guest order the same number of dishes. The experts differ, but the billable count is fixed. ...

The Experts Are Sparse Inside: Why MoE Cost Cuts Stop at 1.2x Cost has a way of making architecture fashionable. Mixture-of-Experts models became attractive because they promise a pleasant bargain: keep a large total parameter count, but activate only a small part of the model for each token. In business language, that sounds like capacity without the full compute bill. In engineering language, it means routing each token to a few expert feed-forward networks instead of running every expert all the time. ...

Rollouts are expensive little creatures. They consume GPU time, produce long reasoning traces, wait for reward computation, and then—if the reward signal is flat—contribute exactly nothing to learning. The GPU was busy. The training dashboard looked serious. The model learned no usable distinction. Very productive, in the same way a meeting with twelve people and no decision is productive. ...

Meta’s experimental ultra-sparse MoE model with 128 experts, designed to explore efficient large-scale scaling and routing strategies for future LLaMA architectures.

Meta’s experimental LLaMA 4-series MoE model with 17 billion parameters and 16 experts, designed to explore sparse routing and scaling strategies.

A 4-bit quantized, instruction-tuned variant of Meta’s LLaMA 4 Scout MoE model, optimized by Unsloth for efficient fine-tuning and deployment.