More FLOPs, Worse Choice: Sparse MoE Scaling Has to Price the Cluster

TL;DR for operators

Suppose a team has already bought the cluster time: 32 B200 nodes for 20 days. It might seem sensible to prefer the sparse-MoE design that manages to execute the most model FLOPs during that window. In the paper’s reported search, that rule would pick the wrong configuration. The loss-optimal candidate uses about $1.23\times10^{23}$ model FLOPs—roughly 36% fewer than the candidate that realizes about $1.94\times10^{23}$—yet reaches the lower predicted loss.

Sarkar, Tang, and Zha’s MOSAIC framework¹ explains why. It does not treat every model FLOP as costing the same amount of cluster time. Instead, it jointly chooses sparse-MoE architecture, training-token budget, and distributed execution layout while accounting for the throughput and memory behavior of each candidate.

For training planners, the shift is consequential: convert a GPU-hour allocation into architecture-specific useful compute before fixing expert count, routing, model scale, token budget, or parallelism. The evidence is credible on the tested B200/Megatron-Core stack, but it does not establish a universal sparsity target or a portable optimum across hardware.

The cluster budget is not a model-FLOPs budget

Sparse MoEs make the distinction between theoretical computation and delivered computation difficult to ignore. Two architectures can require similar model FLOPs per token while imposing very different costs on the hardware through expert dispatch, grouped GEMMs, memory pressure, pipeline bubbles, and collective communication.

The paper formalizes that gap with Model FLOPs Utilization, or MFU: the fraction of nominal cluster peak throughput that becomes the model computation required by a particular architecture under a particular execution plan. It then defines deliverable model FLOPs as

The raw hardware ceiling is fixed by the devices and training window. The useful budget is not. An architecture with poor MFU receives less usable model computation from the same purchased cluster.

That changes architecture search from a two-stage process—choose the model, then optimize its execution—into a coupled problem. MOSAIC searches over the model geometry and training-token count while also enumerating feasible combinations of tensor, expert, context, pipeline, and data parallelism, micro-batch size, and checkpointing. Memory feasibility is part of the constraint, not a cleanup step after the architecture has been selected.

Sparsity looks different once the hardware has a price

The paper’s scaling-law result makes the need for that coupling especially clear. Under a conventional fixed model-FLOPs budget, predicted loss decreases as sparsity rises across the calibrated range. There is no interior optimum: the best point is simply the highest supported sparsity.

That result is not evidence that arbitrarily sparse models are universally better. It says something narrower. Within the observed regime, model-FLOPs accounting does not impose enough cost on increasing sparsity to create an interior trade-off.

Hardware accounting does.

More sparsity can improve statistical efficiency by increasing stored capacity relative to active computation, but the execution penalties can grow through communication, smaller expert kernels, pipeline effects, memory constraints, and the parallel layout required to make the model fit. Once those penalties reduce MFU, the useful compute available during the fixed training window shrinks.

In the reported systems-aware budget sweeps, the selected sparsity becomes interior, occupying a broad range of roughly 0.915 to 0.963 rather than moving monotonically with budget. The expert split factor $G$—dense-FFN width divided by individual-expert width—lands at $G=4$ across most reported budgets, although neighboring values are often close.

The relevant result is therefore not “0.956 is the optimal sparsity.” It is that the optimum becomes hardware-dependent once systems cost enters the objective.

The performance model is accurate enough to change rankings on B200

A co-design framework is only decision-relevant if its systems model can distinguish good layouts from bad ones. The paper tests that directly rather than relying only on analytical timing equations.

The pipeline result is particularly concrete. In the reported sweeps, pipeline fill ratio explains much of the throughput spread: performance collapses near a fill ratio of one and largely saturates once the ratio reaches about four. This is implementation-sensitive evidence, but it shows why “architecture first, systems later” can fail even before more complex communication effects are considered.

What changes for pretraining decisions

The paper directly shows that architecture-dependent throughput can change which sparse-MoE configuration is preferred under a fixed hardware envelope. Cognaptus extends that finding into a planning rule: a team should estimate the useful compute each candidate architecture can actually receive before locking the model specification.

For a training organization, that affects several decisions at once. Expert count and top-$K$ routing alter both capacity and execution cost. Expert granularity changes grouped-GEMM efficiency. Model scale determines memory and parallelism requirements. The token budget depends on what the cluster can finish within the window. Added nodes can even change the best geometry rather than merely accelerating an already chosen model.

This also changes capacity planning. If an infrastructure team evaluates a larger cluster only by multiplying nominal peak FLOPs, it may miss that additional devices alter expert-parallel locality, pipeline structure, collective exposure, and memory feasibility. The marginal node can move the architecture frontier rather than simply rescale the same optimum.

MOSAIC itself is not trying to maximize MFU. The 32-node, 20-day choice has predicted MFU of only 12.38%. Its objective is lower predicted loss under the useful-compute and feasibility constraints. A higher-MFU architecture can still lose if its statistical efficiency is worse.

The ranking result is stronger than the exact optimum

The most defensible conclusion is about ranking, not about one architecture prescription.

The scaling-law dataset is substantial—150 stated from-scratch runs—but some MOSAIC search points extend far beyond it. The 32-node, 20-day optimum realizes model FLOPs about 150 times larger than the largest fitting run. Several scaling-law coefficients also lie on an identifiability ridge, and that uncertainty is not propagated through the discrete architecture search.

The systems model has its own scope. It reflects B200 hardware and a relatively basic Megatron-Core stack. Different kernels, communication-computation overlap, interconnects, accelerators, batch choices, or strongly overtrained regimes can move the optimum. The paper also limits expert split factor to $G\leq8$.

Those constraints do not erase the measured ranking evidence. They narrow what should be transported from it. Organizations using hardware and software similar to the tested environment have evidence that systems-aware architecture search can improve pretraining decisions. Organizations on different stacks have a stronger reason to reproduce the measurement process than to copy the reported geometry.

Price useful compute before committing the model

Sparse-MoE scaling exposes a weakness in treating model FLOPs as a universal training currency. The same nominal hardware allocation can deliver different amounts of useful model computation depending on architecture and execution plan, and those differences can be large enough to reverse which configuration is preferred.

For pretraining teams, the decision sequence should therefore change. Do not select a theoretically compute-optimal sparse architecture and assume systems engineering will recover the promised economics afterward. Estimate the architecture-specific compute the target cluster can actually deliver, and let that constraint participate in the model decision itself.

That is the durable contribution of MOSAIC. The exact optimum is conditional. The accounting correction is much harder to dismiss.

Cognaptus: Automate the Present, Incubate the Future.