Sparse Is Not Cheap by Default: What MoE Efficiency Actually Depends On

TL;DR for operators

Sparse mixture-of-experts models can own hundreds of billions—or even a trillion—parameters while activating only a much smaller subset for each token. That makes activated parameters a more relevant starting point than total parameters when comparing computational burden.

It does not settle the infrastructure question. The surveyed literature shows that tokens still have to be assigned to experts, overloaded experts need capacity controls, unused capacity can create padding, overflow may result in dropped tokens, and experts distributed across devices require substantial network traffic. Dong Pan and colleagues’ survey brings these model-level and systems-level constraints into one view.¹

For teams evaluating an MoE model, four questions deserve attention before treating sparsity as a cost advantage:

1. How many parameters are activated per token, not merely how many exist?
2. How concentrated is the routing, and what happens when experts become overloaded?
3. Where do the experts reside, and how much token traffic must move between devices?
4. If deployment is decentralized, who absorbs heterogeneity, failures, privacy, security, and coordination costs?

The paper supports these as architecture and infrastructure questions. It does not independently measure serving cost, latency, ROI, or production readiness.

The parameter count is only the first number

Suppose a procurement team is comparing two models whose headline parameter counts differ by several hundred billion parameters. The immediate temptation is to translate the larger number into proportionally larger runtime cost.

Sparse MoE complicates that comparison because most parameters are inactive for any given token. The survey’s model table makes the distinction concrete. DeepSeek-v3 is listed at 671B total parameters with 37B activated; KimiK2 at 1T total with 32B activated; Qwen3-Next at 80B total with 3B activated. These figures do not establish that one model is cheaper or faster than another—the models differ in many other respects—but they show why total parameter count alone is a poor proxy for per-token computation.

The mechanism is straightforward. The model decides which small subset of expert subnetworks should process each token. In the survey’s formulation,

where the router weights selected expert outputs and sparse gating leaves most experts inactive for a particular input.

For a model buyer, this changes the comparison unit. Total parameters describe available capacity. Activated parameters describe only part of the work performed per token. Neither number captures the entire deployed system.

Routing determines whether sparse capacity is actually usable

Once tokens are assigned selectively, another problem appears: they do not necessarily distribute themselves evenly.

In token-choice routing, each token selects its preferred experts. The survey synthesizes prior work showing that popular experts can become overloaded while others remain underused. Sparse activation can therefore coexist with inefficient utilization.

Training systems commonly introduce a load-balancing objective:

The coefficient $\alpha$ controls how strongly the model is encouraged to distribute work across experts. The survey emphasizes that this is a trade-off rather than a universally beneficial constraint. If balancing pressure is too weak, routing can collapse toward a subset of experts, creating instability and wasted resources. If it is too strong, the balancing objective can interfere with model quality or specialization.

That matters outside the training team. An infrastructure planner estimating throughput from the fraction of active experts is implicitly assuming that those experts receive work in a manageable distribution. Routing concentration can invalidate that assumption even when theoretical sparsity has not changed.

Expert capacity trades overflow against wasted space

Each expert can also be assigned a maximum number of tokens. The survey describes expert capacity as

where $T$ is the batch token count, $E$ the number of experts, and $f$ a capacity factor.

The capacity factor creates a systems trade-off. More spare capacity reduces the risk that a heavily selected expert overflows, but unused slots create padding and wasted computation or memory. Less spare capacity improves nominal utilization but increases the possibility that excess tokens must be dropped or otherwise handled.

This is why “only a few experts activate” is incomplete as an efficiency statement. Two implementations with similar activated parameter counts can behave differently because their routing distributions, capacity factors, overflow policies, and balancing strategies differ.

For serving-cost estimation, these are configuration variables worth asking a vendor or platform team about. The survey does not provide a standardized cost model that converts them into dollars or milliseconds.

Distributed experts can move the bottleneck into the network

MoE also changes where computation happens. Under expert parallelism, different experts are placed on different devices. Tokens must travel to whichever devices host their selected experts and then return to their original positions.

The survey identifies this all-to-all dispatch as a major communication bottleneck. Its comparison of centralized frameworks shows that expert parallelism is commonly combined with data, tensor, or pipeline parallelism rather than operated as an isolated scaling technique.

This shifts the model-selection question from “How many FLOPs are avoided?” toward “What must cross the network to realize those avoided FLOPs?”

For a platform team operating a high-bandwidth cluster, the answer depends on topology, interconnect performance, routing patterns, batch structure, and implementation. Sparse computation may reduce arithmetic while introducing enough data movement to constrain realized throughput.

The survey reviews this systems problem; it does not benchmark those trade-offs under a common hardware configuration. Cross-framework cost comparisons would therefore go beyond its evidence.

Decentralized MoE replaces one infrastructure constraint with several others

The survey’s most distinctive synthesis concerns decentralized MoE: spreading expert computation across geographically distributed or consumer-grade resources rather than keeping the system inside a tightly controlled data-center cluster.

The potential attraction is clear. Organizations could access dispersed compute instead of concentrating every expert on expensive, homogeneous infrastructure.

The engineering burden also changes sharply. The survey groups decentralized challenges into four categories:

For a team deciding between centralized and decentralized deployment, this is better understood as a change in constraint set than as a straightforward infrastructure reduction.

Centralization concentrates hardware and networking requirements. Decentralization introduces scheduling across heterogeneous devices, variable communication, failure recovery, privacy controls, security mechanisms, and potentially incentives for participants supplying resources.

The paper identifies these requirements but provides no evidence that decentralized MoE is cheaper overall for a given production workload.

What to inspect before selecting an MoE model

The survey suggests a more disciplined procurement and architecture checklist.

For model portfolio owners, compare total and activated parameters alongside expert count and routing policy. A trillion-parameter headline says little about runtime burden without the activation pattern.

For infrastructure teams, inspect expert placement and estimate the resulting token traffic. If experts are distributed across devices, interconnect requirements belong in the capacity model from the beginning.

For ML engineering teams, examine load concentration, expert-capacity settings, overflow behavior, token dropping, and whether balancing mechanisms are likely to interfere with specialization.

For organizations considering decentralized deployments, add hardware variability, churn, failure recovery, malicious nodes, privacy protection, and incentive design to the architecture review.

The survey also gives a useful caution about the word expert. Reviewed studies indicate that specialization can occur at the level of token classes, grammar, modality, or local contextual patterns. An expert should not be assumed to correspond to a stable, human-readable business domain merely because the architecture uses that label.

The survey maps the design space; it does not price it

The paper is a narrative synthesis of 129 references spanning foundational MoE research, recent systems work, model reports, and vertical applications. It connects routing, expert design, parallelism, decentralized infrastructure, and applications more broadly than many earlier surveys.

That breadth should not be mistaken for standardized empirical comparison. The authors report no reproducible search strategy, screening flow, formal study-quality assessment, benchmark replication, or common cost experiment. The evidence base also mixes peer-reviewed work, preprints, technical reports, model reports, and industry material.

Accordingly, the strongest use of the survey is architectural: it identifies the variables that determine whether sparse MoE is operationally attractive.

It cannot tell a procurement team that an MoE deployment will reduce its inference bill by a particular percentage, or tell a platform team that decentralized execution will outperform a conventional cluster. Those questions require workload-specific measurement.

Sparse activation changes the economics of model capacity by separating what a model owns from what it activates. Real efficiency emerges only after routing, capacity management, memory movement, communication, hardware topology, and failure conditions are included in the calculation.

For operators, that is the more useful number to pursue: not parameter sparsity in isolation, but the system cost required to make that sparsity work.

Cognaptus: Automate the Present, Incubate the Future.