Sparse Per Token, Dense Per Batch: XShare Reprices MoE Routing at Inference Time

TL;DR for operators

Sparse MoE routing does not guarantee a sparse serving batch. As more tokens are decoded together, especially under speculative decoding, their individual top-k choices can collectively touch a large fraction of the model’s experts. That increases expert-weight movement during a memory-bound stage of inference.

XShare treats this as a serving-policy problem rather than a model-retraining problem. It ranks experts using router scores from the current batch, limits the shared expert pool, and then lets each token perform its normal top-k refinement within that pool. In the reported GPT-OSS 120B experiments, moderate restrictions improve output-token throughput by roughly 6-13% in standard decoding and around 13-14% in several speculative settings while keeping accuracy comparatively close to baseline. More aggressive restriction can go much faster, but the quality loss can be large.

For inference teams, the relevant control is therefore not “make the MoE as sparse as possible.” It is the expert budget that minimizes weight movement or peak GPU load while remaining inside the product’s acceptable quality envelope.

Sparse routing can become a large batch footprint

A sparse MoE model may activate only four or eight routing experts for one token. Serving systems, however, rarely process one token in isolation. They batch requests to increase accelerator utilization, and speculative decoding can enlarge the effective token batch again.

The paper formalizes the resulting problem with the expected number of distinct experts activated by a batch:

Here, $N$ is the total number of routing experts, $k$ is the number selected per token, and $B$ is the effective batch size. Under the paper’s approximation, the union of experts rises quickly as $B$ grows. A model can remain sparse at the token level while forcing the serving system to touch many different expert weights across the batch.

That changes the unit of optimization. The relevant footprint is no longer only the number of experts one token activates. It is the number of distinct expert weights the whole batch requires.

XShare makes the expert pool a batch-level serving decision

XShare¹ uses the router itself to decide which experts the current batch should share. It sums each expert’s gating scores across tokens, selects a limited subset with the largest aggregate scores, and then performs per-token top-k routing inside that subset.

This is not static pruning. The model weights are unchanged, and the selected pool can change from batch to batch.

The paper’s proxy objective is:

The contribution of each expert is additive, so the objective is modular. For a fixed layer budget $m_l$, sorting experts by their aggregate gating score gives the optimal solution to this proxy problem.

The distinction matters. The proof establishes the best selection under the router-score objective. It does not establish that the resulting subset maximizes benchmark accuracy. XShare relies on the assumption that higher router scores are a useful signal of which experts matter for model quality.

The implementation also protects a small number of each token’s highest-ranked choices before imposing broader sharing. That warm-up step is consequential: it reduces the risk that a batch-wide decision removes an expert that is disproportionately important for one token.

Moderate sharing sits on the useful part of the throughput-quality frontier

The standard-decoding experiments on GPT-OSS 120B at batch size 16 are the paper’s main evidence that reduced expert footprint can translate into end-to-end serving gains rather than merely a lower sparsity count.

A configuration with layer budget $m_l=24$ and warm-up $k_0=1$ improves OTPS by 7.1% on AIME2025, 12.7% on GPQA, and 8.5% on MMLU-Pro. Reported accuracy changes are -0.83, -1.01, and -0.57 percentage points respectively.

Another configuration, $m_l=12$ and $k_0=2$, produces roughly 6-7% throughput gains while the reported accuracy values increase on all three benchmarks.

Those measurements should not be read as evidence that stronger restriction is always better. The paper’s more aggressive settings function as a sensitivity test of the trade-off. With $m_l=0$ and $k_0=1$, AIME2025 throughput rises by 50%, but accuracy falls from 87.50 to 76.67, a 10.83-point decline.

For operators, $m_l$ and the warm-up setting are better understood as runtime policy parameters. Their appropriate values depend on the cost of a quality regression in the specific product and workload.

Speculative decoding creates a different sharing opportunity

Speculative decoding makes the original batch problem more pronounced because several candidate tokens may be routed for each request. XShare adds a hierarchical variant rather than treating all speculative tokens as unrelated.

The reason is empirical: speculative tokens belonging to the same request show greater overlap in expert preferences than unrelated tokens. The method first builds a small expert pool within each request, then combines those pools and refines the selection across the batch.

With GPT-OSS 120B, four requests, and speculation length 3, the reported $(k_0=1,m_l=0,m_r=4)$ configuration improves OTPS by 14.3% on AIME2025, 13.3% on the rendered LCBench row, 14.9% on MMLU-Pro, and 13.4% on GPQA. Accuracy remains competitive in several reported rows, although performance is not uniformly preserved across every benchmark and configuration.

Removing the protective warm-up can be damaging. One no-warm-up configuration loses 22.5 accuracy points on AIME2025 and 19.61 points on the AALCR accuracy row. The mechanism therefore depends not only on finding shared experts, but on preserving enough token-specific routing preference before sharing is imposed.

In expert parallelism, the busiest GPU becomes the constraint

When experts are distributed across GPUs, reducing total expert count is not sufficient. Synchronized execution can be constrained by whichever GPU receives the largest selected expert set.

The GPU-aware XShare variant incorporates physical expert placement into selection. In the reported DeepSeek-R1 IFEval experiment at batch size 16, average activated experts fall from 160.4 to 43.4, while maximum experts on any GPU fall from 25.6 to 8.64. Accuracy changes only from 0.697 to 0.696.

On GSM-8K at batch size 8, the corresponding figures move from 50.7 to 33.2 activated experts and from 11.3 to 5.94 maximum experts per GPU, with accuracy declining from 0.956 to 0.946.

For teams running expert-parallel serving, this reframes routing as partly a placement-aware scheduling problem. Cognaptus infers that the relevant policy target may be peak per-GPU expert pressure rather than total selected experts when synchronization latency dominates. The paper measures that load reduction directly; it does not report a universal conversion from lower peak load to proportional end-to-end speedup.

The control surface is promising, but deployment evidence is still narrow

The results are strong comparative system evidence inside the tested environment: GPT-OSS 120B and DeepSeek-R1, vLLM, and one AWS p5en.48xlarge instance with NVIDIA H100 GPUs. Speculative decoding is evaluated on GPT-OSS 120B with EAGLE-3 and speculation length 3. The structured result tables do not report confidence intervals or repeated-run variance.

That leaves several questions for production adoption. Router scores must remain informative enough for the proxy to work. Speculative-token correlation must persist under the target workload. Different accelerators, expert placements, inference engines, batch schedulers, and model architectures may move the throughput-quality frontier.

The paper nevertheless changes a useful systems decision. MoE sparsity is not only a property of model architecture or nominal top-k routing. Under real batching, it can become a runtime allocation problem. XShare shows that inference teams can intervene at that layer—without retraining—and trade expert footprint against throughput, quality, and peak GPU pressure.

The right operating point is unlikely to be the smallest expert pool. It is the smallest pool that still satisfies the quality requirement of the workload being served.

Cognaptus: Automate the Present, Incubate the Future.