Probe Before You Prune: Measuring Which MoE Experts a Workload Can Lose

TL;DR for operators

A mixture-of-experts model can be too large for an inference budget even though only a small subset of its experts executes for each token. The inactive experts still have to reside somewhere. In the measured Mixtral deployment studied by Janati et al., removing four of eight experts per layer cut 4-bit memory from 24.2 GB to 12.3 GB and per-token latency from 40.3 ms to 25.2 ms on a single A100.¹

The difficult decision is which experts to remove. Random pruning and simple magnitude-based selection deteriorate sharply at aggressive compression. The paper instead uses a brief task-relevant adaptation run as a diagnostic: observe which experts’ routing treatment changes most, keep those experts, structurally remove the least-sensitive ones, and discard the temporary adapter.

For deployment teams, the useful result is not that lightweight fine-tuning makes the final model better. The adaptation serves as a measurement instrument. How that instrument is configured affects what it measures: router-local adaptation gives a cleaner pruning signal than broader LoRA at matched rank, and increasing router-only LoRA rank improves the resulting expert ranking across the tested range.

The boundary is equally operational. Compression depth cannot be chosen from the Mixtral MMLU-Pro curve alone. GSM8K develops a severe accuracy cliff under deeper pruning, while BoolQ behaves non-monotonically. The candidate experts can be identified cheaply; the acceptable pruning ratio still has to be validated against the workload that matters.

The expensive part is choosing experts, not deleting them

Structural expert removal has an intuitive infrastructure effect. Fewer resident experts reduce parameter storage and, in the paper’s measured decoding regime, reduce memory traffic. Mixtral still routes each token to two active experts, so pruning does not primarily save latency by reducing the number of experts executed per token. It saves by shrinking what must remain resident and be moved through the memory hierarchy.

The authors measure an unusually clean deployment progression:

These are direct measurements on one A100 with 4-bit Mixtral weights, not a universal hardware law. Within that setting, however, resource savings are close enough to linear that a deployment team can roughly price the infrastructure benefit of removing another expert before deciding whether the capability loss is acceptable.

That makes expert selection the harder engineering problem.

On the paper’s main MMLU-Pro pipeline, the unpruned model scores 44.83%. After four experts per layer are removed, magnitude-based selection falls to 15.88% and random selection to 16.31%. A task-conditioned routing signal preserves substantially more performance.

A short adaptation run becomes a measurement intervention

The router is the part of an MoE model that decides which experts receive each token. If a task places pressure on the model during adaptation, changes in how the router treats particular experts can reveal which experts the task depends on.

The paper turns that observation into a one-shot pruning procedure. It briefly adapts the model, measures the change associated with each expert, ranks experts within each layer, removes the same number of lowest-sensitivity experts from every layer, then evaluates the original checkpoint with those experts structurally deleted. There is no post-pruning retraining, distillation, or router recalibration.

For directly adapted routers, the practical score is the absolute change in the norm of an expert’s router vector:

Small changes imply that adaptation placed relatively little pressure on that expert’s routing treatment. Those experts are pruned first.

The temporary adapter is then discarded. This point prevents a common misreading of the results: the paper is not demonstrating that router-only LoRA is generally the best way to fine-tune an MoE model. It is testing which adaptation configuration gives the clearest measurement of expert importance.

Broader adaptation makes the diagnostic signal weaker

The clearest controlled result comes from changing where rank-8 LoRA is applied while keeping the main Mixtral setup otherwise fixed.

After pruning four experts per layer, router-only adaptation retains 27.54% MMLU-Pro accuracy. Extending LoRA to the router plus attention projections reduces this progressively, and targeting all linear modules leaves 24.42%.

That pattern is informative because ordinary fine-tuning intuition could suggest the opposite: give the optimizer more places to adapt and it should represent the task more effectively. But representation quality is not the objective here. The adaptation is being used to expose which experts matter.

When adaptation is restricted to the router, task pressure has fewer places to go. Changes must be expressed more directly through routing. Once attention projections and expert computation can also absorb the loss reduction, router movement becomes a less isolated indicator of expert relevance.

Cognaptus interprets this as a broader deployment principle: a diagnostic intervention should be optimized for signal identifiability, not automatically copied from the configuration that maximizes task adaptation performance.

LoRA rank controls measurement fidelity, not deployed capacity

The rank sweep reinforces the same interpretation. With router-only LoRA, higher rank improves post-pruning accuracy monotonically at every tested compression depth.

At four experts removed per layer, MMLU-Pro rises from 23.42% at rank 2 to 25.87% at rank 4, 27.54% at rank 8, and 28.76% at rank 16.

The adapter remains tiny: the tested ranks train approximately 0.0006% to 0.0045% of model parameters. More importantly, none of that adapter capacity remains in production. Higher rank changes the perturbation used to discover the expert ranking; it does not make the deployed pruned model larger.

Replications also provide a useful interpretation threshold. Across twelve paired measurements, reruns differ by 0.76 MMLU-Pro points on average and by at most 2.24 points. The authors therefore treat roughly 2.2 points as the practical resolution of a single comparison. The broad rank ordering is stable, while small adjacent differences deserve less weight.

Router weights do not have to move for routing importance to be measurable

The paper also tests adaptation methods that leave router weights frozen. In that case, parameter change cannot be used as the diagnostic.

Instead, the authors measure how much each expert’s router logit changes across held-out tokens:

Using this behavioral signal, IA3 reaches 28.04% MMLU-Pro after four-expert pruning, versus 27.54% for router-only LoRA. That difference is inside the paper’s estimated measurement resolution.

This experiment is best read as a robustness and mechanism test, not evidence that IA3 is superior. Training budgets are not matched across LoRA, IA3, and Houlsby adapters, so perturbation geometry is not causally isolated. What the result does support is narrower: expert importance can remain observable through changes in routing behavior even when the router parameters themselves are frozen.

Compression depth has to follow the workload

The method transfers beyond the main Mixtral sweep. In the Qwen1.5-MoE mathematics setting, removing half of the 20 experts leaves router-sensitivity pruning at 49.7% mean accuracy across eleven benchmarks, compared with 39.1% for magnitude selection and 4.4% for random pruning.

But the cross-benchmark Mixtral tests also show why a single aggregate retention curve is unsafe for deployment planning.

ARC-Challenge declines gradually from 54.10% unpruned to 46.59% after four experts are removed. GSM8K falls from 51.25% to 12.05%. BoolQ initially drops, partially recovers at intermediate pruning, then falls sharply to 62.1%.

The adaptation corpus also appears to condition what the pruning signal protects. Domain-matched mathematical adaptation transfers strongly in the Qwen study, while capabilities weakly exercised by the adaptation data can be less protected. The evidence is suggestive rather than a complete causal isolation of dataset effects, but it gives deployment teams a concrete control surface to test.

What deployment teams can take from the paper

The resulting workflow is straightforward: choose data representative of the capability that must survive, run a brief diagnostic adaptation, measure per-expert routing sensitivity, prune low-sensitivity candidates, then benchmark progressively deeper compression on the actual workload.

The last step cannot be skipped.

The reported accuracy is a selection test, not a compression ceiling

Several boundaries affect how far the results should be generalized.

The controlled sweeps concentrate on Mixtral-8×7B-Instruct. The Qwen mathematics experiment broadens the evidence, but it changes model family, domain, and training regime simultaneously. Its starting 20-expert checkpoint was also itself selected from a larger 60-expert model using router sensitivity because full fine-tuning exceeded available cluster memory.

Most sweep configurations are single runs, with partial replication used to establish the approximate 2.2-point comparison resolution. Comparisons among LoRA, IA3, and Houlsby use unmatched budgets. Comparisons with other pruning papers are also not controlled head-to-head experiments because evaluation harnesses, calibration data, and layerwise pruning policies differ.

Finally, the authors deliberately exclude post-pruning repair. Router recalibration, retraining, distillation, or iterative pruning could recover performance. The reported curves therefore tell us about the quality of one-shot expert selection, not the maximum accuracy attainable after a full compression pipeline.

Probe the capability you intend to preserve

For MoE deployment, the paper changes where compression engineering should begin. The first question is not how many experts can be deleted. It is how cheaply the model can reveal which experts the target workload depends on.

Lightweight adaptation provides one answer: use a temporary perturbation to make task dependence visible in routing, then remove experts that barely respond. The Mixtral experiments indicate that the probe works better when adaptation is concentrated where the measurement is being taken and when the temporary adapter has enough rank to express the routing revision.

The infrastructure benefit can be substantial. The acceptable pruning depth remains empirical.

That combination suggests a disciplined deployment policy: probe with representative data, prune incrementally, price the resource savings, and stop where the workload-specific capability curve becomes unacceptable.

Cognaptus: Automate the Present, Incubate the Future.