Pooling Resources: UniPool and the MoE Budget Nobody Wanted to Audit

Opening — Why this matters now

AI infrastructure has entered its spreadsheet era. Not the glamorous spreadsheet, where revenue projections grow diagonally upward and nobody asks where the assumptions came from. The other spreadsheet: the one where compute cost, memory footprint, inference latency, training instability, and model quality all insist on appearing in the same row.

Mixture-of-Experts architectures have become one of the industry’s favorite answers to this pressure. Instead of activating every parameter for every token, an MoE model routes each token to a small subset of specialized feed-forward “experts.” The sales pitch is elegant: store many parameters, activate only a few. Scale the model without paying full dense-model compute on every token. Very tasteful. Very expensive. Slightly suspicious.

The suspicion is exactly where UniPool: A Globally Shared Expert Pool for Mixture-of-Experts becomes interesting.¹ The paper asks a deceptively simple question: if every transformer layer owns its own private bank of experts, are we scaling intelligence — or simply duplicating similar expert functions layer after layer?

That distinction matters because much of enterprise AI economics is not about whether a model is impressive in a demo. It is about whether the same quality can be obtained with fewer stored parameters, better routing discipline, and a clearer understanding of what capacity is actually doing. In other words, less “look at the parameter count” and more “please explain why we are paying for this particular pile of tensors.”

The paper’s answer is UniPool: replace layer-private expert ownership with a single global expert pool accessed by independent routers at each layer. The idea is not merely architectural tidiness. It reframes expert capacity as a shared organizational resource — closer to a central specialist team than a redundant specialist desk installed on every floor of the building.

The paper directly shows that, at the tested scales, this shared-pool design improves validation loss and perplexity over matched vanilla MoE baselines, while reduced-pool variants can match or outperform layer-wise MoE using only 41.6%–66.7% of the vanilla expert-parameter budget. My business interpretation is narrower but important: if this direction scales, AI cost optimization will become less about shrinking models after training and more about designing architectures that avoid redundant capacity before the bill arrives. Revolutionary concept: not buying the waste in the first place.

Background — Context and prior art

A standard transformer layer has an attention block and a feed-forward block. In MoE models, the feed-forward block is replaced by a set of expert feed-forward networks. A router decides which expert or experts receive each token. In a conventional layer-wise MoE design, each layer owns its own expert set:

This convention is not irrational. If every layer operates at a different level of representation, it seems natural to give every layer its own specialists. Early tokens need one kind of transformation, later representations need another, and the router can learn the local division of labor.

But the paper argues that this design may be over-rigid. Recent work cited by the authors suggests substantial expert redundancy inside trained MoE models: same-layer expert weights can be highly similar, similar-expert rerouting can preserve accuracy, and pruning can remove large portions of expert capacity with surprisingly tolerable damage. UniPool adds a routing-based probe: in three production MoE models — Qwen1.5-MoE, DeepSeek-V2-Lite, and Qwen3-30B-A3B — the authors randomize the learned router in one deep-half MoE layer at a time. If deep-layer experts were truly sharply specialized, random routing should be painful. Instead, the average downstream accuracy drop is only 1.0–1.6 points.

The paper’s interpretation is careful: this does not mean routing is useless everywhere. It means that, in these tested deep layers, the choice among same-layer private experts carries limited local task signal. When randomizing the expert choice barely hurts, the supposedly specialized expert bank begins to look less like a team of experts and more like a room full of people who attended the same training seminar.

Prior MoE systems such as Switch Transformer, GShard, Mixtral, and the DeepSeek MoE line largely preserve layer-private expert ownership. Other parameter-sharing work, such as Universal Transformers and ALBERT, shares broader model components across depth. UniPool sits between those worlds: it shares the large feed-forward expert pool across layers, while keeping attention blocks and routers layer-specific. That distinction is important. The model does not pretend all layers are identical. It lets layers ask different questions of the same shared expert budget.

Analysis or Implementation — What the paper does

UniPool changes the ownership rule. In vanilla MoE with $L$ layers and $E$ experts per layer, the model stores $L \times E$ layer-private expert feed-forward networks. Each layer $l$ routes token $x$ only to its own local experts:

$$ \text{FFN}l(x) = \sum{i \in \text{Top-k}(r_l(x))} g_{l,i}(x) \cdot e_{l,i}(x) $$

UniPool replaces those private expert sets with one shared pool:

$$ \mathcal{E} = {e_1, e_2, \ldots, e_M} $$

Each layer still has its own router $r_l$, but it routes into the same global pool:

$$ \text{FFN}l(x) = \sum{i \in \text{Top-k}(r_l(x))} g_{l,i}(x) \cdot e_i(x) $$

This is the central move. Attention and routers remain layer-specific; expert parameters become globally reusable. The architecture says: different layers may need different routing policies, but they do not necessarily need separately owned copies of similar expert functions.

The paper then solves two practical problems created by this move.

First, the usual per-layer auxiliary loss no longer matches the ownership structure. In vanilla MoE, if an expert receives no tokens within a layer, that layer’s private expert is wasted. Under shared ownership, an expert unused by layer 7 may be heavily used by layers 3, 12, and 18. Calling it “dead” because one layer ignored it is the architectural equivalent of declaring an employee idle because they did not attend your meeting.

UniPool therefore introduces a pool-level auxiliary loss. Instead of balancing usage separately inside every layer, it aggregates token-to-expert assignments across sharing layers and balances the global pool. In simplified terms, the paper defines the global average token fraction for expert $i$ as:

$$ f_i = \frac{1}{L}\sum_{l=1}^{L} f_i^{(l)} $$

and uses a pool-level balancing objective:

$$ \mathcal{L}{pool} = \alpha{pool} \cdot M \cdot \sum_{i=1}^{M} f_i \cdot P_i $$

where $P_i$ is the global average routing probability for expert $i$. The implementation uses a one-micro-batch-behind statistic to avoid cross-layer tensor dependencies, which is a pleasingly practical detail: the authors are not just drawing arrows between boxes and hoping distributed training forgives them.

Second, routing into a larger shared pool can become unstable. Different layers may produce hidden states with different scales. A standard softmax router can translate those scale differences into inconsistent routing sharpness. UniPool adopts NormRouter, which normalizes router logits and applies ReLU-based scoring:

$$ s_i = \sigma \cdot c \cdot \max\left(0, \frac{z_i}{|z|_2 + \epsilon}\right) $$

Here, $\sigma$ is learnable, $c$ is a fixed calibration constant estimated by Monte Carlo sampling, and $\epsilon$ is a numerical stabilizer. The paper’s logic is straightforward: normalization makes scores less sensitive to layer-dependent magnitude, ReLU induces sparse competition, and the learnable scale lets routers adjust strength during training.

The experimental setup is intentionally matched against vanilla MoE. The main baselines use LLaMA-style transformer backbones at five active-parameter scales: 182M, 469M, 650M, 830M, and 978M. Models are trained on roughly 30B tokens from the Pile, using 8 experts per layer and top-1 routing in the vanilla baseline. In the matched UniPool setting, the global pool size is $M = 8L$, so total expert feed-forward count and per-token expert compute are matched. That matters: the main comparison is not “we secretly gave one model more experts.” The comparison isolates expert ownership, routing, and balancing.

Findings — Results with visualization

The headline result is consistent but not magical. UniPool improves validation loss and perplexity across all five tested scales.

A simple text chart makes the scale of the loss reductions easier to compare:

Validation loss reduction vs. vanilla MoE
182M | ██████████████████████        -0.0288
469M | ██████████████████████████    -0.0346
650M | ███████████████████████       -0.0308
830M | █████████████████████████████ -0.0386
978M | █████████████                 -0.0172

The 830M result is especially interesting. The 830M model is deeper — 48 layers with hidden size 1024 — while the 978M model is wider — 24 layers with hidden size 1536. UniPool achieves a lower validation loss in the deeper 830M configuration than in the wider 978M configuration, despite the latter having more active and stored parameters. The paper interprets this as support for a budget-allocation view: depth creates more opportunities to reuse a global expert pool. Width is not automatically a better use of capacity. Yes, the model architecture has entered capital budgeting.

The downstream evaluation broadly follows the same pattern. On seven zero-shot benchmarks, UniPool improves the average score at every tested scale, although individual task results are not uniformly positive.

This is useful because perplexity improvements can sometimes look elegant while downstream behavior looks unimpressed. Here, the paper provides evidence that the loss improvements translate into modest but consistent average task gains. Not a parade. A signal.

The more business-relevant result is the reduced-pool experiment. The authors shrink the shared expert pool below the matched vanilla expert budget while keeping top-1 active expert compute matched. At every tested scale, a smaller-than-vanilla UniPool variant still surpasses the layer-private baseline.

This is the paper’s most operationally interesting claim: pool size becomes a depth-scaling hyperparameter. Instead of assuming expert parameters must grow linearly with layer count, a shared-pool design can let expert parameters grow sublinearly with depth — at least in these experiments.

The ablation study is also important because it prevents a lazy interpretation. UniPool does not win merely because it shares parameters, nor because NormRouter alone is a better router. At the 182M scale:

The ugly row is the revealing one. Sharing experts while keeping the old per-layer auxiliary loss performs worse than vanilla MoE. That means the architecture and training objective must be redesigned together. Reusing expert capacity without changing the balancing objective is not efficiency. It is just moving the furniture into the hallway and calling it open-plan.

The routing-randomization analysis gives a second lens. On the authors’ own trained models, randomizing one deep-half layer in vanilla MoE causes small average accuracy drops: -1.3 at 469M and -1.5 at 978M. Under UniPool, the matched top-8 randomization causes larger drops of -4.1 at both scales.

The paper’s interpretation is that UniPool makes routing decisions more load-bearing. In vanilla deep layers, private experts are more substitutable. In UniPool, experts are shared across layers and compete globally, so the router’s selected expert appears to matter more. This is a useful sign: the model is not merely reusing capacity; it may be turning redundancy into specialization.

Finally, the utilization analysis shows why the pool-level objective matters. With a shared pool, softmax routing and per-layer auxiliary loss can collapse traffic onto a small subset of global experts. UniPool’s pool-level auxiliary loss plus NormRouter restores more balanced global usage while preserving different layer-specific routing patterns. In management language: the shared resource is actually used as a shared resource, not captured by a few loud departments.

Implications — What changes in practice

The paper directly shows an architectural result under controlled experimental conditions: at 182M–978M active-parameter scales, trained for 30B tokens, UniPool improves validation loss and perplexity versus matched vanilla MoE baselines, improves average zero-shot benchmark accuracy, and can beat vanilla MoE with smaller shared expert pools.

The business interpretation starts after that sentence, not before it.

1. Model capacity should be treated as budget allocation, not decoration

Many AI procurement and build-vs-buy conversations still treat parameter count as a prestige metric. MoE models complicate that habit because stored parameters and active parameters diverge. UniPool complicates it further: even stored expert parameters may be organized inefficiently if every layer owns private experts that could have been shared.

For business leaders, the practical question becomes:

This matters for ROI because the cost of AI systems accumulates across training, fine-tuning, serving, monitoring, and future refresh cycles. A model that reaches a quality target with fewer stored expert parameters may reduce memory pressure and deployment complexity — but only if the throughput and systems implications are also favorable.

2. Efficiency is not just pruning after the fact

A large part of the model-efficiency ecosystem focuses on post-hoc compression: prune, quantize, distill, merge, cache, reroute, or otherwise tidy up the model after it already learned with excess structure. Those techniques are useful. They are also sometimes the technical version of cleaning up after a banquet nobody budgeted properly.

UniPool points toward preemptive efficiency. Instead of asking which redundant experts can be removed after training, it asks whether expert ownership should be global from the beginning. That is a different operating philosophy: design the resource-sharing mechanism before the redundancy crystallizes.

For companies building domain-specific LLMs or fine-tuned expert systems, this suggests a more disciplined architecture review:

3. Shared resources require shared governance

The analogy to business operations is almost too neat, so naturally we should use it carefully. A shared expert pool is like a centralized specialist team serving multiple departments. Done well, the organization avoids duplicated roles and increases specialist utilization. Done badly, everyone fights for the same bottleneck and the queue becomes the strategy.

UniPool’s lesson is that sharing alone is insufficient. The paper’s ablation shows that shared experts with the wrong auxiliary loss perform worse than vanilla MoE. In business terms, centralization without the right allocation rules is not efficiency; it is bureaucracy with better branding.

The technical equivalent of governance is the pool-level auxiliary loss and NormRouter. These mechanisms define how shared capacity is accessed, balanced, and stabilized. For AI operations, the broader principle is clear: whenever a system introduces shared resources — shared tools, shared memory, shared retrievers, shared agents, shared vector stores — it also needs allocation logic. Otherwise the system will either collapse into overuse of a few components or distribute work so evenly that specialization disappears.

4. The missing production question is throughput

The authors are explicit about limitations. They do not report wall-clock throughput comparisons. At the matched setting, UniPool has the same total expert FFN count as vanilla MoE; the architecture changes ownership by reference rather than immediately reducing stored parameters. Storage and memory savings emerge in the reduced-pool regime. The paper also notes that pool-level auxiliary loss introduces overhead from cross-layer statistic accumulation, and routing into a larger candidate pool may affect token-dispatch efficiency under expert parallelism.

That caveat matters. A model architecture can look better on validation loss but become awkward in production if routing, dispatch, memory movement, or parallelization degrade serving economics. For enterprise deployment, the next question is not only “does UniPool improve quality?” It is:

Production ROI = quality gain
               - training overhead
               - routing/dispatch overhead
               - memory and storage pressure
               - engineering complexity
               - operational risk

The paper provides evidence for the quality and stored-parameter side of this equation. It does not close the production-throughput side. That is not a flaw; it is a boundary. Boundaries are useful. They prevent slide decks from becoming theology.

5. Agentic and modular AI systems should pay attention

Although UniPool is a model-architecture paper, its conceptual lesson travels well into business automation and agentic systems. Many organizations are now building multi-agent workflows, retrieval systems, tool-using assistants, and domain-specific automation stacks. These systems often duplicate capabilities across workflow stages: every agent gets its own prompts, tools, memory fragments, validators, and reporting logic.

The UniPool principle asks: should every stage own its own full specialist stack, or should some capabilities be globally pooled and routed to as needed?

This is extrapolation, not a result from the paper. But it is a useful one. Many AI automation projects fail not because the model cannot reason, but because the system architecture quietly duplicates work, scatters ownership, and lacks clear utilization signals. UniPool is a reminder that “modular” does not automatically mean efficient. Sometimes modularity is just redundancy wearing a lanyard.

Conclusion

UniPool is not a claim that all experts should always be shared, or that layer-private MoE is obsolete. The experiments are limited to 182M–978M active-parameter scales, 30B training tokens, and seven zero-shot downstream benchmarks. The authors are clear that billion-scale validation, longer training horizons, broader evaluation, and throughput studies remain open.

What the paper does show is still important: the per-layer expert ownership rule in vanilla MoE is not sacred. It can produce redundancy, especially in deeper layers. A globally shared expert pool, when paired with a pool-level balancing objective and scale-stable routing, can improve quality and parameter efficiency at the tested scales. Reduced-pool variants suggest that expert capacity need not grow linearly with depth.

For business readers, the broader lesson is simple: AI efficiency is not just a hardware problem, nor just a compression problem. It is an allocation problem. Where do we place capacity? Who can reuse it? What governance prevents collapse or waste? How do we know a routed decision actually matters?

UniPool answers those questions inside the MoE architecture. The same discipline belongs in enterprise AI systems, agentic workflows, and every automation project where duplicated capability quietly becomes cost. Not every layer needs its own expert empire. Sometimes the smarter move is to pool resources — and finally make the router earn its salary.

Cognaptus: Automate the Present, Incubate the Future.