When the Next Sparse Billion Should Go to Memory, Not Experts

TL;DR for operators

A sparse-model team with a fixed marginal parameter budget should not assume that the next increment belongs in more experts. In the LongCat-Flash experiments reported by Liu et al.,¹ parameter-equivalent expert scaling performs better earlier, but N-gram embedding scaling takes the lead after the MoE reaches a sufficiently high-sparsity regime.

That does not make embedding capacity a universally superior scaling target. The paper finds several limits: embedding allocation becomes counterproductive when it grows too large; particular hash-table sizes create collision spikes; deeper models weaken the influence of input-level embeddings; and the embedding signal needs explicit amplification during training.

At scale, LongCat-Flash-Lite allocates 31.4B of its 68.5B parameters to N-gram embeddings and dynamically activates 2.9B–4.5B parameters per token. At the reported 1.3T-token checkpoint, it beats a matched expert-scaled baseline on 9 of 11 base-model benchmarks.

For production teams, the relevant test is broader than training loss. Compare expert capacity with lookup-memory capacity under the same total-parameter budget, then measure workload quality, activated parameters, memory traffic, cache behavior, batching, and kernel overhead. The paper’s thresholds should be treated as LongCat-specific starting hypotheses, not portable scaling laws.

The next sparse parameter does not always belong in another expert

Mixture-of-Experts models create a useful separation between stored capacity and per-token computation. A model can hold many experts while routing each token through only a small subset. Once that architecture is in place, the natural way to add sparse capacity is to keep adding experts.

The LongCat experiments show why that rule can eventually become expensive in the wrong dimension. Early in scaling, additional experts remain the better use of a parameter-equivalent budget. At higher sparsity, however, their marginal improvement diminishes. The preferred allocation then reverses: the same additional parameter budget can produce lower loss when placed in sparse lookup memory rather than more routed experts.

The alternative studied here is an N-gram embedding: a set of lookup tables indexed by recent token context. Instead of activating another feed-forward expert, a token retrieves a small number of context-dependent vectors from a much larger stored table. Capacity grows without proportionally increasing routed matrix computation.

This is the central result of the paper’s controlled scaling study, not an inference from the final benchmark table. The authors train parameter-equivalent configurations from scratch at 280M, 790M, and 1.3B activated-parameter scales, using 300B tokens per configuration. The comparison is deliberately between two destinations for the same total parameter budget.

The resulting rule is conditional: fund experts while expert scaling is still efficient; consider lookup memory after marginal expert returns flatten.

Once embeddings win, their budget still has a ceiling

A tempting interpretation is that the allocation should simply keep shifting toward embeddings after the crossover. The ablations argue against that.

The reported advantage is non-monotonic. When too large a share of total parameters is assigned to N-gram embeddings, the matched expert-scaled model can regain the lead. In the tested settings, the authors derive a practical guideline of keeping the embedding allocation at roughly half of total parameters or less. LongCat-Flash-Lite lands close to that boundary: 31.4B embedding parameters account for about 46% of its 68.5B total.

Lookup capacity also introduces a different failure mode from expert routing: collisions. Because recent token sequences are hashed into finite tables, different N-grams can land on the same entry. The paper reports sharp collision increases when N-gram vocabulary sizes approach integer multiples of the base vocabulary size—even when the chosen N-gram vocabulary size itself is prime. Table sizing therefore becomes an architecture parameter rather than a storage afterthought.

Other hyperparameters appear less fragile. Once the maximum N-gram order is at least 3 and the design uses at least two sub-tables, performance is relatively insensitive to moderate choices; maximum orders between 3 and 5 are reported as consistently near-optimal.

These experiments are best read as design ablations. They constrain the main scaling claim by identifying where the mechanism stops behaving like a free capacity expansion.

Width helps the lookup memory; depth slowly hides it

Model geometry changes the allocation decision.

With depth held fixed, increasing width expands the regime in which N-gram embeddings outperform parameter-equivalent expert scaling. At the 1.3B activated-parameter scale, the embedding variant still holds a clear advantage at total-to-activated parameter ratios as high as 50. In plain terms, wider hidden representations appear better able to exploit the additional lookup capacity.

Depth moves in the other direction. The input-level N-gram signal has to survive repeated residual transformations. The paper reports a pronounced contraction in its relative advantage beyond 20 shortcut layers, although the method remains effective through the tested 40-layer configuration.

A separate ablation supports the same mechanism from another direction. Without intervention, the initial embedding signal can be overwhelmed by the first attention output. Amplifying the embedding contribution with either a scaling factor or LayerNorm lowers training loss and both Chinese and English validation losses by 0.02 in the reported experiment.

The paper also tests whether sparse memory should be injected repeatedly across layers. Per-Layer Embedding underperforms the parameter-equivalent input-level N-gram design. Per-Layer N-gram Embedding produces only marginal gains in one setting and no consistent advantage as width or depth increases, while also increasing activated parameters. The authors consequently leave it out of the large-scale system.

For architecture teams, this means the capacity question cannot be separated from width, depth, and signal propagation. A lookup-memory allocation that works in a wide, relatively shallow sparse model may not preserve the same return in a deeper family.

LongCat-Flash-Lite tests the allocation at 68.5B parameters

The paper’s strongest large-scale evidence is the matched comparison between LongCat-Flash-Lite and LongCat-Flash-Lite-Vanilla. The training strategy and data recipe are held constant; the distinguishing choice is whether 31.4B parameters are allocated to N-gram embeddings or additional experts.

At the reported 1.3T-token checkpoint, the embedding model wins 9 of 11 base-model benchmarks. Some of the larger differences are:

The losses are also informative. MMLU falls from 64.81 to 64.01, while MultiPL-E moves from 30.20 to 30.03. This is not uniform domination across tasks.

The external chat-model table is less useful for isolating the architecture effect because the compared models differ in training data, post-training, decoding, and evaluation infrastructure. LongCat-Flash-Lite leads the listed systems across the reported agentic tool-use and agentic coding rows—including 54.4 on SWE-Bench and 33.75 on TerminalBench—but those results support competitive capability, not a controlled causal claim about embedding scaling.

Fewer active experts do not create serving savings by themselves

Moving capacity from experts into lookup tables reduces active MoE load, but it also changes where the system spends time.

The authors therefore pair the architecture with an inference stack that includes an N-gram cache, custom CUDA kernels, EAGLE-3 three-step speculative decoding, wide expert parallelism, operation overlap, kernel fusion, optimized attention combine, and Programmatic Dependent Launch.

These are not peripheral implementation details. Once active expert compute and memory I/O fall, lookup cost, effective batch size, GPU utilization, and kernel-launch overhead become more visible. Architectural sparsity can shift a bottleneck without removing it.

That changes the business case. A team evaluating this design should compare at least five quantities under its actual workload:

An organization that can reproduce the architecture but not the serving optimizations may capture the parameter-allocation benefit without reproducing the paper’s end-to-end inference economics.

Treat the reported crossover as a hypothesis to test internally

The evidence is reasonably strong for a comparative claim inside the LongCat-Flash family: the experiments use matched total parameter counts, shared token budgets, controlled ablations, and a matched large-scale baseline.

The portability of the thresholds is much less certain. The study does not provide repeated-seed uncertainty or significance tests. Its pretraining and validation data are not documented in enough detail for a full independent assessment of representativeness or contamination. Width, depth, tokenization, routing design, data mixture, and serving hardware could all move the crossover point.

Inference portability is narrower still because the reported system depends on specialized implementation work and H800-class GPU deployment.

Cognaptus therefore reads the paper as a change in capacity-allocation procedure, not a new universal scaling ratio. When additional experts show flattening returns, a model team now has a credible alternative experiment to run: hold total capacity and training conditions as constant as possible, shift a marginal block into lookup memory, and test the complete architecture-plus-serving system.

The decision becomes more disciplined because “more sparse capacity” no longer names a single engineering action. It names a budget that can be invested in different mechanisms with different returns and different operational costs.

Cognaptus: Automate the Present, Incubate the Future.