SSD Capacity Is Not Throughput: What FlashMoE Changes About Local MoE Serving

TL;DR for operators

Moving inactive MoE experts to SSD can make an oversized sparse model fit when the full checkpoint does not fit comfortably in DRAM. But every expert missing from fast memory must be fetched before decoding can continue. In FlashMoE’s evaluated system, an expert load takes roughly 3 milliseconds and expert loading accounts for more than 70% of decoding time. SSD therefore solves capacity only if the cache avoids enough expensive misses.

The paper’s diagnostic shows that ordinary recency-based caching leaves substantial I/O avoidable. An ideal future-aware cache reached an 86% hit rate versus about 73% for LRU, while LRU generated nearly 1.9 times as many I/O operations. More tellingly, 34.2% of experts evicted by LRU were fetched again within five steps, versus 0.1% under the oracle policy. FlashMoE¹ responds by combining recency and frequency signals in a small learned replacement model. On OLMoE-1B-7B, it improved hit rate by 21% over LRU, reduced SSD I/O by 22%, and improved decoding speed by about 22%; the Qwen3-30B-A3B decoding gain was about 7%.

For deployment teams, the hardware question is therefore not simply whether to buy enough memory to make the checkpoint fit. SSD performance, VRAM available for cached experts, checkpoint layout, and replacement quality jointly determine usable throughput—and whether smarter caching is worth more than additional DRAM, VRAM, or storage performance.

SSD solves capacity, then exposes an I/O problem

A sparse MoE model activates only part of its expert capacity for each token. That creates an apparent opportunity: leave inactive experts on SSD instead of reserving enough main memory for the entire checkpoint.

The difficulty appears when an expert is needed again. If it is no longer in fast memory, the system must fetch its weights from storage before computation can proceed. FlashMoE reports roughly 3 milliseconds for one SSD expert load, compared with about 158 microseconds for an operation of its learned cache model. Storage access is sufficiently expensive that expert loading exceeds 70% of total decoding time in the reported setup.

The paper’s diagnostic experiment shows that ordinary caching leaves considerable performance on the table. An ideal cache with knowledge of future accesses would evict whichever expert will be needed farthest in the future. Using that oracle as a reference, the authors report an 86% cache hit rate versus about 73% for LRU, with LRU generating nearly 1.9 times as many I/O operations.

More revealingly, 34.2% of experts evicted by LRU were fetched again within five subsequent time steps. Under the oracle policy, the corresponding rate was 0.1%. This diagnostic is not an end-to-end performance result; its purpose is to establish that a meaningful fraction of SSD traffic is created by replacement decisions rather than by unavoidable capacity pressure.

FlashMoE changes what gets loaded

The first contribution is architectural rather than predictive.

FlashMoE separates the checkpoint into a minimal set of non-expert weights and separate expert files indexed by layer and expert. Initialization loads the non-expert components rather than first placing the complete model in system DRAM. During prefill, the system identifies the experts required by the input and loads those experts from SSD as needed.

That decomposition matters because sparse activation applies to storage traffic as well as computation. Across input lengths of 32, 64, 128, and 256 tokens, the paper reports that prefill touched about 47%, 58%, 64%, and 67% of experts respectively. Longer prompts reach more experts, but the fraction rises sublinearly in this experiment.

The startup measurements show the consequence. FlashMoE reports initial loading 4× faster than llama.cpp and 6.8× faster than Fiddler and DAOP under the approximately 1 GB available-DRAM test condition. Including prefill narrows those advantages to 2.5× and 4.1× respectively.

These figures support a specific systems claim: when full-checkpoint DRAM residency is the constraint, decomposing experts into independently loadable objects can remove unnecessary startup work. They do not establish that SSD loading itself is fast.

The cache has to model more than recency

Once only some experts can remain in VRAM, the system repeatedly faces a replacement decision: which cached expert should be evicted when another must enter?

LRU assumes recent use is the strongest signal of near-future reuse. LFU favors experts used frequently over the accumulated history. The paper’s comparison between their individual eviction decisions on Qwen3-30B-A3B found LRU making the better choice only about 56% of the time. Neither signal consistently dominates.

FlashMoE therefore maintains both expert recency and frequency. An access resets recency and increments frequency; without an access, recency increases while frequency remains unchanged. The features are normalized and concatenated for a small three-layer feed-forward network.

The model is trained using targets derived from the future-aware optimal replacement policy. In effect, the system learns whether combinations of “used recently” and “used often” resemble experts that should be preserved rather than hard-coding either LRU or LFU.

The additional computation is intentionally small: the per-layer model is reported at about 113 KB, and its work can execute asynchronously while much slower SSD loading proceeds.

Better eviction decisions translate into fewer SSD reads

The main cache-policy evidence connects prediction quality to system behavior rather than stopping at an ML metric.

The different throughput gains across the two models are worth retaining. A 22% improvement on one model and 7% on another is evidence that the value of smarter caching depends on routing behavior, model geometry, cache capacity, or their interaction. It is not evidence for a fixed performance multiplier.

The paper’s abstract reports a maximum overall inference speedup of 2.6× against existing MoE inference systems. That is a peak result within the evaluated configurations, not a portable expectation for arbitrary local hardware.

Capacity planning becomes a four-variable decision

For an organization considering workstation-class MoE deployment, Cognaptus would treat FlashMoE less as a prescription to “use SSD” than as evidence that four system choices need to be evaluated together.

Checkpoint layout determines whether inactive experts impose startup and DRAM-residency costs. SSD performance determines the penalty when an uncached expert must be fetched. VRAM cache capacity determines how many experts can remain close to the GPU. Replacement policy determines how efficiently that limited cache is used.

That makes hardware sizing more explicit. Additional VRAM can buy more cached experts. Faster storage can reduce the cost of misses. Better replacement can reduce the number of misses without adding memory. More system DRAM may make some offloading unnecessary. The economically preferable choice depends on the required model size and service-level target rather than on any single memory specification.

A workload-specific learned cache is also conceivable for recurring enterprise inference patterns because FlashMoE constructs its policy from routing traces. The paper, however, does not test whether a policy trained on one routing distribution remains effective after task, language, prompt style, or model behavior changes. Production teams would need to measure that rather than assume it.

The benchmark is workstation evidence, not generic edge evidence

The label “edge device” needs a narrow reading here. The evaluated machine uses an RTX 5070 Ti with 16 GB of GDDR7 and a PCIe 5.0 NVMe SSD rated at 7.4 GB/s read bandwidth. It is a capable consumer desktop, not phone-class or embedded hardware.

The experiment also deliberately leaves only about 1 GB of system DRAM available. That is useful for exposing the failure mode the architecture targets, but it is not representative of ordinary memory availability on the same machine.

Evidence breadth is limited further by two MoE models and TriviaQA-derived workloads. The held-out questions were separated from cache-policy training data, which reduces direct training/evaluation reuse, but the evaluation-set size is unspecified. The paper reports no repeated-run variance, confidence intervals, or significance tests. Transfer under routing-distribution shift is untested, as are energy consumption, thermal behavior, SSD endurance, output-quality equivalence, and production workload performance.

Those omissions do not undo the reported benchmark. They determine what decision it can support: FlashMoE provides credible evidence that storage-backed expert capacity plus workload-aware cache replacement can improve local MoE inference under severe memory pressure. It does not yet establish the hardware envelope or workload range over which that design remains superior.

Local MoE serving is becoming a storage-systems problem

FlashMoE’s strongest contribution is to separate model capacity from compulsory memory residency. SSD supplies capacity; fine-grained checkpoint decomposition prevents inactive experts from being loaded unnecessarily; VRAM caching absorbs repeated accesses; and learned replacement reduces avoidable SSD traffic.

That combination changes the deployment question. For local sparse models, “Will the checkpoint fit?” is no longer sufficient. Operators also need to ask how often the serving workload will miss the expert cache, what each miss costs on the chosen storage stack, and whether adding memory or improving replacement produces the better throughput gain.

The paper makes that systems trade-off measurable on one desktop configuration. The next step for production use is to test whether the same economics survive the actual workload and hardware that will carry them.

Cognaptus: Automate the Present, Incubate the Future.