Inference Systems

TL;DR for operators The familiar rule for heterogeneous LLM inference—put attention on one kind of hardware and FFN work on another—becomes less reliable when some attention layers stop accumulating state with context length. The paper’s SubQuadratic Disaggregation design, or SQD, partitions decode around that distinction. Context-growing dense attention and sparse top-k selection stay with the DRAM GPU that already owns the full KV state. Top-k attention, sliding-window attention, linear recurrence, and FFN/MoE work have static resident footprints with respect to context and can move to an SRAM-only decode accelerator. ...

TL;DR for operators Moving inactive MoE experts to SSD can make an oversized sparse model fit when the full checkpoint does not fit comfortably in DRAM. But every expert missing from fast memory must be fetched before decoding can continue. In FlashMoE’s evaluated system, an expert load takes roughly 3 milliseconds and expert loading accounts for more than 70% of decoding time. SSD therefore solves capacity only if the cache avoids enough expensive misses. ...