When Attention Stops Growing, the Hardware Split Should Too
TL;DR for operators The familiar rule for heterogeneous LLM inference—put attention on one kind of hardware and FFN work on another—becomes less reliable when some attention layers stop accumulating state with context length. The paper’s SubQuadratic Disaggregation design, or SQD, partitions decode around that distinction. Context-growing dense attention and sparse top-k selection stay with the DRAM GPU that already owns the full KV state. Top-k attention, sliding-window attention, linear recurrence, and FFN/MoE work have static resident footprints with respect to context and can move to an SRAM-only decode accelerator. ...