When Attention Stops Growing, the Hardware Split Should Too

TL;DR for operators

The familiar rule for heterogeneous LLM inference—put attention on one kind of hardware and FFN work on another—becomes less reliable when some attention layers stop accumulating state with context length.

The paper’s SubQuadratic Disaggregation design, or SQD, partitions decode around that distinction. Context-growing dense attention and sparse top-k selection stay with the DRAM GPU that already owns the full KV state. Top-k attention, sliding-window attention, linear recurrence, and FFN/MoE work have static resident footprints with respect to context and can move to an SRAM-only decode accelerator.

The strongest result is measured rather than projected. On an adjusted eight-B200 proxy, SQD improves mean tokens per joule over the strongest GPU-only baseline by 53% for GLM-5.2, 31% for Nemotron 3 Ultra, and 56% for Gemma 4 31B. The paper also models Rubin GPUs paired with Groq-3 LPX hardware and finds better low-latency throughput, but those numbers should be treated as analytical projections.

For inference operators, the practical decision changes from “Which device runs attention?” to “Which stages own context-growing state, and which can be kept resident in SRAM?” That affects accelerator allocation, SRAM sizing, interconnect design, and whether latency-sensitive requests should use a heterogeneous path at all.

The attention-versus-FFN split no longer matches the memory problem

Suppose an inference service combines expensive DRAM-equipped GPUs with specialized accelerators built around much faster but capacity-constrained SRAM. The placement problem is straightforward only if every attention layer behaves similarly.

It does not.

Dense attention reads a KV state that grows with context length. Sparse top-k selection also needs access to a context-growing state because it must decide which historical entries matter. But after those entries have been selected, top-k attention operates only on the selected subset. Sliding-window attention touches a bounded window, linear or recurrent attention maintains fixed-size state, and FFN or MoE parameters do not grow with the request context.

That distinction is central to Rethinking Heterogeneous System Disaggregation for Subquadratic Attention.¹ The authors classify decode stages along two dimensions: how much computation they perform relative to data moved from memory, and how their resident state scales as context grows.

The result is a different placement rule. What matters is not whether an operator is called “attention.” What matters is whether its required resident state keeps expanding.

SQD keeps growing state with DRAM and moves static state to SRAM

SQD follows that rule directly.

Quadratic work remains on the GPU. For dense attention, that means the attention operation itself. For sparse top-k attention, the GPU performs the selection step because it owns the full growing KV state.

The SRAM-side accelerator receives the work whose state footprint is predictable: top-k attention after selection, sliding-window or recurrent attention, and FFN/MoE computation.

This also corrects a likely misunderstanding of the architecture. SQD is not a proposal to move “attention” wholesale onto an SRAM accelerator. Doing so would recreate the capacity problem that the heterogeneous design is intended to avoid. The split occurs inside the attention family.

That matters operationally because attention-FFN disaggregation can force activations across devices at every layer boundary. By colocating the context-growing operations with the GPU and grouping static-footprint work on the decode accelerator, SQD reduces the number of exposed crossings.

The architecture also colocates prefill and quadratic decode on the GPU side. Because prefill tends to be more compute-intensive while decode attention is strongly memory-bound in the evaluated regime, the system can overlap work that stresses different GPU resources rather than simply adding their execution times.

Sparse attention creates a communication problem of its own

Keeping sparse top-k selection on the GPU solves the state-capacity problem but introduces another dependency: the SRAM-side attention stage still needs the selected KV entries.

If every selected entry had to cross the interconnect synchronously at every layer, the communication delay could erase much of the benefit.

The paper addresses this using a property of GLM-5.2 called IndexShare. Several layers reuse a common selection pattern. Once the GPU has selected indices for a group, later layers provide time in which missing selected-KV entries can be prefetched while earlier work is executing.

The cache is therefore latency-aware rather than merely a conventional capacity cache. The relevant question is whether enough selected state is already resident—or can be transferred soon enough—that communication finishes before the later layer reaches its execution deadline.

This mechanism is narrower than “sparse attention works well with disaggregation.” It depends on cross-layer selection reuse. A model that recomputed independent sparse selections at every layer would expose substantially more blocking communication.

The measured result is primarily an energy result

The paper evaluates GLM-5.2, Nemotron 3 Ultra, and Gemma 4 31B across chat, RAG, and agentic workload regimes on an adjusted eight-B200 system.

The proxy restricts the decode-side GPUs to approximate the compute resources of an SRAM-oriented accelerator and applies corresponding power-accounting assumptions. Under that setup, SQD is charged about 4.11 kW compared with 7.26 kW for the stock eight-GPU system, while retaining 63% to 97% of no-disaggregation decode throughput.

The resulting mean energy-efficiency gains are substantial:

These comparisons are the paper’s strongest operational evidence because they come from executed workloads on the proxy system rather than from the target-hardware model.

They also clarify the economic proposition. SQD is not primarily demonstrating that specialized hardware always produces the most tokens per second. It shows that a model-aware placement policy can preserve much of decode throughput while materially reducing the power attributed to serving.

For latency-sensitive inference at sustained utilization, that difference can translate into lower energy cost per delivered token and greater useful work within a fixed power envelope.

Rubin+LPX extends the argument, but it remains a projection

The authors then model systems combining Rubin R100 GPUs with Groq-3 LPX SRAM-oriented hardware. The analytical model is anchored to measured B200 profiler and transport constants and reproduces the relative B200 tokens-per-joule advantage to within 15% across nine model-baseline comparison points.

Under fixed power budgets, the model projects that SQD can deliver roughly 1.2–1.5× higher achievable user tokens per second than attention-FFN disaggregation across the evaluated models and workloads. For GLM-5.2, projected system throughput is 2.9–3.6× higher over the overlapping service-level range.

Those numbers are not measurements from Rubin or LPX hardware.

That boundary matters because the commercial inference is more credible than the absolute forecast: if future GPU-plus-SRAM systems preserve the modeled resource balance and communication behavior, model-aware disaggregation can make heterogeneous serving especially attractive in low-latency regimes.

At relaxed latency targets, the paper finds that GPU-oriented baselines can again win on peak throughput because larger batches push decode toward more compute-efficient operation. SQD is therefore a workload-routing option, not a universal replacement path.

Provision SRAM for residency and interconnect for latency

Two sensitivity results turn the architecture into more concrete infrastructure guidance.

First, additional SRAM capacity helps mainly until the model’s static state fits. Beyond that point, extra capacity produces diminishing benefit. This is particularly relevant for MoE systems, where resident parameter capacity can be much larger than the parameters actually read for one token.

Second, SQD is more sensitive to interconnect latency than to raw bandwidth. The transfers are relatively small, but some lie directly on the decode critical path. Adding bandwidth does little when the dominant penalty is waiting for a blocking dependency to begin or complete.

For datacenter architects, the implication is specific: an SQD-like system should not be provisioned by maximizing every hardware dimension independently. SRAM should be sized around static residency, and the GPU-to-accelerator fabric should prioritize low blocking latency once adequate transfer bandwidth is available.

The deployment decision is model-specific and latency-specific

The paper leaves several important boundaries.

The SRAM-only accelerator is approximated through adjusted B200 execution rather than measured as physical target hardware. The proxy power accounting intentionally removes components such as HBM-related costs to emulate that target. Rubin+LPX performance is analytical. Some one-million-token geometry points for Nemotron 3 Ultra and Gemma 4 extend beyond the served 262,144-position configurations, and the inference-performance campaign uses dummy weights except for the real-weight sparse-locality traces.

None of these limitations overturns the measured comparison on the proxy. They do restrict how confidently an operator should convert the results into a hardware purchase forecast.

The more durable conclusion is architectural. As LLMs replace dense attention with sparse, windowed, recurrent, and other subquadratic mechanisms, “attention” ceases to describe one memory behavior. Hardware placement policies that continue to treat it as one category risk optimizing for an older model structure.

For operators evaluating heterogeneous inference, the first question should therefore come before accelerator selection: Which parts of this model still own state that grows with context?

That answer determines where expensive DRAM capacity is unavoidable—and where SRAM specialization can begin to pay.

Cognaptus: Automate the Present, Incubate the Future.