The 172× Speedup Has an Evidence Budget

TL;DR for operators

Prefilling-dLLM v2¹ demonstrates that a diffusion language model can learn to select distant context efficiently rather than repeatedly attending to every token. At 128K context, its per-generated-token decoding latency is approximately 172× lower than full dense attention, and it leads the tested sparse methods on three selected retrieval tasks. Yet its LongBench average is 25.4, below sliding-window attention at 30.9 and full dense attention at 44.3.

For operators building document-processing systems, the practical implication is workload-specific: learned routing can be attractive when answers depend on a small number of distant passages, but less reliable when evidence is distributed across many sections. The speedup is not an end-to-end measurement, and the retrieval results do not establish general document comprehension. Architecture selection needs separate measurements of answer accuracy, evidence coverage, prefill cost, and decoding cost.

A faster answer can still miss required evidence

Consider an AI system reviewing a lengthy contract, research report, or operational record. Some questions require locating one specific clause or fact. Others require combining information from several sections, including qualifications and exceptions that may be far apart.

Examining every token repeatedly is expensive. Restricting attention to a smaller set of apparently relevant passages reduces computation, provided those passages contain everything needed for the answer.

The difficulty is that the system must decide what to read before it knows what information will prove necessary. An omitted passage cannot contribute directly to the result, even when it remains present in the original input.

A recent computational study makes this tradeoff unusually visible.

At 128K context, Prefilling-dLLM v2 reduces per-generated-token decoding time from 1,190 milliseconds to 6.91 milliseconds relative to full dense attention. But on LongBench-v1 at 16K, its average score trails even simple sliding-window attention.

These measurements come from different evaluations, not a matched speed-quality experiment. Together, they reveal a specific problem: efficiently locating selected evidence and retaining enough information for broader document analysis are different capabilities.

How learned routing controls what the model can read

A diffusion language model generates text by repeatedly refining masked positions, rather than strictly predicting one next token at a time. Its bidirectional attention can use surrounding information from both directions, but repeatedly processing long sequences makes dense attention costly.

Prefilling-dLLM v2 addresses this through sparse attention learned during continued training, rather than imposing a selection rule only at inference.

The architecture divides context into chunks of 64 positions. Each contains 63 content tokens and one appended routing position holding the model’s existing mask token.

A binary indicator and learned role offset distinguish this routing mask from ordinary masks used for generation. The routing position produces an attention-weighted summary of its chunk’s content keys; it is not itself an answer token or an ordinary projected routing-token key.

Content queries then rank remote chunks using two signals: similarity to the chunk summary and the entropy of the summary’s pooling weights. A rank-64 low-rank residual adapts queries for routing without replacing the pretrained query projection.

Every fourth transformer layer selects the top 32 remote chunks, retaining a bidirectional local window. The remaining 21 of 28 layers use sliding-window attention. The nominal attention budget is approximately 2,560 keys per query.

After selection, attention mass is allocated jointly between the chosen remote chunks and the local window, then distributed among content tokens within each selected chunk.

This design has a consequential constraint. Hard top-K selection determines which chunks are visible, and membership itself is not differentiable. Training can adjust the routing mechanism through attention masses assigned to selected chunks, but evidence excluded by the selection cannot be directly read through that route layer.

The model can therefore become better at choosing passages without acquiring unrestricted access to the full document.

Retrieval improves, particularly when the target is concentrated

The main benchmark comparison uses Dream-v0-Base-7B, with a shared dense context-extension checkpoint, approximately matched sparse-attention budgets, and a common Fast-dLLM decoder.

The evaluated sparse alternatives include bidirectional adaptations of Native Sparse Attention (NSA), DSA, and sliding-window attention (SWA).

On three custom-generated RULER task families, Prefilling-dLLM v2 achieves the highest sparse-model average at every tested context length from 16K to 128K.

With YaRN scaled to evaluation length, its three-task means are 53.8, 54.7, 52.3, and 47.7. NSA, its strongest sparse competitor, obtains 42.7, 43.3, 39.0, and 40.3 respectively (Section 4.2, Table 1).

The 128K breakdown is more revealing:

Scores use length-scaled YaRN. Full dense is a separate reference architecture, not part of the sparse-model ranking.

The learned router is strong at finding a single distant target. Its advantage over NSA persists on more demanding tasks, but absolute accuracy drops sharply as the task requires retrieving or tracking multiple pieces of information.

The dense reference remains substantially stronger on these evidence-intensive tasks. At 16K, for example, full dense scores 100.0 on multi-key retrieval and 91.7 on variable tracking, compared with 42.0 and 19.3 for Prefilling-dLLM v2.

Thus, the main comparison establishes a relative improvement among the tested sparse methods. It does not establish equivalence to unrestricted attention.

The mechanism tests reveal what actually contributes

The paper includes component ablations and diagnostic interventions that go beyond comparing final benchmark scores.

The most consequential component ablation removes the low-rank query residual. Under fixed YaRN at 128K, the three-task RULER mean falls from 46.5 to 9.1. Removing the routing role offset lowers it to 37.3, while removing the entropy contribution lowers it to 27.3 (Appendix B, Table 6).

These tests indicate that adapting queries and distinguishing routing positions are central to the reported implementation.

Routing-token substitutions provide another check. At 128K with fixed YaRN, the native mask achieves a mean of 46.5, versus 22.8 with an EOS token and 30.7 with a newly learned token (Appendix B, Table 7). The alternatives were trained with their respective tokens, so this is not merely an inference-time replacement test.

Diagnostic interventions examine whether the summaries actually guide retrieval. Shuffling chunk summaries disrupts their alignment with content and sharply reduces single-needle accuracy. When the target is relocated to another chunk, native-mask routing retains 68% accuracy on a 100-example probe, while the EOS and new-token variants both score zero (Section 4.3, Figures 4–5).

These are mechanism diagnostics, not independent demonstrations of broad document comprehension.

One ablation also limits the claim that every learned component is preferable to a simpler alternative. Replacing attention-weighted summaries with mean pooling improves the three-task average at 16K, 32K, and 64K under fixed YaRN. It performs worse at 128K, where its mean falls to 35.9 versus 46.5.

The combined result supports learned routing and native-mask representations, while leaving the optimal summary construction dependent on evaluation conditions.

The 172× result measures decoding, not the whole application

The latency experiment makes the efficiency benefit concrete, but its scope matters.

At 128K context, under the common Fast-dLLM decoder:

Section 4.4, Figure 6. Measurements use one synthetic sequence per length with 128 masked answer tokens; reported timings are medians of ten trials.

Prefilling and decoding have different computational profiles, so the much larger decoding improvement cannot be applied to total request latency.

The reported latency comparison also excludes NSA, DSA, and SWA. Consequently, the study does not establish that Prefilling-dLLM v2 is the fastest sparse architecture under matched serving conditions.

For infrastructure teams, the economic interpretation depends on how often long prefixes are processed, how much text is generated, and which stage dominates actual workload time.

LongBench exposes the cost of limited document coverage

The most consequential counterevidence comes from LongBench-v1 at the 16K training length.

Prefilling-dLLM v2 averages 25.4, below NSA at 26.6, DSA at 25.8, and SWA at 30.9. Full dense reaches 44.3 (Section 4.2, Table 2).

This reverses the ranking observed on the selected RULER tasks.

The architecture provides a plausible explanation. Selecting 32 remote chunks can work well when a small set of passages contains the answer. Tasks involving summarization, multiple supporting facts, or document-wide relationships may require information from a larger and more dispersed set of chunks.

The LongBench results are consistent with that information-coverage limitation. They do not isolate missing chunks as the cause of every individual task failure.

For a retrieval-oriented application that repeatedly extracts a specific fact from long records, the method is a credible candidate for further testing. For legal analysis, financial-report synthesis, or compliance review requiring multiple supporting passages, the fixed selection budget creates a measurable omission risk.

This is a Cognaptus operational inference, not a deployment outcome measured by the authors.

What remains unproven for deployment

The evaluation establishes a useful architectural result under relatively narrow conditions.

The model is trained at 16K; results at 32K through 128K involve length extrapolation. The RULER evaluation covers only three of 13 task families, uses the authors’ generator, and overlaps with task families included in continued training. Answer lengths are provided during RULER evaluation.

Each benchmark configuration has one continued-training run, without reported multi-seed variation or uncertainty intervals. The study covers one 7B diffusion backbone, while latency testing uses synthetic sequences rather than heterogeneous production requests.

Those boundaries suggest an application-specific selection process. A deployment team should separately measure factual retrieval, multi-evidence completeness, document synthesis quality, prefill latency, and decoding latency, while comparing routed attention with dense and simpler sparse alternatives under the same workload.

Prefilling-dLLM v2 shows that learned chunk routing can make selected distant information substantially cheaper to access. Its remaining limitation is more fundamental to the application: when a correct answer depends on information outside the selected chunks, faster access to the chosen evidence does not compensate for what was omitted.

The deployment criterion is therefore not retrieval speed alone, but how reliably the architecture preserves the evidence required for the decision being made.

Cognaptus: Automate the Present, Incubate the Future.