Sparse Attention, Decisive Lookup

TL;DR for operators

A hybrid model can contain relatively little attention and still depend on that attention for its decisive retrieval step.

Across 23 measured depth profiles from twelve checkpoints, the intermediate clue needed for a two-hop answer becomes most causally usable immediately before the layer that performs the largest conversion toward the final answer. That conversion occurs in an attention-containing layer in every measured profile.

The paper’s intervention experiments suggest a more specific division of labor. In several sequential hybrids, recurrent layers carry or rebuild the intermediate key, while a small set of attention heads reads the relevant contextual fact. In three targeted models, eight selected heads reproduced roughly 69–100% of the measured key effect.

There is an additional mechanism: in four of seven tested hybrids, recurrent memory after the final attention layer could retrieve a stored binding using the computed key. But that behavior is neither universal across Mamba-2 hybrids nor stable across closely related checkpoints and task variants.

For evaluation, model selection, and regression testing, architecture labels are therefore an unreliable proxy for retrieval behavior. Test the mechanism you depend on. The boundary is equally important: these findings come from controlled generated fact lists, not unrestricted production conversations.

The decisive retrieval step still lands on attention

A two-step question creates a simple computational requirement. The model must first derive an intermediate clue, then use that clue to locate the fact that determines the answer.

Reducing the number of attention layers might suggest that a hybrid model has transferred most of this retrieval work into recurrent state. The Key Handoff: Retrieval in Hybrid Language Models by Kaan Kale, Oguzhan Baser, and Sriram Vishwanath tests that assumption directly.¹

The striking result is consistent across all 23 depth profiles the authors measure. The causal usefulness of the intermediate key peaks at the input to the layer that produces the largest increase in the donor-answer effect. Across that same layer, the key effect falls and the answer effect rises. The largest such conversion step occurs in an attention-containing layer in every tested model.

That pattern matters because it distinguishes carrying information from using it for retrieval. A recurrent layer may preserve a clue for many layers without being the component that actually matches that clue to a contextual fact.

In the sequential hybrids, this concentration is substantial. The positive answer-conversion rise is carried by roughly 1.7–2.0 effective attention layers in the late conversion band. For Nemotron-H on the arrow task, about 95% of the answer-conversion rise falls on attention.

Sparse attention, in these experiments, is still strategically placed attention.

The assay separates a key from an answer and a position

Showing where a candidate answer becomes stronger is not enough. An internal state could already contain the completed answer. It could encode where the relevant fact appeared. Or it could carry the intermediate key that another component will later use.

The paper’s main measurement contribution is designed to separate those possibilities.

The authors construct donor and recipient prompts so that three explanations predict different answers: follow the computed key, follow the donor fact’s row position, or simply carry over the donor’s completed answer. They then transplant the donor’s final-token residual state into the recipient at a chosen layer and measure which candidate logits move.

This is stronger than asking whether a probe can decode the key. Decodability establishes that information is present. The intervention asks what information the state can causally make another computation use.

Several controls support the key interpretation. At the resolving-layer input, rebinding produces seven-candidate top-1 rates of 86% for Nemotron-H, 71% for Granite-H, and 67% for Nano. In causal-scrubbing tests, sharing only the key produces roughly 3.2–3.4× gains, while sharing only the donor answer or row produces effects below 1×.

The state is not pure: the experiments also detect a row-position component. But the dominant causal signal under these tests is the computed key, not merely an already-finished answer.

Recurrence carries the key; a few heads read it

Once attention is identified as the resolving step, the natural question becomes what the recurrent layers are doing before it.

Targeted recurrent-output swaps in four sequential hybrids redirect the later lookup toward the transplanted key and reduce the original lookup. That supports a carrier role: recurrent computation transports key-specific information toward a later reader.

The reader itself can be surprisingly concentrated.

At the resolving layers of Nemotron-H, Granite-H, and Nano, the authors select eight attention heads using separate examples and inject their output changes into the recipient. Those eight heads reproduce approximately 100% of the transplant key effect in Nemotron-H, 69–75% in Granite-H, and 93–96% in Nano. Every selected set beats all tested random eight-head sets.

Clamping all heads at the resolving layer removes approximately all of the key effect in Nemotron-H and Granite-H when no later reader is available. Nano retains 36–39%, consistent with additional downstream retrieval capacity.

This is not evidence that every hybrid concentrates retrieval into eight heads. The component experiments cover only a subset of the broader model panel. It does show why reliability testing based only on layer type can miss the relevant dependency: retrieval may depend on a narrow attention pathway embedded inside a largely recurrent architecture.

Some recurrent memories can retrieve after attention ends

The carrier-reader account is not the whole story.

The paper also tests recurrent Mamba-2 memory after the model’s final attention layer. Here the intervention changes what binding is stored in memory while holding the test prompt’s incoming computation fixed. If the recurrent suffix can retrieve from that memory using the test prompt’s computed key, changing the stored binding should redirect the answer.

Four of seven tested hybrids show such an effect.

For Nemotron-H, transposing the relevant binding multiplies the kitchen-to-garden probability ratio by a geometric-mean factor of 2.7 under ordinary execution. When the final attention update is skipped, the memory-provided answer reaches top-1 more often; in Nemotron-H the increase is 22 percentage points.

A separate row-rotation test addresses the obvious alternative explanation. The retrieved answer follows the queried key more strongly than the position where the fact appeared. In Nemotron-H, the key-over-row probability-ratio factor is 1.9×; in Granite-Small it is 1.4×.

That supports a content-addressed memory interpretation: some recurrent states are not merely transporting earlier computation. They can perform another retrieval after attention is no longer available.

But the qualification is central. Some Mamba-2 hybrids show no such read. Nano-9B-v2 shows it while its parent Nano-12B-v2 does not. Nemotron-H shows the behavior for place words but not for the numeric task variant.

The presence of the architectural mechanism does not tell you whether a particular checkpoint will use it for a particular retrieval problem.

Retrieval behavior should be a regression target

For model developers, evaluators, and reliability teams, the paper suggests a diagnostic shift.

Instead of asking whether a model has attention or recurrent memory, ask where a required retrieval operation is actually performed. A depth profile can identify where an intermediate representation becomes causally useful and where it is converted into an answer. Targeted component interventions can then test whether that conversion depends on specific recurrent outputs, heads, or recurrent memories.

That creates several practical uses.

After a model update, a retrieval-sensitive regression suite could test whether the resolving layer has moved, whether the same small head set remains important, or whether a post-attention memory route has appeared or disappeared. During model comparison, two checkpoints with similar architecture labels should not be assumed to expose the same failure modes. And when designing stress tests, teams can target the components that actually execute retrieval rather than treating every layer of the same architectural class as interchangeable.

These are Cognaptus inferences from the paper’s measurement logic, not demonstrated production outcomes. The experiments do not measure deployment latency, serving cost, or downstream business reliability.

Controlled mechanisms are not production guarantees

The evidence is unusually strong for a mechanistic study because it relies on counterfactual interventions, held-out identities, prespecified configurations, matched controls, causal scrubbing, head interventions, memory transpositions, alternative prompt formats, and statistical comparisons.

Its external scope is narrower.

The primary tasks are generated 17-fact lists run in raw completion mode. The strongest component-level experiments involve only three or four targeted hybrids. Cross-model differences cannot be interpreted as causal effects of architecture because the checkpoints also differ in training and other properties. Intervention methods can also place internal states off their natural trajectories, even though the paper includes diagnostics and controls intended to reduce that concern.

The appropriate conclusion is therefore not that hybrid language models universally implement retrieval through one fixed handoff.

It is that architecture diagrams leave out an operationally consequential question: which component actually performs the lookup in this model, on this task?

Here, the answer is often a handoff. Recurrence carries the clue. Attention performs the decisive conversion. And in some models, recurrent memory gets another chance to retrieve after attention is finished.

That mechanism is measurable. For reliability work, measuring it is more defensible than guessing from the label on the architecture.

Cognaptus: Automate the Present, Incubate the Future.