Sparse Attention, Decisive Lookup
TL;DR for operators A hybrid model can contain relatively little attention and still depend on that attention for its decisive retrieval step. Across 23 measured depth profiles from twelve checkpoints, the intermediate clue needed for a two-hop answer becomes most causally usable immediately before the layer that performs the largest conversion toward the final answer. That conversion occurs in an attention-containing layer in every measured profile. ...