Attention Is a Connection Walk, Not Automatically a Laplacian
TL;DR for operators When a model behaves unexpectedly, teams often inspect attention maps to see where information flowed. Those maps show which source tokens were selected and how strongly, but not how the retrieved features were transformed before reaching the destination token. The paper proves that multi-head attention is exactly representable as a scaled, edge-dependent connection walk: token routing is supplied by attention weights, while feature transport is supplied by an attention-gated mixture of the heads’ value-output maps. Two layers can therefore display similar attention patterns while computing materially different transformations. ...