Two Efficient Attentions, One Denominator Problem
TL;DR for operators Long-context teams face a familiar choice: compute fewer token-to-token interactions, compress global interaction structure, or accept the quadratic cost of exact attention. The harder design question is what happens when two cheaper approximations are combined. If each branch normalizes its own output over a different effective support, simply adding or gating them can give the branches incompatible scales. ...