Linear-Attention

TL;DR for operators Running a sequence model in linear time does not settle the memory-design problem. A recurrent model still has to decide what shape its compressed history should take. RunningTensor1 replaces the matrix memory used by conventional linear attention with a higher-order tensor state. In the paper’s clearest matched ablation, an order-3 tensor state with the same ConcOnlyC projection averages 98.5 F1 on multi-query associative recall, versus 30.6 F1 for the matrix-state version. That gap makes memory geometry a serious architecture variable for workloads dominated by recovering many earlier associations. ...

Latency has a habit of hiding inside words that sound efficient. “Constant decoding cost” is one of those phrases. It suggests a clean engineering promise: linear attention avoids the context-length explosion of softmax attention, so long-context inference should become simpler, cheaper, and less melodramatic. Very nice. The GPU accountants, however, have not retired. ...