Memory Geometry Is a Model Decision
TL;DR for operators Running a sequence model in linear time does not settle the memory-design problem. A recurrent model still has to decide what shape its compressed history should take. RunningTensor1 replaces the matrix memory used by conventional linear attention with a higher-order tensor state. In the paper’s clearest matched ablation, an order-3 tensor state with the same ConcOnlyC projection averages 98.5 F1 on multi-query associative recall, versus 30.6 F1 for the matrix-state version. That gap makes memory geometry a serious architecture variable for workloads dominated by recovering many earlier associations. ...