TL;DR for operators

Running a sequence model in linear time does not settle the memory-design problem. A recurrent model still has to decide what shape its compressed history should take.

RunningTensor1 replaces the matrix memory used by conventional linear attention with a higher-order tensor state. In the paper’s clearest matched ablation, an order-3 tensor state with the same ConcOnlyC projection averages 98.5 F1 on multi-query associative recall, versus 30.6 F1 for the matrix-state version. That gap makes memory geometry a serious architecture variable for workloads dominated by recovering many earlier associations.

The practical constraint is substantial. Sequence-length scaling remains linear in $T$, but state size grows as $\mathcal{O}(W^o)$ with tensor order $o$; the implemented order-3 case therefore has $\mathcal{O}(W^3)$ state size. Only order 3 is tested, and the main reported evidence is a synthetic retrieval benchmark. Architecture teams should read this as a mechanism worth testing when associative memory is the bottleneck, not as evidence that higher-order recurrent state is already a general replacement for attention or matrix-state alternatives.

Compressing history is only half the architecture decision

Long-context systems face a familiar constraint. Full attention can preserve direct access to earlier tokens, but its pairwise attention structure becomes expensive as sequences grow. Recurrent and state-space alternatives attack that problem by summarizing the past into an internal state that can be updated as new tokens arrive.

The usual engineering question is how efficiently that state can be updated.

RunningTensor adds another question: what geometry should the state have?

A matrix is only one possible memory object. If the task requires a model to preserve many associations and later recover the right value from several interacting cues, a second-order matrix may compress those relationships too aggressively. The paper tests whether adding another state dimension can preserve more useful structure without giving up recurrent inference.

The most revealing experiment is not the headline comparison against unrelated architectures. It is the controlled state-geometry ablation. With ConcOnlyC held fixed, the tensor-state variant averages 98.5 F1, while the matrix-state counterpart averages 30.6.

That experiment does not prove tensors are universally better memory. It does show that, in this retrieval setting, changing the state object itself can dominate performance even when sequence complexity remains in the same broad recurrent family.

RunningTensor adds a second address to recurrent memory

Ordinary causal linear attention can be written as a recurrent matrix memory. Each token writes a value associated with a key, and later queries retrieve information from the accumulated state.

RunningTensor generalizes that construction. In its demonstrated order-3 form, the recurrent state is

$$ \mathcal{S}_t = \mathcal{S}_{t-1} + \mathbf{v}_t \otimes \mathbf{k}_t \otimes \mathbf{r}_t, $$

and the model reads from it using two query vectors:

$$ \mathbf{o}_t = \langle \mathcal{S}_t, \mathbf{q}_t, \mathbf{u}_t \rangle. $$

Conceptually, each value is stored against two address axes instead of one. In the parallel formulation, the contribution of an earlier value is weighted by the product of two query-key similarities:

$$ \mathbf{O} = \left( (\mathbf{Q}\mathbf{K}^{\top}) \odot (\mathbf{U}\mathbf{R}^{\top}) \odot \mathbf{M} \right) \mathbf{V}. $$

This is useful because the proposal is not an unrelated recurrent mechanism. If the extra state mode is collapsed to dimension one and its stored and query values are fixed to one, the formulation reduces to causal linear attention.

That containment result changes how to interpret the architecture: RunningTensor expands the memory geometry of linear attention rather than replacing its basic recurrent logic.

The matched ablation is stronger evidence than the perfect score

The paper evaluates associative memory using Multi-Query Associative Recall, a synthetic task that places multiple associations inside a sequence and later asks the model to recover them.

At $d_{\text{model}}=128$, RunningTensor reports 100.0 F1 at sequence lengths 1,024, 2,048, and 4,096. Full attention is also effectively perfect across those lengths. DeltaNet averages 97.7, while Gated DeltaNet averages 61.2; Mamba-2 and GLA perform poorly in the reported configuration.

RunningTensor also uses 176k non-embedding parameters per layer in this table, versus 934k for attention and 249k for DeltaNet. Its reported mean validation forward-pass latency is 311.9 ms per step: faster than attention at 333.5 ms and DeltaNet at 355.7 ms, but slower than Mamba-2, Gated DeltaNet, and GLA.

Those comparisons are informative, but the tensor-versus-matrix ablation is more diagnostic because it changes the memory geometry inside a closely matched configuration.

Configuration Average MQAR F1
Tensor state + ConcOnlyC 98.5
Matrix state + ConcOnlyC 30.6
Full tensor configuration 100.0

The full model adds a tensor delta update, forgetting, query-skip gates, and temporal filtering of the output gate. Removing query-skip gating still leaves the reported average at 100.0, while removing the delta rule but retaining other additions lowers it to 98.6.

So the benchmark strongly implicates the tensor state itself. It provides less support for claiming that every auxiliary component is necessary.

The delta rule turns storage into correction

A recurrent memory cannot only append information. It also needs a way to revise what is already associated with an address.

RunningTensor lifts the delta-rule idea into its tensor state. Before writing a new value, the model asks what the decayed state already predicts at the current pair of addresses. It then writes a gated residual correction:

$$ \mathcal{S}_t = \alpha_t\mathcal{S}_{t-1} + \beta_t \left( \mathbf{v}_t - \langle \alpha_t\mathcal{S}_{t-1}, \hat{\mathbf{k}}_t, \hat{\mathbf{r}}_t \rangle \right) \otimes \hat{\mathbf{k}}_t \otimes \hat{\mathbf{r}}_t. $$

The architecture also introduces data-dependent forgetting and query-skip gates, while ConcOnlyC replaces a dense fused projection with channel tiling and a tied depthwise convolution.

For implementation teams, these additions should be separated conceptually. The tensor state changes memory capacity and addressing geometry. The delta rule changes how existing associations are corrected. ConcOnlyC targets projection parameter cost. The MQAR ablations suggest they contribute differently rather than forming one indivisible package.

Linear in sequence length does not mean cheap memory

The central deployment misconception is easy to make.

RunningTensor preserves linear scaling in sequence length:

$$ \text{sequence-length scaling}=\mathcal{O}(T). $$

But its recurrent state scales as

$$ \text{state size}=\mathcal{O}(W^o). $$

For the demonstrated order-3 model, that means cubic scaling in state width.

The design therefore exchanges one resource problem for another. It avoids maintaining a quadratic attention structure over the whole sequence, but spends more capacity inside the persistent recurrent state.

For a streaming retrieval system, that trade may be attractive because state cost does not grow with the number of processed tokens. For a wide model, however, the tensor state can become expensive even though the sequence-length curve remains linear.

The paper does not provide a broad scaling study across tensor orders, state widths, accelerators, or production workloads. Deployment economics therefore cannot be inferred from the asymptotic sequence-length result alone.

Test higher-order memory where retrieval failure is measurable

The business case is clearest for systems that process long streams and repeatedly need to recover earlier associations: persistent agent context, event streams, document processing, or other workloads where compressed recurrent memory is attractive and retrieval quality can be measured directly.

Cognaptus inference is that architecture teams evaluating linear-time sequence models should add state geometry to their design matrix. A comparison that asks only whether a recurrent alternative is faster than attention can miss whether its compressed state preserves the associations the product actually needs.

The appropriate evaluation is workload-specific: reproduce the retrieval pattern, vary state geometry under comparable conditions, then measure retrieval quality together with persistent-state memory, throughput, and latency.

That recommendation is narrower than choosing RunningTensor outright. The reported body evaluates only order 3, relies primarily on synthetic MQAR, reports the best validation F1 across three seeds and three learning rates rather than uncertainty across runs, and does not provide numerical downstream results for the broader language-understanding and non-synthetic retrieval improvements described elsewhere in the paper.

Memory geometry deserves its own benchmark

RunningTensor’s strongest contribution is not simply another linear-time architecture. It demonstrates that the object used to compress history can itself determine whether earlier associations remain recoverable.

The 98.5-versus-30.6 matched ablation makes that point unusually visible. The next question for deployment teams is empirical: does the same advantage survive the state-size cost on the retrieval patterns, widths, and hardware that matter to their product?

Until that is tested, higher-order recurrent memory is best treated as a promising design option with a clear mechanism and a narrow but striking demonstration—not a settled default.

Cognaptus: Automate the Present, Incubate the Future.

  1. Luca Herranz-Celotti and Vincent Guigue (2026). RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States. arXiv:2609.12814. https://arxiv.org/abs/2609.12814 ↩︎