Zero Is Not Nothing: When an Inactive Token Still Changes the Model

TL;DR for operators

If adding an unused token whose hidden state is exactly zero changes existing tokens or predictions, that token is not actually inactive. Standard softmax attention can produce exactly this behavior: a visible zero source still contributes normalization mass even when its value contribution is zero.

OAttention¹ addresses this at the attention operator by giving each token a norm-derived presence coefficient. The coefficient gates what a receiver emits and, separately, how much support a source contributes to the attention numerator and denominator. Exact zero then means zero participation under the declared operator contract.

The harder result is architectural. The paper’s O-Closure framework requires the same zero-extension property across normalization, metadata injection, feed-forward branches, and other token-coupled computation. In a two-block experiment, an OAttention-only model still let an inserted zero state grow to a maximum magnitude of 7.983; the OTransformer path kept it exactly at zero.

For platform teams managing optional token slots, sparse latent positions, padding-like states, or disabled entities, the practical lesson is to test inactivity across the entire processing path. The evidence supports that structural contract in controlled settings. It does not show that arbitrary pretrained models can be retrofitted without performance loss, or that business-data missingness should be encoded as a zero vector.

A zero source can still change everyone else’s attention

Suppose a model has room for an optional token. When the slot is unused, its hidden representation is set to the zero vector. A reasonable operational requirement is simple: inserting that unused slot should leave every existing token unchanged and should leave the new slot at zero.

Ordinary softmax attention does not provide that guarantee.

For a zero source, the projected value may contribute nothing useful, but the source can still receive a finite attention score. In the simplest zero-score case, it contributes $\exp(0)=1$ to the softmax partition. Existing attention weights are consequently renormalized. The token carries no represented content, yet its presence can alter the computation.

An attention mask answers a different question: is this receiver allowed to attend to this source? It does not make the source representation intrinsically non-participating when that relation remains visible.

The paper reframes the requirement as participation rather than masking.

One coefficient controls both emission and support

The canonical construction begins with a smooth presence coefficient derived from a token’s hidden carrier:

At the exact origin, $p_i=0$. For a fixed nonzero carrier, presence approaches one as $\tau$ becomes small.

OAttention gives this same coefficient two jobs. Receiver presence $p_i$ gates the aggregate emitted at position $i$. Source presence $p_j$ also enters both the attention numerator and its normalization partition.

That second role is essential. Merely zeroing a null receiver’s output would stop that position from emitting an update, but a zero source could still alter the denominator seen by other receivers. By removing its support mass as well as its value contribution, OAttention makes an exact-zero source disappear from the effective attention support.

Under the paper’s stated boundary and metadata assumptions, the resulting operator satisfies null-receiver, null-source-insertion, self-attention-insertion, and empty-support contracts while retaining ordinary dot-product scores, external visibility masks, exponential competition, and value aggregation for active states.

The finite-precision operator study is principally an implementation check of those algebraic claims. Against direct evaluation, maximum output or attention-weight error was at most $8.94\times10^{-8}$. After zero-source insertion, old-output error was at most $4.47\times10^{-8}$ and old-weight error at most $5.96\times10^{-8}$; the inserted receiver output and source weight remained exactly zero.

That is strong evidence for the declared operator. It is not yet evidence for an arbitrary Transformer.

Attention closure does not survive an ordinary Transformer by itself

The paper formalizes the broader requirement as O-Closure: append an exact-zero token, and all existing outputs must remain unchanged while the appended coordinate returns exact zero.

This changes the audit target. A model can use null-consistent attention and still fail the overall property because other components generate or respond to nonzero states.

Additive positional or feature injection can write information into a zero slot. Affine normalization can introduce offsets. A biased feed-forward network can generate a nonzero update from zero input. Statistics computed across tokens can also change when a new token is included unless that token contributes zero statistical support.

The paper’s two-block ablation isolates this distinction:

The first three arms share exact initialization, making this an ablation of compositional closure rather than a broad performance comparison.

This is the paper’s most operationally consequential result. A team cannot establish “inactive token” semantics by inspecting the attention implementation alone. Every path that can modify a token or compute shared support has to satisfy the same participation rule.

Exact closure and pretrained compatibility are separate tests

Structural correctness does not imply that a retrofit leaves active representations untouched.

The paper tests this distinction on a frozen TabPFN v3 regressor. At $\tau=10^{-8}$, no model arm is fine-tuned. Across 18 dataset-seed cases, OAttention changes mean RMSE by +0.088%, while the broader Full-O retrofit changes it by +0.177%. Mean $R^2$ changes are correspondingly small.

These are compatibility observations from one host at a deliberately near-identity scale, not a non-inferiority guarantee.

The appendix stress test explains why calibration deserves its own acceptance criterion. At $\tau=1$, OAttention changes mean RMSE by +0.138%, but Full-O changes it by +47.050% across the same 18 cells. The algebraic zero-extension principle has not failed. Instead, finite presence gates are attenuating active states, and repeated gating across additional branches compounds the perturbation.

For a retrofit decision, the two questions should therefore be tested independently:

1. Does zero insertion satisfy the intended structural invariance contract?
2. Does the chosen presence scale and gate placement preserve acceptable behavior on active inputs?

Passing the first does not answer the second.

Treat NULL invariance as an architecture contract, not a missing-data policy

Cognaptus’ operational inference is strongest for systems where inactive computational slots have an explicit structural meaning: dynamically allocated token positions, optional feature tokens, sparse latent slots, padding-like states, or entities that should be disabled without affecting active context.

For those systems, zero insertion becomes a useful architecture test. Insert one or several exact-zero states at supported locations, then check both the new coordinates and all pre-existing outputs through attention, normalization, injection, feed-forward processing, routing, pooling, caches, and readouts that are supposed to respect inactivity.

The boundary is semantic as well as architectural. The paper does not show that representation norm naturally encodes absence, nor that training learns to place inactive concepts at the origin. An observed value of zero, a NaN, an informative missing feature, a learned mask token, and a structural NULL can require different behavior.

Whole-model closure also remains conditional on compatible metadata. Renumbered absolute positions, sequence-level statistics, token merging, caches, routing, compression, or non-neutral readouts can break the zero-extension law even when the local O-components are correct.

OAttention therefore contributes a precise operator contract, while O-Closure supplies the more demanding systems question: when the model says a state is inactive, which computations are still allowed to notice that it exists?

For architecture auditing, that is a sharper specification than “we masked the token.” The paper shows how to enforce it in controlled token-processing paths and, just as importantly, demonstrates where the guarantee stops.

Cognaptus: Automate the Present, Incubate the Future.