Farther Apart, Same Context

TL;DR for operators

A transformer can perform well when the information it needs appears at familiar positions and still fail when the same dependency is moved farther away inside an unchanged context window.

Nevermann and Gros isolate this problem with a fixed-context test of distance generalization.¹ In their main 8-layer, 8-head, hidden-size-512 configuration, a transformer with no explicit positional encoding, or NoPE, generalizes better to unseen dependency distances than RoPE or ALiBi. But the result does not justify removing positional encoding by default: smaller NoPE models become fragile, while explicit positional schemes remain more stable.

Two other common scaling assumptions also weaken under the test. Showing the model more training distances improves absolute out-of-distribution accuracy, but the gains are sub-linear relative to the additional coverage. Training related tasks together can help at some distances and hurt at others.

For teams evaluating retrieval, memory, or long-context architectures, this suggests adding fixed-context distance stress tests alongside conventional accuracy measures. Architecture choice, distance coverage, model capacity, and multitask composition should be evaluated together rather than optimized independently.

A model can know the task and still fail when the dependency moves

Suppose a system must recover information that appeared earlier in a sequence. It succeeds reliably when the relevant information is 10 or 20 tokens away. Move the same dependency farther back, without increasing the total context window, and performance may deteriorate.

A conventional length-generalization test makes the diagnosis harder because several things change at once. The sequence gets longer, token positions move into regions the model may not have seen, and the dependency itself may span a larger distance. A failure does not tell you which change mattered.

The paper’s first contribution is therefore diagnostic rather than architectural. It keeps the context length fixed and varies the separation between the source information and the point where it must be recalled. The authors call this distance generalization.

That distinction gives architecture teams a narrower question: can the model execute a familiar dependency when the distance between the relevant tokens is unfamiliar?

Explicit position information does not guarantee better extrapolation

The paper tests causal decoder-only transformers on synthetic delay-copy tasks. The main configuration has eight layers, eight attention heads, hidden size 512, and a context length of 256. Models are trained from scratch and compared under three positional treatments: RoPE, ALiBi, and NoPE.

All three can perform well on distances represented during training. Their behavior separates more clearly outside the trained range.

In the main configuration, NoPE produces the strongest out-of-distribution distance generalization, while RoPE performs worst. This matters because the intuitive expectation runs in the other direction: if the model needs to distinguish where information occurred, explicitly encoding position sounds like an advantage.

The experiment instead shows that an explicit positional scheme can also shape what the model learns about distances in ways that extrapolate poorly. The authors’ proposed mechanism is that a sufficiently capable causal transformer can infer useful relative positional structure from the sequence itself. Under NoPE, it may therefore avoid some encoding-specific positional behavior that becomes brittle beyond the trained distances.

That mechanism remains an interpretation, not a demonstrated circuit explanation. The paper does not conduct the attention- or circuit-level analysis required to establish exactly how NoPE resolves distance.

The model-size ablation changes the architectural conclusion

Stopping at the main result would produce the wrong operational rule.

Appendix B varies model dimension and attention-head count while holding depth fixed. As capacity falls, NoPE becomes less reliable. Small NoPE transformers can fail where RoPE and ALiBi remain stable and can outperform it.

This ablation is best read as a robustness test of the main architectural conclusion. It does not overturn the main experiment; it identifies a condition under which that result stops holding.

The resulting decision rule is more constrained: NoPE can generalize strongly without explicit positional encoding when the model has enough capacity to learn the required positional relationships implicitly.

For teams considering a smaller model because of latency, memory, or deployment constraints, positional encoding and model size therefore cannot be treated as separate tuning decisions. An architectural simplification that survives at one scale may fail after compression.

More distance diversity helps, but each increment buys less

The authors next expand the set of source-recall distances represented during training. Absolute out-of-distribution performance improves as the training range becomes more diverse.

The paper also measures relative generalization as

where $P_{\rm out}$ accumulates accuracy over unseen distances and $P_{\rm in}$ accumulates accuracy over trained distances.

As training covers more distances, $P_{\rm out}$ increases, but sub-linearly relative to the expansion of in-distribution support. The relative generalization ratio therefore falls.

Operationally, “add more diversity” is incomplete guidance. More distance coverage has value, but the experiment provides no basis for expecting proportional returns. The appropriate quantity to measure is the marginal improvement on distances that remain outside the training support.

This is particularly relevant when generating or curating training examples across many context configurations carries a material compute or data-production cost.

Related tasks can transfer knowledge—or interfere with each other

The paper also trains full and selective delay-copy tasks jointly.

The results do not support a monotonic transfer story. With RoPE, the basic joint-training setup produces mostly destructive interference relative to single-task training. NoPE and ALiBi often benefit from transfer at moderately unseen distances but can deteriorate farther outside the trained region.

When the two tasks are trained on more widely separated distance ranges, transfer becomes asymmetric. Knowledge from one task can improve the other at some distant ranges while degrading performance around other regions.

These experiments are not a generic indictment of multitask learning. Their narrower result is more useful: task relatedness is insufficient evidence that joint training will improve out-of-distribution behavior. The sign of transfer depends on the positional scheme, the task, and where the evaluated distance lies relative to each task’s training support.

For a shared model serving multiple retrieval or memory behaviors, aggregate validation can hide precisely this kind of localized interference. Distance-conditioned evaluation can expose it.

The practical value is better diagnosis, not a new default architecture

Cognaptus’ operational inference from the paper is a testing recommendation.

When a product depends on recovering, relating, or transforming information across varying separations, evaluate the model while holding total context length constant and moving the dependency itself. Measure at least three things separately:

The affected users are architecture and model-training teams. The decision is whether an architecture, data expansion, or multitask configuration is robust enough for deployment. The condition is a workload where dependency distances vary materially. The boundary is that this paper establishes the diagnostic behavior only in controlled synthetic experiments.

The paper does not establish how production LLMs behave

Several limitations materially constrain application.

The models are trained from scratch on synthetic delay-copy tasks, not natural-language corpora or pretrained LLMs. Evaluation uses teacher-forced next-token accuracy rather than free-running generation, so autoregressive error compounding is outside the experiment. Main curves select the best-performing model among five runs, although the appendix reports relatively minor run-to-run differences.

Joint-task datasets also contain fewer examples of each individual task than their single-task counterparts because task-frequency parameters are unchanged. That leaves an exposure-count confound, even though the authors argue that the distance-specific pattern of degradation is inconsistent with a simple shortage of examples.

Most importantly, the paper demonstrates that positional encoding, capacity, training diversity, and transfer interact under a clean experimental setting. It does not establish that NoPE will outperform RoPE in a pretrained production model.

The stronger conclusion is methodological: when the business requirement depends on using information across unfamiliar separations, test the separation directly. A long context window and strong accuracy on familiar layouts are not evidence that the model will preserve the same behavior when the dependency moves.

Cognaptus: Automate the Present, Incubate the Future.