Farther Apart, Same Context
TL;DR for operators A transformer can perform well when the information it needs appears at familiar positions and still fail when the same dependency is moved farther away inside an unchanged context window. Nevermann and Gros isolate this problem with a fixed-context test of distance generalization.1 In their main 8-layer, 8-head, hidden-size-512 configuration, a transformer with no explicit positional encoding, or NoPE, generalizes better to unseen dependency distances than RoPE or ALiBi. But the result does not justify removing positional encoding by default: smaller NoPE models become fragile, while explicit positional schemes remain more stable. ...