Transformers

A curated guide to textbooks, authors, websites, and papers for readers who want to study transformer internals, attention math, fine-tuning, GPU optimization, and benchmarking in more depth.

TL;DR for operators If adding an unused token whose hidden state is exactly zero changes existing tokens or predictions, that token is not actually inactive. Standard softmax attention can produce exactly this behavior: a visible zero source still contributes normalization mass even when its value contribution is zero. OAttention1 addresses this at the attention operator by giving each token a norm-derived presence coefficient. The coefficient gates what a receiver emits and, separately, how much support a source contributes to the attention numerator and denominator. Exact zero then means zero participation under the declared operator contract. ...

TL;DR for operators Long context becomes expensive for a specific architectural reason: standard self-attention allows every position to interact with every other position. In the usual formulation, that means $O(n^2 d)$ computation and $O(n^2)$ storage for attention-related matrices as sequence length $n$ grows. Hasi Hays’s mathematical monograph on attention1 is useful because it connects that deployment constraint directly to the mechanism that makes attention powerful in the first place. ...

TL;DR for operators A team trying to move a transformer onto a phone, camera, robot, or embedded device has several levers: reduce the model, lower numerical precision, change the runtime, or move to a better accelerator. Hema Hariharan Samson’s survey of lightweight transformers1 suggests these choices cannot be evaluated independently. In the paper’s analyzed batch-size-1 setting, smaller workloads can become limited by how quickly the device moves model data rather than by raw arithmetic capacity; the paper reports roughly 60–75% hardware utilization in a favorable 15–40M-parameter range. Its broader comparisons similarly show that INT8 execution, operator fusion, optimized runtimes, and specialized accelerators change realized latency by very different amounts across devices. For an operator, the practical target is not the smallest model. It is the smallest accuracy loss that satisfies the product’s measured latency and energy budget on the actual deployment hardware. ...

TL;DR for operators A player-centric sports system has to answer three questions together: what happened, when did it happen, and who did it? This paper suggests that the representation used to preserve the “who” can materially affect the other two. Wang, Yang, and Wang’s Entity-Aware Sequence Transduction model, ME-DST, keeps separate player-role slots through sequence encoding rather than merging all players into a single frame representation.1 On the FOOTPASS validation set, it reaches 0.778 Micro F1, compared with 0.675 for the strongest official TAAD+DST baseline. ...

TL;DR for operators Long-context teams face a familiar choice: compute fewer token-to-token interactions, compress global interaction structure, or accept the quadratic cost of exact attention. The harder design question is what happens when two cheaper approximations are combined. If each branch normalizes its own output over a different effective support, simply adding or gating them can give the branches incompatible scales. ...

TL;DR for operators A bike-sharing operator has to reposition bikes before the next demand surge, even when one station is influenced by nearby docks and by commuter corridors elsewhere in the city. In the reported New York and Chicago tests, STAGformer records the lowest error in seven of eight city-month RMSE and MAE cells; GAT retains the lowest Chicago September MAE. ...

TL;DR for operators When a model behaves unexpectedly, teams often inspect attention maps to see where information flowed. Those maps show which source tokens were selected and how strongly, but not how the retrieved features were transformed before reaching the destination token. The paper proves that multi-head attention is exactly representable as a scaled, edge-dependent connection walk: token routing is supplied by attention weights, while feature transport is supplied by an attention-gated mixture of the heads’ value-output maps. Two layers can therefore display similar attention patterns while computing materially different transformations. ...

A document lands in an intake queue. It might be an invoice, a memo, a form, a résumé, or one of those corporate artifacts whose layout says more than the words do. Someone wants the system to classify it instantly, because every downstream workflow—routing, extraction, compliance, archiving—depends on that first label. The fashionable answer is: send it to a large language model. Extract the text, paste it into a prompt, ask for one label, and let the machine be clever. This is attractive because it feels general. It is also how many automation projects quietly turn a visual problem into a text problem, then act surprised when the system starts calling file folders “proposals” because the word proposal appeared somewhere on the page. ...

Weights are expensive twice. First, they cost money to train. Then they cost money every time a model is served, copied, quantized, tuned, monitored, and occasionally blamed for a cloud bill that no one wants to read twice. This is why every architecture paper with the words “efficient,” “low-rank,” “shared,” or “recursive” immediately attracts attention. Some of that attention is deserved. Some of it is merely the industry’s permanent hunger for a cheaper miracle with a nicer benchmark table. ...