Transformers

A curated guide to textbooks, authors, websites, and papers for readers who want to study transformer internals, attention math, fine-tuning, GPU optimization, and benchmarking in more depth.

TL;DR for operators A bike-sharing operator has to reposition bikes before the next demand surge, even when one station is influenced by nearby docks and by commuter corridors elsewhere in the city. In the reported New York and Chicago tests, STAGformer records the lowest error in seven of eight city-month RMSE and MAE cells; GAT retains the lowest Chicago September MAE. ...

TL;DR for operators When a model behaves unexpectedly, teams often inspect attention maps to see where information flowed. Those maps show which source tokens were selected and how strongly, but not how the retrieved features were transformed before reaching the destination token. The paper proves that multi-head attention is exactly representable as a scaled, edge-dependent connection walk: token routing is supplied by attention weights, while feature transport is supplied by an attention-gated mixture of the heads’ value-output maps. Two layers can therefore display similar attention patterns while computing materially different transformations. ...

A document lands in an intake queue. It might be an invoice, a memo, a form, a résumé, or one of those corporate artifacts whose layout says more than the words do. Someone wants the system to classify it instantly, because every downstream workflow—routing, extraction, compliance, archiving—depends on that first label. The fashionable answer is: send it to a large language model. Extract the text, paste it into a prompt, ask for one label, and let the machine be clever. This is attractive because it feels general. It is also how many automation projects quietly turn a visual problem into a text problem, then act surprised when the system starts calling file folders “proposals” because the word proposal appeared somewhere on the page. ...

Weights are expensive twice. First, they cost money to train. Then they cost money every time a model is served, copied, quantized, tuned, monitored, and occasionally blamed for a cloud bill that no one wants to read twice. This is why every architecture paper with the words “efficient,” “low-rank,” “shared,” or “recursive” immediately attracts attention. Some of that attention is deserved. Some of it is merely the industry’s permanent hunger for a cheaper miracle with a nicer benchmark table. ...

A materials lab does not need an AI system that can politely imitate the periodic table. It needs one that can search. That difference sounds small until money enters the room. In materials discovery, every serious candidate eventually asks for simulation time, specialist review, density functional theory validation, and—if it survives long enough—lab synthesis. A model that produces many plausible crystals is useful. A model that pushes candidates toward a target property before the expensive validation begins is more useful. Less glamorous, perhaps. But so is a good spreadsheet, and civilization somehow survives. ...

Serving an LLM is usually discussed in pleasantly managerial language: latency, throughput, context windows, GPU memory, quantization, cache eviction. Nice clean nouns. Then the model ruins the spreadsheet by producing internal activations that are thousands of times larger than ordinary values, while some tokens quietly become attention magnets for reasons that are not exactly semantic. Very professional behavior from a trillion-dollar technology stack. ...

Routes look simple until someone has to pay for them. A delivery van does not care whether an optimization model sounds elegant. It cares whether the assigned route wastes fuel, crosses another vehicle’s territory, violates capacity, or produces a schedule that looks clever in a paper and stupid on the street. The Capacitated Vehicle Routing Problem, or CVRP, is where that mundane reality becomes mathematically unpleasant: multiple vehicles, limited capacity, customer demand, depot returns, and a search space that grows far faster than managerial patience. ...

Traffic control rooms rarely suffer from a shortage of numbers. Sensors count vehicles, lanes report flows, APIs stream updates, dashboards glow politely, and somewhere in the middle of all this a manager is expected to decide whether the next congestion wave is routine, dangerous, or about to become a public complaint. The naive answer is predictable: feed everything into a larger model. If one road sensor helps, fourteen must help more. If a Transformer can learn temporal patterns, give it the whole motorway and let attention perform its usual magic trick. ...

Furnace loading is not the glamorous side of artificial intelligence. No one gives a keynote about choosing which pile of titanium scrap should enter an induction furnace. Which is precisely why it is useful. The paper Enterprise Resource Planning Using Multi-type Transformers in Ferro-Titanium Industry applies a Multi-Type Transformer, or MTT, to two classic combinatorial optimization problems: the Knapsack Problem (KP) and the Job-Shop Scheduling Problem (JSP). It then pushes the method into a real manufacturing allocation case: selecting raw materials for a ferro-titanium furnace batch.1 ...