Compute-Efficiency

TL;DR for operators Improving a language model does not necessarily require proportionally more GPU-resident parameters. COUNT stores corpus-derived continuation statistics in host RAM and converts them into transformer input representations using existing model weights. In controlled experiments, COUNT delivers validation-loss improvements equivalent to 46–89% more standard-transformer training compute, according to fitted scaling laws. An optimized implementation retains more than 93% of baseline training throughput, even with a count table approaching 798 GB. ...

TL;DR for operators A pretraining-data pipeline receives millions of uneven records. Some are unusable, some contain removable noise, some need structural repair or added explanation, and some are already valuable enough that further processing may damage them. The operational problem is therefore not how to apply more cleaning, but how to decide which intervention—if any—each example needs. ...

TL;DR for operators The paper is not really saying “use a smaller speech model.” That would be too convenient, and reality hates convenience. It is saying something more useful: audio-model efficiency is a budget allocation problem. Model size, audio duration, encoder token resolution, and adaptation depth are different ways to spend compute, and they do not buy the same thing. Agarwal, Gangrade, Pal, and Wu study this across automatic speech recognition using Whisper on LibriSpeech and speech emotion recognition using wav2vec2 on CREMA-D.1 ...

TL;DR for operators The usual AI strategy story is simple: whoever spends the most on compute owns the future. The paper behind this article makes a more awkward claim: under current language-model scaling assumptions, massive compute advantage may be a temporary lead, not a permanent moat.1 The mechanism is not magic. It is diminishing returns. Chinchilla-like scaling laws imply that each additional unit of training compute buys a smaller reduction in loss. Meanwhile, hardware improvement and algorithmic progress are shared forces. They do not only help the largest labs. They also make yesterday’s “small” budget more capable. The result is a curve where frontier models pull ahead, peak in relative advantage, and then become less distinguishable from cheaper models. ...

TL;DR for operators Search-contempt is not “AI plays worse so it learns more”. That would be the lazy interpretation, and business strategy already has enough lazy interpretations wearing expensive shoes. The paper introduces a hybrid MCTS method for AlphaZero-like self-play systems. It behaves like standard PUCT search for the player to move, but at opponent nodes it eventually freezes the opponent’s visit distribution after a threshold, $N_{scl}$, and samples from that frozen distribution rather than constantly updating it toward stronger play.1 The effect is subtle but important: the system stops assuming the opponent will always improve its response with more search. ...