Scaling-Laws

TL;DR for operators A useful scaling law does not merely say “bigger is better.” That is not a law; that is a purchasing department with a GPU account. The paper behind this article studies whether the composition of pretraining data can change the compute-optimal balance between model size and downstream data in jet classification.1 The answer, in this setting, is yes. Training from scratch on JetClass produces a nearly balanced scaling rule: as compute grows, the optimal model size and dataset size grow at roughly similar rates. But after pretraining on a JetClass-II corpus augmented with Beyond Standard Model resonance decays, the compute-optimal rule shifts sharply toward downstream data. More of the next compute budget should be spent processing more examples, not inflating the model. ...

TL;DR for operators A new paper by Jose Marie Antonio Miñoza, Erika Fille T. Legara, and Christopher P. Monterola argues that a log-sum-exp neural layer is not merely analogous to a viscous Hamilton-Jacobi equation. Under the paper’s parameterisation, it is exactly the Hopf-Cole solution of one, evaluated at the input point.1 The operational point is not “neural networks are physics now”, although someone will certainly try to put that on a slide. The point is cleaner: one parameter, $\varepsilon$, simultaneously controls softmax temperature, PDE viscosity, and entropy-regularised convex optimisation. That makes smoothness, expressiveness, robustness, attribution sharpness, and scaling behaviour mathematically coupled. ...

Forecasting is where machine learning often learns humility. A language model can sound clever while being wrong. A forecasting model has fewer hiding places. Revenue arrives or it does not. CPU saturation happens or it does not. Demand spikes, latency drifts, inventories rot, turbines fail, and the spreadsheet smiles politely before punishing everyone involved. This is why time-series foundation models have been treated with a particular kind of suspicion: useful, interesting, sometimes impressive, but not yet comfortably scalable in the way large language models became scalable. ...

Cleaning is comforting. Every serious AI team has some version of the same ritual. Remove spam. Remove repetition. Remove bad language detection. Remove low-quality pages. Remove documents that look too weird, too short, too duplicated, too uneducational, too internet. Then hope the model learns from the respectable leftovers. That instinct is not foolish. In small or compute-constrained training runs, filtering often helps. The expensive mistake is treating that local truth as a permanent law. ...

Procurement meetings have a familiar ritual now. Someone opens a leaderboard, sorts by average score, points at a model near the top, and asks why the company is not using that one. It feels empirical. It is neatly ranked. It has decimals. Very scientific-looking decimals, the most seductive species of decimal. The problem is not that leaderboards are useless. The problem is that we often treat them as scoreboards when they are closer to measurement instruments. A scoreboard tells us who won under agreed rules. A measurement instrument first has to prove that it measures the thing it claims to measure. If the instrument mixes model size, benchmark difficulty, contributor practices, post-training choices, item redundancy, and residual artifacts into one number, then the number may still be useful. It is just not self-explanatory. ...

Heart of Scale: Why Bigger ECG Models Don’t Always Beat Better Biases A hospital does not buy an ECG model because it enjoys leaderboard furniture. It buys one because somebody wants a cheap, reliable signal from a noisy waveform: rhythm abnormality, structural heart disease, ICU risk, mortality risk, maybe a demographic or physiological clue that was not explicitly labeled during pre-training. ...

The expensive habit of feeding the machine Data teams have a familiar ritual. The model disappoints. Someone asks for more data. Another person asks for cleaner data. A third person, usually with a spreadsheet and a suspiciously calm face, asks whether the extra labeling budget is approved. Then the pipeline expands. More driving clips. More corner cases. More annotated scenes. More storage. More training runs. More dashboards explaining why the latest model is still not quite where it should be. ...

Budget meetings have a way of making AI theory suddenly less philosophical. Someone asks the simple question: “If we double the model size or the training data, how much better does the system get?” Then someone else opens a spreadsheet, adds a few curves, and everyone pretends the future has become manageable. This ritual has powered a large part of modern AI investment. Scaling laws made model development feel less like guesswork and more like engineering. ...

Mind the Model: When Generative AI Teaches Neuroscience New Tricks A model is not a mind. This should not need saying, but then again, neither should “do not use benchmark scores as a personality test,” and here we are. The more useful point is subtler. Modern generative AI does not matter to neuroscience because transformers are secretly brains in a hoodie. It matters because machine learning has turned several once-vague ideas about cognition into working engineering mechanisms. Not perfect mechanisms. Not biological mechanisms by default. But mechanisms clear enough to test, stress, reject, adapt, or steal with appropriate academic manners. ...