Scaling Laws Without Power Laws: Why Bigger Models Still Win

Budget meetings have a way of making AI theory suddenly less philosophical.

Someone asks the simple question: “If we double the model size or the training data, how much better does the system get?” Then someone else opens a spreadsheet, adds a few curves, and everyone pretends the future has become manageable. This ritual has powered a large part of modern AI investment. Scaling laws made model development feel less like guesswork and more like engineering.

But there is a quieter question underneath the spreadsheet: why do these curves exist at all?

One popular answer has been that scaling laws merely echo the data. Natural language has Zipf-like frequency structure. Images have spectral regularities. If the world gives us power laws, perhaps neural networks simply inherit them. Convenient. Also slightly too neat.

The paper On the Origin of Neural Scaling Laws: From Random Graphs to Natural Language challenges that explanation by taking language apart until almost nothing literary remains.¹ The authors train transformers not only on natural language, but also on random walks over graphs. In the cleanest case, those graphs have no power-law degree distribution, no power-law spectrum, and no Zipf-like token frequencies. Yet the transformer loss still follows scaling-law behavior.

That is the interesting part. Not “bigger models win again,” which is hardly breaking news. The sharper point is that bigger models can keep winning even after the usual statistical alibi has been removed.

The paper asks where the curve comes from, not whether the curve exists

Most business readers encounter scaling laws as planning tools. The common form is roughly:

or, when dataset size is varied:

Here, $L$ is loss, $N$ is model size, $D$ is dataset size, and $L_\infty$ is the irreducible loss: the part that cannot be eliminated merely by adding more model or data under the setup being studied.

The usual operational interpretation is simple. If the exponent is larger, returns improve faster. If the irreducible loss is lower, the curve has more room to fall. If the compute-optimal curve is reliable, you can decide how to allocate budget between model parameters and training tokens.

The paper does not reject this planning value. It asks a more basic question: what produces the power-law curve in the first place?

A tempting answer is “the data.” Natural language has extreme frequency imbalance. A few words and patterns occur constantly; many appear rarely. If model training is mostly about gradually learning rarer and rarer patterns, then power-law loss curves may simply reflect power-law data frequencies. That explanation is intuitive, tidy, and dangerous in the way tidy explanations often are.

The authors’ strategy is to remove complexity step by step. They start with random walks on graphs, which are equivalent to bigram sequence models. Then they introduce power-law structure deliberately. Then they move toward language through bigram language models, transformer-generated synthetic language, and finally FineWeb-edu natural language.

This is why a mechanism-first reading is better than a normal summary. The paper is not just a tour of datasets. It is a controlled demolition of a simple causal story.

A random walk is language after the grammar has been confiscated

The simplest object in the paper is a random walk on a graph.

Imagine each node as a token. From the current node, the sequence moves to a neighboring node according to a transition probability. Predicting the next token means learning those transition probabilities. This is basically a bigram model: the next token depends only on the current token.

That sounds too simple to be relevant to LLMs. It is simple. That is the point.

If scaling laws appear in this stripped-down setting, then they cannot require syntax, semantics, world knowledge, or the full messiness of human text. The authors use this simplicity as a laboratory. They can choose graph families, tune whether the data has power-law structure, and observe how transformer loss behaves as model size and dataset size change.

The central experiment uses Erdős–Rényi graphs. These are random graphs where edges are formed without preferential attachment. In the unbiased random-walk setting, the relevant node and transition distributions do not exhibit the kind of power-law structure associated with natural language. No Zipf. No scale-free network. No poetic long tail. Just a graph that behaves like a graph.

And still, transformers trained on next-token prediction show power-law scaling behavior.

That result is the paper’s main evidence. It does not prove a complete theory of scaling laws. It does something narrower and more useful: it breaks the claim that power-law data structure is necessary for neural scaling laws to appear.

The experiment ladder is not decorative; it isolates what changes

The paper’s datasets form a complexity ladder. Each rung has a different role.

This table matters because otherwise the paper can be misread as a pile of experiments. The Erdős–Rényi result is the main causal wedge. The biased and scale-free graph experiments test what happens when power-law-like structure is added back. The language bigram and TnL datasets build a bridge from artificial sequences toward real text. The FineWeb-edu experiments reconnect the simplified world to familiar LLM scaling discussions.

The argument is not “data structure does not matter.” That would be too strong. The paper shows something more precise: power-law data correlations are not necessary for power-law loss scaling, while data complexity and structure still appear to influence exponents and fit quality.

That distinction is small enough to be missed and large enough to matter.

The likely mechanism is not Zipf; it may be learnability

The discussion section offers a useful clue. In one-epoch training, when dataset size is fixed, the number of optimization steps is also fixed. If increasing model size lowers loss under the same number of steps, then the larger model is not merely more expressive. It is also easier to train toward a low-loss solution under the chosen optimization process.

In plain English: bigger models may open the door wider.

The authors suggest that the basin of low-loss solutions may change with model size in a way that makes gradient-based optimization more efficient. This is not presented as a finished theory. It is a mechanism candidate. But it is a much better candidate than “the data had a Zipf curve, so naturally the loss had a power law,” because the Erdős–Rényi case removes that easy explanation.

There is also a second possibility. Even if the input data lacks power laws, the model’s internal activations might develop power-law-like structure during learning. In that case, power laws would not be inherited from the dataset directly. They would emerge inside the trained system.

That possibility is especially relevant for business interpretation. If scaling laws are partly about optimization geometry and internal representation, then scaling forecasts are not merely dataset statistics. They depend on the full training recipe: architecture, parameterization, optimizer, hyperparameter search, and data complexity.

A spreadsheet that treats the exponent as a universal constant is not a forecast. It is a beautifully formatted superstition.

The bigram result shows why “simple” does not mean “irrelevant”

The language-bigram experiment is one of the paper’s best bridges for non-specialist readers.

The authors tokenize FineWeb-edu using the GPT-2 tokenizer, build a bigram model from token co-occurrences, remove extremely rare bigrams for computational tractability, and train transformers on sequences sampled from this bigram graph. This produces a language-derived random-walk problem: not natural language in the full sense, but not an arbitrary graph either.

The result is telling. The paper reports that the bigram language setting produces especially clean power-law fits. The dataset has a broken power-law structure in bigram probabilities, so this does not replace the Erdős–Rényi result. Instead, it clarifies the relationship between two claims:

1. Power-law data structure is not necessary for scaling laws.
2. Power-law data structure can still make scaling behavior cleaner or change the exponents.

That is exactly the kind of distinction business users need. “Not necessary” does not mean “not useful.” A dataset’s statistical structure may affect scaling quality and return curves even if it is not the root cause of scaling laws.

The authors also compare the bigram model with transformer-generated language datasets, called TnL. These are sampled from one-layer, two-layer, and four-layer transformers trained on FineWeb-edu. The point is not that TnL is a new product category, though someone in a strategy deck will probably try. The point is to create intermediate datasets between bigram language and real natural language.

The paper finds that scaling exponents interpolate between bigram language and natural language. In particular, one of the main exponent patterns changes monotonically as estimated dataset complexity increases. This supports the idea that complexity matters, but not in the cartoonish way where “more Zipf” mechanically equals “more scaling.”

The fitting lesson is less glamorous and more operational

The paper also spends serious attention on how scaling laws are fitted. This part is less catchy than random graphs, but for practitioners it may be more immediately useful.

First, the authors emphasize the irreducible loss term, $L_\infty$. Dropping this offset can make the fitted scaling exponent look much smaller and can make the fit worse. This matters because raw loss on a log-log plot does not need to be a straight line. With an offset, the curve can bend while still being well described by a power law in the reducible loss.

That sounds like a technical detail until someone uses the wrong exponent to justify a training budget.

Second, the authors criticize the widely used two-dimensional Chinchilla-style parametric fit for $L(N,D)$. They compare it with neural network regression and kernel regression for fitting the loss surface. Their reported conclusion is that the more flexible regression methods fit substantially better, although they are less interpretable.

This creates a practical trade-off:

The quiet message is that scaling-law practice has a model-risk problem. Organizations do not only need better models; they need better models of their models.

Shallow transformers can reproduce more than expected

Another notable result is that basic language-scaling behavior can be recovered with surprisingly small architectures: two-layer transformers with short context length.

This does not mean shallow transformers are enough for modern capabilities. Obviously not. A two-layer model with context length 100 is not going to become the operating system of the enterprise, unless the enterprise has very modest dreams.

The point is methodological. Some scaling-law phenomena do not require frontier-scale depth or long-context training to appear. That means researchers can study parts of the scaling puzzle in cheaper controlled settings. For businesses, the implication is not “train tiny models and call it a day.” It is that small-scale scaling probes can be informative when designing data strategy, optimizer choices, and parameterization experiments.

The paper also revisits the famous Kaplan-versus-Chinchilla discrepancy over compute-optimal scaling. In their shallow FineWeb-edu experiments, whether embedding parameters are included helps recover different historical conclusions about how parameters and tokens should scale. This is a reminder that “model size” is not a neutral measurement. Counting conventions can change the apparent law.

That is not pedantry. If procurement, engineering, and research teams use different definitions of parameter count, they may think they are disagreeing about strategy when they are partly disagreeing about accounting.

μP appears as an efficiency hint, not a miracle button

The paper includes preliminary experiments with maximal update parameterization, or μP. The authors find evidence that μP may be more parameter efficient than standard parameterization in their FineWeb-edu settings. More specifically, loss appears to decrease more sharply with model size, and compute-optimal training may not require a fixed token-per-parameter ratio.

This is best read as an exploratory extension, not the main thesis.

The useful business interpretation is not “switch to μP and collect your margin expansion.” The result is small-scale and preliminary. The more defensible point is that parameterization can alter scaling efficiency. That makes scaling laws less like natural constants and more like operational measurements under a given recipe.

For AI teams, this shifts attention toward controlled internal scaling studies. The question should not be “what exponent did the internet say we have?” It should be:

• Under our architecture, data mixture, optimizer, parameterization, and evaluation target, what exponent do we observe?
• How sensitive is it to the training range?
• Does a different recipe move the curve enough to matter economically?
• Are we measuring parameter count consistently?
• Is the forecast being validated outside the fitted window, or merely admired inside it?

The last question is especially healthy. Forecasting curves are at their most charming just before they extrapolate into nonsense.

What the paper directly shows, and what Cognaptus infers

It is useful to separate the evidence from the business extrapolation.

The paper’s business value is not that it gives a new universal scaling recipe. It does almost the opposite. It makes universal recipes look suspicious.

The better takeaway is diagnostic: scaling laws can emerge from the interaction between data, architecture, optimization, and learnability. Therefore, serious compute planning should measure that interaction directly.

The boundary: this is controlled science, not a frontier-model oracle

The paper’s limitations are not decorative. They affect how the result should be used.

Most experiments are small compared with frontier LLM training. The models are shallow, often two or four layers. Context lengths are short. Training is generally one epoch. Random-walk tasks, even when large, are much simpler than language understanding. Hyperparameter search matters, especially for the random graph experiments. The authors also note that fitted exponents can be sensitive to the window over which the fit is performed, and that bootstrap confidence intervals do not fully capture this window sensitivity.

Those boundaries do not weaken the central contribution. In fact, they clarify it.

The paper is not saying: “Here is the final theory of scaling.” It is saying: “The simplest popular explanation is insufficient.” That is a cleaner and more durable result.

For businesses, the danger is to overgeneralize in either direction. One bad reading is: “Scaling laws are universal, so just buy more compute.” Another bad reading is: “Scaling laws are fragile artifacts, so forecasting is useless.” The paper supports neither.

The more disciplined reading is this: scaling laws are real enough to be useful, but local enough to require measurement.

Bigger models still win, but the reason is less convenient

The old comfort story was that scaling laws ride on the back of power-law data. Natural language has Zipf. Models learn the frequent patterns first and the rare patterns later. Loss falls predictably. Spreadsheet smiles. Budget approved.

This paper makes that story too small.

When transformers trained on Erdős–Rényi random walks still exhibit scaling laws, the origin cannot simply be power-law correlations in the data. Something about neural sequence learning itself is contributing: optimization geometry, internal representations, sample efficiency, parameterization, or some combination we do not yet fully understand.

That is good news and bad news.

The good news is that scaling behavior may be more general than the Zipf story suggests. The bad news is that it is also less reducible to a simple dataset statistic. You cannot read a corpus frequency plot and know your scaling future. You have to run the experiment.

For companies building domain-specific models, this is the practical message: do not borrow public LLM exponents as if they were exchange rates. Build small scaling ladders on your own data. Include irreducible loss in the fit. Compare fitting methods. Test sensitivity to the training window. Track whether optimization or parameterization changes the curve. Then decide whether the next compute step is justified.

Bigger models still win. But if this paper is right, they may win not because the data secretly handed them a power law, but because learning itself opens differently as scale changes.

That is less tidy. It is also more useful.

Cognaptus: Automate the Present, Incubate the Future.