When Benchmarks Break: Why Bigger Models Keep Winning (and What That Costs You)

Budget.

That is where the benchmark story usually becomes less elegant. A vendor shows a model card with better reasoning scores, stronger multi-task accuracy, and a leaderboard position polished to a mirror finish. Then someone in operations asks the rude question: what does this improvement cost per customer case, per analyst hour, per compliance review, or per failed escalation?

The answer is rarely on the leaderboard. Naturally.

David Owen’s paper, How predictable is language model benchmark performance?, is useful because it does not simply repeat the familiar scaling sermon. It asks a narrower and more operational question: can benchmark performance itself be forecast from compute scaling?¹ The answer is partly yes, and that “partly” is where the business meaning lives.

The paper shows that average benchmark performance can be reasonably predictable as models scale. In one key result, extrapolating BIG-Bench Hard performance across one order of magnitude in compute produces an average absolute error of about 6 percentage points. But individual task performance is much noisier, with average extrapolation errors around 18 percentage points. That gap is not a statistical footnote. It is the difference between “the industry will probably improve” and “your workflow will probably improve.”

Those are not the same sentence. Procurement teams keep pretending they are.

Scaling made benchmarks look scientific

The modern benchmark economy rests on a simple bargain. If language-model loss improves predictably with model size, data, and compute, then benchmark scores should also rise as models scale. The first part of that bargain has serious empirical support. Kaplan and colleagues showed that language-model loss follows power-law relationships with model size, dataset size, and training compute across large ranges of scale.² Later, the Chinchilla work refined the economics: many large models were not just “too small” but undertrained relative to their compute budget, and better allocation between parameters and training tokens could produce stronger models at lower inference cost.³

That history matters because it explains why benchmarks became the industry’s favorite scoreboard. Scaling offered a clean narrative: train larger models, lower loss, improve downstream performance, win the chart. Researchers got a tractable measurement culture. Vendors got a marketing asset. Buyers got something that looked like due diligence.

The problem is not that this story is false. The problem is that it is too aggregated to answer the questions that businesses actually need answered.

A benchmark average is a portfolio measure. It tells you how a model performs across many tasks, prompts, formats, and difficulty regimes. That is valuable if your question is, “Where is the frontier moving?” It is less valuable if your question is, “Should this model handle insurance claims, legal intake, financial reconciliation, or production support tickets next quarter?”

Benchmarks did not become useless. They became over-interpreted.

What the paper directly shows

Owen’s paper studies whether benchmark results can be predicted using scaling-estimated loss. Instead of treating benchmark scores as isolated snapshots, it connects them to a scaling-law view of model improvement. The core move is simple: if loss falls predictably with compute, and benchmark performance relates smoothly to loss, then one should be able to forecast benchmark performance for larger models from smaller-model trends.

That works better at the aggregate level than at the task level.

This is the paper’s most important practical lesson: scaling remains informative, but the unit of interpretation matters.

At the level of “AI capability as a broad industry phenomenon,” scaling is still a powerful signal. At the level of “will this model reduce my customer-service escalation rate by 20%,” the signal becomes much weaker. The model can improve on average while failing to improve on the subset of tasks that pays your invoices.

Averages are polite. Operations are not.

Why bigger models keep winning even when benchmarks become fragile

There is a common misconception here: if benchmarks are fragile, then larger models should stop winning them. That is not how measurement systems work.

Larger models can keep winning because benchmark averages reward broad incremental gains. A model does not need to become meaningfully better at every task. It only needs enough improvements across enough items to lift the aggregate score. If easy, familiar, or training-adjacent tasks improve steadily, the total score rises even when harder reasoning pockets remain unstable.

BIG-Bench was designed precisely because earlier benchmarks were becoming too easy and too saturated. It assembled a wide set of tasks across linguistics, mathematics, common sense, software, social bias, and other domains, with the explicit aim of probing capabilities beyond then-current systems.⁴ That diversity is a strength, but it also creates the aggregation problem. Once a benchmark becomes a basket, the average can rise while the contents move unevenly.

This is why the paper’s 6-point versus 18-point contrast matters. Six percentage points says the basket is somewhat predictable. Eighteen percentage points says the apples, bolts, glassware, and suspiciously labeled “reasoning” items inside the basket are not all moving together.

For businesses, the benchmark average is useful as a market signal. It tells you whether the frontier is improving fast enough to revisit automation plans. But it is a weak replacement for local evaluation. If your task depends on precise instruction following, domain-specific judgment, rare-event handling, or calibrated refusal, the aggregate score may be directionally encouraging and operationally insufficient.

That is not a contradiction. It is just what happens when a scoreboard is asked to do a procurement officer’s job.

The expensive part is not the model score; it is the mismatch

The business cost of broken benchmarks is not that companies buy large models. Sometimes they should. The cost is that companies buy large models for the wrong reason.

A higher benchmark score can justify three very different decisions:

The paper directly supports the first two uses more than the last two. It shows that benchmark performance can be forecast at a useful aggregate level. It does not show that benchmark gains translate into positive deployment ROI.

That distinction sounds boring until the invoice arrives.

A model that is 6 percentage points better on a broad benchmark may still be overkill for summarizing routine emails. A smaller model, retrieval system, or task-specific workflow may deliver the same business value with lower latency and easier governance. Conversely, a larger model may be justified for high-ambiguity tasks where failures are costly and smaller models collapse under edge cases. The benchmark does not decide this. The task economics do.

This is where Cognaptus would interpret the paper less as a warning against scale and more as a warning against lazy substitution. Do not substitute aggregate benchmark scores for workflow evidence. Do not substitute model size for process design. Do not substitute leaderboard rank for governance.

Very large models are not magic. They are expensive probability machines with better manners.

What benchmark gains hide from operators

Benchmarks usually compress several dimensions into one score. Production systems do the opposite. They decompose performance into cost, latency, reliability, escalation, monitoring, review, and user trust.

HELM was important because it made this problem explicit: evaluation should not collapse into accuracy alone, and language models need to be assessed across scenarios and metrics such as calibration, robustness, fairness, bias, toxicity, and efficiency.⁵ That is closer to how organizations actually experience AI. A model that improves accuracy while worsening calibration may look better in a leaderboard and worse in a decision workflow.

The benchmark-scaling paper reinforces this point from another direction. If aggregate benchmark scores are more predictable than task-level outcomes, then buyers should treat benchmark gains as a starting hypothesis, not an answer.

A practical evaluation should ask four questions:

The second question is often the most neglected. Many organizations test a model on a blended sample and celebrate a higher average. Then deployment reveals that the model improved on cases humans could already handle quickly, while the difficult cases still require expert review. Congratulations: the automation now performs beautifully where automation was least needed.

This is how benchmark thinking leaks into business practice. It encourages average-case optimism.

Main result versus robustness: do not confuse the two

The central result is the relationship between scaling-estimated loss and benchmark performance. The robustness work around this kind of analysis exists to test whether the relationship survives reasonable changes in assumptions, subsets, and fitting methods. It should not be read as a second thesis claiming that every capability is smoothly predictable.

That distinction matters because readers often overcorrect. One reader sees predictable aggregate scaling and concludes that task-level forecasting is solved. Another sees noisy task-level prediction and concludes that scaling laws are useless. Both are too dramatic, which is convenient for social media and inconvenient for thinking.

A better reading is this:

The paper supports a middle position. Scaling is neither dead nor destiny. It is a strong statistical regularity that becomes less decisive as the question becomes more local, more operational, and more economically specific.

That is less catchy than “bigger is over” or “scale is all you need.” It also has the benefit of being closer to true.

What buyers should change immediately

The first change is to stop asking, “Which model is best?” Ask, “Best under which task mix, cost structure, latency constraint, and review policy?”

The second change is to evaluate by strata. Separate easy, medium, hard, rare, adversarial, ambiguous, and high-liability cases. A single accuracy number hides too much. If larger models mainly improve the easy stratum, they may raise benchmark-style averages without changing business outcomes. If they improve the hard stratum, they may be worth paying for even when the average gain looks modest.

The third change is to price the full system, not the model call. A larger model may reduce prompt engineering effort, retrieval complexity, or human escalation. It may also increase inference cost, monitoring cost, and vendor dependence. The relevant unit is not “tokens per answer.” It is “total cost per acceptable completed task.”

A simple procurement table is enough to prevent a surprising amount of nonsense:

This is where bigger models may still win. But they should win after surviving operational comparison, not because a leaderboard gave everyone emotional support.

Boundaries: what this paper does not settle

The paper does not prove that benchmark performance perfectly predicts real-world usefulness. It studies benchmark predictability, not enterprise ROI. That is an important boundary.

It also does not make scaling irrelevant. If anything, the aggregate results show the opposite: scale remains a meaningful driver of broad benchmark performance. The mistake is interpreting broad predictability as local certainty.

There are also measurement limits. Public benchmark datasets are incomplete, model reporting can be uneven, prompting conditions vary, and modern models may have unknown training exposures. These issues do not invalidate the analysis, but they make precision harder. For operators, that reinforces the central lesson: public evaluation is useful for orientation, not substitution.

Finally, business workflows are not benchmark tasks wearing a tie. They involve messy inputs, changing policies, user incentives, compliance constraints, and downstream accountability. A model’s benchmark score is one component of system design. It is not the system.

Bigger models, smaller excuses

The most useful reading of the paper is not “benchmarks are broken.” That line is too easy, and therefore suspicious.

The better reading is that benchmarks are becoming more specialized instruments. They can tell us something meaningful about the direction of frontier progress. They can support capability forecasting at the aggregate level. They can help identify whether scaling is still producing measurable returns.

But they cannot tell a company whether a larger model is worth deploying into a specific workflow at a specific price under a specific risk policy. That requires local evidence.

So yes, bigger models may keep winning. They may keep lifting benchmark averages, improving broad capability measures, and justifying further frontier investment. The uncomfortable part is that the buyer’s problem is not whether the model is generally stronger. The buyer’s problem is whether that strength appears where the business actually bleeds time, money, or risk.

A benchmark can point toward the frontier. It cannot tell you whether the frontier is worth your cloud bill.

Cognaptus: Automate the Present, Incubate the Future.