Measure for Measure: Why AI Evaluation Must Follow the Failure

TL;DR for operators

• A lower model bit width is not automatically a speedup. A lower training loss is not automatically a reliable policy.
• GRINQH evaluates quantization through the mechanism it is meant to change: decoding-stage memory traffic, kernel throughput, end-to-end generation speed, and retained task accuracy.¹
• Kolmogorov regression evaluates diffusion policies through trajectory geometry, a PDE-based inference residual, rollout behavior, anomaly detection, and an external safety filter.²
• The shared lesson is not that the two forms of “precision” are technically equivalent. They are not. The lesson is that fidelity and evidence should be allocated according to the actual failure structure of the system.
• A production evaluation should connect four things explicitly: the intervention, the mechanism it changes, the diagnostic that observes that change, and the operational outcome that justifies deployment.
• Composite scores are useful only when their weights reflect real business priorities and their components remain separately visible. Otherwise, they are merely spreadsheets wearing authority.

The dashboard is not the system

AI evaluation has developed an awkward habit: optimize a convenient number, improve that number, and declare the system improved.

A compressed model occupies fewer nominal bits, therefore inference must be faster. A policy reaches a lower validation loss, therefore its trajectories must be safer. A benchmark average rises, therefore customers will notice. A composite score increases, therefore the project is ready for production.

This is an efficient way to produce slides.

It is less effective at producing reliable systems.

The problem matters more now because AI projects are moving from capability demonstrations into operating environments. Once a model sits inside a customer service workflow, an inference platform, a robot, or a production line, its failures acquire structure. Latency may arise during one execution stage but not another. A trajectory may look accurate point by point while accumulating physically destructive oscillations. A forecaster may be slightly worse on average error yet far better at identifying a bottleneck that shuts down a factory.

At that point, “model quality” stops being a single variable. It becomes a chain.

The two papers examined here occupy very different technical territories. GRINQH studies dynamic low-bit inference for large language models. Kolmogorov regression studies diffusion policies over continuous trajectories. One is concerned with GPU memory traffic; the other with covariance operators, temporal regularity, and safety.

Their shared contribution is methodological: evaluation becomes credible only when it follows the failure mechanism from the intervention inside the model to the behavior of the deployed system.

That sounds obvious. It is also routinely ignored.

Two meanings of precision, one evaluation problem

The word precision appears in both papers, but it refers to different objects.

The papers are therefore not proposing interchangeable versions of adaptive precision. Applying a Cameron–Martin norm to an LLM kernel would be an imaginative use of everyone’s time.

What they share is the decision to stop treating every part of a system as equally important.

GRINQH does not allocate the same bit width to every channel or optimize prefill and decoding identically. Kolmogorov regression does not assign the same penalty to errors in every trajectory mode or treat batch loss as sufficient evidence of deployment fidelity.

Both ask a more useful question:

Which errors or resources actually dominate the failure we care about?

The evaluation follows from the answer.

A bit width is not a speed measurement

Quantization papers often report a nominal representation size—4-bit, 3-bit, perhaps an attractively alarming 2-bit—and invite the reader to mentally convert that number into cheaper inference.

The conversion is not automatic.

An algorithm can reduce an arithmetic representation without reducing the bytes fetched from memory. It can introduce packing, reconstruction, branching, or indexing overhead. It can accelerate one inference stage while slowing another. It can produce an impressive isolated kernel result that disappears once prompt processing, scheduling, and framework overhead enter the measurement.

GRINQH starts with the distinction that matters operationally: prefill and decoding have different bottlenecks.

During prefill, many prompt tokens are processed in parallel. Loaded weights can be reused, producing relatively high arithmetic intensity. This stage is generally more compute-bound.

During batch-size-one autoregressive decoding, each new token requires the model’s weight matrices to be accessed again. Compute units can spend much of their time waiting for weights to arrive from off-chip memory. This stage is largely memory-bandwidth-bound.

Uniform quantization treats these stages as though they were the same job. GRINQH does not.

It preserves higher precision for prefill, then dynamically allocates weight-channel precision during decoding according to the magnitude of the current input activations. Channels associated with large activations receive more bits. Less influential channels receive fewer bits or zero bits, which turns part of the mechanism into structured sparsity.

For a precision distribution $p_b$ over available bit widths $b$, the average effective width is approximately

But this average is not the final evidence. It is only a control setting.

The actual mechanism requires a storage layout capable of fetching only the selected bit segments. GRINQH therefore stores weights in bit planes, allowing its custom Triton kernel to load higher-order segments incrementally and reconstruct the selected precision on chip.

That implementation detail is the difference between a compression claim and a hardware claim.

The evaluation chain in GRINQH

The paper’s evidence follows a fairly disciplined sequence:

1. Bottleneck identification: batch-size-one decoding is memory-bound.
2. Intervention: activation-dependent channel precision reduces the weight data required for each decoding step.
3. Mechanism verification: roofline and profiling results show the kernels remain in the memory-bound regime and runtime scales with effective bit width.
4. Local performance: the custom kernel is compared with optimized fixed-precision implementations.
5. End-to-end performance: the implementation is integrated into vLLM and measured across complete prompt-and-generation workloads.
6. Capability retention: language-model accuracy and perplexity are evaluated across multiple tasks and model families.

The reported kernel results illustrate why all six steps matter. An effective 2-bit configuration reaches approximately 204.6 decoding tokens per second, compared with 160.4 for the 4-bit MARLIN baseline in the reported setup. The authors describe decoding gains of up to 28%.

Prefill, however, is roughly 20% slower than the optimized 4-bit baseline because the bit-planar representation must be unpacked. End-to-end value therefore depends on the ratio of generated tokens to input tokens. A long-generation workload benefits more than a prompt-heavy, short-output workload.

This is the paper’s most useful business lesson. The relevant question is not:

How many bits does the model use?

It is:

Under our actual traffic shape, hardware, batch size, and output length, which bytes disappear, which operations are added, and what happens to total request latency?

“Three-bit model” is a product label. The answer to that second question is an operating result.

A training loss is not a trajectory guarantee

The Kolmogorov paper attacks an analogous shortcut in diffusion policies.

Standard diffusion models generally add isotropic white noise and train a network to predict or remove that noise using mean-squared error. For images, a pixel-grid formulation may be acceptable. For control trajectories, the coordinates are related across time. A brief error and a persistent oscillation can have similar pointwise magnitudes while producing very different physical consequences.

The paper represents trajectories as functions in a Hilbert space and introduces a Matérn covariance structure that encodes temporal regularity. Rather than perturbing the training trajectories with independent white noise, it uses colored noise whose covariance links nearby time points.

It then replaces ordinary mean-squared error with a Cameron–Martin loss. For an error trajectory $e$ and covariance operator $C$, its simplified form is

The inverse covariance is sometimes called a precision operator. Errors are weighted according to the geometry implied by the trajectory distribution.

This is not merely “pay more attention to large errors.” In the paper’s spectral interpretation, different modes have different variances. An equal numerical error in a tightly concentrated mode may be more damaging than the same error in a high-variance mode. Ordinary MSE ignores that asymmetry.

The paper also derives an inference-time diagnostic from the backward Kolmogorov equation. A learned value function $\hat u$ should approximately satisfy the governing PDE. Its violation can be summarized by a residual of the form

A low training loss measures agreement with training targets under the selected objective. The residual asks a different question: does the learned function satisfy the structural equation it is supposed to approximate at this point in the trajectory?

That distinction is the paper’s strongest idea.

From loss to deployment behavior

On the PushT manipulation benchmark, the paper reports a maximum episodic reward of 0.95 for the Cameron–Martin formulation versus 0.78 for the MSE baseline. It also reports a 67.6% reduction in the Kolmogorov residual, with lower residuals associated with smoother and more successful rollouts.

The paper then moves into a simulated six-station manufacturing line. There, the residual is used as an anomaly score to identify a deliberately constrained station. Across 100 simulation runs, it ranks the true bottleneck first, with Precision@1 of 1.0 and a reported signal-to-noise ratio of roughly 13.

Finally, the learned forecast is combined with a Hamilton–Jacobi reachability filter. In the simulation, the filter blocks dispatch decisions outside a calculated safe set and prevents 351 deadlock events across 100 runs.

Notice the progression:

• the loss shapes the learned trajectory;
• the residual interrogates structural fidelity at inference;
• the task outcome measures whether the policy succeeds;
• the anomaly score supports operational diagnosis;
• the reachability filter enforces a safety constraint that learning alone does not guarantee.

Once again, the headline model metric is only the first link.

A residual is not reality either

Mechanism-aligned evaluation is better than aggregate evaluation. It is not infallible.

The Kolmogorov residual measures violation of the assumed backward Kolmogorov equation under the selected covariance model. A high residual can reveal that the learned function is inconsistent with that structure.

A low residual does not prove that the structure faithfully represents every relevant feature of the real environment.

This distinction matters because the paper describes the residual as a universal failure detector. The construction may be architecture-independent, but its operational meaning remains assumption-dependent. It can detect departures from the encoded PDE. It cannot detect every failure whose cause lies outside that PDE, such as sensor corruption, actuator wear, an omitted constraint, a poorly chosen covariance kernel, or a genuinely novel operating regime that remains mathematically smooth.

Conformity is not truth. It is conformity.

The paper’s theoretical result also requires boundaries. Its total-variation convergence guarantee is stated under assumptions including finite second moments, full support, and a trace-class covariance operator. The experiments examine qualitative consequences such as smoother rollouts and lower residuals; the paper explicitly notes that they do not directly measure the theorem’s total-variation distance.

That is a meaningful theoretical contribution and an encouraging empirical result. It is not a universal certificate for arbitrary autonomous systems.

The awkward manufacturing numbers

Full-paper reading occasionally produces the impolite discovery that a paper disagrees with itself.

The abstract and conclusion of the Kolmogorov paper state that the manufacturing model achieves a 28.4% reduction in normalized WIP RMSE relative to the LSTM baseline. Appendix E’s final table instead reports normalized RMSE values of 0.124 for InfDiff+HJ and 0.092 for LSTM+CONWIP. The surrounding text explicitly says the infinite-dimensional diffusion system “loses on pointwise root mean-square error.”

Those statements cannot all describe the same metric in the same direction without an omitted transformation or reporting error.

The paper’s broader operational result remains interpretable: its composite score is 0.637 versus 0.564, driven by bottleneck detection and the safety component. But that composite score introduces another attribution problem.

The evaluated system is not only the diffusion forecaster. It includes:

• the InfDiff forecaster;
• the Kolmogorov residual bottleneck detector;
• a Hamilton–Jacobi safety filter.

The comparator includes an LSTM forecaster and a CONWIP heuristic without an equivalent safety-certification layer.

The integrated pipeline may indeed be operationally preferable under the authors’ chosen priorities. But the result does not isolate how much of the improvement comes from the learned policy, how much from the residual diagnostic, and how much from adding a formal safety filter unavailable to the baseline.

This is not a reason to discard the experiment. It is a reason to label it correctly.

It is a system comparison, not a pure forecasting-model comparison.

The mechanism-aligned evaluation stack

The two papers suggest a reusable evaluation architecture for production AI.

Most weak evaluations stop at layer three.

A team identifies a plausible mechanism, changes something associated with it, and reports the control variable as though it were the outcome. Fewer bits become “lower latency.” Lower loss becomes “safer control.” More retrieved documents become “better answers.” A larger context window becomes “better long-context reasoning.”

The system is politely asked not to reveal the missing links.

What the papers show—and what businesses should infer

What the papers show

GRINQH provides evidence that stage-aware, input-dependent precision can translate into measurable decoding acceleration when paired with a storage format and kernel that actually reduce memory movement. It also shows that a single deployed weight representation can offer multiple operating points along an accuracy-throughput frontier.

The Kolmogorov paper provides evidence that covariance-aware trajectory learning can improve rollout behavior in the tested PushT setup, and that a PDE residual can serve as a useful anomaly signal in the paper’s simulated environments. Its manufacturing study also illustrates the value of combining learned prediction with independent formal verification.

What businesses should infer

The business implication is broader than either method.

Evaluation design should begin with the decision the organization must make, not with the benchmark the model team already has.

For an inference platform, that decision might be whether to deploy a quantized model on a particular GPU fleet. The evaluation must therefore reflect concurrency, prompt lengths, generated-token distributions, memory capacity, quality thresholds, and tail latency.

For a control system, the decision might be whether an autonomous policy can act without human approval. The evaluation must therefore include trajectory stability, anomaly sensitivity, false alarms, constraint violations, recovery behavior, and the authority granted to any safety filter.

A benchmark average may contribute evidence. It cannot define the decision by itself.

Five operating rules for better AI evaluation

1. Write the failure pathway before selecting metrics

Document the causal chain:

Every arrow needs evidence. A missing arrow is not filled by enthusiasm.

2. Match benchmarks to workload shape

GRINQH’s gains depend partly on the ratio of decoding to prefill. A deployment dominated by long outputs and batch-size-one generation is not equivalent to one dominated by long prompts, short responses, and high batching.

The same principle applies outside inference. A policy tested on short, smooth episodes has not been evaluated for long-horizon drift merely because both scenarios produce an average reward.

3. Keep mechanism metrics and outcome metrics separate

Memory traffic can explain throughput, but it is not throughput. A PDE residual can explain failure risk, but it is not task success. Retrieval recall can explain answer quality, but it is not customer resolution.

Measure both. Do not merge them prematurely.

4. Attribute integrated-system gains by component

When a model is combined with guardrails, tools, filters, or human review, evaluate:

• the model alone;
• each added component;
• the complete system;
• interactions among components.

Otherwise, a deterministic safety filter may be credited to the model, while the filter’s latency and conservatism quietly belong to nobody.

5. Treat composite scores as policy documents

The Kolmogorov manufacturing composite assigns explicit weights to RMSE, starvation recall, safety violations, and bottleneck identification. That is preferable to hiding priorities, but the score is still a management decision expressed mathematically.

Change the weights and the winner may change.

Every composite score should therefore include:

• the unaggregated component metrics;
• the rationale for each weight;
• sensitivity analysis across plausible weights;
• minimum acceptable thresholds that cannot be compensated for by other components.

A catastrophic safety rate should not be redeemable through excellent average accuracy. Some failures deserve veto power.

The deeper lesson: allocate fidelity where failure concentrates

The most interesting connection between these papers is not “adaptive precision.” It is selective fidelity.

GRINQH assigns representational fidelity to channels whose current activations make them influential. It spends fewer bits where memory traffic can be reduced with limited effect on output quality.

Kolmogorov regression assigns error sensitivity according to covariance structure. It penalizes mistakes more heavily in modes where the assumed trajectory distribution is tightly concentrated and where deviations are more structurally significant.

Neither system assumes that every dimension, stage, channel, or mode deserves equal treatment.

That is a useful design principle, but it also creates an evaluation obligation. Once fidelity is allocated selectively, the system must prove that its importance model is correct enough:

• Are activation magnitudes a stable proxy for channel importance under the deployed workload?
• Does the chosen covariance kernel represent the trajectory regularities that matter?
• Do calibrated thresholds transfer across tasks?
• Does a low residual remain associated with success under distribution shift?
• What happens when the supposedly unimportant component becomes decisive?

Selective allocation can produce efficiency and robustness. It can also produce elegantly optimized blind spots.

Stop asking one metric to do five jobs

A credible production evaluation does not require an enormous dashboard. It requires a coherent one.

The objective is not to accumulate metrics until every stakeholder finds a preferred number. It is to select a small set whose relationships are explicit:

• a metric for the intervention;
• a metric for the mechanism;
• a metric for end-to-end behavior;
• a metric for operational value;
• a boundary condition describing where the conclusion stops.

GRINQH is persuasive where it links effective precision to actual memory loading, kernel behavior, end-to-end throughput, and retained accuracy. Its result is narrower where batch size, hardware architecture, or workload shape changes.

Kolmogorov regression is persuasive where it distinguishes trajectory-aware learning from pointwise loss and adds an inference-time structural diagnostic. Its claims require greater caution where simulation replaces deployment, where formal safety tooling drives the integrated score, and where the paper’s RMSE reporting is internally inconsistent.

Both papers are more valuable when read this way—not as declarations of universal superiority, but as demonstrations of how evaluation should be attached to a mechanism.

The practical rule is simple:

Do not ask whether the metric improved. Ask whether the metric sits on the path between the intervention and the failure you are paying to prevent.

Everything else is measurement theatre, albeit sometimes with very attractive charts.

Cognaptus: Automate the Present, Incubate the Future.