Same Request, Different Batch, Different Bits

TL;DR for operators

A GPU kernel can produce numerically acceptable results yet return different output bits for the same sample when the surrounding batch changes. That distinction matters when inference infrastructure dynamically groups requests.

BIEvolve adds two checks to existing AI-driven kernel optimization: static analysis of batch-dependent execution decisions and dynamic comparison of outputs across batch contexts. In controlled experiments, final-solution batch-invariance pass rates increased from 67.6% to 91.9%, while retaining 91.8% of baseline geometric mean kernel speedup. A separate ablation found 92% detection of confirmed violations using both checks, versus 83% with static analysis alone and 55% with dynamic testing alone.

For teams deploying AI-generated kernels, the evidence supports a separate batch-invariance acceptance gate alongside correctness and speed. Passing that gate demonstrates consistency under tested conditions, not guaranteed deterministic inference.

A correct kernel can change its answer when the batch changes

An LLM inference service may process a request alone or alongside other requests. The request’s input remains unchanged, but the GPU workload surrounding it can differ.

Ordinary kernel validation asks whether the implementation produces an acceptably accurate result relative to a reference. That comparison typically allows numerical tolerance. It does not necessarily require identical output bits across different execution contexts.

The distinction has measurable consequences.

In God Does Not Play Dice: Enforcing Batch Invariance in Agentic Kernel Generation, Youwei Bao, Chengpeng Wang, and Simin Chen audit 131 released GPU kernel artifacts. Of the 83 with meaningful semantic batch dimensions, 19 exhibit confirmed violations, or 22.9% of applicable artifacts.¹

This released-artifact audit establishes that the problem occurs in actual agent-generated implementations, although the selected artifacts cannot establish its prevalence across all GPU kernels.

The relevant property is batch invariance (BI): under a fixed hardware and software environment, an unchanged sample must produce bitwise-identical output across admissible batch sizes, compositions, and positions.

The failure can originate below the mathematical definition of the operation. Changing batch conditions may alter work partitioning, tiling, reduction order, or algorithm selection. Because floating-point addition is not associative, different accumulation orders can produce different bits even when both results fall within the allowed correctness tolerance.

For an infrastructure team, passing reference correctness and preserving exact outputs across contexts are therefore separate release decisions.

BIEvolve makes batch dependence a reason to reject a candidate

BIEvolve integrates into existing kernel-generation agents rather than replacing their optimization algorithms.

Its central design is to make batch invariance an explicit candidate-acceptance condition. A generated kernel must satisfy structural restrictions, reference correctness, observed cross-batch consistency, and performance evaluation.

The evaluation sequence is:

1. Static dependency analysis: Examine whether changing batch context can influence protected execution parameters.
2. Reference correctness testing: Check the candidate against the task’s existing numerical requirements.
3. Dynamic cross-batch validation: Execute an unchanged target sample within selected batch contexts and compare its output bits.
4. Performance benchmarking: Evaluate the candidate’s speed when earlier acceptance conditions are satisfied.

The static component uses a contract language to specify batch-context variables and execution parameters that should not depend on them. Protected parameters can include split-reduction counts, recurrence segmentation, and random-number seeds.

For Python and Triton, the analyzer uses syntax-tree dependency tracking; for C/C++/CUDA, it applies targeted source patterns. Contracts can describe global rules and more specialized kernel-family restrictions.

This matters because an execution dependency can be suspicious even when a finite set of tests happens not to expose an output difference. Conversely, a runtime mismatch may reveal behavior outside the dependency patterns that static analysis recognizes.

When a candidate fails, BIEvolve sends diagnostics through the agent’s existing feedback channel. The agent can then produce another implementation within its normal search procedure.

The intervention changes which candidates are accepted and, consequently, which optimization trajectories the agent explores. It does not require changing the underlying model weights.

The measured reliability gain comes with a smaller performance cost

The main controlled comparison evaluates K-Search, Iterative Agent, and Evolve Agent with and without BIEvolve.

Each framework retains its native optimization procedure and search-turn budget: 20 turns for K-Search and 25 for the other agents. The reported generation configuration uses Gemini 3.7 Flash and an NVIDIA H100 NVL GPU.

Source: Section 4.2, Table 2. BI pass rates concern final correctness-passing kernels under the reported evaluation protocol. Speedup retention compares geometric mean kernel speedups against task references.

The overall improvement is 24.3 percentage points in BI pass rate. Meanwhile, geometric mean speedup declines from 28.91× to 26.54×, retaining 91.8% of the baseline figure.

This is a kernel benchmark trade-off. It does not mean end-to-end inference throughput declined by 8.2%.

The framework variation is consequential. K-Search and Iterative Agent retain less speedup, while Evolve Agent slightly exceeds its baseline geometric mean performance. The additional constraint therefore does not impose a uniform optimization penalty.

BIEvolve intervened in 30.8% of tasks overall, meaning at least one candidate was rejected during optimization. For Iterative Agent, the intervention rate reached 43.6%.

These results suggest that reliability constraints can materially reshape search behavior without necessarily eliminating competitive kernels. The observed trade-off remains specific to the tested frameworks, tasks, and budgets.

Static and dynamic checks catch different violations

A separate component ablation tests the detector rather than comparing final agent-generated solutions.

It uses 200 independently labeled candidates: 100 with confirmed BI violations and 100 without confirmed violations. All component configurations are evaluated against this shared pool.

Source: Section 4.3, Table 3; Appendix C.2.

Among the 100 confirmed violations, static analysis uniquely detects 37, dynamic validation uniquely detects nine, and both identify 46.

The ablation therefore supports the architectural choice to combine structural dependency checks with behavioral execution tests. Neither component subsumes the other.

The 2% false-rejection figure requires care: candidates labeled without confirmed violations have not necessarily been proven invariant. Similarly, the balanced pool was deliberately constructed for component comparison, not to estimate real-world violation prevalence.

The reported millisecond evaluation times also exclude generation, compilation, correctness testing, and performance benchmarking. They cannot be interpreted as total deployment or optimization overhead.

Kernel release policy should distinguish three forms of evidence

Cognaptus inference: Organizations using AI-generated GPU kernels in inference infrastructure should treat numerical correctness, batch invariance, and execution performance as separate acceptance dimensions.

That distinction changes the release workflow.

An infrastructure team approving an automatically optimized attention or reduction kernel can require its generation agent to report reference correctness, contract violations, tested cross-batch mismatches, and benchmark performance. Rejected candidates should retain the relevant dependency path or failing batch context so the optimization agent can attempt a targeted repair.

The team can then assess reliability-performance trade-offs at the kernel-family and agent-framework level rather than applying a single assumed performance penalty.

BIEvolve’s results justify investigating this workflow. They do not establish the production return on investment: integration expense, real serving workloads, and system-wide latency effects were not measured.

Passing the gate is not proof of deterministic inference

The formal batch-invariance requirement covers all admissible contexts and sample positions. The implemented dynamic validator checks only selected contexts and preserves the target sample at position zero.

Static analysis also has incomplete coverage of cross-file dependencies, opaque dispatch, and undeclared execution decisions.

Consequently, the reported 91.9% final-solution BI pass rate measures success under the study’s validation conditions, not universal mathematical invariance.

The experiment additionally uses one reported generation model and GPU configuration, with selected BI-relevant workloads. It does not measure downstream changes in token generation, complete LLM serving behavior, or reinforcement-learning outcomes.

The resulting assurance should be stated narrowly: the kernel passed defined structural checks and demonstrated bitwise consistency across the tested batch contexts.

The extra release criterion changes what optimization means

BIEvolve’s central contribution is not a new way to make kernels faster. It shows how an existing optimization agent can be redirected toward implementations that satisfy a numerical reliability constraint that conventional correctness testing may overlook.

The empirical result is substantial: higher tested batch-invariance pass rates with most geometric mean kernel speedup preserved. The component ablation also explains why structural analysis and runtime testing belong together.

For teams adopting agent-generated GPU code, the decision is whether batch-context consistency belongs in the release contract. This study provides evidence for adding that check while keeping its assurance conditional on the contexts, dependency rules, and hardware environment actually evaluated.

Cognaptus: Automate the Present, Incubate the Future.