Greedy Decoding Has a Precision Clause

TL;DR for operators

Greedy decoding removes random token sampling, but it does not guarantee identical responses across numerical configurations. Du and colleagues¹ show that the same model, prompt, and GPU can produce different token sequences under BF16 and FP16 inference, even when repeated runs within one precision are identical.

The decisive mechanism is often a small difference between the scores of competing tokens. When numerical perturbations reverse their ranking, one changed token can redirect subsequent generation. Selectively recomputing the final output projection in FP32 improves exact sequence agreement by 22–36 percentage points in selected configurations, with less than 4% latency overhead in the reported low-batch experiments.

For operators, numerical precision belongs in the inference configuration used for reproducibility testing and audit replay. The correction is worth evaluating when output-head arithmetic is a substantial source of disagreement. It does not guarantee identical outputs, generalize uniformly across hardware and batch sizes, or establish greater task accuracy.

The same model can reproduce itself but disagree with another deployment

Suppose an organization evaluates a language model, records its responses, and later moves inference to another serving configuration. The model checkpoint, prompts, and decoding settings remain unchanged. The new deployment uses greedy decoding, selecting the highest-scoring token at every step without sampling.

Should the recorded responses be reproduced exactly?

Greedy decoding is deterministic with respect to the token scores it receives and a fixed tie-breaking rule. It does not guarantee that different numerical implementations produce identical scores. Two independently repeatable serving configurations can therefore generate different responses.

The researchers isolate this problem by comparing inference under two 16-bit floating-point representations, BF16 and FP16. These formats represent numbers differently, affecting stored model values and intermediate calculations.

In their primary TinyLlama-1.1B experiment, repeated BF16 inference achieved 100% within-format consistency. Yet BF16 and FP16 produced identical complete sequences on only 41% of 100 GSM8K prompts, with the checkpoint, prompt, decoding algorithm, and GPU held fixed.

This was not an isolated result. Cross-precision divergence affected 49–100% of prompts across the reported model and benchmark evaluations. On TinyLlama, disagreement occurred on 59% of GSM8K prompts, 64% of HumanEval prompts, and 70% of MBPP prompts.

The study measures Exact Agreement Rate (EAR): the percentage of prompts for which two configurations generate identical complete token sequences. A single differing token is enough to classify a response pair as divergent.

The consequences can extend beyond one position. Among the 59 divergent TinyLlama GSM8K prompts, 24 produced different sequence lengths, with an average absolute length difference of approximately 34 tokens and a maximum of 236.

Once a token changes, subsequent generation is conditioned on a different sequence. Precision-sensitive decisions can consequently produce much larger downstream differences than the original arithmetic perturbation.

Token-ranking margins explain why numerical error sometimes matters

A transformer assigns a numerical score, or logit, to every possible next token. Greedy decoding selects the highest score.

The difference between the two highest scores is the top-two logit margin. A large margin makes a token decision relatively resistant to small perturbations. A narrow margin makes the ranking easier to reverse.

The paper formalizes this relationship.

Let $v^{(1)}$ be the original highest-scoring token, $z(v)$ its reference logit vector, and $\Delta z(v)$ the perturbation caused by changing numerical configuration. Under the proposition’s unique-maximum and deterministic tie-handling assumptions, the original winner changes exactly when at least one competing token satisfies:

The left side measures how much a competitor gains relative to the original winner. The right side is the gap it must overcome.

The condition applies to every competing token, not necessarily just the original runner-up. It establishes an exact algebraic criterion for a ranking reversal; it does not provide a universal bound on floating-point error.

The experimental traces reinforce the importance of decision margins.

Source: Section 7.5, Table 7; divergence-conditioned numerical tracing.

The accumulated numerical differences are comparable between the groups and are actually slightly larger in the agreeing group under these measurements. Their token-ranking margins, however, differ by approximately 150 times.

This explains why monitoring total numerical error alone is insufficient for identifying unstable greedy decisions. Error direction and distance from the token decision boundary determine whether the selected token changes.

The researchers then examine where the perturbations originate. Layer-level tracing, activation grafting, KV-cache substitution, and logit-lens analysis point to the importance of late computational stages, particularly the lm_head—the projection converting the final transformer representation into vocabulary scores.

These experiments support the output head as a significant source of correctable disagreement in some configurations. They do not show that all numerical differences originate there. Transformer-body arithmetic, accumulated cache differences, and weight-storage precision also contribute.

The paper’s estimates of error propagation and relative output-head dominance rely on simplifying assumptions about residual connections, rounding errors, and their accumulation. Unlike the directional token-flip condition, these quantitative estimates are heuristic.

Correct the vulnerable token decisions, not every calculation

The proposed intervention leaves ordinary inference unchanged unless the top-two logit margin falls below a calibrated threshold.

At such a step, the system recomputes the final output projection in FP32, a higher-precision format, before selecting the next token. The model is not retrained, and its transformer-body computation remains in the original precision.

The authors compare integer-logit quantization, selective top-K FP32 recomputation, and selective full-vocabulary FP32 output-head recomputation.

The primary results show a substantial improvement in exact agreement:

Source: Section 6.3, Table 3; TinyLlama-1.1B, 100 prompts per benchmark, margin threshold $\tau=0.001$.

The gate activates on approximately 0.7–1.4% of decoding steps in the primary experiments. The measured A10G overhead for gated full-vocabulary recomputation is 1.4% additional latency and 11% higher peak GPU memory use.

A top-K alternative achieves the same agreement rates on these benchmarks with less than 2% reported overhead. The tested ablations find that recomputing the top two candidates is sufficient in the evaluated gated configurations, although this is not a universal guarantee.

The comparison also reveals why increasing precision indiscriminately can perform worse on the agreement objective.

Integer quantization alone leaves agreement unchanged. Recomputing the FP32 output head at every step, without the margin gate, achieves only 60% GSM8K agreement rather than 63%, while taking approximately 2.5 times the baseline latency. Broader precision interventions can also introduce additional changes to autoregressive trajectories.

Higher arithmetic precision does not necessarily mean greater agreement between two already distinct inference paths. Selective recomputation limits changes at high-margin positions while targeting decisions most vulnerable to numerical perturbations.

Correction effectiveness changes with the inference stack

The mechanism suggests a specific applicability condition: correcting the output head is most effective when its numerical error materially contributes to token-ranking reversals. If two configurations have already developed substantially different hidden states, output-head recomputation cannot generally reconcile them.

The cross-model experiments demonstrate the resulting variability.

On A10G GSM8K, Intervention C improves exact agreement by 36 percentage points for Llama-3.2-3B, 22 for TinyLlama, 10 for OLMoE, eight for Mistral-7B, three for Qwen2.5-3B, and zero for DeepSeek-R1-Distill-Qwen-7B.

These are model-specific experimental outcomes, not a monotonic relationship with parameter count. The researchers observe severe early divergence and unstable FP16 intermediate activations in some Qwen-family configurations. Their proposed connection to training-time numerical stability remains a hypothesis rather than an experimentally identified cause.

Hardware and batching also change the outcome. TinyLlama’s measured improvement is 22 percentage points on A10G, 21 on L4, and 12 on A100. A separate batch-size robustness test finds improvements of approximately 17, seven, seven, and zero percentage points at batch sizes one, two, four, and eight.

The FP8 experiments provide a particularly informative mechanism test. When the transformer body is held fixed in BF16 and only output-head precision changes, a suitably adjusted margin threshold produces 100% agreement on TinyLlama. But when FP8 quantization extends through every linear layer, output-head-only correction improves agreement by just one percentage point.

The head-isolated experiment tests whether the proposed mechanism can work under controlled upstream conditions. The end-to-end experiment establishes why that result should not be generalized to ordinary FP8 serving.

Exact agreement and task correctness require separate measurements

Exact token agreement is a demanding reproducibility metric, but it cannot directly measure whether two responses accomplish the same task.

For Qwen2.5-3B on the full HumanEval benchmark, 73.8% of BF16 and FP16 generations differ at the token level, yet execution outcomes agree on 97.6% of problems. Many different code sequences therefore lead to the same correctness outcome under the benchmark tests.

The reverse concern also appears: similar aggregate accuracy can conceal changes in which individual prompts are answered correctly. In a separate Qwen2.5-3B GSM8K comparison, aggregate accuracies are 27% and 28% across BF16 and FP16, while correctness changes on 19 of 100 prompts.

These results make paired, per-example evaluation more informative than aggregate accuracy alone when an organization changes serving precision.

The intervention’s quality tests do not show statistically significant degradation in the reported comparisons. A 600-prompt Qwen2.5-3B non-inferiority analysis supports a five-percentage-point margin, with measured accuracy moving from 11.7% to 11.0%. This supports a limited non-inferiority claim, not proof of zero accuracy effect across workloads.

The study also reports improved agreement with FP32 reference trajectories, although unmodified FP16 remains closer to FP32 than corrected BF16 in the tested comparisons. Greater agreement between two serving formats should not be interpreted as verified correctness or universal fidelity to a reference.

Reproducibility needs configuration-level controls

For organizations using exact output replay, the immediate operational implication is to treat inference configuration as part of the reproducible artifact.

Checkpoint identity, prompt formatting, numerical format, GPU implementation, kernels, batch configuration, and deterministic execution settings all deserve versioned records. Preserving only the model identifier and generation parameters is insufficient for asserting exact cross-environment replay.

For teams evaluating numerical corrections, a defensible validation procedure is to establish within-configuration repeatability, measure paired cross-configuration agreement, identify low-margin decisions, and test gated recomputation against latency, memory, and task-correctness requirements. Calibration should be repeated when hardware, quantization scope, or batching changes.

The evidence is strongest for controlled greedy inference on the evaluated models, primarily at or below 7B parameters and at low batch sizes. The larger 12.2B model receives divergence characterization but not the full intervention comparison. Sampling-based decoding, much larger systems, and unrestricted production workloads remain outside the demonstrated effectiveness range.

For systems requiring bit-exact historical replay, pinning the complete execution configuration remains more defensible than relying on an intervention that only partially reconciles numerical trajectories.

The study ultimately establishes a narrower but more actionable finding than universal determinism: greedy decoding can be repeatable within an inference configuration while remaining sensitive to numerical changes across configurations. Its proposed correction works by targeting the token decisions where numerical perturbations can change the selected output, provided the relevant error remains sufficiently localized.

That makes reproducibility a property to measure and validate for a particular deployment, rather than something guaranteed by choosing greedy decoding.

Cognaptus: Automate the Present, Incubate the Future.