FP32

TL;DR for operators Greedy decoding removes random token sampling, but it does not guarantee identical responses across numerical configurations. Du and colleagues1 show that the same model, prompt, and GPU can produce different token sequences under BF16 and FP16 inference, even when repeated runs within one precision are identical. The decisive mechanism is often a small difference between the scores of competing tokens. When numerical perturbations reverse their ranking, one changed token can redirect subsequent generation. Selectively recomputing the final output projection in FP32 improves exact sequence agreement by 22–36 percentage points in selected configurations, with less than 4% latency overhead in the reported low-batch experiments. ...