Precision Has a Map: TileMix Routes INT8 Inside Dense Attention
TL;DR for operators Long-context prefill creates an uncomfortable infrastructure choice. Lowering attention precision can reduce compute cost, but applying INT8 everywhere may degrade retrieval and question-answering quality. Sparse attention attacks the same cost problem differently by avoiding some interactions altogether. TileMix proposes another control surface: keep every legal attention interaction, but choose which regions of the attention-score computation run in FP16 and which run in INT8. The paper1 reports that this mixed routing can recover part of the quality lost under uniform INT8 while retaining much of the throughput advantage. On an NVIDIA A100 40GB, LLaMA 3.2 3B-Instruct at 4k tokens reaches 31.80 K tokens/s with the SpTrans75 configuration, versus 14.33 K tokens/s for FlashAttention and 29.80 K tokens/s for the matched all-INT8 path. ...