Mixed Precision

TL;DR for operators A fixed MoE memory budget does not tell you which experts can safely absorb the lowest precision. Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees1 argues that the experts most frequently or strongly used are not necessarily the ones that need the most bits. In its theory, experts associated with less-prevalent but still task-relevant features develop weaker activations and smaller router-norm changes, leaving less margin for quantization error. ...

TL;DR for operators Long-context prefill creates an uncomfortable infrastructure choice. Lowering attention precision can reduce compute cost, but applying INT8 everywhere may degrade retrieval and question-answering quality. Sparse attention attacks the same cost problem differently by avoiding some interactions altogether. TileMix proposes another control surface: keep every legal attention interaction, but choose which regions of the attention-score computation run in FP16 and which run in INT8. The paper1 reports that this mixed routing can recover part of the quality lost under uniform INT8 while retaining much of the throughput advantage. On an NVIDIA A100 40GB, LLaMA 3.2 3B-Instruct at 4k tokens reaches 31.80 K tokens/s with the SpTrans75 configuration, versus 14.33 K tokens/s for FlashAttention and 29.80 K tokens/s for the matched all-INT8 path. ...