Four Bits, Fewer Failure Modes
TL;DR for operators Post-training quantization methods that look similar on normal 4-bit benchmarks can separate sharply when the numerical problem becomes difficult. In one Mistral-7B v0.1 stress test, QuIP-family variants abort with NaNs and GPTQ reaches WikiText-2 perplexity of 379.73 ± 15.71. Kashin-DCT+H reaches 8.92 ± 0.01, close to the FP16 reference of 8.63. ...