LLM Compression

TL;DR for operators Compression is usually sold as a tidy pipeline: pick a smaller architecture, prune some layers, quantize the result, then call procurement and explain why the GPU bill is still rude. This paper argues that the pipeline itself is the problem.1 The authors propose a joint compression framework for Llama-3.1-8B that searches architectural choices and quantization choices together. That means the system does not first decide “how much model” it wants and only afterward decide “how many bits” each part deserves. It treats width, depth, layer importance, weight precision, activation precision, and latency as interacting deployment variables. ...

TL;DR for operators Compression is usually sold as a clean engineering bargain: smaller model, lower memory, cheaper inference, acceptable accuracy loss. This paper asks the more operationally annoying question: after compression, does the model still know when it should hedge? The answer is: not reliably. Tong et al. benchmark compressed LLMs using conformal prediction, a framework that converts model probabilities into prediction sets with target coverage.1 In this setup, the important uncertainty metric is prediction set size: if the model needs to include more answer options to maintain coverage, it is less certain, even if its top-1 accuracy still looks respectable. ...

Packing is easy until one object is much larger than everything else. A warehouse can fit hundreds of ordinary boxes onto neatly spaced shelves. Add one grand piano, however, and the spacing plan becomes rather less elegant. Either the piano does not fit, or every shelf is redesigned around an object that appears once. ...

Cost has a way of making architecture less romantic. In diagrams, a Transformer block looks clean: attention mixes tokens, the MLP transforms features, residual connections keep information flowing. In deployment, the same diagram becomes an invoice. Attention is especially expensive because its cost grows with sequence length. In the paper’s LLaMA-7B timing example, an attention layer has roughly half the parameters of an MLP layer, yet runs nearly twice as long at sequence length around 3,000 and about three times as long around 7,000. Attention is elegant. It is also very good at charging rent. ...

Compression needs a rulebook, not just a diet plan Compression is the least glamorous part of the LLM business until the bill arrives. A model works beautifully in a cloud demo. Then someone asks whether it can run on a device with limited memory, limited energy, limited connectivity, and limited patience. Suddenly the elegant system becomes a logistics problem. Quantize it. Prune it. Shrink it. Hope it still speaks like the original model and not like a sleep-deprived intern summarizing a legal contract from memory. ...