The Model Got Smaller. The Risk Got Wider.

TL;DR for operators

Compression is usually sold as a clean engineering bargain: smaller model, lower memory, cheaper inference, acceptable accuracy loss. This paper asks the more operationally annoying question: after compression, does the model still know when it should hedge?

The answer is: not reliably. Tong et al. benchmark compressed LLMs using conformal prediction, a framework that converts model probabilities into prediction sets with target coverage.¹ In this setup, the important uncertainty metric is prediction set size: if the model needs to include more answer options to maintain coverage, it is less certain, even if its top-1 accuracy still looks respectable.

The paper’s practical message is not “never compress.” That would be tedious, and also wrong. The real message is that compression should be evaluated on two axes: task accuracy and uncertainty behavior. Weight-only 4-bit quantization with 16-bit activations, or W4A16, is usually the safest regime in the benchmark. Weight-and-activation 4-bit quantization, or W4A4, is much more disruptive. Pruning is more method-dependent: SparseGPT and Wanda are generally more stable than simple magnitude pruning, while structured pruning can be surprisingly effective in one architecture and quietly disastrous in another.

The most important business finding is the decoupling: a compressed model can keep accuracy near baseline while requiring larger prediction sets, or lose accuracy while prediction set size barely moves. Either case breaks the lazy assumption that “accuracy passed, therefore deployment risk passed.” Congratulations, the dashboard is green. The warning light was never wired.

The second finding is scale. Larger models tend to absorb compression-induced uncertainty better than smaller models. This matters because a smaller compressed model that barely meets an accuracy target may still be a worse operational choice than a larger compressed model with better uncertainty stability.

The third finding is threshold behavior. Uncertainty inflation is not always gradual. A model can look stable across moderate pruning levels, then suddenly need much larger prediction sets after another compression step. That means a single compression setting is not an evaluation. It is a coin toss wearing a lab coat.

The boundary is clear: this is a multiple-choice benchmark, with six answer options, a fixed 90% target coverage level, sampled calibration and test splits, and averaged prompting/scoring strategies. It does not prove the same behavior for open-ended generation, tool use, agent workflows, or domain-specific enterprise tasks. But it gives operators a useful diagnostic: do not compress the model and only ask whether it still gets the answer right. Ask whether it now needs a wider safety net to say so.

Compression is not only a cost knob

Every enterprise AI deployment eventually meets the same unpleasant spreadsheet. The full model is capable, but expensive. Latency is irritating. GPU memory is not a charitable institution. The obvious response is compression: quantize the model, prune the model, move the same product promise onto cheaper infrastructure, and hope the user never notices.

Most compression evaluations make this trade look deceptively clean. They ask whether the compressed model preserves accuracy. If accuracy stays close to the full-precision baseline, the compressed model is treated as deployment-ready. That is a reasonable first question. It is not a sufficient one.

A production model does not only choose answers. It also produces a confidence structure around those answers. In retrieval, triage, customer support, compliance review, medical pre-screening, contract analysis, and decision support, the important operational question is often not “Can the model answer?” It is “When should the model answer alone, when should it ask for evidence, and when should it escalate?”

Compression can disturb that layer without making the accuracy table scream. This is the paper’s central contribution: it makes the disturbance measurable.

Tong et al. do this by using conformal prediction. Instead of treating the model’s top answer as the whole story, conformal prediction builds a set of plausible labels. With an error rate $\alpha$, the goal is coverage close to $1 - \alpha$. In this paper, $\alpha = 0.1$, so the target coverage is 90%.

For an operator, the intuition is simple. If a six-option question can usually be answered with a prediction set of two labels, the model is giving a sharper signal than a model that needs five labels to maintain the same coverage. Both may satisfy the coverage target. One is more useful. The other is basically saying, “The answer is somewhere in this room.” Technically valid. Commercially awkward.

The paper therefore tracks three metrics:

The benchmark spans 12 LLMs, four model families, scales from 1B to 70B parameters, dense and Mixture-of-Experts architectures, five NLP tasks, and several compression regimes. The tasks are reformulated as six-option multiple-choice problems, using MMLU, CosmosQA, HellaSwag, HaluDial, and HaluSum. Each dataset contributes 2,000 sampled instances, split evenly between calibration and test. Results are averaged over two conformal scoring methods, LAC and APS, and three prompting strategies.

That setup matters because it prevents the paper from being just another leaderboard. It is not asking, “Which compression method wins?” It is asking, “Which compression method preserves the model’s ability to remain usefully uncertain?”

That is a better question. Also a more expensive one. Naturally, it was postponed until someone built a benchmark.

W4A16 behaves like compression; W4A4 behaves like a personality change

The first major comparison is between two quantization regimes.

W4A16 quantizes weights to 4-bit while keeping activations at 16-bit. W4A4 quantizes both weights and activations to 4-bit. The difference sounds like a technical footnote until the uncertainty results arrive with a small hammer.

In the task-averaged results, W4A16 is generally stable. Llama2-70B under AWQ reaches 73.20% accuracy with prediction set size 2.18, compared with the FP16 baseline at 72.48% accuracy and set size 2.17. Qwen3-8B under AWQ is similarly close to baseline: 69.61% accuracy and 2.68 set size, versus 70.44% and 2.68 for the dense model.

Those are the kind of numbers that make infrastructure teams quietly happy. Lower memory, similar accuracy, similar uncertainty. Nobody needs to perform interpretive dance in front of the risk committee.

W4A4 is different. On Llama2-7B, QuaRot drops average accuracy from 47.09% to 22.04% and raises set size from 3.09 to 5.65. That is not graceful degradation. That is the model asking to select nearly the entire six-option menu while also getting much worse at choosing the entrée.

The same method is less damaging on larger models. Llama2-70B under W4A4 QuaRot still drops from 72.48% to 66.49% accuracy, but its set size rises only from 2.17 to 2.54. The model is hurt, but it does not collapse into broad uncertainty the way the 7B version does.

This is the first practical comparison: not all 4-bit compression means the same thing. Weight-only quantization can be a relatively disciplined deployment tool. Activation quantization is a more aggressive intervention. The paper’s results do not say W4A4 is useless. They say it is not a casual default for uncertainty-sensitive systems.

The ablation on Llama2-7B strengthens the point. W8A8 and W6A6 settings remain relatively tolerable compared with W4A4. The FP16 baseline averages 47.09% accuracy and set size 3.09. W8A8 QuaRot slightly raises average accuracy to about 47.20%, though set size increases to about 3.33. W8A8 SmoothQuant reduces accuracy modestly to about 45.70% and keeps set size near 3.16. W6A6 QuaRot gives about 45.36% accuracy and 3.35 set size. But W4A4 SpinQuant drops to 26.26% accuracy and 4.52 set size.

The likely purpose of this ablation is not to introduce a second thesis about every possible precision level. It isolates the precision threshold. The broader lesson is that uncertainty degradation is not merely a smooth function of fewer bits. At some point, the representation damage changes regime.

This is exactly the kind of result operators need before they ship a “cost optimized” model into a workflow where abstention, escalation, or review routing matters.

Accuracy and uncertainty do not fail in the same direction

The paper’s most important finding is also the easiest one to misunderstand.

A reader might expect accuracy and uncertainty to move together. If compression damages the model, accuracy should fall and uncertainty should rise. If accuracy remains stable, uncertainty should remain stable. That belief is tidy. Reality has declined the invitation.

The paper shows that compression often decouples accuracy from prediction set size. This is visible in the scatter comparison of accuracy change versus set-size change across models, tasks, and compression methods. If the two metrics were tightly linked, the points would form a consistent pattern. Instead, they scatter.

Two cases matter operationally.

First, accuracy can remain close to baseline while uncertainty expands. For Llama2-13B on commonsense inference, W4A16 AWQ increases accuracy from 59.63% to 61.17%, while set size rises from 2.83 to 3.13. That is not a disaster. It is worse: it is easy to miss. The compressed model looks fine if you only check accuracy. But it now needs a wider conformal set to maintain reliability.

Second, accuracy can drop sharply while set size barely changes. For Llama3.1-8B under Wanda pruning on question answering, accuracy drops from 62.93% to 45.97%, while set size changes only from 2.98 to 3.09. In other words, the model becomes much less correct without proportionally broadcasting greater uncertainty.

Both patterns are dangerous in different ways.

The useful business interpretation is not that set size replaces accuracy. It is that accuracy alone cannot describe deployment readiness. Accuracy tells you whether the model got the selected answer right. Set size tells you how much uncertainty the system must carry to preserve coverage.

Those are different costs. A model with acceptable accuracy but inflated set size may still increase human review load. A model with falling accuracy and stable set size may require stricter routing because it is no longer self-separating hard cases properly. The spreadsheet that only tracks inference cost and benchmark accuracy will miss both.

This is where conformal prediction becomes practical rather than decorative. It converts uncertainty into an operational quantity: how many plausible labels does the model need to keep on the table?

Larger models absorb compression better, but scale is not magic

The paper’s second comparison is scale. Larger models tend to absorb compression-induced uncertainty better than smaller ones.

The cleanest example is Llama2 under W4A4 QuaRot. Average set-size inflation falls sharply as model scale increases:

The same qualitative pattern appears at task level. On question answering, Llama2-7B rises from set size 3.20 to 5.73 under W4A4 QuaRot. Llama2-13B rises from 3.10 to 3.36. Llama2-70B rises from 2.64 to 2.80.

The interpretation is not mysterious. Larger models have more redundancy, richer internal representations, and more capacity to absorb approximation noise. Compression damages them too, but it often does not push them over the same uncertainty cliff.

For business deployment, this complicates the simplistic “smallest model that passes accuracy” strategy. A compressed 7B model may be cheaper than a compressed 70B model, but if it inflates uncertainty enough to increase review load, fallback calls, retrieval retries, or user friction, the apparent infrastructure saving can leak elsewhere.

This is not a moral argument for giant models. It is a cost-accounting argument. The unit economics of inference do not end at tokens per second. They include escalation rate, abstention quality, wrong-answer risk, and how often downstream systems must compensate for degraded confidence.

Still, scale is not a universal shield. The paper notes that Llama3.1-70B under magnitude pruning shows a large commonsense-inference set-size increase. In the pruning table, Llama3.1-70B on commonsense inference moves from 81.33% accuracy and 1.89 set size in the dense baseline to 46.47% accuracy and 3.25 set size under 50% magnitude pruning. Large model, ugly result.

The replacement belief should be precise: scale improves compression robustness on average, especially under aggressive quantization, but method and task still matter. A 70B model is not a sacred animal. It is just harder to break.

Pruning is not one method; it is a family argument

Quantization has a relatively clear pattern in the benchmark: W4A16 is generally safer; W4A4 is generally more disruptive. Pruning is messier.

The paper evaluates unstructured pruning at 50% sparsity using magnitude pruning, SparseGPT, and Wanda. It evaluates structured pruning at 20% sparsity using LLM-Pruner and SliceGPT.

Simple magnitude pruning is usually the weakest. SparseGPT and Wanda are generally more stable, but their behavior depends on model scale, task, and architecture. Structured pruning is even more volatile because it removes larger components rather than individual weights. That can produce real efficiency benefits, but it can also remove parts of the model that matter disproportionately for uncertainty behavior.

Consider Llama2-70B. Under Wanda pruning, average accuracy is 66.63% and set size is 2.42, compared with the dense baseline at 72.48% and 2.17. The model degrades, but not catastrophically. On Llama2-7B, Wanda drops average accuracy from 47.09% to 31.13% and raises set size from 3.09 to 3.58. Same method, less forgiving scale.

Structured pruning is more dramatic. LLM-Pruner gives Llama3.1-70B an average 74.25% accuracy and 2.13 set size, not far from the dense baseline of 77.60% and 1.92. But SliceGPT on Qwen3-8B produces a degenerate result in the main table, marked as unreliable. In the detailed Qwen3 dense-model pruning table, SliceGPT also causes severe collapses for larger Qwen dense models: for Qwen3-32B, several task accuracies fall near the low 20s, with set sizes around 4–5.

This is not a contradiction. It is the point. Pruning methods encode assumptions about which parameters, rows, columns, heads, neurons, or structures are safe to remove. Those assumptions may preserve top-line accuracy in one architecture and disrupt uncertainty in another.

For operators, the decision rule should be boring and strict:

The unpleasant lesson: “50% sparse” or “4-bit” is not a reliability specification. It is a compression description. Reliability still has to be measured.

The dangerous part is the cliff, not the slope

The third comparison is compression intensity. Operators often assume that more compression creates gradually more damage. Ten percent pruning should be mild. Thirty percent should be more visible. Fifty percent should be worse. This is broadly plausible, and often operationally false in the detail that matters.

The paper tests progressive Wanda pruning from 0% to 50% sparsity. The likely purpose of this experiment is a robustness and sensitivity test: it asks whether uncertainty changes smoothly as compression increases or whether there are tipping points.

The result is threshold-like behavior.

For Llama3.1-8B on reading comprehension, set size moves only from 1.89 at baseline to 2.06 at 40% sparsity, then jumps to 2.53 at 50%. On commonsense inference, it moves from 2.80 at baseline to 2.63 at 30%, then 3.04 at 40% and 3.42 at 50%. On dialogue response selection, it is 2.68 at baseline, falls slightly to 2.54 at 40%, then jumps to 3.16 at 50%.

Llama2-7B shows a similar pattern on reading comprehension. Its set size rises from 2.46 at baseline to 2.89 at 40%, then jumps to 3.48 at 50%. The last 10 percentage points of sparsity contribute more than half of the total set-size inflation from baseline to 50%.

Llama2-70B is flatter. On reading comprehension, its set size moves from 1.79 at baseline to 1.94 at 50%. That is the scale effect again. But even large models can show task-specific cliffs: Llama3.1-70B on commonsense inference rises from 1.89 at baseline to 2.21 at 30%, then 2.66 at 40% and 3.16 at 50%.

This matters because many compression evaluations test a single convenient operating point. That is not enough. If the uncertainty curve has cliffs, a model that looks safe at 40% pruning can become operationally unattractive at 50%. The difference may not be a gentle degradation; it may be the point where the conformal set widens abruptly.

A compression pipeline should therefore sweep compression ratios. Not because researchers enjoy extra tables, though they do, but because threshold behavior changes the deployment decision. The safe operating point is not the highest compression ratio that preserves accuracy. It is the highest compression ratio before uncertainty behavior becomes commercially inconvenient.

The MoE case is architecture-specific, not a free lunch

The paper includes Qwen3-30B-A3B as a Mixture-of-Experts case. This is an exploratory extension relative to the main dense-model comparisons. MoE models activate only a subset of experts per input, so compression can interact with routing, expert capacity, and task-specific specialization.

The results are not clean enough for a slogan, which is good. Slogans are where nuance goes to die.

The dense MoE baseline averages about 46.52% accuracy and 3.32 set size across the five tasks. Under W4A16 RTN, average accuracy is about 45.64% and set size improves to about 3.09. That looks stable. AWQ is worse, reducing average accuracy to about 37.17% and increasing set size to about 3.59. GPTQ appears strong on average, with about 48.40% accuracy and 2.75 set size, but the document summarization entry has only 12.68% coverage and a degenerate set size of 1.00, so that average is not trustworthy as a deployment signal.

Pruning is more polarized. Magnitude and Wanda preserve or improve average accuracy in some respects but increase set size. SparseGPT lowers accuracy while keeping set size closer to baseline. Structured pruning is the most dramatic contrast: LLM-Pruner improves both average accuracy and set size in the MoE table, while SliceGPT collapses to about 16% average accuracy and inflates set size to roughly 5.57.

This section should not be read as “LLM-Pruner is magic for MoE.” The safer interpretation is narrower: MoE compression is highly method-dependent, and aggregate compression descriptors do not predict reliability. Routing architecture changes the failure surface. If a business is using MoE models, dense-model compression assumptions should not be copy-pasted into production.

Copy-paste remains undefeated as the cheapest path to expensive mistakes.

What the paper directly shows versus what operators should infer

The paper directly shows that, in this benchmark:

The inference for business practice is straightforward: compression should become a governed model-change process, not a one-line infrastructure optimization. A compressed model is a new model from the standpoint of risk behavior. It deserves re-evaluation of confidence, escalation thresholds, and review economics.

This is especially true when uncertainty controls workflow routing. If the model’s prediction sets become larger, more cases may need human review or additional evidence. If accuracy drops without set size expanding, the model may be under-signaling its degradation. Either way, the business process changes.

The model may fit on cheaper hardware. The uncertainty may not fit inside the old operating procedure.

A practical compression evaluation checklist

The paper implies a useful evaluation sequence for teams deploying compressed LLMs in decision-support settings.

First, keep the full-precision model as the reference point. Measure accuracy, coverage, and prediction set size before compression. A compressed model should be compared against this baseline, not judged in isolation.

Second, evaluate W4A16 before more aggressive regimes. The benchmark suggests that weight-only 4-bit quantization with 16-bit activations is often the most uncertainty-preserving option. It may not be the cheapest possible configuration, but it is a sensible starting point when reliability matters.

Third, do not approve W4A4 only because the accuracy loss is “acceptable.” Check set size. The paper shows that activation quantization can materially widen uncertainty, especially for smaller models.

Fourth, treat pruning methods as separate candidates. Magnitude pruning, SparseGPT, Wanda, LLM-Pruner, and SliceGPT should not be bundled under one pruning label. They remove different things, and the uncertainty consequences differ.

Fifth, sweep compression ratios. This is non-negotiable when pruning is involved. The paper’s progressive Wanda results show that the difference between 40% and 50% sparsity can be much larger than the difference between 10% and 30%.

Sixth, evaluate by task type. A model can be stable on question answering but fragile on commonsense inference or summarization-style selection. Average metrics are useful for screening. They are not sufficient for workflow design.

Seventh, translate set size into operating cost. A set-size increase is not just an academic uncertainty metric. It may mean more review, more abstention, more fallback retrieval, more user clarification, or lower automation coverage.

That last step is where many model evaluations stop too early. They report benchmark degradation and call it a day. Operators need to know whether the degradation changes staffing, latency, compliance posture, or customer experience.

The boundary: this is a useful diagnostic, not a universal deployment certificate

The paper’s limits are important, but they should not be inflated into dismissal.

The benchmark uses multiple-choice tasks. That is necessary for the conformal setup used here, but it does not cover free-form generation. Open-ended generation has different uncertainty problems: semantic equivalence, sequence-level uncertainty, hallucination risk, tool-use uncertainty, retrieval uncertainty, and answer usefulness. A model can behave well under six-option classification and still produce confident nonsense in long-form generation. Enterprise AI, being enterprising, likes to do both.

The target coverage is fixed at 90%, with a 50/50 calibration-test split. Different coverage levels, calibration sizes, or domain distributions may change the observed trade-offs. The paper also samples 2,000 instances per dataset due to computational cost. That is reasonable for a large benchmark, but production teams should not treat the paper’s numeric thresholds as universal constants.

The tasks are standardized with added “I don’t know” and “None of the above” options. This helps compare uncertainty across tasks, but it is still a stylized evaluation environment. Real workflows often have messy labels, missing context, ambiguous goals, and users who consider “None of the above” an invitation to open a support ticket.

There is also a conformal assumption: calibration and test data should be exchangeable. In production, distribution drift is common. If the calibration data no longer resembles current traffic, coverage guarantees weaken. That does not make conformal prediction useless. It means calibration governance becomes part of the deployment process.

So the correct boundary is this: the paper does not certify any compression method for all LLM use. It demonstrates that compression can alter uncertainty behavior in measurable, decision-relevant ways, and it gives a practical framework for detecting that change.

That is enough to change how compression should be evaluated.

The conclusion is not “compress less.” It is “measure the thing you are about to break.”

The easy article would end with “uncertainty matters.” It does. Everyone now nods solemnly. Very productive.

The sharper conclusion is this: compression changes the model’s risk interface. It can preserve top-line accuracy while widening the conformal set needed to maintain coverage. It can reduce accuracy without producing enough extra uncertainty. It can behave gently until a compression threshold is crossed. It can be absorbed by scale in one family and exposed by task-specific fragility in another.

That means model compression is not just an inference optimization. It is a reliability intervention.

For businesses, the immediate move is modest: add conformal prediction set-size checks to compression evaluation where tasks can be framed as classification or selection. Use W4A16 as a safer default candidate, not as a sacred rule. Treat W4A4 and pruning as method-specific changes requiring uncertainty validation. Sweep compression levels. Compare dense and compressed models not only on accuracy but on the cost of uncertainty.

The model got smaller. Good. The bill may go down.

But if the prediction set got wider, the risk did not disappear. It was merely moved from the GPU budget into the operating model. Very efficient. Very modern.

Cognaptus: Automate the Present, Incubate the Future.