TL;DR for operators
Combining more compression techniques can make a small language model unusable without producing additional deployment savings. In a controlled study, all 16 tested configurations that applied 4-bit NF4 quantization before unstructured pruning collapsed. Reversing the operation order preserved viable alternatives. Compression sequence therefore belongs in model validation, not merely pipeline implementation.
Recovery is possible but has limits. LoRA adaptation reduced pruning-induced perplexity by as much as 97.4%, yet the recovered model remained worse than its uncompressed baseline. A separate ablation found no consistent benefit from adding self-distillation over ordinary LoRA fine-tuning.
Deployment measurements also separate theoretical compression from realized efficiency. INT8 quantization reduced GPU memory consumption by 40–45% with less than 0.5% average benchmark accuracy degradation. On CPU, GGUF q4 inference generated tokens 46–54% faster than q8, while unstructured pruning delivered no measurable memory or throughput advantage under dense execution.
For deployment engineers, the selection process should be sequential: establish model viability, verify task accuracy, and measure actual resource improvements on the intended inference backend before adding compression complexity.
Adding compression steps can destroy the model
A team deploying a language model within a fixed memory budget faces several apparently compatible options. It can represent weights with fewer bits, remove low-importance parameters, and fine-tune the resulting model to recover lost quality. Each operation has a plausible efficiency objective.
The difficulty is that these operations change the inputs to subsequent operations. A compressed model may already have lost information that another transformation expects to preserve. Even when the resulting model remains accurate, its numerical compression may not translate into lower runtime memory consumption or faster inference.
Bhushit Nema, Siddharth Krishnamurthy, and Anjali Kadiyala examine these interactions in Compression Technique Interactions in Small Language Models: A Factorial Ablation Study.1
Their primary experiment varies three factors: quantization at FP16, INT8, or NF4 INT4 precision; unstructured magnitude pruning at 0%, 30%, or 50%; and post-compression LoRA recovery enabled or disabled. Four models, ranging from Phi-2 at 2.7 billion parameters to Qwen2.5-7B-Instruct, produce 72 primary configurations. Another 48 runs investigate compression ordering.
The main experiment classifies a configuration as collapsed when its WikiText-2 perplexity reaches or exceeds 1,000. Seventeen of the 72 configurations cross this threshold. Sixteen belong to one particular combination: NF4 quantization applied before 30% or 50% unstructured pruning, with or without recovery.
The finding concerns an interaction between operations, rather than an inherent inability of either technique to work individually.
Compression ordering changes model viability
Quantization reduces the numerical precision used to store model weights. NF4 is a particular 4-bit representation. Unstructured magnitude pruning sets selected low-magnitude weights to zero, attempting to remove relatively less important parameters.
Both transformations discard information. Their order determines the representation on which the second operation acts.
The ordering experiment provides the decisive comparison. Applying pruning before NF4 quantization produces viable configurations with reported perplexity around 20–21. Applying NF4 first and pruning afterward causes collapse in the tested combinations.
That distinction matters when interpreting another reported result: the ordering analysis finds no statistically significant perplexity difference among viable configurations (ANOVA F = 0.099, p = 0.96).
There is no contradiction. The statistical comparison concerns model quality conditional on remaining viable. Collapsed models are excluded from the averages of surviving configurations. Consequently, a nonsignificant difference among survivors does not establish that ordering has no effect on failure risk.
The authors interpret the collapse as a possible consequence of successive information loss: coarse weight representation followed by magnitude-based removal leaves insufficient effective model information. This is plausible, but the study does not isolate that mechanism through a dedicated representation-level experiment.
For compression-pipeline design, the observed failure boundary is sufficient to justify testing operation sequences independently. It does not establish that every 4-bit quantization method or pruning algorithm will exhibit the same behavior.
LoRA can repair substantial damage without restoring baseline quality
Pruning damage is not always permanent.
Low-Rank Adaptation (LoRA) introduces small trainable matrices to adapt a model without updating its full original parameter set. The study applies recovery using rank-8 adapters, 200 training steps, and 256 C4 samples.
For Qwen2.5-7B, 50% FP16 pruning raises perplexity to 920.6. LoRA recovery reduces it to 24.3, a 97.4% reduction.
The magnitude is striking, but the reference point matters. The original model’s baseline perplexity is approximately 11.8. Recovery brings a severely damaged model back to a usable range, not back to its original quality.
Other reported recovery reductions are 70.9% for Qwen3-4B, 56.0% for Nemotron-Mini-4B, and 43.2% for Phi-2 under their respective tested pruning conditions. Recovery effectiveness is therefore architecture- and configuration-dependent.
The study also isolates the contribution of self-distillation, which trains the recovered model to match outputs from an uncompressed teacher. In a separate full-corpus ablation, ordinary LoRA cross-entropy fine-tuning without a teacher performs comparably.
For example, teacher-free LoRA achieves perplexity of 39.71 versus 46.45 with self-distillation for Qwen3-4B; for Qwen2.5-7B, the corresponding values are 27.48 and 28.80. Self-distillation performs better on the other two tested models, so the result is not universal teacher-free superiority.
Cognaptus inference: when recovering a pruned model, an engineering team should benchmark LoRA-only adaptation before provisioning a teacher-based distillation workflow. The experiment does not establish the total training-cost difference, but it gives no consistent quality justification for the additional teacher signal under the tested settings.
Deployment efficiency depends on the inference backend
A model can have fewer effective weights without occupying less memory during execution.
The distinction becomes apparent in the study’s deployment measurements.
On the NVIDIA L40S GPU, INT8 quantization alone reduces memory consumption by 40–45%, with less than 0.5% degradation in average accuracy across MMLU, HellaSwag, ARC-Challenge, and BoolQ. This is a comparatively conservative option when GPU memory is the binding constraint.
On CPU, the researchers convert models to GGUF and evaluate them using llama.cpp. Their q4 inference configurations achieve 46–54% higher generation throughput than q8 across the four models, with differences below 0.5 perplexity points in the reported within-engine comparisons.
| Model | q4 tokens/s | q8 tokens/s | q4 throughput gain |
|---|---|---|---|
| Phi-2 | 17.5 | 11.5 | 52% |
| Qwen3-4B | 12.5 | 8.1 | 54% |
| Nemotron-Mini-4B | 14.4 | 9.8 | 46% |
| Qwen2.5-7B | 7.3 | 4.8 | 52% |
Source: Appendix F, Table 10. Measurements use CPU llama.cpp inference; q4 and q8 are not directly interchangeable with the GPU experiment’s NF4 and INT8 implementations.
Unstructured pruning produces a different result. Setting weights to zero does not eliminate their storage or computation when the runtime uses dense representations and kernels. The paper observes no CPU resident-memory or throughput improvement from unstructured pruning under its tested llama.cpp execution.
For infrastructure investment, pruning only has a plausible efficiency return if the chosen execution system actually exploits sparsity. This study does not demonstrate such a return using a sparsity-aware engine.
Selecting a compression recipe by deployment constraint
The measurements suggest different starting configurations rather than one standard recipe.
| Deployment condition | Candidate approach | Decision boundary |
|---|---|---|
| GPU memory constraint | INT8 without pruning | Verify architecture-specific accuracy and runtime memory |
| Severe pruning damage | LoRA-only recovery first | Compare recovered quality against the uncompressed baseline |
| CPU generation latency | GGUF q4 | Validate throughput and task quality on the target processor |
| Sparse execution support | Consider pruning with recovery | Require measured sparsity-aware memory or latency gains |
The architecture-specific results reinforce this separation. Combining INT8, 30% pruning, and recovery changes average benchmark accuracy by −1.1% for Phi-2, −0.8% for Qwen3-4B, 0.0% for Qwen2.5-7B, and +0.5% for Nemotron-Mini-4B. The study does not establish a uniform accuracy tradeoff or additional efficiency benefit for every architecture.
Its evaluation also offers a screening approach. Perplexity on 20 WikiText-2 samples correlates strongly with the full 1,940-sample evaluation across 16 compared configurations (Pearson r = 0.985). Three-seed robustness checks on 16 selected configurations report perplexity standard deviations below 0.7 points.
These are measurement-validation and robustness tests, not independent demonstrations of production capability. Perplexity can help eliminate poor compression candidates cheaply, but the final selection still requires downstream task tests and deployment measurements.
The CPU memory-tier study offers additional context: Phi-2 fits the tested 4GB tier, both 4B models fit within 8GB, and all four models fit within 12GB. None fits within the tested 2GB tier. These are feasibility observations for the evaluated environment, not measurements on every device advertised with those memory capacities.
Where these results stop applying
The experimental contrasts are informative but have specific limits.
The study evaluates four models between 2.7B and 7B parameters, principally on one NVIDIA L40S GPU and one CPU machine class. The primary configuration sweep uses one random seed, with three-seed replication restricted to selected cases. The pruning method is unstructured L1 magnitude pruning; the study does not establish efficiency gains under sparsity-aware deployment.
All language benchmarks are English-only. Mobile ARM processors, embedded NPUs, multilingual performance, and fairness-related behavior are not directly evaluated. The CPU and GPU experiments also use different quantization backends, preventing simple numerical extrapolation between their perplexity or memory results.
There is a smaller reporting qualification: the source describes benchmark validation across 30 compressed configurations, while the displayed Appendix A table contains 25 identifiable rows, including baselines. The principal comparisons remain reported, but this inconsistency limits reconstruction of the complete benchmark-evaluation set.
Compression should earn its place in the pipeline
The study changes how compression complexity should be justified.
For a team deploying small language models, a technique deserves inclusion when it preserves sufficient task quality and creates a measurable advantage in the target environment. Operation order must be evaluated because it can determine whether a model remains functional. Recovery must be assessed against original quality, not merely percentage improvement from a damaged state. And pruning must deliver demonstrated runtime savings rather than numerical sparsity alone.
The strongest operating rule is therefore to evaluate the ordered compression pipeline as a complete deployment configuration, not to assume that individually effective techniques will produce cumulative benefits when stacked.
Cognaptus: Automate the Present, Incubate the Future.
-
Bhushit Nema and Siddharth Krishnamurthy and Anjali Kadiyala (2026). Compression Technique Interactions in Small Language Models: A Factorial Ablation Study. arXiv:2611.00464. https://arxiv.org/abs/2611.00464 ↩︎