Model-Compression

TL;DR for operators Combining eight specialized AI adapters into one deployable adapter can reduce infrastructure complexity, but it can also discard capabilities when the adapters encode poorly aligned parameter updates. Di Wu and Shuaidong Pan’s Group-wise Rank-Aware Model Merging (GRAM)1 addresses this by clustering compatible task updates before merging them. It also allocates limited representational capacity according to the strength of each task’s learned directions. The merging procedure requires model weights, not task training or calibration data. ...

TL;DR for operators Transformer compression should account for how changes to intermediate representations affect final predictions, not merely how accurately those representations can be reconstructed. Research across ten pretrained language models finds that equally sized hidden-state perturbations can produce orders-of-magnitude differences in prediction error. A local Fisher-based measure helps identify directions and token updates worth preserving. It improves layerwise compression decisions for several models and offers competitive structured pruning. Geometry-regularized distillation also improves most completed reverse-KL and skew-KL comparisons, but not consistently under forward KL. These are promising engineering signals rather than guaranteed deployment savings: the measurements are expensive, and their theoretical protection is local. ...

TL;DR for operators Post-training quantization methods that look similar on normal 4-bit benchmarks can separate sharply when the numerical problem becomes difficult. In one Mistral-7B v0.1 stress test, QuIP-family variants abort with NaNs and GPTQ reaches WikiText-2 perplexity of 379.73 ± 15.71. Kashin-DCT+H reaches 8.92 ± 0.01, close to the FP16 reference of 8.63. ...

TL;DR for operators Fracast-01 reduces a probabilistic time-series foundation model to 85,001 parameters. Against TinyCast, that is a 42.0% parameter reduction, while TinyCast retains 4.2% lower aggregate MASE and 3.3% lower aggregate WQL. The smaller model is therefore an efficiency point, not an accuracy winner. The trade-off is uneven. Fracast-0 is close to TinyCast at short horizons, especially on probabilistic forecast quality, but the gap widens at medium and long horizons. Its weakest reported frequency group is secondly data, where normalized MASE rises above the Seasonal Naive baseline. ...

TL;DR for operators Wang et al. study a deployment problem that is easy to underestimate: an RL-trained agent can work well before compression or modification and then fail badly after its policy is perturbed.1 Their proposed method, Stable Perturbation-Robust Policy Optimization (SPrPO), trains the agent under perturbations while allocating less noise to locally sensitive FFN channels and adjusting total perturbation strength using a gradient-based improvement proxy. ...

TL;DR for operators A fixed MoE memory budget does not tell you which experts can safely absorb the lowest precision. Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees1 argues that the experts most frequently or strongly used are not necessarily the ones that need the most bits. In its theory, experts associated with less-prevalent but still task-relevant features develop weaker activations and smaller router-norm changes, leaving less margin for quantization error. ...

TL;DR for operators A deployment team can reach an awkward point after quantization: the checkpoint is small enough to ship, but quality is below target, and replacing the compression pipeline would mean another round of calibration, validation, packaging, and operational risk. The paper on Activation-Weighted Seeded Residual Coding, or AWSRC, asks whether some of that lost quality can instead be bought back with a small additional payload while leaving the existing low-bit reconstruction untouched.1 In its cleanest comparison, all tested residual codecs receive exactly 49,245,876 extra serialized bytes on the same RTN-SDQ parent. AWSRC reaches perplexity 7.04 and mean zero-shot accuracy 0.69—the best perplexity and a top accuracy result among the tested sparse, low-rank, learned-vector-quantized, and AWSRC repairs. Learned vector quantization has the best unrounded KL result, so the evidence supports a quality-per-byte advantage rather than dominance on every metric. ...

TL;DR for operators A mixture-of-experts model can be too large for an inference budget even though only a small subset of its experts executes for each token. The inactive experts still have to reside somewhere. In the measured Mixtral deployment studied by Janati et al., removing four of eight experts per layer cut 4-bit memory from 24.2 GB to 12.3 GB and per-token latency from 40.3 ms to 25.2 ms on a single A100.1 ...

TL;DR for operators A small model can perform acceptably after task specialization and still lose more accuracy than expected when compressed. The paper examined here suggests that part of that loss may be shaped earlier, during fine-tuning, rather than being determined by the quantizer alone. Paper evidence: Muon-optimized models lost less accuracy after quantization than Adam-optimized models on six of eight benchmarks. On ARC-e, for example, accuracy fell by 0.55 percentage points after quantization with Muon versus 3.16 points with Adam. The integrated pipeline also produced a 2.86 GB quantized model from a 6.01 GB pre-quantized model while improving reported serving throughput and roughly halving per-token latency. ...

TL;DR for operators A team trying to move a transformer onto a phone, camera, robot, or embedded device has several levers: reduce the model, lower numerical precision, change the runtime, or move to a better accelerator. Hema Hariharan Samson’s survey of lightweight transformers1 suggests these choices cannot be evaluated independently. In the paper’s analyzed batch-size-1 setting, smaller workloads can become limited by how quickly the device moves model data rather than by raw arithmetic capacity; the paper reports roughly 60–75% hardware utilization in a favorable 15–40M-parameter range. Its broader comparisons similarly show that INT8 execution, operator fusion, optimized runtimes, and specialized accelerators change realized latency by very different amounts across devices. For an operator, the practical target is not the smallest model. It is the smallest accuracy loss that satisfies the product’s measured latency and energy budget on the actual deployment hardware. ...