Keep the Directions That Change the Prediction
TL;DR for operators Transformer compression should account for how changes to intermediate representations affect final predictions, not merely how accurately those representations can be reconstructed. Research across ten pretrained language models finds that equally sized hidden-state perturbations can produce orders-of-magnitude differences in prediction error. A local Fisher-based measure helps identify directions and token updates worth preserving. It improves layerwise compression decisions for several models and offers competitive structured pruning. Geometry-regularized distillation also improves most completed reverse-KL and skew-KL comparisons, but not consistently under forward KL. These are promising engineering signals rather than guaranteed deployment savings: the measurements are expensive, and their theoretical protection is local. ...