Fisher-Information

TL;DR for operators Transformer compression should account for how changes to intermediate representations affect final predictions, not merely how accurately those representations can be reconstructed. Research across ten pretrained language models finds that equally sized hidden-state perturbations can produce orders-of-magnitude differences in prediction error. A local Fisher-based measure helps identify directions and token updates worth preserving. It improves layerwise compression decisions for several models and offers competitive structured pruning. Geometry-regularized distillation also improves most completed reverse-KL and skew-KL comparisons, but not consistently under forward KL. These are promising engineering signals rather than guaranteed deployment savings: the measurements are expensive, and their theoretical protection is local. ...

TL;DR for operators Most business AI failures on imbalanced data do not look like dramatic model collapse. They look quieter: the system performs well on common cases, under-serves rare cases, and then someone discovers that “rare” was another word for “expensive when wrong”. The OSDTW paper tackles this long-tailed recognition problem by treating head and tail classes as two related tasks rather than one flattened classification problem.1 Its practical message is not “care more about minority classes”, although that would make a pleasant conference slogan. The message is sharper: imbalance is a structural design problem. You must decide which representation layers should be shared, which parts should specialise, and how much head versus tail supervision should shape the shared model. ...