When the Proxy Wins: Attention Sensitivity Can Hit the Ceiling and Still Miss ICL
TL;DR for operators An internal metric can track a capability well enough to be useful for diagnosis and still fail once the training loop is told to maximize it. In this study, an attention-based signal intended to reflect whether demonstrations matter to a model rises from about 0.516 to 1.413 under direct optimization—within 0.5% of its theoretical maximum. Yet the corresponding behavioural effect is essentially absent on the controlled QA probe, and MMLU accuracy falls to 0.279.1 ...