When the Proxy Wins: Attention Sensitivity Can Hit the Ceiling and Still Miss ICL

TL;DR for operators

An internal metric can track a capability well enough to be useful for diagnosis and still fail once the training loop is told to maximize it.

In this study, an attention-based signal intended to reflect whether demonstrations matter to a model rises from about 0.516 to 1.413 under direct optimization—within 0.5% of its theoretical maximum. Yet the corresponding behavioural effect is essentially absent on the controlled QA probe, and MMLU accuracy falls to 0.279.¹

The failure is not merely a mismatch between two headline numbers. The optimized model learns attention patterns that are sharper and more different between matched and mismatched prompts, but increasingly point to formatting and demonstration-body tokens rather than answer-relevant labels. The metric improves because its geometry is being satisfied, not because the desired capability has been preserved.

For teams deciding whether to turn an internal representation or attention metric into a fine-tuning objective, the operational requirement is stronger than correlation at baseline: deliberately optimize the metric and test whether the intended external behaviour follows. In this experiment, reference-model anchoring preserves broader capability better than maximizing the diagnostic itself, although the evidence does not identify one uniquely superior anchoring mechanism.

A nearly perfect internal score can coexist with worse behaviour

Suppose a fine-tuning team has found an internal model signal that appears to track a capability it wants to preserve. Using that signal as an auxiliary objective is a natural next step: if the metric indicates that the model is using demonstrations, rewarding more of that signal might appear to protect in-context learning during training.

The paper tests exactly that transition—from observing a diagnostic to optimizing it.

The authors define In-Context Sensitivity (ICS) by comparing last-token attention patterns under two otherwise similar four-shot prompts. One contains matched demonstrations; the other keeps the prompt structure but permutes demonstration labels. Larger attention differences produce higher ICS.

Behaviour is measured separately. ICL-GAP is the accuracy under matched demonstrations minus accuracy under the mismatched version. A positive gap indicates that correct demonstrations actually improve the model’s answers.

The stress test makes the distinction clear. Directly maximizing ICS pushes the metric from a pretrained level of roughly 0.516 to 1.413. Because the L2 distance between two attention probability vectors is bounded above by $\sqrt{2}\approx1.414$, this is effectively saturation.

The behavioural result does not follow. At step 5,000, controlled-QA ICL-GAP is -0.010. Across 50 logged checkpoints, its mean is 0.005 with a standard deviation of 0.020. MMLU accuracy declines to 0.279, compared with 0.375 for unrestricted cross-entropy fine-tuning in the controlled comparison.

The result is therefore narrower than “attention does not measure ICL.” It shows that an attention diagnostic can lose behavioural faithfulness when optimization pressure is applied directly to the diagnostic.

The optimization rewarded separation, not useful information

The central question after proxy saturation is not whether ICS increased. It is what kind of model behaviour can produce such a high score.

ICS measures geometric separation between two attention distributions. Its maximum is reached when the distributions become one-hot and point to different token positions. Nothing in that geometry requires those positions to contain information useful for answering the query.

The endpoint analysis shows the model moving toward exactly this solution. Mean top-1 attention concentration rises from 0.18 to 0.94. The overlap between matched and mismatched top-1 attention locations falls from 61% to 4%.

At the same time, the semantic destination of that attention becomes less aligned with the intended capability. Top-1 routing to label tokens falls from 58% in the pretrained model to 26%. By step 5,000, formatting or punctuation tokens account for 31% of top-1 destinations and demonstration-body content for 41%.

The optimization has therefore discovered a high-scoring configuration: make the two prompt conditions trigger sharply different attention locations. That satisfies the metric whether those locations carry answer-relevant information or not.

This is the paper’s concrete Goodhart channel. The proxy is informative when inspected, but once the training objective rewards its numerical value, the optimizer can exploit degrees of freedom that the original interpretation ignored.

Adding behaviour to the objective helps only partially

A natural response is to condition the attention metric on behaviour. The paper tests this with B-ICS, which multiplies ICS by a smooth gate based on a differentiable behavioural signal.

The tested gate improves the endpoint ICL-GAP from -0.010 to +0.020. But ICS still reaches 1.397—98.8% of its ceiling—and MMLU remains low at 0.283.

B-ICS is consequently better interpreted as a constructive sensitivity test than as a solved guardrail. The experiment evaluates one relatively mild smooth gate, with $\beta=10$ and $\delta=0.05$. Stronger thresholds or different gating estimators remain open questions.

Anchoring changes what optimization is asked to preserve

The more promising comparison in this study comes from objectives that constrain drift from the pretrained model rather than reward ever-greater attention divergence.

AnchorTune penalizes changes in selected attention rows relative to a frozen pretrained checkpoint. At the selected $\lambda=0.05$, three seeds produce ICS of $0.500\pm0.001$ and MMLU of $0.370\pm0.058$.

The broader result is more informative than the specific mechanism. Logit-level KL anchoring produces MMLU of 0.383, while weight-space L2 anchoring reaches 0.363. An LM-head-only anchor reaches 0.375. Those ablations weaken any claim that preserving particular mid-layer attention rows is uniquely responsible for the result.

What the paper directly supports is a contrast between optimization directions: in this setup, constraining movement relative to pretrained computation preserves MMLU much better than maximizing an internal divergence proxy.

For a model platform or fine-tuning team, this changes the design question. The choice is not simply which diagnostic correlates with the desired capability. It is whether the diagnostic’s high-scoring states are themselves behaviourally acceptable.

Treat metric promotion as a governance decision

The business implication is strongest for teams converting internal diagnostics into training objectives, capability-preservation penalties, or automated guardrails.

A monitoring metric is evaluated mainly on whether it reveals useful information about a model. A training target faces an additional requirement: the optimizer must not be able to improve the score through states that violate the metric’s intended interpretation.

Cognaptus infers three operational checks from this paper. First, keep an external behavioural measure beside the internal metric when deciding whether a fine-tuning run is preserving capability. Second, stress-test the metric by intentionally optimizing it and examining what happens near its maximum. Third, inspect the structure of those maximizers before deployment: a bounded metric deserves particular scrutiny because its ceiling may be achievable through semantically empty configurations.

The paper also gives a useful warning about dashboards. F_ICS-Max ends with a lower Expected Calibration Error than several comparison arms, but its predictions are simultaneously becoming less accurate and closer to chance. An apparently improved calibration statistic is therefore not sufficient evidence of improved reasoning quality.

The evidence supports a design principle, not a universal prescription

The proxy-behaviour dissociation is clear within the experiment, but its external scope is limited. The main study uses one Llama-2-7B checkpoint, one principal matched-versus-mismatched probe design, four demonstrations, and a 240-item MMLU subset. Most baseline comparisons are single-seed.

There is also an important measurement boundary. The controlled QA ICL-GAP is near zero for the unregularized checkpoints as well, so this experiment does not show that ICS maximization uniquely destroys a strong QA-level ICL effect. The authors separately validate behavioural sensitivity using random-label classification episodes: the pretrained model shows a matched-minus-swapped gap of 0.306, and all four principal fine-tuned checkpoints retain positive gaps with confidence intervals excluding zero.

That distinction limits the claim while strengthening the methodological point. The experiment is best read as a stress test of proxy faithfulness under optimization pressure, not as a comprehensive verdict on attention-based ICL diagnostics.

The operational standard that follows is straightforward: a diagnostic that survives observation has not yet passed the harder test of becoming an objective. Once optimization can see the score, behavioural validation has to remain in the loop.

Cognaptus: Automate the Present, Incubate the Future.