One Correction, Every Case: When LLMs Actually Update the Rule

TL;DR for operators

An AI system receives one signal that an operating rule has changed. The important test is not whether its average performance eventually recovers, but whether it immediately applies the revised rule to cases it has not yet revisited.

Many models fail this test quietly. They correct each stimulus only after encountering it again, producing gradual recovery without inferring that one hidden rule changed for every stimulus at once. For teams deploying agents, that distinction matters whenever a policy change, workflow update, or exception rule must propagate across related cases.

The study finds that successful transfer is associated with representing the hidden operating rule separately from the individual stimulus. That representation remains usable in excluded conditions and shifts in a similar direction across different stimuli—properties the paper measures through cross-condition generalization performance and parallelism score.

The intervention evidence is especially relevant. Geometry-only training increased inference-present sessions from 16.2% to 43.2% and raised first-inference-trial accuracy from 0.45 to 0.63, despite using no language-modeling objective.

The operational lesson is straightforward: after one correction, test unrevisited cases immediately. Eventual recovery may show adaptation; immediate transfer provides stronger evidence that the system updated the rule.

Performance recovery can conceal local relearning

Consider an agent that has learned how to route four categories of customer requests. The organization then changes the routing policy. After one rejected action, the agent should update the policy for every category governed by the same rule, including categories it has not yet seen since the change.

A system can appear adaptive while failing this test. It may correct the first category, then make the old mistake on the second, learn again, and repeat the process until aggregate performance recovers. The final accuracy curve looks acceptable, but the system never represented the policy change as a global change.

Yunan Zeng and Yuwang Wang study this distinction through a text-based contextual reversal-learning task.¹ Humans and LLMs repeatedly choose left or right for four arbitrary pseudowords. The correct mappings depend on one of two unannounced contexts, and all mappings reverse together when the context switches. Correct responses produce rewards of $25 or $5; incorrect responses produce $0.

The arbitrary labels prevent the model from relying on ordinary word meaning. The global reversal creates the central diagnostic: after receiving negative feedback for one stimulus, does the model transfer the revised rule to another stimulus that has not yet appeared in the new context?

The authors classify sessions into two behavioral regimes. Inference-present sessions show stable learning before the switch and above-chance performance on the first eligible unseen-stimulus trial afterward. Inference-absent sessions recover more gradually, consistent with stimulus-by-stimulus relearning.

Both humans and LLMs exhibited both regimes, but LLMs entered the rapid-inference regime less often. Among the evaluated open-weight Llama and Qwen models, inference prevalence rose strongly with parameter count: Pearson correlation was 0.827 and Spearman correlation was 0.888. That relationship does not isolate scale as the cause, since architecture, training data, and post-training vary across models. It does show that successful structural transfer was not evenly distributed across the tested systems.

A readable context signal is not enough

Once the behavioral split is established, the mechanistic problem becomes more precise. A model may encode the current context somewhere in its hidden states without organizing that information in a form that supports transfer.

The paper therefore distinguishes three representational properties.

\ast\astDecoding accuracy\ast\ast asks whether a linear classifier can recover the context from internal activations. High decoding shows that information is present. It does not show that the information is represented consistently across different stimuli.

\ast\astCross-Condition Generalization Performance, or CCGP,\ast\ast trains a classifier on some task conditions and tests it on excluded conditions. A context representation scores well only when the same distinction remains usable outside the examples from which the classifier learned it.

\ast\astParallelism Score, or PS,\ast\ast measures whether switching context moves representations in a similar direction across different stimulus backgrounds. A high score indicates a regular coding direction rather than a collection of unrelated, case-specific separations.

This distinction changes the interpretation of the hidden-state results. Inference-present sessions did not simply contain more information about everything. Stimulus identity remained strongly and stably represented. The selective difference concerned the organization of latent context.

At the stimulus stage, inference-present sessions increased context CCGP by 0.090 and context PS by 0.081 relative to inference-absent sessions. At baseline, before the current stimulus arrived, context was already strongly encoded; successful sessions showed an additional PS increase of 0.104. The carried-over task state was therefore not only readable but more regularly organized.

The useful result is the separation. Stimulus identity stays available, while context becomes more invariant to which stimulus is currently shown. That organization provides a plausible route for transferring one feedback event across cases.

The strongest context organization appears higher in the model

The depth analysis adds architectural specificity. Inference-related context geometry was concentrated in higher transformer layers, where context decoding increased by 0.180, CCGP by 0.155, and PS by 0.064. Lower and middle layers showed weaker or less consistent differences in the observational comparison.

This does not imply that lower layers are irrelevant. Stimulus identity was already represented strongly there. The pattern instead suggests a division of labor: lower representations preserve the current item, while later processing organizes the hidden operating context into a form that can govern multiple items.

That distinction becomes more consequential in the intervention experiments. When the authors directly constrained representational geometry, middle- and higher-layer supervision produced substantially larger behavioral improvements than lower-layer supervision. Higher-layer constraints increased inference accuracy by 0.206, and middle-layer constraints increased it by 0.166. The lower-layer gain was weak and did not remain significant after correction.

For teams designing adapters or representation-level objectives, layer placement is therefore part of the hypothesis rather than a neutral engineering setting. A training intervention aimed at global rule transfer may have limited effect when it primarily modifies layers occupied with stimulus-level encoding.

Training moved the geometry and the behavior together

The observational result links successful inference with abstract context geometry. The paper then uses two interventions to test whether that relationship is functionally meaningful within the task.

The first intervention is task-sequence language-model fine-tuning. Models learn independently generated reversal-task trajectories using disjoint pseudoword pools for training, validation, and evaluation. This is the broader intervention: the model learns the sequential structure of the task, and representational geometry changes alongside behavior.

Inference-present prevalence rose from 16.2% to 76.1%. Accuracy in the inference window increased from 0.45 to 0.77. Context decoding, CCGP, and PS also increased across layers. This experiment shows that learning structured interaction sequences can induce more reusable context representations, although it cannot separate which parts of the language-modeling objective produced the effect.

The second intervention, GeoStruct, targets that ambiguity. It applies a geometry-only objective that aligns context-, response-, and outcome-related difference vectors across task backgrounds. There is no language-modeling loss.

GeoStruct increased inference-present sessions from 16.2% to 43.2% and raised first-inference-trial accuracy from 0.45 to 0.63. The result is smaller than full sequence fine-tuning but mechanistically sharper. Directly changing the organization of hidden representations improved behavioral transfer even without training the model to predict the task sequence.

That intervention does not establish that a single geometric objective governs all reasoning. It supports a narrower conclusion: within this reversal-learning task family, consistent coding directions are not merely decorative correlates of good performance. They are a plausible intervention target.

The robustness tests protect the mechanism claim

Several additional analyses serve as robustness and sensitivity tests rather than separate theses.

The context-geometry effects were not explained by overall representation norm. Removing high main-effect features did not eliminate the pattern. Context geometry weakened on error trials, tying the measurements more closely to successful behavior rather than stable model identity alone.

The authors also report that attention reallocation around context switches was positively associated with context CCGP, PS, and inference success, particularly for attention linked to recent outcome information. This provides a possible process account: feedback changes what information the model attends to as its context representation is reorganized.

The attention result remains correlational. GeoStruct provides the cleaner intervention evidence because the training objective directly manipulates representational organization and produces a behavioral change.

A practical evaluation should test transfer, not eventual recovery

The paper directly demonstrates a controlled behavioral and representational distinction. Applying it to deployed systems requires an additional step of operational inference.

A model owner evaluating policy-update behavior could construct a family of cases governed by one hidden rule, establish stable pre-change performance, introduce a rule reversal, and provide corrective feedback on only one case. The decisive measurement would be performance on an unrevisited case immediately afterward.

That design would separate three system behaviors:

1. \ast\astNo adaptation:\ast\ast the model continues applying the old rule.
2. \ast\astLocal repair:\ast\ast the model corrects cases only after receiving case-specific feedback.
3. \ast\astStructural update:\ast\ast the model transfers the revised rule to unseen or unrevisited cases.

For open-weight models, internal diagnostics could then test whether the revised context generalizes across case backgrounds and whether its coding direction remains consistent. These metrics might help model engineers compare fine-tuning methods, select adapter locations, or identify models whose post-feedback recovery depends on repeated exposure.

The affected decision is specific: a model-evaluation or assurance team deciding whether an agent can propagate a changed policy across related cases after sparse feedback. The relevant condition is a workflow governed by a shared latent rule. The boundary is equally specific: the paper does not validate CCGP or PS as production safety thresholds, nor show that a passing score predicts reliable behavior in natural-language workflows.

The hippocampal comparison is computational, not biological

The study adapts task logic and geometric measurements previously used in research on human hippocampal representations. The comparison concerns a shared computational pattern: abstract task variables can be encoded through approximately parallel directions that generalize across conditions.

It does not show that transformer layers reproduce hippocampal anatomy, biological learning, subjective understanding, or human reasoning as a whole. The human sample was also modest: 41 sessions passed the main inclusion criteria, with no reported demographic or geographic characterization.

External validity is the larger constraint. The environment contains four pseudowords, two contexts, binary actions, and global reversals. The interventions are evaluated on held-out labels and trajectories from the same task family, not unrelated reasoning domains. Internal analyses cover open-weight Llama and Qwen models; proprietary systems contribute behavioral results but not hidden-state evidence.

The study therefore establishes a strong task-local mechanism and a promising measurement strategy. It leaves open whether the same geometry predicts reliable updating in policy compliance, software agents, clinical workflows, or other environments where rules are incomplete, exceptions overlap, and feedback may itself be ambiguous.

From model correction to rule propagation

Post-feedback accuracy compresses several behaviors into one number. A system that revises a global rule and a system that relearns every case can eventually arrive at similar average performance, while creating very different operational risks.

Zeng and Wang provide a controlled way to separate those systems. Rapid transfer is associated with a context representation that remains stable across stimuli, generalizes to excluded conditions, and follows a consistent coding direction. Sequence training strengthens that organization, and direct geometric regularization improves inference without a language-modeling objective.

The near-term value is not a new claim that geometry certifies reasoning. It is a better diagnostic target: after one meaningful correction, test the cases the model has not yet had a chance to relearn.

\ast\astCognaptus: Automate the Present, Incubate the Future.\ast\ast