Anchors Aweigh? Why Small LLMs Refuse to Flip Their Own Semantics

A label looks harmless until you ask it to lie.

Tell a model that a glowing movie review should be labeled POS, and few-shot prompting behaves like a useful intern: it studies the examples, picks up the pattern, and usually gets better. Tell the same model that a glowing review should now be labeled NEG, and the intern becomes less useful. It does not smoothly learn your private code. It does not politely invert its semantic universe. It mostly produces a muddle.

That is the central lesson of Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels, a paper by Anantha Padmanaban Krishna Kumar that tests whether small and mid-sized open LLMs can override the meaning of label tokens through few-shot demonstrations.¹ The answer, in the paper’s setting, is brutally clean: no.

Not “not very well.” Not “only with some instability.” Across eight classification tasks, eight open-source models from 1B to 12B parameters, five random seeds, and few-shot prompts with up to eight demonstrations, the semantic override rate stays at exactly zero.

This article is not about whether prompt examples help. They do. It is about when they help. The paper’s useful business lesson is that few-shot prompting is not a meaning editor. It is a prior refiner. When your examples move with the model’s pre-trained semantics, they sharpen performance. When your examples fight the label semantics, they do not create a new classifier. They create conflict.

A classifier prompt is not a meaning editor

The ordinary business instinct is reasonable: if a model sees enough examples, it should infer the rule.

For a customer-support classifier, examples might teach that “refund request” means BILLING; for compliance triage, examples might teach that a complaint belongs to HIGH_RISK; for sentiment analysis, examples might teach that positive reviews get POS and negative reviews get NEG. This is the everyday promise of in-context learning: no training run, no new weights, just demonstrations in the prompt.

The paper tests a more uncomfortable question: what happens when the demonstrations are internally consistent but semantically inverted?

For sentiment, the prompt shows positive examples labeled NEG and negative examples labeled POS. For natural language inference, the labels cycle through an inverted mapping among ENTAILMENT, NEUTRAL, and CONTRADICTION. For topic classification, labels such as WORLD, SPORTS, BUSINESS, and TECHNOLOGY are cyclically permuted. The instruction remains the same; only the example labels are flipped.

This creates a sharp test between two views of in-context learning.

One view treats ICL as flexible task learning. Under that view, labels are close to arbitrary symbols. If the prompt consistently says that positive means NEG, the model should eventually learn the local convention. Strange convention, yes. But businesses invent strange conventions every day, usually in spreadsheets named “final_v7_really_final.xlsx.”

The other view treats ICL as prior refinement. Under that view, label tokens are not blank tags. POS, NEG, HATE, NOT_HATE, ENTAILMENT, and CONTRADICTION already carry semantic weight from pre-training. Demonstrations can adjust how inputs map into this space, but they cannot easily rewrite what the labels mean.

The paper’s evidence strongly favors the second view for 1–12B open models in the tested few-shot setting.

The mechanism: natural examples sharpen the prior, inverted examples split it

The paper’s cleanest contribution is not simply the accuracy table. Accuracy tells you whether performance rose or fell. It does not tell you why.

To expose the mechanism, the author decomposes model behavior into three alignments:

The distinction between prompt alignment and semantic override is important. A model can partially follow the inverted prompt without learning a coherent inverted classifier. That is exactly the trap. If a model starts outputting more flipped labels, a dashboard may report “greater prompt adherence.” But if accuracy collapses at the same time, that adherence is not learning. It is semantic damage with a nicer name.

The LLaMA-3.1-8B-Instruct results make this visible. With natural demonstrations, average accuracy rises from 69.5% in zero-shot mode to 79.3% at eight shots. QQP, where the zero-shot prior is weak, jumps from 40.6% to 78.4%. MNLI and SNLI gain roughly 14–17 points. Even strong sentiment baselines improve modestly: SST-2 moves from 90.4% to 92.5%, and IMDB from 92.4% to 93.7%.

That is the friendly side of ICL: examples help when they reinforce the model’s semantic grain.

The inverted condition is the unfriendly side. At eight shots, average accuracy falls to 52.8%. SST-2 drops from 90.4% zero-shot accuracy to 47.4%; IMDB drops from 92.4% to 48.4%; AG News falls from 76.0% to 43.9%. The model is not learning a new private language. It is being pulled away from both the ground truth and its own prior.

The one interesting exception is QQP. Inverted ICL reaches 71.6%, above its weak zero-shot baseline of 40.6%, though still below natural ICL at 78.4%. This does not prove override. It suggests that when the zero-shot prior is poor, even corrupted demonstrations may contain some task-relevant signal. The signal can help identify the task structure, but it still does not let the model redefine the label semantics.

That distinction matters. The model may learn “this is a paraphrase-style problem.” It does not reliably learn “SIMILAR now means DIFFERENT.”

The zero is doing real work

The headline result is the semantic override rate: exactly 0.0% across the reported experimental conditions.

This number is easy to quote and easy to misunderstand. It does not mean the models never produce inverted labels. They do. In Table 4, under inverted demonstrations, LLaMA-3.1-8B-Instruct matches the demonstrated inverted label on 51.6% of IMDB examples and 52.6% of SST-2 examples. So the model is not ignoring the prompt completely.

The failure is sharper. The model partially follows the inverted examples, but that prompt-following does not become a coherent anti-semantic classifier. Accuracy and prior alignment decay as prompt-following rises. The model gets tugged in two directions: the examples say one thing, the label tokens mean another, and the output lands in a compromised zone that satisfies neither.

This is why the paper’s mechanism-first interpretation is more useful than a simple “small LLMs are bad at flipped labels” summary. The model is not merely underpowered. It is constrained by the semantic geometry of its label tokens. In the author’s interpretation, ICL adjusts how inputs project onto stable semantic directions learned during pre-training; it does not reshape the semantic manifold itself.

Less poetically: if your label names already mean something, the model brings that meaning into the prompt whether you invited it or not.

The appendix is robustness evidence, not a second thesis

The main body focuses closely on LLaMA-3.1-8B-Instruct because it is easier to see the mechanism in one model. The appendix then tests whether the pattern survives broader variation.

It does.

The paper evaluates eight models: LLaMA-3.1-8B-Base, LLaMA-3.1-8B-Instruct, LLaMA-3.2-3B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-7B, Gemma-3-1B-IT, Gemma-3-4B-IT, and Gemma-3-12B-IT. The model families differ in architecture, tokenization, scale, and instruction tuning. The tasks cover sentiment, NLI, paraphrase detection, hate speech detection, and topic classification.

The likely purpose of the appendix tables is robustness and sensitivity checking: do the main effects survive model family, model size, dataset type, seed choice, and number of demonstrations?

The answer is yes. The appendix reports universal zero override across the tested setup. It also shows a repeated pattern where inverted demonstrations often become more damaging as the number of examples increases from 1 to 8. More examples do not rescue the inverted mapping. They often intensify the contradiction.

That matters because a common product instinct is to “just add more examples.” Here, more examples are not a generic cure. If the examples agree with the model’s label semantics, more shots can sharpen classification. If the examples oppose label semantics, more shots can make the conflict louder.

A practical evidence map

The paper’s results are easiest to use if we separate main evidence, robustness checks, and implementation details.

This is also where the paper is more disciplined than the usual prompt-engineering folklore. It does not merely say “labels matter.” It builds a diagnostic separation between truth, prior, prompt adherence, and semantic override. That separation is exactly what enterprise evaluation often lacks.

Business translation: label design is part of model design

The obvious business takeaway is “do not use inverted labels.” Fine, yes, please do not build a risk system where SAFE means dangerous and DANGER means fine. We have enough governance problems without turning compliance into a word game.

But the deeper implication is broader: label design is not cosmetic. It is part of the model interface.

Many AI workflows quietly assume that labels are internal database codes. The business team defines categories, the prompt includes examples, and the model is expected to adapt. That assumption is safe only when the labels are semantically natural or at least semantically neutral.

Consider three cases.

First, natural labels. A customer ticket labeled REFUND_REQUEST, a review labeled NEGATIVE, or a policy document labeled COMPLIANCE_RISK gives the model a label whose surface meaning supports the task. Few-shot examples should work in the intended direction, especially when the model already has a reasonable prior.

Second, ambiguous labels. Internal labels such as Tier 1, Bucket C, or Case Type 17 do not strongly fight natural language semantics because they carry little public meaning. These may be safer when the taxonomy is proprietary. The model still needs explanation and examples, but at least the label token is not actively arguing with the task.

Third, anti-semantic labels. If an internal workflow defines GREEN as high risk, APPROVED as rejected, or POS as negative, small LLMs may not respect the private convention through prompting alone. They may appear to follow the examples sometimes, while producing brittle or incoherent decisions. This is the dangerous category because the failure can look like partial compliance.

The Cognaptus inference is straightforward: for small and mid-sized open LLM deployments, taxonomy engineering should happen before prompt engineering. Use labels that match ordinary meaning, or use neutral symbols with explicit mapping. Do not make the model fight public semantics and then blame it for losing the fight.

When prompt engineering should stop

The paper gives a useful stopping rule for enterprise teams.

If few-shot examples improve performance while predictions remain aligned with strong zero-shot behavior, prompting is probably refining the prior. That is a good use of ICL. Keep improving examples, improve retrieval, add representative cases, and calibrate thresholds.

If examples increase prompt-following while accuracy falls, the prompt is probably fighting semantic anchors. More examples may not solve the problem. They may simply produce a more confident mess.

At that point, the intervention should change:

This is not a rejection of small LLMs. It is a more realistic operating manual. Small open models are attractive because they are cheaper, easier to host, and often good enough for narrow workflows. But their controllability is not infinite. They come with semantic commitments baked into pre-training. Pretending otherwise is not strategy. It is procurement optimism wearing a lab coat.

Governance implication: evaluate the prior before trusting the prompt

Most enterprise LLM evaluations begin with prompted performance: give the model the desired instruction, add examples, test accuracy, then iterate.

The paper suggests a better sequence.

First, measure zero-shot behavior. What labels does the model naturally prefer? Where does it already agree with the intended taxonomy? Where does it resist?

Second, compare zero-shot predictions with few-shot predictions. If few-shot success mostly coincides with zero-shot agreement, the prompt is refining the model’s prior. That is not bad. In fact, it may be exactly why the prompt works. But it means the system depends on the model’s inherited semantic structure.

Third, test adversarial or counter-semantic cases when the business taxonomy is non-standard. Do not assume consistent demonstrations will override label meaning. The paper’s core result says that, for the tested 1–12B models, they do not.

This turns prior alignment into a governance diagnostic. It answers a question that accuracy alone hides: is the model learning the business rule, or is the business rule lucky enough to align with what the model already believes?

That distinction becomes important in regulated workflows. A compliance classifier that succeeds because HIGH_RISK means what everyone thinks it means is easier to document. A classifier that relies on a private inverted convention is harder to audit and easier to break.

Boundaries: what the paper shows, and what it does not

The paper’s result is strong, but its scope is not universal.

It studies open models from 1B to 12B parameters: LLaMA, Mistral, Qwen, and Gemma variants. It does not establish that much larger frontier models cannot learn inverted semantics. In fact, the paper situates itself against prior work suggesting that GPT-3-scale models may eventually flip labels under some conditions.

It studies few-shot prompting with $k \in {1,2,4,8}$ demonstrations. It does not settle many-shot prompting with hundreds of examples, although the related literature discussed in the paper suggests that smaller models can still struggle even in many-shot regimes.

It uses greedy decoding with temperature 0, generates up to three tokens after the label stub, and maps the first generated word to labels with task-specific heuristics. Different decoding strategies, constrained decoding, calibration, or post-processing could change operational behavior. They would not necessarily prove that ordinary ICL has rewritten semantics; they may simply route around the problem.

It focuses on semantically loaded labels. POS, NEG, HATE, NOT_HATE, ENTAILMENT, and CONTRADICTION are not neutral symbols. The result should not be lazily extended to every arbitrary code or every learned tool-use convention. A model may handle neutral labels differently from anti-semantic labels.

Finally, the paper is primarily behavioral. It offers a geometric interpretation—semantic labels occupying stable regions in the representation manifold—but it does not provide a mechanistic circuit-level proof. The interpretation is useful, and it fits the evidence, but business readers should treat it as an explanatory frame rather than an MRI scan of the model’s brain. No, the manifold did not sign a confession.

The practical rule: flow with semantics or pay for adaptation

The useful rule is simple:

Use prompting when your labels match the model’s semantic priors. Use training, calibration, neutral symbols, or external logic when your workflow needs to redefine them.

This rule is not glamorous. It will not sell a magical “prompt library” that turns any model into any classifier. It does, however, reduce failure.

For AI automation teams, the decision path should look like this:

1. Start with natural-language labels that mean what they say.
2. Measure zero-shot behavior before few-shot behavior.
3. Use few-shot examples to refine, not contradict, the model’s semantic prior.
4. If the taxonomy is proprietary, prefer neutral labels or explicit structured outputs.
5. If the mapping must be non-standard and stable, stop prompting and use a stronger intervention.

The paper’s contribution is valuable because it replaces a vague prompt-engineering belief with a testable mechanism. Few-shot examples are not magic instructions. They are signals injected into a model that already has a semantic map. When the signal points along the map, the model follows. When the signal points against the map, the model does not become flexible. It becomes conflicted.

Conclusion: the anchor is the interface

The phrase “semantic anchor” sounds abstract, but the operational meaning is concrete. In small and mid-sized LLMs, labels are not just output tokens. They are part of the interface contract between business logic and pre-trained language knowledge.

That contract can help you. Natural labels make few-shot prompting cheaper and more reliable. They let the model reuse what it already knows.

The same contract can also block you. Inverted or unnatural labels turn prompting into a fight against pre-training. The model may partially obey, but the paper’s evidence suggests it will not coherently flip its semantics in the tested regime.

So the practical message is not “small LLMs are useless.” It is sharper: small LLMs are useful when you design workflows that respect their semantic anchors. If you need those anchors moved, prompting is the wrong tool. Bring training, calibration, symbol design, or external control logic.

The anchor is not a bug. It is part of the interface. Build around it.

Cognaptus: Automate the Present, Incubate the Future.