The Model That Knows It Knows: When Introspection Hides in the Logits

Audit.

That is the word enterprises prefer when they want something to sound measurable, serious, and safely boring. You audit model outputs. You audit prompts. You audit logs. You audit whether the assistant said the forbidden thing, leaked the private thing, or hallucinated the regulatory thing.

The problem is that models are not only output machines. They are also representation machines. Between the input and the final answer, they build intermediate signals, suppress some of them, amplify others, and then hand management a neat little sentence pretending the whole internal mess never happened.

A new paper, Latent Introspection: Models Can Detect Prior Concept Injections, makes that gap unusually concrete.¹ The authors show that Qwen2.5-Coder-32B-Instruct can contain recoverable information about prior concept-vector injections in its KV-cache representations, even when its sampled output denies that anything was injected.

That sentence needs careful handling. It does not mean the model is conscious. It does not mean it has a little dashboard where it watches its own neurons glow. Please spare us the Silicon Valley séance.

It means something narrower and more useful: under a controlled intervention, the model appears to have access to information about a previous internal perturbation, and that information is visible in logits and intermediate layers before being weakened near the final output.

For business readers, this is not a metaphysical scandal. It is an evaluation problem. If your safety process only asks, “What did the model say?”, you may miss what the model internally represented before it decided what it was allowed, likely, or trained to say.

The trick is to inject first, ask later

The paper’s core design is simple enough to explain without ritual incense.

First, the researchers train steering vectors for concepts such as cats, bread, love, fear, death, truth, creativity, programming, and music. A steering vector is a direction in activation space associated with a concept. Add it during inference, and the model’s internal processing is nudged toward that concept.

Second, they apply the vector only while generating the KV cache for an earlier conversational turn. The KV cache stores key-value representations from prior tokens so later tokens can attend back to them efficiently. This matters because the authors are not continuously steering the model while asking the diagnostic question.

Third, they remove the steering vector. Only after that do they ask the model whether some concept was injected.

That sequencing removes two cheap explanations. The model cannot simply react to an ongoing intervention, because the intervention is gone. It also cannot infer the injection from its own visibly steered output, because the earlier response is fixed and no tokens are sampled under active steering in the main protocol.

So the practical question becomes sharper:

Did the prior internal perturbation leave information in the cached representations that the model can later access?

That is why the experiment is more interesting than a normal prompt test. It is not asking whether the model can say impressive things about introspection. Models can say impressive things about anything; that is their most economically destructive hobby. The test asks whether a known causal intervention inside the model can be detected later.

The obvious evaluation says “nothing to see here”

If we only inspect the model’s final sampled answer, the default result looks unimpressive.

Under a direct mechanistic prompt with no extra background information, asking whether a concept was injected barely changes the final “yes” probability. The paper reports a shift from about 0.2% false positives to about 0.3% true positives in the basic setting. A standard behavioral evaluation would probably conclude: no introspection.

That conclusion would be too fast.

With a pro-introspection document explaining how transformer K/V streams can carry information from prior internal states, the model’s detection under accurate mechanistic framing rises substantially: the authors report 39.9% detection with only a 0.8% false positive rate. In other words, the model does not become a random “yes” machine. Its “yes” probability increases mainly when there actually was an injection.

The first business lesson is already visible here. The same underlying system can look incapable under one evaluation surface and partially capable under another. That is inconvenient for compliance checklists, which are allergic to surfaces.

The important correction is not “models are secretly self-aware.” It is more operational: sampled outputs are a lossy endpoint.

The signal appears after the injection site, then fades near the output

The most important evidence is not the final answer. It is the layer-by-layer trajectory.

The injection is applied to Qwen2.5-Coder-32B-Instruct across layers 21–42, roughly the middle third of the 64-layer model. But the introspection signal does not immediately appear as a clean answer. Using logit-lens analysis, the authors trace “yes” probability through intermediate layers. The detection signal emerges later, around layer 48, peaks around layers 58–62, and then sharply attenuates in the last few layers.

That pattern is the article’s central mechanism.

The model does not simply receive an injected concept and instantly report it. Instead, the signal seems to become more readable downstream, after later computation has integrated cached information. Then, near the output, the signal is weakened.

A simplified reading looks like this:

This is why the paper’s title-level claim is stronger than “prompting affects answers.” Prompting affects answers, yes; thank you, 2023. But here the evidence says the model’s internal layers can carry a detection signal that is later attenuated before final output.

That is a different governance problem. You are not merely testing an answer. You are testing an answer after post-training priors, assistant-persona constraints, refusal habits, distributional defaults, and output selection have all had their turn at the microphone.

Concept identification makes the noise explanation much harder

A critic could still object: perhaps concept injection just perturbs the model, and the perturbation makes it more likely to answer “yes” to strange introspective questions.

The paper addresses this in two ways.

First, the authors test control questions unrelated to introspection. Always-yes and always-no factual controls show negligible shifts, about 0.2%. Ambiguous varied-baseline questions shift more, at most 8.0%. Confusing questions shift more still, up to 25.8%, which suggests injection may interact with uncertainty. That is not nothing, and the authors do not pretend it is nothing. But the control results make a simple generic yes-bias explanation unattractive.

Second, and more importantly, the model can sometimes identify which concept was injected.

The authors run a concept-identification setup with nine injected concepts. They prompt the model with a numbered list and inspect logits over the digit tokens using the logit lens. They also shuffle answer order across trials to reduce position-bias concerns. The result is a confusion matrix: injected concept in rows, predicted concept in columns.

The best reported condition reaches 1.36 bits of mutual information between the injected and recovered concepts. With nine equiprobable concepts, the theoretical maximum is:

So 1.36 bits is about 43% of the maximum possible information. That is not full recovery. It is also not random noise.

The model identifies some concepts much better than others. In the main text’s reported confusion matrix, programming is identified at 92%, death at 75%, and truth at 71%, while cats and bread are not identified correctly in that displayed condition. Appendix analysis under a different prompt setting shows another useful point: the signal is uneven, concept-dependent, and sometimes distorted by background attraction to certain concepts.

That unevenness matters. It prevents the lazy interpretation that “the model can introspect” as if introspection were a single switch. What the paper shows is more granular: some injected semantic directions leave recoverable traces under some prompts at some layers.

That is less dramatic. It is also more believable.

The vague prompt beats the accurate explanation, because of course it does

One of the stranger findings is that accurate mechanistic description is not always the best way to elicit the signal.

The study tests prompting conditions by crossing different framings with different information documents. The main 4×4 grid varies how the intervention is described—accurate mechanism, wrong mechanism, vague mechanism, and poetic no-mechanism—and what supporting document is provided—none, pro-introspection, matched lorem ipsum filler, or poetic salience text.

The result is not what a tidy engineering mind would prefer. Vague phenomenological framing performs very well. Asking whether some concept seems “particularly salient” reaches 68–84% balanced accuracy across information conditions. Meanwhile, the accurate mechanism framing performs weakly without scaffolding and improves substantially only when paired with the pro-introspection document.

This is the sort of result that makes prompt engineering look less like engineering and more like asking a Victorian ghost the right question. Still, there is a serious interpretation.

The model may have better access to a representation like “something is salient” than to a representation like “my activations were manipulated by an external steering vector during KV-cache generation.” The latter is a technical description imposed by researchers. The former may be closer to the internal variable that actually becomes readable downstream.

There is another possibility. Directly saying “your internals were manipulated” may trigger a trained assistant prior: deny unusual internal access, deny special self-awareness, deny weird capabilities. Vague salience framing may avoid that defensive corridor. The model is not asked to endorse a grand claim about its own mind; it is asked whether anything stands out.

The paper does not resolve these explanations. That unresolved point is not a footnote. It is part of the practical story: model evaluation is prompt-sensitive not only because models are fragile, but because different prompts may align with different internal descriptions of the same phenomenon.

The appendix mostly strengthens the interpretation, not the headline

The appendix is useful because it separates “nice story” from “tested irritation.”

Several appendix components function as robustness or measurement support rather than a second thesis.

The emergent misalignment appendix deserves special discipline. The authors test whether misalignment-associated activation patterns can be detected through related methods. The effects are smaller and less consistent than concept-vector results. That section should be read as exploratory, not as proof that models can reliably diagnose their own alignment state.

The additional-model appendix is also important but bounded. The authors run limited replications on Llama 3.3 70B Instruct and Qwen2.5-72B Instruct. Both show introspection signals and final-layer attenuation. But their prompt behavior differs sharply. Llama 3.3 70B performs strongly under Accurate Mechanism + No Document, while the pro-introspection document reduces performance in that setting. Qwen2.5-72B responds more strongly to the pro-introspection document, reaching 88.8% accuracy under Accurate Mechanism + Pro-Introspection Document. Peak concept-identification MI appears at layer 78 in both larger models, with 0.58 bits for Llama 70B and 1.2 bits for Qwen 72B.

So yes, the phenomenon generalizes somewhat. No, the prompting recipe does not.

That distinction matters for any company tempted to convert this paper into a five-point audit template by Monday morning. Monday morning templates are how nuance goes to die.

What the paper directly shows

The safest reading is compact.

The paper directly shows that, in Qwen2.5-Coder-32B-Instruct, concept-vector injections into earlier KV-cache representations can leave recoverable traces. These traces can shift output probabilities, appear strongly in intermediate layers, support above-chance concept identification, and then attenuate near the final output. The effect is specific enough that simple generic noise or yes-bias explanations are not sufficient.

It also shows that prompting changes access to the signal. Mechanistically accurate descriptions are not always best. Vague salience framing can outperform them. That is annoying, but scientific evidence is under no obligation to flatter our naming conventions.

The strongest contribution is not any single accuracy number. It is the causal chain:

1. inject a known concept into prior internal representations;
2. remove the intervention;
3. ask about the injection;
4. inspect not only the sampled answer but also logits and layers;
5. recover both detection and concept-specific information.

That chain is why the study matters.

What Cognaptus infers for business use

Now the business interpretation.

For enterprise AI governance, the paper is a warning against output-only assurance. If a model’s final answer can deny a signal that is visible in its intermediate representations, then a clean transcript is not the same thing as a complete evaluation.

This matters most in systems where internal state is operationally relevant:

The practical implication is not that every company needs logit-lens dashboards tomorrow. Many companies cannot access intermediate layers because they use closed APIs. Even when they can, representation-level evaluation is expensive, specialized, and easy to misuse.

The more realistic takeaway is layered evaluation.

Start with behavioral tests. Add prompt-variation tests. Add control questions. Track probability distributions where available, not only sampled text. For open-weight or high-risk deployments, add representation-level probes and targeted activation analysis. Most importantly, do not treat “the model said no” as the end of the audit.

A better evaluation stack would distinguish four layers:

That stack is not glamorous. It is instrumentation. Which is precisely why it is useful.

The boundary: this is not proof of general self-awareness

The paper is careful about scope, and the article should be too.

The result is about functional access to prior internal states under controlled concept injections. It is one operational slice of introspection, not a full theory of consciousness, agency, or selfhood. The main model is Qwen2.5-Coder-32B-Instruct, with limited larger-model replications. Prompt sensitivity remains poorly understood. The authors observe where signals emerge and fade, but they do not identify the exact circuits responsible. The emergent misalignment extension is exploratory and less robust than the concept-vector experiments.

There is also an enterprise boundary. Most production teams using hosted LLM APIs cannot inspect intermediate representations. They can still learn from the paper’s logic, but they cannot directly reproduce the strongest measurements unless they use open-weight models or obtain deeper instrumentation from vendors.

So the honest business conclusion is not:

“Your model is secretly self-aware.”

It is:

“Your model’s final answer may understate what its internal computation represented.”

That is enough. Actually, for governance, it is more than enough.

The quiet lesson: outputs are the press release, not the whole company

The most useful image from this paper is not a model waking up and noticing its own thoughts. It is more mundane: an organization where middle management knows something, legal edits it, compliance sanitizes it, and the public statement says, “We have no comment.”

The model’s intermediate layers appear to carry a signal. The final layers reduce it. The sampled answer may deny it.

That does not make the model conscious. It makes the transcript incomplete.

For AI safety researchers, the paper adds evidence that self-relevant information can exist latently and be suppressed before output. For enterprise users, it says that model assurance should not stop at the surface. For executives, it offers a simple but unpleasant audit principle: the answer is not the system.

The model that says “no” may still have passed through “yes” on the way there.

And if introspection can hide in the logits, other capabilities may hide there too.

Cognaptus: Automate the Present, Incubate the Future.