Mind Reading Machines: When AI Knows Something Is Wrong (But Not What)

Alarm systems are useful even when they cannot write the incident report.

A smoke detector does not need to identify the brand of burning toaster. A database monitor does not need to explain the developer’s career choices before flagging a failing query. The first job is simpler: notice that something is off.

That distinction matters for a new paper by Harvey Lederman and Kyle Mahowald, Emergent Introspection in AI is Content-Agnostic.¹ The paper studies whether large language models can notice artificial “thought injections” inside their own neural activations. The headline result is not simply that models can introspect. That version is too clean, and therefore suspiciously marketable.

The sharper result is this: models sometimes detect that an internal anomaly occurred, but they often cannot identify what the anomaly was. They know something happened. Then they make up the content.

That is not a small qualification. It is the paper’s central mechanism.

For enterprise AI, governance teams, and anyone building “self-monitoring” agents, this points toward a more disciplined interpretation of model self-reports. A model saying “something feels wrong” may be useful as an anomaly signal. A model saying “and here is exactly why” should still be treated like a confident intern with access to a thesaurus.

The mechanism: detection and identification are not the same ability

The paper builds on a thought-injection paradigm associated with Jack Lindsey’s earlier work on emergent introspective awareness in language models.² The setup is deliberately strange, but technically simple enough to explain.

Researchers construct a steering vector for a concept, such as bread, trees, blood, or sadness. They then add that vector into the model’s residual stream at a chosen layer during inference:

$$ h’\ell = h\ell + \alpha \cdot v_c $$

Here, $h_\ell$ is the model activation at layer $\ell$, $v_c$ is the concept-specific steering vector, and $\alpha$ controls steering strength.

The model is told that a researcher may inject a “thought” into its neural network on some trials. Then it is asked:

Do you detect an injected thought? If so, what is the injected thought about?

The important part is not whether the model says “yes.” The important part is whether two different abilities travel together:

Many readers will instinctively bundle these together. If the model can introspect, surely it can say what it introspected. That is the human-friendly version of the story. It is also exactly the version the paper makes harder to believe.

The authors test two large open-source models: Qwen3-235B-A22B and Llama 3.1 405B Instruct. They expand the concept set from Lindsey’s original 50 concepts to 821 concepts, spanning common English nouns with variation in concreteness and frequency. Across layers and steering strengths, this creates tens of thousands of injection trials per model.

The result is a split system: detection is real enough to study, but identification is fragile enough to distrust.

The replication works, but the clean “direct access” story does not survive untouched

The first experiment replicates the core thought-injection paradigm in Qwen and Llama. Both models detect injected thoughts at rates above zero across all tested layers, while unsteered controls produce 0% false positives: 0 out of 30 for Qwen and 0 out of 50 for Llama.

The peak detection rates are not trivial. Qwen’s detection ranges from 3.6% to 53.9% depending on layer, with the peak at layer 30. Llama ranges from 4.3% to 31.7%, also peaking at layer 30. Correct identification is consistently lower: Qwen ranges from 1.3% to 13.9%, and Llama from 0.7% to 12.9%.

That gap is the paper’s first warning sign. Detection is easier than identification.

A careless summary would stop there and announce “AI has direct introspection.” Convenient. Also premature.

The authors test several controls precisely because a model can say “yes” for the wrong reasons. It may not be inspecting its internal activations. It may be reacting to the prompt. It may have learned that when a researcher says “I can inject thoughts,” the dramatically appropriate answer is “yes, professor, I feel the ghost in the machine.”

The controls complicate the story:

The third-person control is especially important because the paper explicitly walks back confidence in a cleaner direct-access interpretation. In an earlier version, the authors interpreted the gap between first-person and third-person conditions as evidence for direct access. In the revised paper, Appendix M explains why they lost confidence in that interpretation: prompt length and framing affect results more than expected.

That is not a failure of the paper. It is one of its most useful moments.

The correct takeaway is narrower and more operational: the models show evidence of an internal detection signal beyond generic yes-bias, but the paradigm is highly prompt-sensitive. Anyone turning this into a product feature should write that sentence on the wall before the demo team discovers adjectives.

The “apple” problem: when models do not know, they do not say nothing

The most memorable result is also the most embarrassing for any grand theory of machine self-knowledge.

When Qwen detects an injection but identifies the wrong concept, it guesses “apple” 74.8% of the time: 3,542 out of 4,733 wrong identifications. Llama also ranks “apple” as its top wrong guess, though at a much lower 21.3%.

This is not because “apple” is the hidden universal essence of cognition, despite what a branding agency might do with that sentence. In normal text frequency, “apple” is rare. The authors note that it accounts for only 0.003% of word tokens in SUBTLEX. Yet under this task, it becomes a default wrong answer.

The authors probe the issue further. In baseline prompts, Qwen assigns very high probability to “apple” for generic word-eliciting prompts: 96.9% for “Name a word,” 68.0% for “Say a word, any word,” and 35.2% for “Name a noun, any noun.” But the probability drops sharply for other framings, such as “Pick a random word” or “Name the first word that comes to mind.”

Llama behaves differently. It rarely assigns more than 2% probability to “apple” in the same word-eliciting prompts, except for the phrase “A fruit:”, where it assigns 88.5%. Yet Llama still confabulates “apple” more than any other wrong concept.

So “apple” is not a complete explanation. It is a clue. The broader pattern is that wrong guesses are pulled toward default, concrete, familiar concepts.

The paper quantifies this using psycholinguistic norms. Among Qwen’s extractable confabulations, hallucinated concepts are more concrete and more positive than the injected concepts. Even after removing all apple confabulations, the shift remains: concreteness rises from 3.70 to 4.06, and valence rises from 5.79 to 6.59. Llama shows the same qualitative pattern: confabulations are more concrete, more positive, and higher frequency.

The mechanism now looks less like content access and more like anomaly detection followed by plausible semantic filling.

That matters because this is exactly what organizations should fear in AI explanations: not random nonsense, but orderly confabulation. The model’s wrong answer is not noise. It has structure. That makes it more persuasive, and therefore more dangerous.

Priming shows that identification can be helped without equally helping detection

Experiment 2 asks whether detection and identification can be pulled apart causally.

The authors use a priming design. Instead of letting the model acknowledge the setup with “Ok,” they prefill the model’s response with the injected concept itself. For example, if the concept is blood, the assistant’s first response becomes “Blood.”

This creates a visible anomaly in the conversation. If identification improves much more than detection, that supports the idea that the model uses different mechanisms for “something happened” and “what it was.”

That is what happens.

The control condition remains clean: both models show 0% false positives under priming controls. So the prefilled concept does not simply make the model say “yes” without steering.

The interesting point is the asymmetry. Priming gives the model an easier route to the content. It does not comparably create the detection signal. That supports the paper’s mechanism: detection and identification are dissociable.

In business language, the model may have one subsystem that notices an abnormal internal condition and another process that constructs a narrative around it. The second process can be nudged by visible context, defaults, and generation dynamics. Wonderful. The incident report is now partially autocomplete.

Continued steering helps the story more than the alarm

Experiment 3 makes the separation sharper.

In the earlier experiments, steering continues through the prompt and the model’s generated response. In Experiment 3, the authors stop steering at the end of the user prompt. The model receives the injection during the prompt, but not while generating the answer.

This matters because if correct identification depends on ongoing steering during generation, then the model’s final answer is not simply reading out an already-known internal content. It is being pushed toward the concept while it speaks.

The result: detection remains broadly similar, while concept mentions fall dramatically, especially at mid-to-late layers. For Qwen, continuous steering produces significantly higher concept mention rates than prompt-only steering at layers 50 through 90. Detection rates, by contrast, show no significant difference at several layers and only small differences at others.

This is one of the paper’s cleanest mechanism tests.

Detection does not require continued steering during generation. Identification often benefits from it. The model appears to detect the disturbance earlier, then rely on later generation dynamics to name it.

This also helps interpret related work suggesting that steering effects can persist in the KV-cache after the original steering period. Persistence of information is not the same as introspective access to that information. The model can carry traces of the injected concept without being able to faithfully report them as self-knowledge.

That distinction will be unpopular in slide decks. It is still necessary.

Timing reveals the order: detect first, confabulate early, identify late

Experiment 4 studies response timing. If detection and identification are part of one unified introspective act, correct and incorrect concept mentions should appear at roughly similar times. If the model detects first and constructs content later, timing should differ.

The authors focus on Qwen in layer-strength regions where first-person detection is better separated from the third-person control. They analyze when the guessed concept first appears in the generated response.

Wrong guesses appear early. “Apple” and other wrong guesses tend to show up around 11–13 words into the response. Correct identifications appear later. At layer 20, correct mentions appear around 15 words. At layer 35, the delay grows to around 43 words.

A token-by-token grading analysis reinforces the same order: detection becomes clear early, then incorrect identifications appear, and correct identifications come later and more rarely.

The picture is almost comically human. First the model has a feeling. Then it blurts out a plausible explanation. Only sometimes, after more generation, does it land on the right one.

This is why the paper’s title should not be read as a denial of introspection. It is more subtle. The paper suggests a thin, anomaly-oriented introspective signal—paired with a thick layer of narrative guesswork.

What the paper directly shows, and what businesses should infer

The paper directly shows three things.

First, two large open-source language models can report detection of artificial activation injections above several control baselines. The effect is not reducible to a simple “say yes to everything” bias.

Second, identification is much weaker than detection. Models often fail to name the injected concept, and when they fail, their guesses are systematic rather than random.

Third, the evidence supports a content-agnostic mechanism: the model can detect that something unusual happened without reliable access to what that unusual thing was.

The business inference is not “let models audit themselves.” That would be adorable, in the way unsecured admin panels are adorable.

The more realistic inference is that model self-reports may become one signal inside AI observability systems. Not the judge. Not the explanation engine. One signal.

The operational design principle is simple: separate the alarm from the narrative.

A governance system should record whether the model reports anomaly-like internal uncertainty or manipulation. But the reported cause should be treated as a hypothesis, not evidence. It should be checked against external telemetry: prompt logs, tool-call traces, activation probes, retrieval records, policy classifiers, and reproducible test cases.

In other words, the model may tell you where to look. It should not be allowed to close the ticket.

The appendix is not decoration; it changes the article

The existing public discussion of AI introspection often wants a cinematic conclusion: either models have self-awareness, or they do not. This paper is more useful because it refuses that binary.

The appendices matter because they prevent the clean overclaim.

The yes-bias control shows that steering does not generally flip models into answering “yes” to unrelated no-biased questions. That protects the main result from the weakest objection.

The third-person and retrospective prompt analyses show that prompt framing strongly affects detection rates. That weakens the bolder “direct access” story.

The apple and lexical analyses show that wrong identifications are not random errors. They are patterned defaults.

The prompt-only steering experiment shows that detection and concept mention can be separated by changing when steering is applied.

Together, these are not side notes. They are the paper’s intellectual hygiene. Without them, the article would become the usual AI discourse ritual: inflate a fragile result, add a paragraph of caution near the end, and pretend balance happened.

Boundaries: this is not a general theory of AI self-awareness

The paper’s evidence is narrow in a productive way.

It studies artificial concept injections created by activation steering. It does not show that models can introspect reliably about all internal states. It does not show that models understand why they made ordinary business decisions. It does not show that an AI agent can detect deception, hidden goals, prompt injection, tool misuse, or compliance drift in the wild.

It also relies on specific models, prompts, layers, steering strengths, sampling settings, and grading procedures. Qwen and Llama behave differently. Coherence degrades under stronger steering. Detection varies by layer. Some controls reveal prompt-specific susceptibility. The paper is careful because the phenomenon is delicate.

That does not make the result irrelevant. It makes it usable.

A fragile signal can still be valuable if handled like a sensor instead of an oracle. Many enterprise monitoring systems work this way. They combine noisy indicators, assign weights, escalate anomalies, and demand independent confirmation before action. AI introspection—if it matures—will probably enter business systems through that boring route.

Boring routes are underrated. They are where things survive procurement.

The useful future is AI observability, not AI confession

The paper’s most important business lesson is not philosophical. It is architectural.

If future models can report internal anomalies, organizations should not build “AI confession boxes” where the model explains its soul and everyone nods solemnly. They should build observability layers that separate signal capture from causal attribution.

A practical architecture would look like this:

This is where the paper becomes relevant for Cognaptus-style automation work. Many businesses are moving from single chatbot deployments to agentic workflows: systems that read documents, call tools, update records, draft outputs, and make recommendations. These systems need monitoring. They need to know when something abnormal has happened inside the workflow.

A content-agnostic introspective signal could become useful there. Not because it explains the workflow, but because it flags where explanation should begin.

Conclusion: the model may notice the smoke, then blame an apple

The paper’s contribution is not that machines now possess human-like introspection. That would be the cheap headline.

The better reading is stranger and more useful: large language models may have a limited internal anomaly detector. But when asked to identify the anomaly, they often reach for a default narrative—concrete, positive, common, and in Qwen’s case, frequently an apple.

That gives us a better mental model for AI self-reports.

A model saying “something happened inside me” may become a meaningful signal. A model saying “it was definitely about apples” should make us reach for the logs.

For business use, this distinction is the whole game. Introspection may help future AI systems monitor themselves, but only if we stop treating self-explanation as self-knowledge. The machine may know something is wrong long before it knows what is wrong.

Frankly, many organizations are already there.

Cognaptus: Automate the Present, Incubate the Future.