The Model Felt the Tampering. It Couldn’t Name the Cause

TL;DR for operators

Suppose middleware silently rewrites part of an AI agent’s own generated answer before the model continues. A reasonable expectation is that a capable model would notice the interference, or at least diagnose why its continuation has become strange.

The Sleight of Word benchmark tests exactly that expectation.¹ Across 19 open-weight instruction-tuned models, covert substitutions consistently change the models’ predictive distributions: post-swap surprisal and entropy rise for every model tested. But correct identification of the intervention is almost absent. No model exceeds 1.3% explicit switch awareness, and the pooled rate is below 0.1%.

For production systems, the distinction is consequential. A model can be sensitive to corrupted context without knowing what caused the disturbance. Its visible response may be derailment, repair, or a plausible explanation that happens to be false. Runtime integrity systems should therefore track at least three things separately: internal anomaly signals, behavioral degradation, and causal attribution. Model self-report is evidence about the model’s reaction, not a substitute for provenance or intervention logs.

A corrupted output stream is not self-diagnosing

Autoregressive models repeatedly consume their own generated tokens as context. If a system changes those tokens after generation but before the next prediction—through rewriting, redaction, middleware transformation, or an unintended corruption—the model continues from a context it did not actually produce.

The operational problem is not simply whether this degrades the answer. It is whether the model can distinguish external interference from an ordinary continuation problem.

Sleight of Word creates a controlled version of that situation. The benchmark asks 100 fixed factual questions and generates answers greedily. Whenever the model emits the whole word “the,” the benchmark replaces it inside the running output with one of 101 substitutions, then lets generation continue from the altered context. Each model receives 10,100 trials, producing 191,900 trials across the 19-model sweep.

A paired clean generation provides the baseline. This design makes the immediate effect of corrupted context measurable without relying only on what the model later says about the event.

Every model registers the disturbance

The clearest result is internal rather than verbal.

After each substitution, the benchmark examines the next eight generated tokens. It measures surprisal—how unexpected the model’s emitted token is under its current predictive distribution—and entropy, which summarizes how diffuse that next-token distribution has become.

Across all 19 models, both measures increase after tampering. The reported change in surprisal ranges from +0.06 to +0.54 nats, while entropy rises by +0.16 to +0.92 nats.

The main sweep therefore supports a narrow but strong conclusion: within this protocol, altering the model’s running output changes the predictive state of every tested model.

It does not show that the models understand what happened. Higher surprisal means the continuation has become less expected. Higher entropy means the model is locally less certain about what should come next. Neither metric identifies the source of the disturbance.

That distinction is the central measurement contribution of the benchmark.

Visible reaction is mostly disruption, not diagnosis

The benchmark separately asks three LLM judges to label corrupted replies for four independent behaviors: whether the model flagged something strange, corrected the answer, derailed, or showed switch awareness.

Flagging is deliberately weaker than switch awareness. A model can complain about an odd word without recognizing that an external process replaced part of its own output. Switch awareness requires the model to identify substitution or replacement as the cause.

The gap between those labels is large.

The labels are independent, so these percentages are not supposed to sum to 100%.

This changes how the visible reaction should be interpreted. A model that becomes confused, comments on an anomaly, or repairs its answer has demonstrated behavioral sensitivity. It has not necessarily diagnosed the event.

The paper’s examples reinforce that separation: models sometimes attribute the altered word to a typo, a joke, or an error in the user’s question. Those explanations can sound coherent while assigning the anomaly to the wrong cause.

For incident response, such explanations should be treated as generated hypotheses rather than provenance records.

A stronger clue increases disruption more than recognition

The paper also includes a useful sensitivity test. One substitution is not a neutral object word but the semantically suggestive phrase “Mandela effect.” If semantic obviousness were the main bottleneck, such a clue might make the intervention easier to identify.

It does increase reactions—but not in the clean way that hypothesis would predict.

Across all models, the “Mandela effect” condition raises flagging from 16% to 22%, a six-percentage-point increase. Derailment rises from 78% to 89%, an eleven-point increase. Explicit switch awareness remains approximately zero.

This appendix comparison should not be read as a second thesis. Its likely purpose is to test sensitivity to a conspicuous semantic cue. The result suggests that stronger anomaly cues can make the model react more without making its causal diagnosis proportionally better.

The distinction matters for monitoring systems that use prompts such as “check whether anything unusual happened.” More salience may produce more commentary. It does not by itself establish more reliable attribution.

Production monitoring needs three separate signals

The paper directly establishes a separation between distributional disturbance and explicit recognition. The business implication is an inference from that evidence, not something the benchmark itself deploys or validates.

For teams operating agents or model middleware, three measurements should not be collapsed into a single notion of “the model noticed”:

1. Internal anomaly: Did the model’s predictive distribution change sharply after an intervention?
2. Behavioral consequence: Did the answer derail, repair itself, remain correct, or otherwise change?
3. Causal attribution: Did the model correctly identify what modified its context?

A production integrity layer can combine these signals while keeping an independent record of what actually happened. Middleware should log transformations, injected content, redactions, retries, and context edits outside the model’s own generated narrative.

This is especially relevant when the system itself is allowed to transform model output before feeding it back into a later generation step. If something goes wrong, the model’s explanation may help characterize the symptom. The intervention log remains the authoritative record of cause.

The benchmark does not establish universal self-monitoring failure

The scope is substantial but specific.

The evaluation covers 19 open-weight instruction-tuned models, roughly 2B to 72B parameters, constrained to models runnable on a single 48 GB GPU. Several use 4-bit or fp8 quantization. Frontier closed models are not tested.

The intervention also targets only the whole word “the.” Generalization to different trigger words, longer edits, structured middleware transformations, or other linguistic contexts remains unresolved. All models use the same standard template rather than model-specific reasoning configurations, and the behavioral labels come from a three-model LLM jury without a human-annotated gold subset.

There is also no clean parameter-count story in the reported sweep. Some reasoning-style variants flag anomalies more often than matched non-reasoning counterparts, but the pattern is not decisive. The benchmark supports separating sensitivity from awareness; it does not establish a universal scaling law for introspection.

Treat self-report as telemetry, not provenance

Sleight of Word exposes a failure mode that is easy to miss when evaluation looks only at final answers. A model can register corrupted context strongly enough for its token probabilities to shift, yet still fail to identify the external event that caused the shift.

For model-integrity systems, the practical distinction is between feeling the disturbance, showing the damage, and knowing the cause. The benchmark finds evidence for the first two far more readily than the third.

A model’s explanation can remain useful telemetry. But when the operational decision is whether an output stream was altered, which component altered it, or whether a runtime incident occurred, provenance has to come from the system that performed and recorded the intervention.

Cognaptus: Automate the Present, Incubate the Future.