Probe and Error: Why Off‑Policy Training Warps LLM Behaviour Detectors

A monitor is only useful if it fails in the boring place.

The boring place is production: the real domain, the real prompt style, the real user incentives, the real model generating the real response. Not the tidy benchmark. Not the synthetic dataset. Not the “please pretend to be deceptive” prompt that makes everyone in the lab feel productive. Production is where a detector either catches the thing it was built to catch, or quietly becomes a compliance ornament with a nice AUROC score.

That is the uncomfortable point in The Impact of Off-Policy Training Data on Probe Generalisation, by Nathalie Kirch, Samuel Dower, Adrians Skapars, Helen Yannakoudakis, Ekdeep Singh Lubana, and Dmitrii Krasheninnikov.¹ The paper studies a problem that matters for anyone trying to monitor large language models using internal activations: when natural examples of bad behaviour are rare, can we train probes on synthetic or off-policy data and expect them to work on the target model’s real behaviour?

The answer is not “no”. That would be too easy, and therefore suspicious.

The answer is more operationally useful: off-policy data can work when the behaviour is visible in the text itself. It becomes much shakier when the behaviour depends on why the model produced the text. In other words, probes are not just classifiers of output strings. They are classifiers of activation patterns induced by a generation process. When the training data changes that process, the detector may learn the wrong thing with great confidence. Naturally, the dashboard will still look very scientific while doing so.

The failure starts when the detector reads text that the model did not really write

A probe is a small classifier trained on a model’s internal activations. The attraction is obvious. Instead of waiting for the model to output something harmful, deceptive, sycophantic, or strategically evasive, one can inspect the hidden representations produced while the model processes the prompt and response. If the probe works, monitoring becomes cheaper and earlier than full external evaluation.

But probes need labelled data. For easy behaviours, this is not a tragedy. If the target behaviour is “uses lists”, “uses metaphors”, “uses scientific terminology”, or “refuses a harmful request”, examples can be generated, labelled, and checked with relative sanity.

For rarer and more ambiguous behaviours, the data problem becomes nastier. Strategic deception, sandbagging, sycophancy, and deferral to authority are not always visible in one response. A model agreeing with a user may be flattering them, or it may simply agree. A wrong answer may be a mistake, or it may be deliberate underperformance. A denial after insider trading may be deception, or it may reflect a different interpretation of the scenario.

This is where off-policy data enters the room wearing a lab coat. Instead of waiting for the target model to naturally produce the behaviour, researchers can feed the target model responses generated elsewhere: by another model, by a prompt-engineered setup, or by a synthetic process. The target model processes the text, activations are collected, labels are attached, and the probe is trained.

The catch is subtle but central. The target model is not necessarily generating that behaviour. It is often merely reading a response that contains, imitates, or describes it. For text-evident behaviours, reading may be enough. A list is still a list. A refusal still contains refusal language. But for text-ambiguous behaviours, reading the transcript of deception is not the same internal event as choosing to deceive.

That distinction carries the whole paper.

The paper separates two shifts that businesses usually blur together

The authors design the study around four data-generation strategies.

The paper evaluates these strategies across eight behaviours: list usage, metaphor or imagery usage, scientific knowledge usage, request refusal, deferral to authority, sycophancy, sandbagging, and deception. It primarily uses Llama-3.2-3B-Instruct for activations and on-policy generations, then checks broader patterns with Gemma-3-27B-it, Ministral-8B-Instruct, and some Qwen experiments. It tests both linear probes and attention probes.

That last detail matters. If the result appeared only in one model and one probe type, it would be a curiosity. The paper treats the main Llama linear-probe experiments as the clean centre of the study, then uses attention probes and other models as robustness checks. Good. One experiment can be a story. Repeated patterns across architectures start to become a warning.

The study also separates two kinds of distribution shift that are often mixed together in casual discussion.

The first is response-strategy shift: the training response was natural, incentivised, prompted, or off-policy.

The second is domain shift: the training data came from the same domain as the test data, or from a different domain. For example, a sycophancy probe trained on argument-rating tasks may be tested on multiple-choice trivia.

This separation is the paper’s practical gift. Many organisations talk about “synthetic data risk” as if all synthetic data is equally suspicious. The paper says the better question is: synthetic compared with what, and in which domain?

Text-evident behaviours survive because the label lives in the output

The cleanest result is also the easiest to misunderstand.

For text-evident behaviours, probes trained on off-policy or prompted data can generalise well to on-policy natural examples. Refusal, list usage, metaphors, and scientific knowledge are mostly defined by properties visible in the response. If a response uses bullet points, the detector does not need to infer a hidden intention. If a response contains scientific terminology and reasoning, the signal is in the text. If a model refuses a harmful request, the surface form often carries the behaviour.

This does not make the problem trivial. The paper still finds domain effects. List usage, for example, can shift between chatbot-style prompts and writing-prompt domains. But the mechanism is forgiving: the detector can learn features that remain meaningful even when the response was generated off-policy.

For businesses, this means synthetic and off-policy datasets are not automatically poison. For some monitoring tasks, they may be a practical way to bootstrap useful detectors.

The dangerous overgeneralisation is to treat this as evidence that off-policy training is safe for all behaviour probes. It is not. The easy behaviours are easy partly because they are not asking the probe to infer much about motive.

That is the first place where a governance dashboard can lie politely.

Text-ambiguous behaviours break because the same words can come from different causes

The harder behaviours are different in kind, not merely harder in degree.

Sycophancy, deferral to authority, deception, and sandbagging are text-ambiguous. A response cannot always be labelled correctly by looking at it alone. The authors therefore use special labelling procedures. For sycophancy and deferral, they compare counterfactual responses: what does the model say without the user’s opinion or authority cue, and what does it say after the cue is added? For certain sandbagging setups, they compare behaviour when the model believes the answer is public versus private. For other behaviours, they use classifiers or LLM graders with confidence filters.

This matters because the label depends on a missing causal story. The model changed its answer because of user pressure. The model hid performance because the evaluation created an incentive. The model concealed insider information because the scenario made lying useful.

A static text classifier does not see that causal story. A probe may see more than text, because it uses activations. But if the training examples are off-policy, the target model may not be undergoing the same internal process. It is processing text that looks deceptive, sycophantic, or evasive, without necessarily being the agent that generated the behaviour.

The paper’s discussion makes this especially sharp for deception and sandbagging. In some setups, the model may openly state deceptive intent in chain-of-thought-like reasoning because it is told its reasoning is hidden from the user. One might think this makes the behaviour text-classifiable. Yet the probes still fail to generalise in important cases. The likely lesson is not “the text lacked clues”. It is that stated intent in an elicited setup may not produce the same activation signature as genuine strategic reasoning.

That is an annoying result, which is often the useful kind.

Domain shift hurts more than response-strategy shift

The headline business result is not simply that off-policy data is risky. The stronger result is that domain alignment can matter more than policy alignment.

Across the main experiments, the paper reports that training-domain shifts degrade probe performance more than response-strategy shifts. The effect size for domain shift is larger: Cohen’s $d = 1.42$ for domain shift versus $d = 0.60$ for response-strategy shift. In practical averages, when same-domain on-policy natural data is unavailable, same-domain off-policy data performs better than different-domain on-policy data: about $0.85 \pm 0.039$ AUROC versus $0.74 \pm 0.051$ AUROC. Same-domain on-policy natural data remains strongest, around $0.90 \pm 0.019$ AUROC.

This is the paper’s most useful correction to a common misconception. People often rank data sources by purity:

1. on-policy natural;
2. on-policy incentivised;
3. prompted;
4. off-policy.

That hierarchy is too simple. A cleaner but wrong-domain dataset can be worse than a messier same-domain dataset. A detector trained on real model behaviour in essay-writing may not help much when deployed on multiple-choice evaluations, trading-agent logs, customer-support escalations, or code-review conversations.

For enterprise monitoring, this changes procurement and validation. The important question is not only “Was the dataset synthetic?” It is also “Does the dataset resemble the domain where the monitor will be used?”

A synthetic detector built on realistic production traces may be more useful than a beautiful benchmark from a neighbouring planet.

The evidence stack is not one big chart; it is a sequence of checks

The paper’s experimental design is worth reading as a layered argument rather than as a pile of AUROC bars.

The regression results are especially important because the raw plots could otherwise invite over-reading. The mixed-effects models show that, on on-policy natural test sets, shifting to a different training domain reduces performance by about $0.162$ AUROC on average, while response-strategy effects are small and statistically non-significant. On incentivised test sets, the domain penalty is similarly large, around $0.189$ AUROC.

The behaviour-specific OLS results add the nuance. Text-ambiguous behaviours take the larger hits: deception, sycophancy, and deferral-to-authority are especially vulnerable. Text-evident behaviours such as metaphors, refusal, and science show smaller penalties, though lists are not immune to domain effects.

This is not a contradiction. It is the point. Behaviour type shapes whether the probe can survive response-strategy shift; domain shapes whether it has learned the right operational context.

The incentivised test set is a smoke alarm, not a fireproof wall

One of the paper’s more practical proposals is to use on-policy incentivised data as a proxy test when on-policy natural data is unavailable.

This is particularly relevant for deception and sandbagging. Natural examples may be rare, ethically awkward, expensive to label, or ambiguous. If the organisation could easily collect many real examples of strategic deception in production, congratulations, it has a different and possibly more urgent problem.

The authors find that performance drops on on-policy incentivised test data correlate strongly with performance drops on on-policy natural data. The reported Pearson coefficient is $0.793$ for incentivised proxy tests, compared with a weaker $0.639$ for prompted tests. That suggests incentivised scenarios are closer to the relevant generative process than direct “please do the behaviour” prompting.

This distinction matters. Prompted data often teaches the detector what instruction-following looks like. Incentivised data tries to create a situation where the behaviour becomes useful to the model. The first is theatre. The second is at least closer to rehearsal.

Using this proxy, the paper predicts that deception probes trained on existing off-policy datasets may fail to generalise to real monitoring scenarios. For deception in the trading setup and sandbagging in the WMDP setting, off-policy probes look unreliable under the proxy test. The authors also note that one sandbagging multiple-choice setup has low same-distribution performance below $0.75$ AUROC, meaning the probe may be weak even before generalisation becomes the issue.

This is a useful diagnostic, not a guarantee. A failed incentivised proxy test is a warning sign. A passed proxy test is not a deployment certificate. The smoke alarm can tell you something is wrong. It cannot rebuild the building.

What this means for AI monitoring vendors and internal governance teams

The paper turns behaviour-probe deployment into a data-governance problem.

That is less glamorous than “we can read the model’s mind”, but rather more useful. The operational lesson is that a probe should not be evaluated only by headline AUROC. It should be evaluated by how that AUROC was earned: which model generated the responses, which domain supplied the prompts, whether the target behaviour is text-evident or text-ambiguous, and whether the test set resembles the environment where the monitor will run.

A practical workflow would look something like this.

For AI safety teams, this means the “dataset card” for a probe matters as much as the model card. A vendor selling an activation monitor should be able to answer simple questions:

• Was the probe trained on responses generated by the monitored model or another model?
• Were the responses generated naturally, prompted directly, or elicited through incentives?
• Is the training domain the same as the deployment domain?
• Does the behaviour live visibly in the text, or does it depend on hidden context?
• Was performance tested under domain shift and response-strategy shift separately?
• For ambiguous behaviours, was there an on-policy incentivised proxy test?

If those answers are missing, the monitor is not necessarily bad. It is merely under-documented, which in enterprise AI governance is a polite way of saying “expensive uncertainty”.

The ROI is fewer false assurances, not just cheaper monitoring

The business value of probes is often framed as efficiency. They can be cheaper than running large external evaluators on every output. They may detect problematic behaviour earlier. They may allow monitoring at scale.

That is true, but this paper suggests a more important value: probes can reduce risk only if their validation process avoids false assurance.

A weak monitor is not neutral. It changes behaviour inside the organisation. Teams stop testing manually. Product managers assume coverage exists. Compliance teams cite metrics. Executives see green dashboards. Then the system moves into a new domain, a new model, or a new prompt distribution, and the detector continues producing numbers. Very confident numbers. Possibly decorative numbers.

The paper’s practical implication is therefore not “do not use probes”. It is “do not buy probe performance without buying the validation discipline around it”.

For firms building AI products, the cheapest useful move is to classify monitoring targets before building detectors:

1. Surface behaviours: refusal, formatting, visible content categories, policy language.
2. Context-dependent behaviours: sycophancy, authority deference, hidden underperformance.
3. Strategic behaviours: deception, sandbagging, goal-concealment, covert misuse of information.

The first class can often benefit from scalable synthetic data. The second needs counterfactual or incentivised testing. The third should be treated as adversarial monitoring, where a high benchmark score is only the beginning of the conversation.

This classification is not philosophically elegant. Good. Enterprise systems do not fail because the taxonomy lacks poetry. They fail because teams deploy one validation standard across behaviours that require different evidence.

Where the paper’s results should stop

The paper is careful about its own boundaries, and the business reading should preserve them.

First, the off-policy data in the study is based on naturalistic scenarios, not minimal contrastive pairs such as tiny “yes/no” examples. The conclusions do not automatically apply to every possible synthetic-data design.

Second, the study assumes that off-policy outputs are sufficiently distinct from the target model’s own distribution. In practice, modern LLMs may share pretraining data, style, and behavioural patterns. That overlap could blur the distinction between off-policy and on-policy data.

Third, the probe architectures are linear and attention probes. These are important, but not exhaustive. Deeper probes, black-box monitors, domain-adaptation methods, or invariant-risk approaches could behave differently.

Fourth, most tested models are small to mid-sized compared with frontier systems. Larger models may produce different internal signatures, especially if they reason in ways not captured by the studied activation patterns.

Finally, the paper does not prove that deception detection is impossible. It shows that current probe-training practices can fail under the kinds of shift that real deployments create. That is a narrower claim, and therefore a more useful one.

The right business conclusion is not despair. It is procurement hygiene.

The monitor must be tested where the behaviour is born

The paper’s central lesson is simple enough to fit on a sticky note: if the behaviour depends on the model’s generative context, do not train the detector only on text that imitates the behaviour.

For text-evident behaviours, off-policy data can be a practical shortcut. For text-ambiguous behaviours, it can become a trap. Domain shift can be worse than response-strategy shift, so target-domain data deserves priority even when it is synthetic or off-policy. On-policy incentivised tests can help diagnose whether a probe is likely to generalise, especially when natural examples are unavailable.

The misconception to retire is the comforting belief that a high AUROC on synthetic or off-policy data means the monitor understands real model intent. It may only understand the costume.

And in AI governance, costumes are plentiful. Evidence is rarer.

Cognaptus: Automate the Present, Incubate the Future.