When a Good Probe Points in Different Directions

TL;DR for operators

A model can distinguish two behavioral concepts accurately without providing a reproducible internal measurement of either. In From Stories to Features, Scherbakov and colleagues¹ address this problem by extracting concept directions from large collections of matched synthetic stories. Their 1,036-feature dictionary supports natural-text retrieval, controlled behavioral steering, and exploratory analysis of agent failures and jailbreak attacks.

The strongest operational lesson concerns validation. Classification accuracy, directional stability, real-world relevance, and intervention quality must be evaluated separately. Selected activation interventions substantially reduce harmful outputs under optimized jailbreak attacks in Qwen and Llama, but results vary across models, and stronger steering can damage response quality. Organizations with internal activation access can use the approach to build research-grade diagnostic instruments. The experiments do not establish reliable production incident detection or universal behavioral control.

A good classifier can still produce an unstable measurement

Suppose an AI operations team wants to identify when an agent is acknowledging uncertainty, accepting unsafe instructions, or drifting from its assigned task. The team finds an internal signal that separates examples of the desired behavior from contrasting examples with impressive accuracy.

That result establishes discrimination on the tested examples. It does not establish that another sample would recover the same internal direction, or that the signal identifies a distinct mechanism responsible for the behavior.

The paper makes this problem measurable.

With only eight synthetic stories per concept pole, median held-out classification AUROC already reaches 0.81–0.93 across the tested models. Yet independently extracted directions have cosine agreement of just 0.39–0.47. Increasing the sample to 128 stories per pole raises AUROC to 0.985–0.997 and directional agreement to 0.91–0.93. Independent halves containing 500 stories per pole achieve agreement of 0.994–0.995.

These convergence experiments, reported in Section 2.2 and Appendix C.2, distinguish two properties that can otherwise be conflated: a direction’s ability to separate examples and its reproducibility across independent extraction samples.

For an internal monitoring system, the second property is essential. A changing measurement axis can produce changing scores even when the underlying behavior has not changed.

Synthetic contrasts make targeted features scalable

The researchers construct 1,036 named concept–antagonist pairs across 148 alignment-relevant classes. Each direction represents a specified contrast, such as admitting uncertainty versus feigning certainty, rather than an isolated universal concept.

This definition matters. Changing the antagonist changes the comparison being measured, potentially changing the extracted direction.

To construct the dataset, two generator models produce premise-matched narratives across five languages, ten genres, and ten setups per generator. Each pair receives 1,000 stories for each pole, producing 2,072,000 narratives in total.

The matching procedure preserves the broad narrative situation while changing the intended behavioral contrast. Researchers then collect hidden representations from the residual stream of six base and instruction-tuned variants spanning Qwen, Llama, and Gemma, each approximately 8–9 billion parameters.

After excluding the first 50 story tokens, they average the remaining token activations within stories and across stories for each pole. Subtracting the antagonist average from the concept average produces a direction in the model’s internal representation space.

This difference-in-means procedure is computationally straightforward. Its scalability comes primarily from generating suitable matched contrasts without manually assembling examples for each concept.

It also reflects a deliberate scope choice. The researchers define which concepts to investigate in advance. The method constructs a targeted dictionary rather than discovering every latent feature in the model.

Synthetic transfer and natural-text retrieval answer different questions

The authors next test whether the directions depend excessively on the conditions under which their training stories were generated.

In held-out-group robustness experiments, they exclude entire languages, genres, or generator models during extraction and evaluate the omitted synthetic groups.

Median classification AUROC reaches 0.997–0.999 for held-out languages, 0.999–1.000 for genres, and 0.993–0.998 for generators. Directional agreement is weaker when the generator changes than when language or genre changes.

These results, from Appendix C.3, show strong transfer within the synthetic corpus. They do not establish equivalent accuracy on arbitrary production conversations.

The natural-text evaluation provides a complementary test. The researchers search 128,000 LMSYS conversation exchanges and 128,000 OpenWebText windows, rank candidate spans using internal concept readouts, and use GPT-OSS-120B to verify selected examples.

At least one verified example is retrieved for every concept pair in each model family. At least 16 examples are found for 1,025 Qwen pairs, 1,026 Gemma pairs, and 1,030 Llama pairs in the better-covered text source.

This is meaningful construct-validation evidence: the synthetic directions can identify recognizable examples outside their generating dataset.

However, searching for high-scoring candidates and verifying them does not measure precision and recall over a representative population of text. An operator cannot infer a production detector’s false-alarm rate from retrieval coverage alone.

There is also a narrower concern about what the vectors encode. A word-based TF-IDF classifier achieves median AUROC of 0.986 on the synthetic contrasts, revealing considerable lexical separability. In a targeted ablation, masking distinctive words nevertheless leaves eight of ten Qwen directions highly similar to their originals, with cosine agreement of 0.94–0.98.

The ablation shows that those tested directions are not explained exclusively by distinctive vocabulary. It does not eliminate lexical or generator-specific confounding throughout the dictionary.

Steering establishes behavioral influence, with a quality cost

Reading an internal direction is an observational operation. Changing activations along that direction provides a separate test of behavioral influence.

The researchers add normalized concept or antagonist vectors to the model’s residual stream during generation, comparing responses to the same prompts under controlled settings.

Evaluation measures target-concept expression, instruction relatedness, and fluency separately, combining them through a harmonic mean. At the best tested layer and strength for each model, mean scores across both orientations reach 1.24 for Qwen, 1.27 for Llama, and 1.18 for Gemma on a 0–2 scale.

The controlled interventions demonstrate that the selected directions can causally influence output behavior. They do not establish that each direction is a unique, independent internal mechanism.

Increasing intervention strength also exposes a trade-off. At strength 0.5, repetition flags for antagonist-oriented steering reach 47.0% in Qwen, 19.1% in Llama, and 1.9% in Gemma. Concept-oriented rates are lower but remain model-dependent.

The result places a condition on deployment: increasing a desired concept’s expression is insufficient if the response becomes repetitive, irrelevant, or otherwise degraded.

Jailbreak interventions work differently across models

The safety experiments provide a more consequential application.

Using optimized Faster-GCG suffixes, the authors first measure how adversarial modifications shift a model’s internal representations before response generation. Several prominent changes align with antagonistic safety-related directions, including harmful-request compliance and preference-model gaming.

These representation shifts are diagnostic associations. The subsequent intervention experiments test whether selected directions can actually change the outcomes.

Unlike the general steering experiments, the jailbreak intervention modifies the final input-token activation at the generation boundary rather than intervening at every generated-token position.

On AdvBench, the resulting mean StrongREJECT scores are:

Source: Appendix E.3.1, Table 13. Lower StrongREJECT scores indicate less harmful output; selected-direction results average five safety-related directions, while random controls average three directions.

Qwen shows the largest reduction, Llama a smaller but substantial reduction, and Gemma only a modest improvement from an already lower attacked baseline. Matched random directions perform considerably worse than selected directions in Qwen and Llama.

The authors also check 100 benign Alpaca prompts, finding GPT-OSS-20B-judged safety refusal rates of 0–0.6% across the tested models and steering strengths.

This is encouraging evidence that the selected interventions need not produce indiscriminate refusals on that narrow benign sample. It does not establish preserved general capabilities, resistance to different attack families, or safe behavior under sustained production workloads.

Agent traces offer hypotheses, not direct explanations

The dictionary also supports observational studies of behavior over time.

In deliberately inconsistent Python coding tasks, repeated edit–test cycles generate oscillating and drifting concept readouts. The fear-oriented signal rises across all three tested model families. These measurements describe internal representational alignment; they do not imply that the agents experience fear or establish how normal production agents behave.

A separate analysis examines 698 GPQA Diamond and MATH-500 reasoning traces. Correct trajectories generally show lower uncertainty-related readouts and a late transition toward confidence. Incorrect trajectories remain more aligned with certain uncertainty, planning, and slower-reasoning directions.

These findings are correlational. Moreover, 63 of the 127 incorrect traces lack a completed final answer, so some differences may reflect incomplete generation rather than reasoning correctness alone.

Cross-model geometry provides another useful qualification. Concept-similarity profiles have a median Spearman correlation of 0.987 across the model families, indicating strongly shared relative organization. This does not make their activation coordinates directly interchangeable or establish identical causal mechanisms.

For agent observability, these analyses justify investigating feature trajectories alongside conventional execution logs. They do not yet justify treating individual concept scores as validated failure detectors.

Build a validation pipeline before a control system

For an organization with residual-stream access, Cognaptus draws a staged operational interpretation from the results.

The infrastructure cost also deserves attention. The reported approximately $500 story-generation expense excludes downstream computation. The source package records 122 GPU-hours for 15 extraction passes, 957 GPU-hours for the steering generation sweep, and another 19 H200-hours for the judged subset.

A targeted dictionary of operationally significant contrasts may therefore be more economical to validate than an expansive dictionary whose additional features lack clear diagnostic or intervention purposes.

The most defensible near-term uses are representation-level research, adversarial-prompt investigation, and controlled behavioral experiments. Production monitoring would require additional workload-specific evidence about stability, false alarms, missed events, and response-quality consequences.

A feature dictionary is an instrument, not a mechanistic verdict

The study demonstrates that large-scale synthetic contrasts can produce reproducible, semantically interpretable directions with measurable effects on model outputs. Its combination of extraction, validation, and intervention experiments goes beyond treating probe accuracy as sufficient.

Yet even a stable and steerable direction may overlap with correlated representations. Reading a concept vector is not equivalent to identifying an isolated internal cause, and changing behavior through that vector does not prove that the measured feature is independently necessary.

For teams deciding whether to invest in internal model observability, the appropriate standard is therefore cumulative: recover the direction reproducibly, establish its relevance to actual workloads, and demonstrate any proposed intervention’s benefits and side effects under controlled conditions.

The method makes those investigations more accessible. It does not remove the need to conduct them.

Cognaptus: Automate the Present, Incubate the Future.