A Signal You Can Decode Is Not Yet a Direction You Can Trust

TL;DR for operators

A model can expose enough information for a linear classifier to recover sentiment without giving you a stable sentiment vector that will reappear when the data sample changes.

Said et al.¹ test this directly across English, Hausa, and Yoruba. Every tested probe remains above chance, yet final-token direction agreement ranges from 0.737–0.870 for English, 0.589–0.762 for Hausa, and only 0.101–0.399 for Yoruba. Across repeated topic allocations, the English > Hausa > Yoruba ordering holds in all 77 complete final-token comparisons.

For teams using activation directions in interpretation, steering, monitoring, or safety controls, probe accuracy should therefore not serve as the validation criterion for the direction itself. Replicate the direction on independent topic samples, check nuisance-feature overlap, record representation readout and layer-selection rules, and validate each deployment language separately.

Tokenizer fertility rises in the same order that reliability falls in these experiments. That makes it a reasonable audit variable, not an established explanation: fertility is entangled with language structure, training exposure, and other factors the study does not causally separate.

A readable signal can still move underneath you

Suppose an interpretability team finds that sentiment is cleanly recoverable from a model’s internal activations. A classifier trained on those activations performs well, so the team extracts a vector representing the contrast between pleased and annoyed text and begins using it for monitoring or activation steering.

The vulnerable assumption is not that sentiment exists in the representation. The experiments show that it often does. The assumption is that the particular direction extracted from one sample is a dependable object that another suitable sample would recover.

Those are different questions.

A concept direction is a vector in activation space intended to capture a semantic contrast. Said and colleagues ask what happens when that vector is independently estimated from different groups of topics rather than judged only by whether a classifier can predict sentiment from the representations.¹

The separation is visible in the numbers. In the primary final-token analysis, Gemma E2B reaches 0.983 probe accuracy for English with direction agreement of 0.852. For Yoruba, its probe still reaches 0.625, but agreement falls to 0.101. AfroLlama shows the same underlying distinction: Yoruba probe accuracy is 0.742 while direction agreement is 0.329.

The model contains predictive information. What changes is whether two samples identify approximately the same direction through that information.

Split-half agreement tests the object, not merely the prediction

The study builds 100 pleased-versus-annoyed contrast pairs across 25 topics for each language. Ten topics are reserved for layer selection and 15 separate topics for evaluation.

For an evaluation split, all examples from a topic stay together. The researchers estimate the sentiment direction from one set of topics, estimate it again from the complementary set, and measure the cosine similarity between the two vectors. Repeating this over 100 topic partitions gives the split-half agreement score.

That design changes the question from “Can sentiment be decoded?” to “If the evidence changes, do we recover the same representation direction?”

The main empirical pattern is unusually consistent within this dataset. Final-token agreement is 0.737–0.870 for English, 0.589–0.762 for Hausa, and 0.101–0.399 for Yoruba across the four models in the primary analysis. The authors then repeat the topic allocation 20 times. Among the 77 comparisons in which all three languages have eligible layers, every one retains the English > Hausa > Yoruba ordering.

The repeated allocations are sensitivity analyses, not 77 independent replications. They reuse the same underlying observations. Their value is narrower: the ordering is not an artifact of one fortunate division between layer-selection and evaluation topics.

Reliability depends on choices around the direction

The paper does not treat split-half agreement as a self-validating metric. Several additional tests ask whether an apparently stable direction survives changes in how the representation is constructed and whether it might be tracking something other than sentiment.

The sentence-length constraint is especially relevant operationally. High reproducibility can be misleading if the same nuisance feature appears consistently in both samples. The study therefore selects layers only when sentiment-direction overlap with sentence length stays below a threshold, and checks alternative thresholds as sensitivity tests.

Reliability is consequently not one scalar property waiting to be read off a dashboard. It depends on what representation is sampled, how the direction is estimated, what data are allowed to define it, and whether competing features have been ruled out.

Tokenizer fertility identifies where to inspect, not why reliability falls

The cross-language ordering invites an explanation. One candidate examined in the paper is tokenizer fertility: roughly, how many model tokens are required to encode a unit of text.

For native text, fertility is about 1.14 for English. Hausa is higher—about 1.78 in the Gemma models and 2.03 in AfroLlama—while Yoruba reaches roughly 2.57 and 2.95 respectively. Reliability declines in the same language ordering.

That association is worth recording in an audit because heavier fragmentation could plausibly affect how linguistic information is distributed through model representations. But this experiment cannot assign the observed reliability differences to tokenization.

Fertility covaries with pretraining exposure, language family, structural characteristics, and other unisolated differences. Only three languages and one sentiment contrast are tested. Three native authors per language also wrote different topic blocks, leaving writer and topic effects partly entangled.

A multilingual team can therefore use high fertility as a reason to run stronger reliability checks. It should not treat fertility reduction as a demonstrated remedy.

Operational validation should happen before a direction becomes infrastructure

The paper directly establishes a measurement distinction: predictive recoverability does not guarantee reproducibility of the extracted direction.

Cognaptus extends that result to an operational rule. If a direction will influence model behavior or a governance decision, its validation target should match that use.

For an interpretability report, an unstable direction can make explanations sample-dependent. For activation steering, it can mean that an intervention derived from one topic set does not represent the same semantic axis on another. For monitoring or safety systems, it can turn an apparently semantic control into a brittle rule tied to the data and representation choices used during construction.

A practical validation record should therefore include the language, model and layer, representation readout, direction estimator, topic partitioning procedure, nuisance-feature checks, and replication agreement on held-out samples. Probe accuracy belongs in that record, but it answers a different question.

This study does not establish a universal reliability threshold, and its confidence intervals are approximate and conditional on the selected layers and observed topic sample. Nor do three languages determine what should be expected across multilingual models generally.

What it does establish within its tested setting is more foundational: before calling an activation vector a reusable concept direction, test whether the direction itself comes back.

Cognaptus: Automate the Present, Incubate the Future.