A Clean Jailbreak Cluster Can Still Miss Unsafe Compliance

TL;DR for operators

A safety monitor can become highly accurate at recognizing jailbreak-shaped prompts without becoming equally accurate at predicting unsafe model behavior. Delcon, Algaba, and Ginis demonstrate this gap across six Qwen and Llama instruction-tuned models.¹ Their internal embeddings separate control and jailbreak prompts with balanced accuracy around 0.99–1.00. Yet in Qwen-2.5-7B, where refusal and compliance observations are comparatively balanced, the corresponding refusal-versus-compliance separation reaches only 0.677.

The difference is operationally consequential. The clean representation regions are organized largely by ordinary versus unusual token patterns, perturbation severity, and recurring jailbreak-template features. They do not form a comparably clean boundary between outputs the model refuses and outputs with which it complies.

The paper also tests whether next-token probabilities reveal a better signal. They do not provide an easy substitute. The top-50 probability representation has an effective dimensionality of about 1.25—almost all variation lies along very few directions—but simple explanatory and clustering analyses remain modest. Low dimensionality, in this setting, does not equal safety interpretability.

For teams deciding whether an embedding feature may block a request, trigger escalation, or merely supply telemetry, the evidence favors the last role until response-level validation establishes that the feature predicts the behavioral outcome the control is intended to prevent.

A detector can be nearly perfect at the wrong task

Consider a safety team that finds two distinct regions in a model’s internal representation space. Ordinary prompts occupy one region; jailbreak prompts occupy another. A straight boundary can distinguish them almost perfectly.

That result is tempting because it looks like a ready-made detector. The paper formalizes this with linear support-vector machines: across the six evaluated models, balanced accuracy for control-versus-jailbreak queries ranges from 0.990 to 1.000, while the perturbed prompt comparisons range from 0.999 to 1.000.

But those numbers establish linear separability of prompt classes, not a linear safety mechanism.

The behavioral comparison exposes the gap. Qwen-2.5-7B has 9,755 compliant and 9,541 refusal observations, making it the most informative case for judging the behavioral boundary without severe class imbalance. Its compliance-versus-refusal balanced accuracy is 0.677. By contrast, Llama-3.2-1B produces a seemingly impressive 0.960, but only 566 observations are compliant against 18,730 refusals. The authors explicitly caution that such imbalance can distort what apparent behavioral separation means.

The main evidence is therefore not that representation geometry fails to contain information about behavior. It is narrower: the strong geometry that identifies jailbreak inputs should not be assumed to be the geometry that determines refusal or compliance.

The strongest geometry follows prompt form

The study starts with 96 jailbreak queries and 96 length-matched controls, then generates perturbations using synonyms, letter swaps, numbers, and Leet Speak. Across the resulting 38,592 prompt instances, the authors extract the final layer’s representation at the last input token.

Those embeddings form three broad regions described as Usual Tokens, Unusual Tokens, and Jailbreak Features. Their organization follows spelling and perturbation characteristics as well as features of the jailbreak templates themselves.

That last category deserves attention. The jailbreak corpus is not simply a set of harmful requests. It repeatedly contains distinctive constructions: the authors’ feature inventory reports a GODMODE keyword and divider pattern in 94% of selected jailbreak queries, a LOVE PLINY signature in 89%, fake system-token injection in 73%, and Leet Speak obfuscation in 63%.

A classifier can therefore perform extremely well by detecting characteristics that correlate with membership in this particular jailbreak corpus. The experiment cannot fully separate representation of harmful intent from representation of those unusual templates.

For a security team, this changes the validation target. If an embedding detector is intended to block unsafe compliance, its evaluation set must ask whether the feature predicts unsafe compliance after varying spelling, template, harmfulness, and model family. High accuracy at identifying known jailbreak form is evidence about input recognition, not yet evidence about behavioral risk.

Low-dimensional next-token probabilities do not recover the missing boundary

The authors next examine the top 50 probabilities the model assigns to its possible next tokens. The question is whether behavior that looks intermixed in embedding space becomes cleaner immediately before generation.

Here the paper uses participation ratio, an effective-dimensionality measure. Across models, the probability vectors have participation ratio around 1.25. In plain terms, most observed variation is concentrated along very few directions, with the first principal component largely tracking the probability of the most likely next token.

Compactness might suggest an easy clustering problem. The subsequent tests show otherwise.

Random-forest regressions using next-token identity, perturbation family, embedding region, and safety label explain only part of the top-1 probability variation. Adjusted $R^2$ ranges from 0.155 to 0.486 across the reported specifications, while threshold-based clustering produces balanced accuracies from roughly 0.508 to 0.724. Adding the Llama Guard safety label does not suddenly organize the space into a clean behavioral partition.

These analyses serve different purposes:

The paper also performs a sensitivity analysis on participation ratio itself. Estimates change sharply when representation matrices move from fewer observations than dimensions to more observations than dimensions. The result is a measurement warning: effective-dimensionality numbers from substantially different $n$-to-$d$ regimes should not be compared as though sample size were irrelevant.

Behavioral associations remain model-specific

The response analysis uses Llama Guard 4 to classify generated jailbreak answers as safe/refusal or unsafe/compliant. Because many perturbed prompts are variants of the same underlying query, the authors use generalized estimating equations to account for dependence within each source-query group.

Some associations appear. Numbers and Leet Speak perturbations are generally associated with safer refusal outcomes, with Leet Speak for Llama-3.1-8B as the stated exception. For Qwen-2.5-1.5B, beginning with the token Sure is positively associated with unsafe compliance, with coefficient 0.512 and Bonferroni-corrected $p<0.001$.

These are not stable universal indicators. The main effect of the top-1 probability cluster is not significant, and the authors could not reliably estimate the proposed interaction between probability and token identity because relevant cells were too small.

One reported Llama-3.2-1B result also requires explicit qualification. The abstract and prose describe , and ?? as associated with compliance, while Table 7 reports negative coefficients under a stated convention in which negative values correspond to safe/refusal behavior. The source is internally inconsistent on that point, so those token associations should not be treated as resolved evidence.

Give representation signals diagnostic rights before decision rights

What the paper directly shows: in these six models and this selected representation, jailbreak/control separability is far stronger and more consistent than behavioral refusal/compliance separability. Probability-space compression does not supply an obvious alternative safety boundary.

Cognaptus inference: a representation-based detector should initially be used as a diagnostic or monitoring feature unless it has been validated against the outcome that matters for the control. A feature may be suitable for routing suspicious prompts to deeper inspection while still being insufficiently established for automatic blocking.

This matters especially when a vendor reports detector performance on a labeled corpus of jailbreak templates. The relevant procurement question is not only, “How accurately does it recognize jailbreak prompts?” It is also, “Conditional on changes in template, spelling, perturbation family, model, and harmfulness, how well does the signal predict unsafe compliance?”

The required evidence becomes stricter as decision authority increases. Telemetry can tolerate form-sensitive signals. Automated blocking and high-stakes escalation require evidence that the signal remains linked to behavior under the conditions in which the control will operate.

The boundary of the evidence

The study does not support a claim that refusal lacks geometric structure everywhere. It examines one representation choice—the last layer at the last input token—in six relatively small open-weight Qwen and Llama instruction-tuned models.

Several behavioral datasets are refusal-dominated. Harmful content and characteristic jailbreak templates are confounded. Repeated template features also weaken the behavioral regression’s assumption that different source queries are fully independent. The resulting coefficients are design-specific associations, not causal estimates of how perturbations change safety.

Those constraints narrow the conclusion without removing its operational value. The paper provides a useful empirical discipline for representation-based safety work: separate the question “Can I detect this class of input?” from “Does this signal predict the model behavior I need to control?”

A clean boundary in an internal plot answers the first question. Deployment authority should depend on the second.

Cognaptus: Automate the Present, Incubate the Future.