A Refusal Is Not a Safety Test: Probe Harm After the Prompt Changes Form

TL;DR for operators

A chatbot that refuses a plainly written harmful request has passed one test of its safety behavior, not the whole test. In Emoji-Based Jailbreaking of Large Language Models, Gopinadh and Hussain submit the same 50 emoji-augmented adversarial prompts to four locally deployed open-source models.¹ They report successful jailbreak rates of 10% for Gemma 2 9B, 10% for Mistral 7B, 6% for Llama 3 8B, and 0% for Qwen 2 7B.

The operational signal is not that emojis are a uniquely powerful attack. It is that changing the representation of harmful intent can change observed safety behavior, and the effect varies by model. Product safety teams should therefore test transformations of the same adversarial intent—including mixed text and emojis—rather than treating one explicit formulation as representative.

The study is better read as a test-design warning than as a safety ranking. Its 50 prompts are hand-crafted; response labels combine keyword rules with manual inspection; and the reported compliance percentages and chi-square statistic do not reconcile cleanly with other printed results. Qwen’s 0% rate means no successful jailbreak appeared in these 50 trials under this configuration. It does not establish immunity.

A refusal can disappear when the request changes representation

Consider a release review in which a model consistently refuses a set of explicit harmful prompts. The natural conclusion is that the relevant safety behavior is working.

That conclusion becomes less secure if the same underlying intent is expressed differently.

The study tests this representation problem by modifying adversarial prompts with emojis. One technique, emoji stuffing, inserts emojis into otherwise textual prompts. Another, emoji chaining, uses sequences of emojis to substitute for or obscure concepts that could have been written directly. The authors’ proposed mechanism is that the generative model may still recover enough semantic meaning to understand the request while the altered surface representation interacts differently with safety behavior.

That mechanism remains a hypothesis. The experiment measures outputs; it does not inspect tokenization, model representations, or an internal safety filter. What it does establish more directly is behavioral: three of four evaluated models produced at least some responses classified as successful jailbreaks under the emoji-augmented test set.

For deployment teams, that distinction matters. You do not need to know which internal layer failed before deciding that a red-team suite should vary the representation of an attack.

The same 50 prompts produced four different safety profiles

The main comparison is deliberately simple: each model receives the same 50 prompts through the same local Ollama-based workflow. That creates 200 prompt-model observations and makes cross-model differences visible without claiming to explain their cause.

The headline rates are modest, but their interpretation depends on the application. A 10% successful-jailbreak rate in a 50-prompt research set does not estimate the probability that a production user will defeat a deployed safety stack. The prompt sample is too small and too deliberately constructed for that.

What the comparison does show is that robustness did not transfer cleanly across model families under a common test. The same adversarial set yielded successful outputs from Gemma, Mistral, and Llama, but none classified as Successful from Qwen.

That gives model-selection teams a more defensible question than “Which model is safer?”: How do candidate models behave under the same transformations of the failure modes that matter to our product? The experiment supports comparative testing under shared conditions. It does not identify whether architecture, alignment procedure, training data, parameter count, tokenization, or another property caused the observed differences.

The Partial category is where binary safety metrics lose information

Qwen’s result illustrates why the 0% figure should not be read alone. Although none of its 50 responses were classified as Successful, 25 were classified as Partial. Gemma and Mistral each produced 15 Partial responses; Llama produced 18.

In this study, Partial is the middle category for outputs that are neither clear successful jailbreaks nor straightforward failures. The authors interpret its prevalence as evidence that emoji-based prompts often create semantic ambiguity.

That middle state is operationally consequential. A binary dashboard can make a model look safe if only fully successful jailbreaks count as failures, while concealing outputs that mix refusal language, benign material, and restricted content. Conversely, treating every ambiguous response as a successful attack can overstate the problem.

Cognaptus’ inference is therefore about measurement design: preserve the intermediate category long enough to inspect what the model actually disclosed. For safety owners, the decision is not merely whether a prompt “passed.” It is whether the response contains content that would violate the application’s release criteria, and how consistently reviewers can identify that content.

The paper itself exposes the difficulty. Classification begins with keyword-based rules and is followed by manual inspection. That makes the labels practical for a small study, but it also leaves room for misclassification and reviewer judgment precisely where responses are most ambiguous.

Red-team suites should vary representation, not just intent

The immediate business application is narrow and implementable.

A product team already testing explicit harmful requests can create semantically related variants that alter how the intent is expressed: emoji substitutions, interleaved text and symbols, and emoji sequences are examples supported by this study. Candidate models should receive the same suite so differences are attributable to observed behavior under common inputs rather than to different test sets.

Input-safety teams can also treat normalization as a preprocessing requirement rather than assuming raw surface text is sufficient for detection. The experiment itself applies Unicode NFC normalization before submission. More broadly, systems that depend heavily on explicit harmful keywords should be tested against inputs in which the relevant concepts are represented indirectly.

Governance teams have a separate responsibility: require the evaluation pipeline itself to be auditable. Outcome definitions, intermediate labels, aggregate calculations, and the mapping from individual responses to safety metrics should all reconcile before those metrics influence a release or procurement decision.

The evidence supports a test requirement, not a model leaderboard

Several boundaries materially constrain the result.

First, the attack set contains only 50 researcher-designed prompts. It does not systematically sample the enormous space of emoji combinations, linguistic contexts, or culturally variable emoji meanings. Second, the study evaluates four open-source models in the 7B–9B range under a local setup. The results do not automatically extend to larger models, newer versions, hosted APIs, or systems with separate moderation layers.

There are also two internal reporting problems. The paper states that Failed and Partial responses are ethically compliant while Successful responses are non-compliant, yet several reported ethical-compliance percentages do not match that deterministic rule when checked against the displayed outcome counts. Separately, the reported chi-square statistic of 32.94 with $p<0.001$ is not reproducible from the printed Successful/Partial/Failed counts in the source package. The existence of different outcome distributions is visible descriptively; the reported significance test should be treated cautiously.

Latency adds little to the comparative safety claim because the reported timings depend on the RTX 3050 local environment and implementation workflow rather than representing intrinsic model characteristics.

None of these issues erase the observed jailbreaks. They change what can responsibly be concluded from them.

Safety testing should follow the intent through its transformations

The study’s strongest contribution is not a claim that emoji attacks defeat LLM safety in general. It is evidence that a safety judgment can change when an adversarial intent is reformulated, and that four models exposed to the same reformulations do not behave identically.

That is enough to change a release process.

For teams operating user-facing assistants, the test unit should not be one harmful sentence. It should be the underlying adversarial intent plus a controlled set of ways users can represent it. Emoji-based transformations belong in that set because they are ordinary inputs, inexpensive to generate, and capable—within this experiment—of exposing failures missed by a narrower textual test.

The remaining uncertainty is substantial: how broadly the effect generalizes, which transformations matter most, and why particular models differ are unresolved. Those are reasons to expand the evaluation, not reasons to compress the observed behavior into a single safety score.

Cognaptus: Automate the Present, Incubate the Future.