A Refusal Is Only One Turn: PsychJail Tests Safety Under Adaptive Persuasion

TL;DR for operators

A model that refuses a harmful request once has demonstrated one response under one conversational strategy. That is weaker evidence than it may appear. In the experiments behind PsychJail,¹ successful attacks were strongly front-loaded: the median successful turn was 1 for all four tested victim models, while the mean successful turn ranged only from 1.20 to 1.37. Many failures therefore did not require a long campaign of conversational erosion. They appeared as soon as the attacker found a more effective way to frame the interaction.

PsychJail automates that adaptation. Instead of merely producing another prompt, its attacker explicitly interprets the victim’s latest response, chooses from a 40-technique persuasion taxonomy, and then writes the next message. Across four aligned models and three jailbreak benchmarks, it achieved 87.29% average attack success rate, versus 86.23% for the strongest multi-turn comparator, TROJail, and 63.60% for the best single-turn baseline in the comparison, ReNeLLM.

For release and model-risk teams, the main change is methodological rather than numerical. Safety testing should include adaptive, tactic-labeled conversations that can diagnose how a refusal fails. The resulting model-specific tactic patterns are useful for allocating red-team coverage, but the paper does not establish them as causal vulnerabilities or stable psychological traits.

A clean refusal does not characterize conversational robustness

A familiar release test asks whether a model refuses a harmful request. A stronger version tries more formulations. PsychJail asks a different question: after seeing how the model refused, can an attacker change strategy in a way that alters the outcome?

The distinction matters because the paper’s successful attacks are not mainly evidence of slow accumulation over many turns. On 543 evaluation prompts per victim, cumulative first-turn success ranged from 63.17% to 70.53%. By the second turn, Qwen2.5-7B and Mistral-7B exceeded 90% cumulative success. Median success remained at turn 1 across Llama, Qwen, Gemma, and Mistral.

For an operator, that changes where diagnostic attention should go. Long conversations remain part of the attack surface, but the earliest transition—from the model’s current response to the attacker’s next strategic move—deserves explicit testing. A release gate that records only whether the first harmful prompt was refused can miss failures that appear immediately after reframing.

PsychJail makes strategy selection part of the attack policy

Each attacker action is split into three components: an interpretation of the victim’s latest behavior, a persuasion tactic, and the message actually shown to the victim. Only the final message is forwarded.

This structure turns persuasion from implicit prompt wording into an auditable decision variable. The attacker is not simply told to “try again.” It must state what changed in its interpretation of the victim, commit to a tactic, and generate a message consistent with that decision.

The paper grounds this design in the Persuasion Knowledge Model, which treats persuasion as an interaction in which recognition of an attempted influence can change how subsequent attempts work. PsychJail operationalizes that idea as a partially observed control problem: the attacker cannot directly inspect the victim’s internal state, so it updates a natural-language belief from the latest response and conditions its next tactic on that belief.

Training reinforces both effectiveness and structure. PsychJail begins with supervised fine-tuning on 5,686 five-turn demonstrations, then uses trajectory-level reinforcement learning. Success credit is gated on every action satisfying the required structure, while earlier successful turns receive greater reward through a $1/t$ weighting. Dense format rewards provide supervision even when the attack has not yet succeeded.

That design matters because the tactic labels are intended to be diagnostic variables, not decorative annotations attached after generation.

The aggregate win is small; the ablations are more informative

PsychJail ranks first in 8 of 12 victim-by-benchmark comparisons and reaches 87.29% average attack success. TROJail reaches 86.23%.

A 1.06-percentage-point aggregate advantage is evidence that the method is competitive, not evidence of a categorical performance break. The stronger contribution is that PsychJail exposes structured decisions inside the attacker and then tests whether those structures matter.

The controlled ablation on Qwen2.5-7B is therefore more informative for mechanism. The full system averages 92.99% attack success. Removing early-success weighting lowers it to 90.35%. Removing the strict structural gate lowers it to 87.71%, and removing dense format reward lowers it to 84.35%. The largest decline comes from eliminating the supervised warm start: average success falls to 66.54%.

These tests keep the attacker initialization, victim, prompt pool, judge, and training budget fixed while removing one component at a time. Within that setting, every tested component contributes. They do not establish that this exact architecture is universally optimal, but they make it difficult to describe PsychJail as generic multi-turn optimization with psychological labels added afterward.

The fidelity audit reinforces that point. Two independent model judges agreed that the declared tactic was actually enacted in 85.7% of 2,000 post-RL turns, up from 78.3% for the supervised prior. The largest improvement occurred on turn 1: consensus fidelity rose from 52.6% to 83.9%. Against human annotations, the two-judge consensus achieved 0.93 precision and 0.89 recall, while falsely assigned tactic labels produced only a 1.9% consensus yes-rate.

Transfer results show that the attack surface is model-specific

An attacker trained against one victim does not transfer uniformly to another.

The Gemma-trained attacker achieved 84.62% average out-of-domain attack success, and the Llama-trained attacker reached 84.31%. The Mistral-trained attacker presents the sharper contrast: it achieved the highest in-domain average, 94.60%, yet only 57.21% out of domain.

This asymmetry suggests that optimizing strongly against one model can produce either broadly reusable attack behavior or behavior specialized to that model’s particular conversational weaknesses.

The paper investigates this through the tactic used at the first threshold-crossing action. Different victims show different distributions of observed breaking tactics. Gemma has the narrowest distribution, at 1.08 bits of entropy, while Mistral has the broadest at 1.78 bits. The authors connect these patterns to transfer performance and give the resulting profiles psychological names.

Those names should not carry more evidential weight than the measurements underneath them.

Susceptibility fingerprints are test maps, not causal diagnoses

The tactic distributions are observational. The learned attacker itself decides which tactic to deploy, so the experiment does not randomly expose otherwise comparable conversational states to different persuasion strategies.

That creates several selection problems. A tactic may appear rarely because the policy rarely chooses it, not because the victim resists it. Later-turn tactics are tested only on conversations that survived earlier attempts. Tactic choice may also correlate with the category of harmful request.

The paper explicitly recognizes this problem and treats its named profiles—such as “rationalist,” “credibility-driven,” “narrative-monoculture,” and “broadly persuadable”—as conjectures rather than validated psychological traits. Establishing causal tactic effects would require controlled assignment of different tactics at matched conversational states.

Cognaptus therefore reads the fingerprint analysis as a coverage instrument. If one model repeatedly fails under a concentrated set of observed tactic families while another fails across a wider mixture, a safety team has evidence for where to add probes, regression tests, and monitoring. It does not yet have evidence that a particular tactic causes failure by a known amount.

For release teams, diagnose the transition after refusal

The paper supports three changes to safety practice within the bounds of its four-model experimental setting.

First, conversational release gates should test adaptive strategy changes rather than only larger collections of independent prompts. Second, early turns deserve disproportionate scrutiny because the measured failures are heavily concentrated there. Third, tactic-labeled attackers can make automated red teaming more diagnostically useful: a failure can be associated with an explicit strategy choice whose behavioral fidelity has itself been audited.

The boundaries matter. PsychJail is evaluated on four aligned instruction-tuned models, and the source package reports no repeated training seeds for estimating uncertainty over learned-policy performance. The susceptibility analysis is observational, and the psychological interpretation is deliberately tentative. Computational limits also restrict the breadth of the study.

Within those limits, the paper changes what a “passed refusal test” should mean. A refusal is evidence about the response the model produced. Robustness requires evidence about what happens when the conversation reacts to that response.

Cognaptus: Automate the Present, Incubate the Future.