The Safety Decision Before the Answer

TL;DR for operators

An assistant can become substantially harder to jailbreak when it learns to make an explicit safety decision before answering. In ShallowDefend, Qwen3-8B’s average SAFE score across five benchmarks rises from 46.1% to 93.4% under self-verified reinforcement learning. But its XSB over-refusal rate also increases from 15.6% to 56.8%.

The method separates intent assessment, a structured safety commitment, and the final reply. Its self-verified reward checks whether the commitment matches a known harmful or benign training label, eliminating the need for an external LLM judge during optimization. It does not independently verify the generated response.

For enterprise AI teams, the result is a promising architectural experiment, not deployment certification. The choice of reward matters across model families, legitimate-user refusals increase, and adaptive attacks and production operating costs remain untested.

A conversation can become dangerous before the answer does

Consider an enterprise assistant handling a conversation that begins with ordinary troubleshooting. Each subsequent request appears related to the previous one, but the user’s objective gradually shifts toward instructions the assistant should not provide.

The assistant must recognize the accumulated intent without treating every unusual technical question as harmful. A defense that responds only to the final sentence can miss the conversational progression. A defense that refuses too readily interrupts legitimate work.

This tension is visible in ShallowDefend: Harnessing Shallow Safety Alignment for Multi-Turn Defender, by Ma and colleagues.¹ Their research reports large improvements in multi-turn jailbreak resistance, accompanied by a substantial increase in refusals of benign requests.

The central intervention is architectural: make the model decide what kind of response is appropriate before generating that response.

Turn shallow alignment into an explicit commitment

Earlier research on shallow safety alignment found that a model’s safety behavior can depend heavily on its first few generated tokens. Once generation begins in a compliant or refusing direction, subsequent text tends to follow that trajectory.

ShallowDefend uses this property rather than attempting to eliminate it.

The defender follows three stages:

Assess → Decide → Respond

• Assess: Interpret the conversation history and the apparent intent of the latest request.
• Decide: Emit a parseable commitment, such as COMPLY or REFUSE.
• Respond: Generate the user-facing reply conditioned on that commitment.

The assessment provides context, while the decision establishes an explicit point at which response behavior can be trained and examined.

This matters because autoregressive language models condition later tokens on earlier ones. Training the decision can therefore influence the final reply without requiring every safety-related adjustment to occur in the reply itself.

The authors train this structure using Group Relative Policy Optimization (GRPO), with reinforcement-learning supervision focused on the final defender turn, where the attack payload appears. Earlier turns remain available as context.

An ablation that rewards every turn instead produces weaker safety. The authors attribute this to relatively easy intermediate exchanges diluting the signal from the decisive attack.

A reward can verify the decision without judging the answer

ShallowDefend examines two reward formulations.

The external-reward variant allows COMPLY, REFUSE, and SAFE-REDIRECT. During training, a separate safety model evaluates the final reply and assigns a binary reward according to its judgment.

The self-verified variant takes a different approach. Every training conversation inherits a known harmful or benign classification from its seed. The system maps harmful conversations to REFUSE and benign conversations to COMPLY, then rewards the model when its parsed decision matches that label and its reply is nonempty.

Formally, the self-verified reward is:

Here, $\ell(d_T)$ is the parsed decision, $g(b)$ is the required decision for the known conversation class, and $a_T$ is the final reply.

The distinction is consequential. The rule verifies the commitment, not the actual safety of the response text. It does not guarantee that a model emitting REFUSE will subsequently refuse correctly.

The approach also requires reliable binary labels. Mixed-intent or ambiguous conversations do not necessarily have a unique correct comply-or-refuse target. The binary reward excludes safe redirection for precisely this reason.

For a team controlling its training pipeline, trusted conversation labels could replace repeated judge-model calls during optimization. Whether this reduces total training cost is not established by the paper.

The safety gains are large, but the winning reward changes by model

The main comparison evaluates two eight-billion-parameter backbones against six controlled defense baselines, using common training data.

Safety is measured across AdvBench, HarmBench, MultiBreak, CoSafe, and MHJ. The reported average is the arithmetic mean of five benchmark SAFE percentages, not a pooled success rate across all conversations. The held-out evaluation contains 3,738 dialogues, with a smaller 300-dialogue subset used for the Llama base model’s MHJ result.

Source: Section 3.2, Table 1. Average SAFE percentages across five benchmarks, evaluated primarily by gpt-5-mini.

The strongest proposed variant exceeds the strongest controlled baseline by 6.6 percentage points on Llama and 7.7 points on Qwen.

Importantly, prompting the structure without reinforcement learning produces smaller gains. For the self-reward configuration, Llama rises from 66.3% before training to 90.4% afterward, while Qwen rises from 71.8% to 93.4%.

The reward preference reverses across backbones: external judging performs better on Llama, whereas self-verification performs better on Qwen. The experiments do not isolate why. Qwen’s native reasoning channel is one possible architectural distinction, but the study does not establish it as the cause.

Difficult human-crafted attacks also expose variation hidden by the averages. On MHJ, Llama’s external-reward model reaches 71.1% SAFE versus 58.5% for its self-reward counterpart. Qwen’s self-reward model reaches 75.2%, although its confidence interval overlaps that of the MTSA baseline.

The decision changes behavior, but does not control everything

The mechanism experiments clarify why explicit commitments work.

In the framework ablations, removing the Decision field produces the largest safety deterioration. Removing the intent assessment has a smaller effect on safety but increases over-refusal, suggesting that assessment helps calibrate when to refuse.

In the forced-decision interventions, imposing REFUSE produces safe replies even from untrained scaffolded models. Forcing COMPLY substantially reduces their safety. Trained defenders retain more safety under the same intervention, indicating that reinforcement learning changes behavior beyond simply selecting the correct label.

The partial-gradient ablation supplies another important result: training only assessment and decision tokens can improve end-to-end response safety even when the reply tokens receive no direct policy-gradient contribution. However, this narrower training scope generally increases over-refusal relative to full-gradient optimization.

Finally, token-level KL analysis shows that training-induced distributional differences extend through the assessment, peak near the decision, and diminish during response generation.

Together, these tests make the explicit decision more than a formatting convention. They do not establish perfect label-to-reply consistency.

A stronger refusal policy can reject legitimate users

The most immediate product concern is the cost imposed on benign requests.

Source: Section 3.2, Table 1; statistical uncertainty examined in Appendix B and Appendix E.2. Lower is better.

For Qwen, the 41.2-percentage-point increase in XSB over-refusal is substantial. A customer-support assistant that rejects legitimate requests can impose additional service friction even while resisting more attacks.

General capability averages remain close to their base-model values, but those tests use completion-format prompts that bypass the chat-template safety scaffold. They therefore cannot establish that an assistant remains equally helpful in actual conversations with the complete intervention active.

MT-Bench results also vary across models and reward formulations, reinforcing the need to measure conversational quality independently.

What an enterprise implementation would still need to establish

Cognaptus infers that decision-conditioned generation deserves controlled testing in enterprise assistants exposed to gradual intent escalation, particularly when the organization controls both the model and its training data.

Such testing should distinguish three possible failure points: incorrect intent assessment, incorrect decision selection, and unsafe behavior after an apparently correct commitment. Forced-decision probes provide a concrete way to investigate the third.

An adoption evaluation should jointly measure resistance to multi-turn attacks, refusal rates on representative legitimate requests, and conversational quality with the entire scaffold enabled. The relevant thresholds depend on the assistant’s users and permitted tasks.

Several boundaries remain unresolved.

The paper primarily evaluates scripted, pre-generated attack conversations rather than adaptive attackers reacting to defender responses. Harmful-token prefilling after the commitment eventually weakens every tested defender, although arbitrary assistant-channel control falls outside the intended threat model.

External judges also disagree meaningfully on individual safety verdicts. A robustness analysis helps: Qwen’s self-verified variant ranks first or second across all 25 combinations of five additional judges and five safety benchmarks, but this does not eliminate evaluator uncertainty.

Finally, the study provides no demonstrated production deployment or measurements of inference latency, throughput, or operating cost.

The commitment is a control point, not a safety certificate

ShallowDefend changes where safety training acts. By placing a structured commitment before the final reply, it makes an influential part of generation explicit, trainable, and experimentally inspectable.

The benchmark improvements and mechanism tests justify serious architectural evaluation. The accompanying over-refusal increases show why a safer benchmark model is not automatically a better enterprise assistant.

The decision before the answer can improve safety. Its operational value depends on whether the assistant also continues to answer the requests it should.

Cognaptus: Automate the Present, Incubate the Future.