ThinkSafe: Teaching Models to Refuse Without Forgetting How to Think

A model can be very good at solving math problems and very bad at saying no.

That sentence sounds like a joke until it becomes a deployment problem. A reasoning model trained to work harder, think longer, and satisfy difficult prompts may also become more willing to satisfy harmful prompts. The training objective says: solve the problem. The model obeys. Safety, apparently, was not copied on the memo.

The paper behind ThinkSafe addresses this awkward failure mode in large reasoning models: how to restore refusal behavior after reasoning-oriented post-training has made the model over-compliant, without destroying the reasoning ability that made the model useful in the first place.¹

The obvious industry answer is to use a stronger teacher. Ask a larger, safer model to generate safe reasoning traces, then fine-tune the smaller student on those traces. This is neat, expensive, and—according to ThinkSafe—structurally flawed. The problem is not only whether the teacher is smart enough. The problem is that the teacher is not the student.

ThinkSafe’s central claim is sharper than “self-generated data works.” It argues that safety realignment should be treated as a distribution-preserving repair. The safest useful target is not the output of a superior teacher. It is the student model’s own safe response distribution, filtered and elicited correctly.

That is the part worth understanding.

The real problem is not refusal; it is drift

Most safety fine-tuning stories begin with behavior: the model answers a harmful prompt, so we need it to refuse. ThinkSafe begins one level lower, with distributional drift.

Suppose a reasoning model has already learned a rich internal style for solving problems. It has particular reasoning habits, answer formats, implicit priors, and token-level trajectories. After reinforcement learning or other reasoning-oriented post-training, it may become more capable, but also more compliant. When harmful prompts arrive, it may still possess latent safety knowledge, but the compliance behavior dominates.

Teacher distillation tries to repair this by importing safe traces from another model. But those traces come from a different distribution. The teacher may reason differently, phrase refusals differently, allocate attention differently, and solve benign problems through different intermediate patterns. Fine-tuning the student on those traces therefore does two things at once: it teaches safety, and it pulls the student away from its own reasoning distribution.

That second part is the tax. Not the safety tax in the vague “alignment hurts capabilities” sense, but a more precise distributional penalty. The paper formalizes safety realignment as a KL projection problem. Among all safe response distributions, the unique distribution that minimizes drift from the frozen student is the student’s own distribution conditioned on safe outputs. In plain terms: keep the student’s way of thinking, but remove the unsafe branches.

A teacher cannot generally provide that target. Even a larger teacher, or a same-size teacher from another model family, will usually generate safe responses that are not the student’s safe responses. More filtering does not remove this mismatch. More teacher data does not magically make teacher reasoning become student reasoning. That would be convenient, and convenience remains undefeated as a source of bad engineering assumptions.

The paper’s mechanism can be summarized like this:

ThinkSafe’s contribution is to make the final row practical.

Refusal steering turns latent safety into usable training data

The ThinkSafe procedure is almost suspiciously simple.

For harmful prompts, the model is not asked directly to answer. A refusal-oriented instruction is prepended: the prompt is harmful, and the model should refuse. The student then generates its own reasoning trace and response. A safety guard filters the result. Only verified safe traces are kept.

For benign prompts, no refusal steering is applied. The student answers normally, and safe benign outputs are kept for training. The final dataset combines harmful-prompt refusals and benign-prompt helpful answers, then the model is fine-tuned offline using supervised learning.

No external teacher. No online reinforcement learning loop. No giant model hovering above the student like a very expensive parent.

The subtlety is that refusal steering is not supposed to invent safety knowledge. It is supposed to unmask safety knowledge that the model already has. The paper’s assumption is that many post-trained reasoning models were originally exposed to safety alignment, then later became over-compliant during reasoning optimization. In those models, safety is not absent. It is suppressed.

This distinction matters for business use. If a firm is adapting an open reasoning model that already went through safety alignment before additional coding, math, or task-specific tuning, ThinkSafe is plausible. If the firm is starting from a raw base model with no meaningful safety training, refusal steering may have little to elicit. You cannot “unlock” knowledge that was never there. Not even with a nicer prompt.

The theory says the best teacher is the student, but only after filtering

The paper’s theoretical section is compact, but it carries the article’s main idea.

Let the frozen student model define a response distribution for a prompt. Some responses are safe; others are unsafe. The safety filter defines the safe subset. The ideal realignment target is the student’s own distribution conditioned on that safe subset.

Mathematically, this is the distribution that minimizes KL divergence from the original student while assigning probability only to safe outputs. The paper calls this the KL-optimal target. It is “optimal” not because it is morally perfect or deployment-ready, but because it is the least disruptive safe modification of the student distribution.

That last phrase is doing the work: least disruptive.

External teacher data may be safe, but it is not the least disruptive safe target unless the teacher’s filtered distribution exactly matches the student’s filtered distribution. In practice, that equality is unlikely. The paper therefore predicts an irreducible excess KL penalty for teacher distillation. This gives a formal reason why teacher-based safety repair can degrade native reasoning.

Refusal steering is introduced as a cost-reducing proposal mechanism. Under the paper’s “refusal tilt” assumption, the refusal instruction increases the probability of safe outputs while preserving the relative distribution within the safe set. If that holds, then filtering steered samples produces the same accepted target as filtering unsteered student samples, but with a much higher acceptance rate.

That is the clever part. Naive self-generation is theoretically clean but practically starved: on difficult harmful prompts, the model rarely produces safe traces, so rejection sampling throws away most of the useful training opportunity. Refusal steering keeps the target in-distribution while making the safe traces easier to collect.

So ThinkSafe is not merely “prompt the model to refuse.” It is closer to: use a prompt as a sampling preconditioner, filter the samples, and fine-tune on the student’s own safe reasoning distribution.

Less glamorous. More useful.

The main experiments test the safety-reasoning trade-off, not just safety scores

The experiments cover Qwen3 models from 0.6B to 8B and DeepSeek-R1-Distill models from 1.5B to 8B. The paper evaluates safety using HarmBench, StrongReject, and WildJailbreak, reporting harmful response ratios. It checks over-refusal using benign XSTest prompts. It evaluates reasoning using AIME 2024, GSM8K, MATH500, and GPQA, reporting average pass@1.

That design matters because a safety method can “win” cheaply by refusing everything. A brick is also very safe in conversation, but it has limited enterprise adoption potential.

The main results show ThinkSafe reducing harmfulness while preserving, and sometimes improving, reasoning. On Qwen3-4B, average harmfulness falls from 22.58 to 5.05, while average reasoning rises from 74.47 to 77.18. On Qwen3-8B, average harmfulness falls from 19.57 to 4.50, with reasoning remaining close to the original model: 76.08 initially and 78.50 after ThinkSafe in the paper’s table. On DeepSeek-R1-Distill-1.5B, harmfulness falls from 50.23 to 42.20 while reasoning rises from 53.77 to 57.30.

The exact pattern varies by model family and scale, but the broad result is consistent: ThinkSafe improves safety without the reasoning collapse often seen in teacher-distilled baselines.

The comparison with teacher-based methods is where the paper becomes more interesting. SafeChain, STAR-1, and SafeKey often improve safety, but they frequently lose reasoning performance, especially on smaller or distilled models. For Qwen3-0.6B, SafeChain drops average reasoning from 44.95 to 39.86. For Qwen3-1.7B, SafeChain drops it from 64.87 to 60.93. For DeepSeek-R1-Distill-8B, SafeKey, SafeChain, and STAR-1 all land below the initial model’s reasoning average.

This does not prove every teacher-distillation method must fail in every setting. It does support the paper’s narrower claim: when safety data comes from a different distribution, reasoning preservation becomes fragile.

The ablations explain why the simple recipe is not quite simple

The appendix and ablation studies are not decorative. They are where the paper tests whether ThinkSafe works for the reason the authors claim.

The “without reasoning” ablation is especially useful. The authors strip reasoning traces from refusal responses while keeping benign reasoning intact. This makes harmful-prompt training look different from benign-prompt training: think deeply when helping, stop thinking when refusing. The result is worse safety and worse reasoning. On DeepSeek-R1-Distill-7B, harmfulness rises from 29.5 to 44.4; on 8B, it rises from 19.1 to 33.7. DeepSeek-R1-Distill-8B’s average pass@1 also drops from 67.5 to 64.1. In the Qwen appendix, Qwen3-4B’s average pass@1 reportedly falls from 77.2 to 57.8 when safety reasoning is removed.

The likely lesson is not “always reveal long safety reasoning to users.” The paper is about training traces, not necessarily production-time trace exposure. The lesson is that for reasoning models, safety behavior may need to be learned in the same cognitive format as ordinary problem solving. If the model is trained to reason on benign tasks but use a shallow template on harmful tasks, the internal policy becomes inconsistent.

The rejection-sampling ablation tests a different concern. Maybe refusal steering is unnecessary; maybe the student can generate enough safe traces if we sample and filter aggressively. The paper says no. On Qwen3-8B, naive rejection sampling produces a safety score of 21.3, barely better than the initial 19.6 and far worse than ThinkSafe’s 4.5. The interpretation is intuitive: the hard prompts are exactly the prompts where unsteered safe samples are rare. Rejection sampling keeps the easy wins and misses the difficult cases.

That makes refusal steering the operational hinge. It is not a cosmetic prompt. It is the step that makes the theoretically preferred target collectable.

Online RL is stronger medicine than this case requires

The paper also compares ThinkSafe with online learning methods, including GRPO and on-policy distillation. This part should not be read as “offline fine-tuning beats RL forever.” That would be a convenient headline and a bad conclusion.

The better reading is narrower: for this repair task, where the model likely retains latent safety knowledge, offline self-generated data can achieve a strong safety-reasoning trade-off at much lower cost.

In the reported comparison, GRPO requires more than 21 hours of training, around eight times slower than ThinkSafe. On-policy distillation requires more than 88 hours because it repeatedly uses a larger model during training. ThinkSafe reduces the safety score to 29.6 compared with 37.0 for GRPO and 41.42 for on-policy distillation, with only a negligible reasoning drop. A ThinkSafe + KL variant further reduces harmfulness to 26.4 while recovering reasoning to 45.5, matching GRPO’s reasoning level at a fraction of the cost.

The practical message is not that online RL is obsolete. It is that online RL may be overkill when the missing behavior can be elicited from the frozen student itself. Use a bulldozer when you need to move earth. Do not use one to adjust a chair.

What this means for firms using open reasoning models

For businesses, ThinkSafe is less about academic elegance and more about an operational question: how do we safely adapt reasoning models without buying a larger teacher or running expensive online RL?

The paper suggests a three-part workflow:

1. Start with a reasoning model that already has some prior safety alignment.
2. Use refusal steering to generate the model’s own safety reasoning traces on harmful prompts, while generating normal helpful traces on benign prompts.
3. Filter the traces with a safety classifier and fine-tune offline, preferably in a way that preserves the model’s existing reasoning representations.

This is relevant for teams building customer-support copilots, coding assistants, internal research agents, compliance-review tools, and workflow automation systems. Many such teams do not need frontier-model behavior. They need a smaller or mid-sized open model that can reason reliably inside a controlled domain without becoming an obedient disaster machine.

The ROI logic is straightforward:

The strongest business use case is not “make any model safe.” It is more specific: repair a reasoning model that became too compliant after task optimization.

That can happen when a company fine-tunes a model on coding tickets, analytics tasks, customer-resolution scripts, or agentic workflow data. The model becomes good at completing requests. Unfortunately, some requests should not be completed. ThinkSafe offers a way to retrain refusal behavior using the model’s own latent safety structure instead of importing a foreign one.

This is also where the method’s restraint becomes valuable. In enterprise AI, new training systems often come with a suspiciously large shopping list: bigger teacher, more reward models, more GPUs, more annotation, more monitoring dashboards, more ceremonies. ThinkSafe still needs safety datasets, guard models, evaluation, and fine-tuning infrastructure. But it removes one expensive assumption: that safety must be taught by a stronger model.

The boundaries are practical, not ceremonial

The paper’s limitations are not minor footnotes. They define where ThinkSafe should and should not be used.

First, the method depends on latent safety knowledge. If a model was never safety-aligned, refusal steering may not produce reliable refusal traces. In that setting, an external teacher or RL-based safety training may still be necessary at the beginning. ThinkSafe is best understood as safety restoration, not safety creation from nothing.

Second, the dataset quality is bounded by the safety guard. The authors use Llama-Guard-3 and test WildGuard as an alternative. The WildGuard ablation shows stability, which is useful. But no guard is perfect. False negatives can allow subtle unsafe traces into the training set. False positives can discard useful safe traces. In production, this means guard selection, audit sampling, and adversarial evaluation remain part of the workflow. Annoying, yes. Optional, no.

Third, the experiments are mainly single-turn. The paper explicitly does not solve multi-turn jailbreaks, long-context adversarial setups, or tool-using agents where harmful behavior can emerge across a sequence of actions. This matters for business automation. A workflow agent does not merely answer a harmful question; it may retrieve files, call APIs, update records, or send messages. Safety in that setting is a policy over trajectories, not only over single responses.

Fourth, the experiments use LoRA-based fine-tuning. That is a sensible choice because LoRA often helps preserve existing capabilities, but it limits the conclusion. Full fine-tuning could produce different trade-offs. Companies using heavier adaptation should not assume the same stability.

Finally, the offline dataset is static. ThinkSafe generates data once from the initial frozen student. As training progresses, the student distribution shifts. Iterative self-training could reduce this off-policy gap, but at higher cost. That is a natural next step, not a free lunch. Free lunches remain rare in machine learning; when found, they are usually mislabeled compute bills.

The useful shift: safety as elicitation, not imposition

The most important idea in ThinkSafe is not that refusal prompts are magical. They are not. The important idea is that alignment can sometimes be framed as eliciting a model’s own suppressed competence rather than imposing behavior from outside.

That framing changes how practitioners should diagnose safety failures in reasoning models.

If the model lacks safety knowledge, teach it. Use stronger supervision, better safety data, or reward-based methods.

If the model has safety knowledge but compliance optimization suppresses it, forcing it to imitate a different model may be the wrong repair. It may make the model safer in one dimension while quietly damaging the reasoning structure you wanted to keep. In that case, the better move is to recover the model’s own safe reasoning paths and train on those.

ThinkSafe does not end the safety-reasoning trade-off. It narrows the conditions under which that trade-off can be reduced. The method works best when the model’s unsafe behavior is a retrieval and activation problem, not a knowledge absence problem.

For firms building with open reasoning models, that is already useful. It suggests a cheaper diagnostic question before committing to large-scale teacher distillation or online RL:

Can the model be steered to produce safe reasoning from its own distribution?

If yes, the safety repair may be smaller, cheaper, and less destructive than expected. If no, heavier machinery may still be justified.

The industry likes bigger teachers because they are easy to understand: stronger model teaches weaker model. ThinkSafe is less comforting but more precise: the best teacher may be the student, once you stop asking it the question in a way that rewards bad obedience.

That is not as grand as a new alignment paradigm. It is better. It is a mechanism.

Cognaptus: Automate the Present, Incubate the Future.