When Helpful Examples Still Lead to Refusal

TL;DR for operators

Correctly labeled helpful examples do not necessarily eliminate false refusals in safety-aligned language models. Training updates from harmful requests can interfere with learning to answer legitimate requests that contain sensitive language.

Chen and colleagues introduce GOAL, a fine-tuning method that selectively removes conflicting gradient components from helpful-response training and separately encourages the model to distinguish harmful from harmless-but-sensitive prompts.¹ Unlike simply reducing safety supervision, the method leaves harmful-example training gradients intact.

Across three primary language models, GOAL leads or ties on 13 of 15 over-refusal benchmark settings. In a blinded human evaluation, useful compliance increases from 66.5% under conventional supervised fine-tuning to 76.0%, while unwarranted refusals fall from 21.5% to 10.0%.

Operational implication: Teams adapting customer-facing assistants should evaluate legitimate-request completion, harmful-request safety, and response usefulness independently. GOAL offers a targeted training option where model gradients and labeled examples are accessible, but its safety results are benchmark-specific and its measured first-run training overhead is 23.0%.

Correct supervision does not guarantee correct refusal behavior

An assistant may reject a legitimate customer request because its wording resembles a harmful instruction. The refusal may satisfy a superficial safety check, yet prevent the customer from completing an authorized task.

One response is to fine-tune the assistant on more examples pairing legitimate questions with helpful answers. Another is to relax refusal behavior. The first assumes that correct training labels will produce the intended behavior; the second risks weakening protection against genuinely harmful requests.

The research exposes a problem with the first assumption.

In a blinded evaluation of 200 held-out prompts using Llama-2-7B-Chat, conventional supervised fine-tuning (SFT) still produced a 21.5% unwarranted-refusal rate. GOAL reduced that rate to 10.0%, an 11.5-percentage-point improvement. More importantly, useful compliance—responding without refusal while providing a sufficiently useful answer—rose by 9.5 percentage points.

These results concern over-refusal: rejecting harmless requests because their language appears risky. They do not measure greater willingness to comply with genuinely harmful requests.

Why would correct helpful-response targets fail to resolve the problem?

Harmful training gradients can reinforce refusal on legitimate prompts

Training examples contribute gradients that determine how model parameters change. Although harmful requests receive refusal targets and harmless requests receive helpful targets, their gradient directions need not be independent.

The authors find that harmful-example gradients concentrate strongly in a relatively low-dimensional region of parameter space. Across the three primary models, the dominant extracted directions capture approximately 82%–89% of aggregated harmful-gradient energy.

They call this region a refusal subspace, constructed using singular value decomposition of per-sample training gradients. It describes parameter-update directions, not a single refusal activation vector in the model’s hidden representations.

The central mechanism is asymmetric gradient coupling: harmful-example gradients align more strongly with hypothetical refusal-target gradients than with correct helpful-target gradients for harmless-but-sensitive prompts.

Consequently, safety training can increase a harmless prompt’s relative likelihood of refusal even when that prompt has been assigned a helpful target elsewhere in training.

The authors formalize this through a refusal preference margin: the log-likelihood of a refusal response minus the log-likelihood of the correct helpful response. Their sufficient condition for harmful-gradient attraction holds for 70.3%–83.7% of evaluated diagnostic samples across the tested models and checkpoints.

This is substantial but not universal. The proposition concerns local, first-order changes under specified gradient conditions, not a proof that harmful gradients explain every false refusal.

A separate representation analysis identifies another difficulty. On Llama-2-7B-Chat, a classifier distinguishing harmful from harmless-but-sensitive prompts achieves an AUC of 0.858 using layer-13 representations but only 0.678 using layer-29 representations (Appendix T, Table 15).

Deeper processing therefore retains some information about the distinction, but makes it less readily separable. The optimization and representation findings describe related problems without establishing that one fully causes the other.

GOAL changes where helpful learning is allowed to act

GOAL addresses the two mechanisms with complementary interventions.

First, it constructs a low-rank projector from harmful-example gradients. For harmless-but-sensitive training examples paired with helpful targets, the method removes the gradient component lying inside that harmful-gradient subspace.

The category-specific rule is:

Here, $\mathbf{P}$ projects onto the extracted harmful-gradient subspace.

The harmful-example gradients remain unchanged by this projection. GOAL modifies how helpful-target learning for over-refusal examples interacts with those directions, rather than simply weakening the harmful-request training objective.

Under the paper’s first-order assumptions, the remaining projected gradient still provides a descent direction for the correct helpful-response loss, provided it is nonzero. This does not guarantee an absolute reduction in refusal preference at every optimization step.

Second, GOAL uses safety-aware contrastive learning to encourage harmful and harmless-but-sensitive prompts to occupy more distinguishable hidden-state representations. A class-balanced memory bank supplies comparison examples, with Circle-style adaptive weighting emphasizing harder similarity relationships.

The implementation restricts LoRA updates to transformer layers selected through benign-versus-harmful representation probes. On Llama-2-7B-Chat, the selected block covers layers 12–22 rather than all 32 layers.

The design therefore intervenes in three places: gradient direction, representation separation, and layer selection.

The controlled tests distinguish targeted repair from generic adjustment

The main benchmark comparison spans Gemma-7B-it, Llama-2-7B-Chat, and Llama-2-13B-Chat. Five benchmark collections measure compliance with harmless requests; separate evaluations measure safety on harmful instructions and general multiple-choice accuracy.

GOAL leads or ties on 13 of 15 primary over-refusal settings, while safety scores remain strong and general-capability results change relatively little.

Several additional tests clarify what produces the improvement.

The matched-target control is particularly relevant to the proposed mechanism. Because harmful and benign examples share a refusal-response pool, the remaining gradient concentration difference cannot be attributed entirely to repeated refusal targets.

Further checks improve confidence without eliminating uncertainty. Five-seed hierarchical paired bootstrap tests find corrected statistical significance against ACTOR on four of five benchmarks for each 7B primary model and all five for Llama-2-13B-Chat. A separate Qwen3-8B evaluation places GOAL first on four of five over-refusal benchmarks, although gains are smaller against an already high-compliance base model.

The blinded human results also establish that the improvement is not merely more frequent non-refusal. On the 200-prompt Llama-2-7B-Chat evaluation, GOAL improves helpfulness, correctness, completeness, and useful compliance compared with conventional SFT.

What assistant teams should measure before adopting the method

For teams fine-tuning customer-support or enterprise assistants, the actionable lesson extends beyond the particular GOAL implementation.

An assistant serving legitimate but sensitive-language workflows needs at least three distinct evaluation outcomes:

1. False-refusal performance: Does it complete authorized requests that resemble risky instructions?
2. Harmful-request safety: Does it continue rejecting genuinely harmful instructions, including adversarial variants?
3. Useful compliance: Are non-refusal answers correct, sufficiently complete, and practically usable?

Cognaptus infers that treating these as separate release criteria would make fine-tuning decisions more informative than tracking a single refusal or alignment score.

Where teams have access to per-example gradients and labeled prompt categories, a GOAL-like intervention becomes a candidate for targeted experimentation. Its value should be compared with conventional SFT, alternative alignment methods, and the organization’s actual workload.

The training economics also require precise interpretation. On one NVIDIA A800, GOAL’s online training stage takes 9.55 minutes versus 9.53 minutes for SFT—approximately 0.21% overhead. Extracting the refusal subspace adds 2.17 minutes, making measured first-run end-to-end overhead 23.0%.

Reusing the subspace may change subsequent costs, but compatibility depends on the model checkpoint and trainable parameter configuration.

Benchmark safety is not deployment safety

The primary training set contains only 500 curated examples: 200 harmful, 200 benign, and 100 over-refusal pairs, with GPT-4o-generated targets. GPT-5.1 supplies the principal compliance and safety judgments.

Human evaluation strengthens validation, but covers one model, three methods, and 200 prompts. Neither these experiments nor the additional Qwen3-8B results establish performance across arbitrary architectures, operational domains, or evolving attack patterns.

The mathematical orthogonality result also has an implementation boundary. AdamW preconditioning does not preserve exact Euclidean orthogonality. The authors measure residual subspace leakage of 3.6%–4.7%, alongside substantial coupling reduction, rather than claiming an optimizer-independent guarantee.

For organizations considering deployment, these boundaries require independent testing on the target model, task distribution, and safety requirements. Some general-capability scores decline slightly, and the measured preservation of harmful-request safety does not establish comprehensive resistance to novel attacks.

The training signal needs more than the correct label

GOAL demonstrates that reducing false refusals need not begin by reducing harmful-request supervision. Correct examples alone may leave conflicting optimization directions and poorly separated representations unresolved.

The paper’s strongest contribution is a testable, targeted intervention supported by component ablations, mechanism controls, and response-quality evaluations. It does not prove a universal explanation for over-refusal.

For model teams, the resulting decision is more precise: determine whether legitimate requests are being rejected, diagnose whether training geometry contributes to the problem, and require independent evidence that any correction improves useful responses without degrading safety.

Cognaptus: Automate the Present, Incubate the Future.