The Safety Leaderboard Has Conditions: Choose Moderators by Harm, Context, and Cost

TL;DR for operators

A moderation team rarely needs a model that is merely “best at safety.” It needs a model that catches the relevant harms at the point where moderation occurs, without creating unacceptable false positives or latency.

A large benchmark by Afshin Orojlooyjadid and Hitesh Patel compares 53 specialized moderators and general-purpose language models across 11 public safety datasets.¹ Its strongest operational finding is not a new overall winner. It is that the ranking changes with the safety problem and with what the moderator is allowed to inspect.

That matters because the apparent precision of a leaderboard is easy to overread. Across 18 dataset-and-input-setting comparisons, only four top-ranked models beat the runner-up by a margin whose paired-bootstrap 95% confidence interval excludes zero. For procurement, many nominal first-place finishes should therefore be treated as ties rather than dependable superiority.

The benchmark also weakens model size as a shortcut for safety quality. Frontier general-purpose models lead several prompt-only settings, while specialized and mid-sized systems become especially competitive when the generated response is available. The right deployment question is consequently conditional: what harm is being screened, at which checkpoint, under what latency and over-refusal constraints?

The leaderboard is less decisive than the point estimates suggest

Suppose two moderation systems score 81.0 and 80.4 on the same test. The higher number is a ranking, but it is not yet evidence that the first model will perform better on another sample drawn from the same setting.

The paper addresses that distinction with paired non-parametric bootstrap tests. The two leading systems on each dataset are re-evaluated across 1,000 resamples using the same sampled records for both models. If the resulting confidence interval around their F1 difference includes zero, the observed gap is compatible with sampling variation.

That happens often. HarmBench in prompt-only evaluation has a top-two gap of 0.6 percentage points with a 95% interval from -2.4 to +3.8. XRTest shows a 1.0-point gap with an interval from -4.1 to +5.7. The paper reports statistically distinguishable top-two gaps in only 4 of 18 dataset-and-setting comparisons.

This analysis is best understood as a robustness check on the leaderboard, not as a separate thesis. It says that procurement should not be driven by a 0.5-point advantage when the evidence cannot distinguish that advantage from noise. The relevant choice can then move to variables the raw rank suppresses: latency, integration complexity, over-refusal, or fit to the product’s dominant harm class.

Where moderation happens changes which model looks strong

The study distinguishes two operational checkpoints before assigning them technical labels.

A system can screen the user request before generation. Or it can inspect the user request together with the model’s generated response. The first is prompt-only moderation, or Q. The second is prompt-with-response moderation, or QA.

These are different safety jobs. A prompt may look ambiguous before generation but become clearly harmful once the generated answer is visible. That additional evidence particularly benefits moderators designed to classify input-output interactions.

The aggregate results reflect the difference. In Q evaluation, the strongest reported average macro F1 scores are 76.9% for GPT-5, 76.8% for GPT-4.1, and 76.0% for GPT-4o-mini. Macro F1 gives balanced attention to classification performance across the safe and unsafe classes rather than letting a dominant class determine the score.

In QA evaluation, the ordering changes. Llama-Guard-3-8B reaches 78.6% average F1, BingoGuard-Llama-8B 77.7%, Gemma-2-27B-it 77.1%, and Gemma-3-27B-it 76.7%.

The paper does not establish that post-generation moderation is universally superior. QA uses responses supplied by the benchmark datasets, and those response distributions may overlap unevenly with moderators’ training data. What the evidence does establish is narrower and operationally consequential: a model selected for pre-generation screening should not be assumed to remain the best choice when the safety layer can inspect outputs as well.

Harm category is part of the model specification

The 11 datasets are grouped into four challenge categories: adversarial and jailbreak resistance, standard policy enforcement, refusal and over-refusal, and conversational or real-world safety.

Different model families lead in different categories. Specialized systems such as BingoGuard perform strongly on standard policy enforcement. Several frontier general-purpose models lead refusal and over-refusal tests. No model family produces a stable ordering across the full benchmark.

The hardest category is conversational and real-world safety, where average F1 is roughly 52% in both Q and QA evaluation. That low ceiling is more informative than minor movement among the leading systems elsewhere. It indicates that product teams operating in context-rich, ambiguous conversations still face a substantial measurement and capability gap even if a chosen moderator performs well on more standardized safety tests.

Cognaptus therefore reads the benchmark as evidence against using parameter count, vendor status, or aggregate F1 as a procurement proxy. The affected user is the team designing the safety layer; the decision is which moderator to deploy; the relevant conditions are harm profile and available context; and the boundary is that the benchmark measures binary decisions on English, single-turn examples.

Prompt design is part of the moderator, not just packaging

The paper also tests three prompts for repurposing general-purpose open models as moderators: a simple binary instruction, a Llama Guard-style template, and the authors’ structured prompt.

This is an ablation, designed to isolate the effect of prompting rather than establish the benchmark’s primary cross-model result. On Gemma-3-4B, Llama-3.2-3B, and Phi-4-mini, the authors’ prompt produces the highest reported average F1 in almost every Q and QA comparison.

For example, Llama-3.2-3B rises from 57.4 under the Llama Guard template to 64.8 with the optimized prompt in Q mode, and from 62.5 to 69.9 in QA mode. Against the simple prompt, the gaps are larger.

The optimized template assigns an explicit moderation role, defines safety categories, provides borderline-case guidance, and constrains the output protocol. It was optimized using a 10% sample of ToxicChat and WildGuard training splits rather than the evaluation data.

The business interpretation should remain narrow. Prompt design is an evaluation variable when a general-purpose model is being used as a moderator. The paper does not show that the same gains transfer to closed frontier systems because the ablation covers only three open-source models.

More reasoning compute produces little gain in this task

A separate GPT-5.2 reasoning-effort ablation tests whether additional reasoning budget materially improves binary moderation. Moving from no reasoning effort to high reasoning effort raises average F1 by only about one percentage point.

This is an efficiency test rather than evidence that reasoning is broadly unhelpful for safety. The evaluated task compresses moderation into a categorical safe-or-unsafe judgment. Richer investigations involving policy interpretation, multi-turn context, severity estimation, or explanation were not tested.

Within this task, however, the result weakens the case for buying additional reasoning compute by default. The benchmark’s latency measurements for 31 open-source systems reinforce the same procurement logic: some comparatively small models occupy attractive accuracy-latency positions, so scale alone is a poor deployment criterion.

A moderation stack should be selected conditionally

For operators, the evidence supports a more structured selection process than choosing the highest aggregate score.

One further variable belongs in the decision even when an aggregate F1 score looks strong: over-refusal. A moderator can improve harmful-content detection by becoming more aggressive and still degrade the product by blocking legitimate requests. The benchmark’s inclusion of refusal and over-refusal tests makes that trade-off visible rather than allowing safety quality to collapse into harmful-content recall alone.

The benchmark is a selection aid, not a deployment certificate

The evidence is broad for a moderation benchmark but deliberately normalized. All content is English. Examples are single-turn. Each dataset is capped at 1,000 records. Richer annotations are collapsed into binary safe and unsafe labels, eliminating severity and multi-label structure. Categorical outputs also prevent calibration and threshold-sweep analysis.

The QA evidence has an additional constraint: responses come from the datasets rather than from every prospective deployment model. A moderator familiar with similar response distributions could therefore benefit for reasons that would not reproduce in another application.

Closed-source APIs create a temporal boundary as well. Provider-side updates can alter behavior after evaluation, which means a benchmark result for a commercial model should be treated as a dated measurement rather than a permanent product property.

These limitations do not erase the comparison. They define what it can support: conditional shortlisting followed by validation on the organization’s own traffic, policies, languages, response distributions, and latency budget.

Safety procurement starts with the moderation job

The benchmark replaces a deceptively simple question—Which moderator scores highest?—with a more useful specification problem.

A team screening jailbreak prompts before generation may rationally choose a different system from a team checking completed outputs. A product exposed to ambiguous conversational risk needs different evidence from one enforcing a narrow policy taxonomy. And when two models are separated by a leaderboard fraction that the bootstrap cannot distinguish, deployment cost and user impact deserve more weight than rank order.

The paper’s strongest contribution is therefore not identifying a safety champion. It is showing that the moderation job itself has to be specified before a model ranking becomes meaningful.

Cognaptus: Automate the Present, Incubate the Future.