When the Rule Changes but the Verdict Doesn't

TL;DR for operators

An AI compliance checker can classify more than 90% of cases correctly while leaving most verdicts unchanged when the supplied regulation is removed or modified. In a five-model audit spanning 20 regulatory domains, the four general-purpose models achieved 90.3–91.9% accuracy but only 7.0–9.4% aggregate sensitivity to rule changes.

For teams procuring or deploying compliance automation, benchmark accuracy does not establish that the system’s decisions depend on the customer’s governing rules. Counterfactual tests that delete, substitute, or negate rules provide an additional diagnostic. But low sensitivity is not proof that a model ignored the regulation: some scenarios independently reveal the likely verdict, and sensitivity changes substantially across datasets.

The practical response is to evaluate labeled correctness and rule responsiveness together, using representative documents and complete rules. Requiring rule quotations or manipulating compliance-related internal activations did not reliably improve accuracy in the tested settings.

A correct verdict can survive the removal of its governing rule

Consider a company testing an AI compliance checker before deployment. The system receives case documents alongside relevant regulations, predicts whether each case complies, and matches the labeled outcome in roughly nine out of ten evaluations.

That result appears promising until the company considers a policy update. If the regulation governing a case changes, will the checker adjust its verdict? Or will it continue producing the same answer based largely on the facts and language of the case?

In Verdict Without the Rule, Saisab Sadhu and colleagues investigate this problem through paired counterfactual evaluations of large language models.¹ Their central finding is that high classification accuracy can coexist with very limited observable responsiveness to the supplied regulation.

For operators approving systems that must accommodate jurisdictional differences or updated policies, the distinction changes what a successful evaluation needs to demonstrate.

Measure what happens when only the rule changes

The researchers introduce Output Compliance Sensitivity (OCS), a black-box diagnostic that measures how frequently a model changes its binary compliance verdict when the document stays fixed but the supplied rule changes.

They test three perturbations:

1. Deletion: Remove the governing rule entirely.
2. Substitution: Replace it with a canonical rule from another regulatory domain.
3. Obligation negation: Reverse mandatory expressions such as must and shall, where those expressions are present.

For each applicable perturbation, OCS records whether the verdict changes. Aggregating these indicators yields a verdict-change rate.

An OCS of 0.08 means approximately 8% of eligible evaluated predictions changed under the specified perturbations. It is not an 8% probability of faithful reasoning, nor does it indicate that the changed verdict is legally correct.

This design offers a practical advantage: an operator can measure behavioral rule sensitivity without access to the model’s internal activations or additional reference labels. Original-rule and perturbed-rule predictions provide the comparison.

Nine out of ten correct, fewer than one in ten changed

The primary experiment covers 3,946 distinct cases from 20 OmniCompliance-100K domains, evaluated across five models. The main evidence is concentrated in the four general-purpose instruction models.

Source: Section 5.1, Table 1; Appendix B, Table 5.

Across all five models, deleting a rule produces a mean verdict-change rate of 5.3%, compared with 8.2% for substitution and 7.0% for obligation negation. Accuracy following deletion remains within four percentage points of baseline for every general-purpose model.

These results provide direct evidence of substantial verdict stability. An additional independence-null comparison strengthens that observation: each model’s observed OCS falls significantly below the flip rate expected from independently paired predictions with the same empirical verdict frequencies (permutation p < 0.0001).

The null comparison accounts for response bias, which matters because a model that almost always predicts compliance would produce few changes even under independent predictions.

Still, neither the low flip rates nor their statistical significance establishes how much genuine regulatory reasoning occurs inside the models.

The cases that actually need a rule behave differently

A scenario may contain enough information for a model to infer its compliance label without consulting the accompanying regulation. In that situation, an unchanged correct verdict after rule deletion is ambiguous evidence of grounding failure.

The authors investigate this through a rule-necessity control: for each model, they identify cases that were classified correctly with the original rule but incorrectly when that rule was deleted.

Within these deletion-critical subsets, rule-swap sensitivity jumps to 0.73–1.00 across the five models, compared with aggregate rates of 0.013–0.094 in the full sample.

The contrast is substantial, but its interpretation requires care. These subsets are defined by observed model behavior, not independently verified legal necessity. Moreover, only Llama-3.1-8B shows an above-null swap result that survives the reported Bonferroni correction. The adapted guard model’s 1.00 sensitivity also coincides with a 1.00 independence-null rate.

This control qualifies the main finding. Some models respond strongly to altered regulations on cases where removing the original rule has demonstrably damaged their accuracy.

Cognaptus inference: A compliance test suite should distinguish cases whose answers are readily recoverable from the scenario from cases where the governing rule materially affects prediction correctness. Aggregate accuracy or aggregate sensitivity alone can conceal this composition effect.

Rule sensitivity depends on the dataset, not just the model

If OCS were a stable characteristic of a model, operators could potentially carry one sensitivity score across workflows. The paper’s external comparison challenges that assumption.

For Llama-3.3-70B, aggregate OCS rises from 8.1% on OmniCompliance-100K to 25.5% on LegalBench and 31.8% on ContractNLI. Baseline accuracy also differs: 90.9%, 49.7%, and 78.6%, respectively.

These are external evaluation checks, not controlled estimates of a single dataset characteristic. The documents, rule formulations, and entailment tasks differ, and only one model is tested externally.

The operational implication is that OCS must be measured against the kinds of rules and documents a deployed system actually encounters. A low score on one benchmark is not a universal model property.

The authors also examine whether regulatory language predicts sensitivity. Mean syntactic dependency depth has a positive domain-level association with OCS (Spearman ρ = 0.570, BH-adjusted p = 0.044). However, the five-feature joint regression is not conventionally significant (p = 0.091). This remains an exploratory association, not evidence that more complex syntax causes stronger rule grounding.

Internal activation changes provide a narrower diagnostic

The paper extends the audit beyond observable verdicts by measuring ICS-delta: the change in a model’s projection onto a fixed, compliance-predictive residual-stream direction after the rule changes.

Across the four general-purpose models, these changes equal approximately 12.8–22.9% of their respective baseline absolute probe magnitudes. Domain-level rankings of behavioral OCS and representational ICS-delta show no statistically significant convergence.

The findings do not establish that models encode little regulatory information. ICS-delta examines one direction at one selected layer and final-token position. Other representations can carry information outside that measurement.

For operators with internal model access, the probe offers complementary observability. It does not replace behavioral testing or identify a complete internal decision mechanism.

Requiring a rule quotation did not restore accuracy

The study tests two possible interventions on cases classified as low in rule sensitivity.

First, it asks models to quote the relevant regulatory clause before producing a verdict. No model achieves a statistically significant accuracy improvement on the LOW-OCS partition. Qwen2.5-7B instead loses 2.8 percentage points of accuracy (p < 0.0001).

Second, the researchers steer residual-stream activations along the compliance-predictive direction in two Llama models. None of the tested coefficients produces a statistically significant accuracy improvement. Several negative-steering settings show nominal accuracy decreases, although those comparisons are not adjusted for multiple testing.

These are intervention findings, not proof that all grounding techniques fail. The prompting experiment also changes rule completeness alongside the instruction, so its effects cannot be attributed solely to requiring a quotation.

For a compliance-product owner, a cited regulation should therefore be assessed as an output artifact, not accepted as independent evidence that the citation determined the verdict.

What the audit can and cannot establish

Three boundaries particularly affect deployment interpretation.

First, the original rules are truncated. Standard prompts include only the first 300 characters of each rule. A missing operative clause can limit sensitivity even when the model would have responded to the complete provision. Any production test should verify the rule actually reaches the model intact.

Second, a changed verdict is not automatically a better verdict. OCS measures responsiveness, not legal validity. Replacing a rule with an unrelated one may produce an altered output without demonstrating appropriate regulatory interpretation. Conversely, an unchanged verdict may remain correct under both rules.

Third, the adapted guard-model result has a restricted meaning. Llama-Guard-3-8B achieves 51.1% baseline accuracy and 1.3% aggregate OCS, but it is being adapted from a fixed safety taxonomy to arbitrary regulatory rules. This is evidence about that adaptation, not a controlled comparison against purpose-built regulatory compliance models.

The study also does not perturb the case document itself, leaving sensitivity to misleading or irrelevant scenario details untested.

Production approval requires two kinds of evidence

The paper supports a concrete extension to compliance-system evaluation.

Before approving a model or vendor, an operator can measure labeled accuracy on representative cases, then run paired rule perturbations to identify verdicts that remain invariant under relevant policy changes. Cases where deleting the rule harms correctness warrant separate analysis, and changes produced by substituted or negated rules should be assessed against their actual legal meaning.

This procedure can also inform regression tests following regulatory updates. The same document can be evaluated under successive rules to determine whether a model changes its answer when the applicable obligation changes.

The research establishes the measurement gap, not an automatic pass-or-fail threshold. The appropriate standard for rule responsiveness remains dependent on the business workflow, validity of the perturbations, and governing legal requirements.

A compliance system that predicts past enforcement outcomes accurately may be valuable. But when the business expects it to apply an explicitly supplied and potentially changing rule, its evaluation must demonstrate more than agreement with historical labels.

Reference

Cognaptus: Automate the Present, Incubate the Future.