Paper Tigers or Compliance Cops? What AIReg‑Bench Really Says About LLMs and the EU AI Act

The gist

AIReg‑Bench proposes the first benchmark for a deceptively practical task: can an LLM read technical documentation and judge how likely an AI system complies with specific EU AI Act articles? The dataset avoids buzzword theater: 120 synthetic but expert‑vetted excerpts portraying high‑risk systems, each labeled by three legal experts on a 1–5 compliance scale (plus plausibility). Frontier models are then asked to score the same excerpts. The headline: best models reach human‑like agreement on ordinal compliance judgments—under some conditions. That’s both promising and dangerous.

Why this matters (beyond leaderboard dopamine)

Compliance assessments are expensive and slow; pre‑market checks for high‑risk AI can take days and cost thousands per system. If LLMs can triage or pre‑screen documentation credibly, you reduce review queues, focus scarce counsel time where it matters, and cut vendor assurance latency. But a wrong “green light” is worse than a delay. This benchmark finally gives us a ruler for that tradeoff.

How AIReg‑Bench is built

Input surface: Only technical documentation excerpts—exactly what assessors say they lean on most—keeps the task crisp and reproducible.

Generation pipeline: A small, staged prompting process creates varied system overviews → targeted “(non‑)compliance profiles” against Article 9/10/12/14/15 → realistic write‑ups. The goal is subtle violations, not cartoonishly bad ones.

Human labels: Six legally trained annotators score each excerpt (3 per item) for compliance and plausibility. Inter‑rater reliability is Krippendorff’s α ≈ 0.65—moderate agreement that mirrors the subjectivity of real compliance work. Median plausibility is 4/5, so the documents read like something a Fortune‑500 provider would actually hand over.

What the numbers actually say

Two takeaways stood out on first principles:

1. Frontier models can rank compliance like humans. Gemini 2.5 Pro tops the chart on agreement (κw ≈ 0.86; ρ ≈ 0.86) and keeps mean absolute error to ~0.46. More importantly: 60% exact matches to the median human score. That’s enough fidelity for prioritization and second‑pair‑of‑eyes use, not sign‑off.

2. Prompting and context control matter—materially. Ablations on GPT‑4o show that removing the actual article text or pushing a “harsh” tone drops overall agreement even as bias shrinks. Translation: You can’t fix calibration with tone alone; you need the right legal context in the prompt and should accept a modest bias if it buys consistency.

Mini‑scorecard (selected models)

(Bias > 0 means a tendency to over‑score compliance.)

How to use this (responsibly) in an org today

1) Treat as triage, not verdicts. Use the top model as a queue‑router: auto‑flag excerpts predicted 1–2 (likely non‑compliant) and 4–5 (likely compliant) for fast human pass; funnel 3s (murky) to senior reviewers.

2) Always pass the relevant article text. The ablation shows sharp drops without it. Build your templates so the exact AIA article snippet is in‑context for every assessment.

3) Panelize to reduce bias. Combine a slightly under‑scoring model (e.g., Gemini 2.5 Pro) with a known over‑scorer (e.g., GPT‑4o) and require concurrence for 5/5 green‑lights; escalate disagreements.

4) Log rationales, not just scores. Humans gave textual justifications; make models do the same. Store these to train evaluators and to explain deltas when auditors ask why you accepted a vendor.

5) Calibrate with a house set. Before production, have counsel score ~25 of your own artifacts; measure model κw and MAE vs. your own medians. Adjust thresholds until false‑negative/false‑positive rates match your risk appetite.

6) Don’t skip plausibility checks. Median plausibility is high in the benchmark; your vendors may not be. Add a model‑scored plausibility gate (structure, coherence, evidence) before compliance scoring to avoid garbage‑in.

Where this benchmark is honest about its limits

• Subjectivity is real: α ≈ 0.65 means there is no single “ground truth.” The Likert framing (“probability of compliance”) is the right call, but you must expect variance across reviewers—and models.
• Single‑turn, doc‑only: Real assessments are multi‑turn and draw on more than documentation. Expect performance to dip on messy, partial, or contradictory corpora unless you invest in retrieval and dedup.
• Standards are moving targets: Harmonized standards under the AI Act are still maturing. A model well‑tuned today may drift from tomorrow’s conformance yardsticks.

Operator’s playbook (copy/paste)

• Prompt skeleton: [Article text] → [system summary] → “Score 1–5 for compliance, justify in 4–6 sentences. Be rigorous but fair; cite concrete evidence from the excerpt.”
• Acceptance policy: Auto‑accept only if (i) compliance ≥4 by two diverse models, (ii) plausibility ≥4, (iii) rationales reference concrete evidence (datasets, controls, tests), and (iv) no contradiction flags.
• Escalation cues: Over‑confident 5s with thin rationales; any 2‑point spread between models; “difficult to assess” flags on high‑stakes systems; or gaps in human oversight descriptions (Article 14) and data governance (Article 10).

What I’d like to see next (and why it matters for you)

• Multi‑turn evaluations: Benchmarks that test whether models can ask for missing evidence before scoring. That’s closer to how real audits work and could halve human back‑and‑forth.
• Standards‑aware scoring: Inject the emerging CEN/CENELEC/ETSI specs into context and see whether κw rises without increasing bias.
• Real‑world docs (anonymized): Even a small corpus would pressure‑test construct validity, especially for messy lineage and adversarial‑testing sections.
• Cost‑quality frontiers: Because you care about throughput, not just κw. The paper’s Pareto view is the right mental model for procurement.

Executive takeaway

• Yes, LLMs can help: top models align closely enough with legal experts to prioritize and surface red flags.
• Only with guardrails: always pass the article text, log rationales, calibrate on your own data, and require model concurrence for green‑lights.
• Plan for drift: re‑calibrate quarterly as standards and models evolve.

Bottom line: AIReg‑Bench moves compliance AI from opinion to measurement. Use it to build a measured, auditable triage layer—never a rubber stamp.