Cover image

The Judge Still Needs an Audit

TL;DR for operators An AI evaluation system does not necessarily become cheaper when another LLM replaces human reviewers. A judge can classify individual answers accurately yet require enough human calibration to make population-level accuracy estimation less efficient than direct auditing. A statistical study of text and multimodal benchmarks found that only 6 of 56 MMLU-Pro target–verifier pairs and none of 56 MMMU-Pro pairs met an idealized human-label efficiency criterion. Meanwhile, sampling based on imperfect model confidence proved effective: with 200 human labels, population-proportional allocation reduced estimation RMSE from 4.56 to 2.46 percentage points relative to equal-per-stratum allocation. ...

November 20, 2026 · 8 min · Zelina