TL;DR for operators
An AI evaluation system does not necessarily become cheaper when another LLM replaces human reviewers. A judge can classify individual answers accurately yet require enough human calibration to make population-level accuracy estimation less efficient than direct auditing.
A statistical study of text and multimodal benchmarks found that only 6 of 56 MMLU-Pro target–verifier pairs and none of 56 MMMU-Pro pairs met an idealized human-label efficiency criterion. Meanwhile, sampling based on imperfect model confidence proved effective: with 200 human labels, population-proportional allocation reduced estimation RMSE from 4.56 to 2.46 percentage points relative to equal-per-stratum allocation.
For evaluation operators, the practical baseline is randomized, population-weighted human auditing. An automated judge should be assessed using its sensitivity, false-positive rate, calibration uncertainty, and total inference costs. The paper establishes conditional statistical and economic advantages, not realized production savings.
The review queue does not measure population accuracy
An AI product team may need to assess thousands of generated answers while having capacity to verify only a small fraction. It can prioritize suspicious answers for error discovery, but that review queue cannot automatically tell management what fraction of all answers are correct. The suspicious cases may be disproportionately wrong.
Adding another LLM as a reviewer appears to remove this constraint. Automated judgments can be produced at scale, but their agreement with ground truth is imperfect. Estimating the accuracy of the original model then requires accounting for the reviewer’s mistakes, including mistakes that are not obvious from its overall judgment accuracy.
Harine Choi and colleagues examine these two routes in From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing.1 Their framework compares human auditing organized by the target model’s confidence with automated verification corrected using human-calibrated judge error rates.
The main cross-model comparison is restrictive for automated judging: just six of 56 directed target–verifier combinations on the text benchmark meet the paper’s idealized human-label savings criterion. All 56 multimodal combinations fall short. These results concern estimation efficiency, not whether automated judges are generally capable of recognizing correct answers.
Confidence can organize an audit without predicting correctness accurately
A model reporting 90% confidence need not be correct 90% of the time. The experiments document systematic miscalibration, including average overconfidence across all eight evaluated models on MMMU-Pro.
That does not make confidence useless for sampling.
The direct-auditing method divides the complete output population into confidence groups, randomly selects answers within each group, and calculates the audited accuracy of each group. The final estimate weights each group’s accuracy by its actual share of the full population:
Here, $N_j/N$ is the population share of confidence group $j$, and $\hat{\theta}_j$ is its observed audited accuracy.
Under the paper’s sampling-design assumptions, this estimator is unbiased even when the confidence values are poorly calibrated. Correctness is determined by the audited labels; confidence determines only how the population is partitioned.
The allocation of human checks still matters. Equal numbers of checks per group can oversample small confidence groups and undersample large ones, increasing uncertainty in the population estimate.
In the paper’s main MMLU-Pro auditing simulation, GPT-5.5 had a true benchmark accuracy of 80.54%. With 200 audited outputs, the allocation methods produced:
| Audit allocation | RMSE |
|---|---|
| Equal allocation across strata | 4.56 percentage points |
| Population-proportional allocation | 2.46 percentage points |
| Oracle Neyman allocation | 2.41 percentage points |
Source: Section 4.3, Figure 3. RMSE is measured against complete benchmark ground truth.
Population-proportional allocation reduced RMSE by approximately 46% relative to equal allocation. Its performance was also close to oracle Neyman allocation, which uses otherwise unavailable within-group correctness variances.
These are main simulation results, based on 5,000 randomized audit replications per budget–allocation condition, rather than observed gains in a deployed human-review operation.
The allocation objectives also differ. Proportional allocation minimizes the denominator of the paper’s distribution-free Hoeffding concentration bound under the stated idealized constraints. Neyman allocation instead minimizes sampling variance by accounting for within-stratum correctness variability.
For an operator without reliable prior variance estimates, proportional allocation is therefore a defensible starting point. Pilot-based adaptive allocation may improve precision, but Appendix C establishes an essential implementation condition: the final estimator must account separately for pilot observations and the adaptively sampled remainder. Naively pooling them can introduce bias.
A judge’s agreement rate is not its estimation efficiency
Automated verification has a different statistical problem.
Suppose a judge accepts some correct answers and mistakenly accepts some incorrect ones. Its overall acceptance rate depends jointly on the target model’s actual accuracy and the judge’s two error characteristics.
The paper expresses this as:
where $q$ is the judge’s acceptance rate, $s$ is its sensitivity—the probability of accepting a correct answer—and $\alpha_v$ is its false-positive rate—the probability of accepting an incorrect answer.
Rearranging gives:
The denominator, $s-\alpha_v$, is the verifier discrimination gap. It measures how differently the judge treats genuinely correct and incorrect answers.
When that gap is small, errors in estimating the judge’s acceptance rate or calibration parameters are amplified. A judge can have a high overall agreement rate because many answers are correct while still having difficulty discriminating between the two correctness classes.
The MMLU-Pro comparisons make this distinction concrete. Claude Sonnet 5 judging GPT-5.5 achieved approximately 84% judgment accuracy but an idealized human-label efficiency ratio of only 0.25. GPT-5.5 judging Llama-3.1-8B achieved approximately 89% judgment accuracy and a ratio of 1.45.
The efficiency ratio compares the number of labels needed for ordinary direct random auditing with those needed for optimally allocated verifier calibration at equal asymptotic estimation precision. A value above one favors the verifier route under the paper’s assumptions.
That ratio depends on target accuracy, judge sensitivity, and false-positive behavior—not raw judgment agreement alone.
The multimodal results reinforce the constraint. For the pooled GPT-5.5 MMMU-Pro verifier operating point, judgment accuracy was 0.700, sensitivity was 0.844, and the false-positive rate was 0.402. The resulting discrimination gap was 0.442. None of the evaluated multimodal pairs exceeded the idealized efficiency threshold.
The cross-model experiments are descriptive comparisons based on observed judgment populations and oracle calibration formulas. They do not independently demonstrate that a prospective calibration workflow will achieve those label requirements.
More automated judgments cannot erase calibration uncertainty
The judge-assisted estimator contains three sources of uncertainty: the estimated overall judge acceptance rate, its sensitivity on correct answers, and its false-positive rate on incorrect answers.
Increasing the number of automated judgments reduces uncertainty in the first component. It does not eliminate the uncertainty in the latter two, which depend on finite human calibration samples.
This creates an important constraint for evaluation-platform design. Even inexpensive, high-volume inference cannot indefinitely substitute for trusted calibration data when the objective is a precise estimate of population accuracy.
The paper also derives a total-cost condition incorporating the price of each automated judgment and each human label, along with their respective contributions to estimation uncertainty.
Under its modeled costs, the six qualifying MMLU-Pro pairs have finite break-even human-label prices ranging from approximately $0.09 to $21.55 per question. Above the relevant pair-specific threshold, optimized judge-assisted estimation is modeled as cheaper than direct random auditing at equal precision.
These thresholds are not universal purchasing benchmarks. They use observed token consumption and API prices dated October 10, 2026, continuous asymptotic allocation assumptions, and idealized calibration. Additional screening to obtain the necessary correct and incorrect calibration examples is excluded.
Moreover, pairs that fail the initial idealized label-efficiency criterion have no finite positive break-even human-label price under the model, regardless of how much human verification costs.
What an evaluation operator can implement—and what remains untested
The immediate operational implication is to distinguish three activities that an evaluation system might otherwise combine: selecting outputs for review, measuring their correctness, and estimating uncertainty.
For teams reporting population accuracy across large output collections, confidence-stratified random auditing offers a statistically grounded baseline when population membership and reliable reference labels are available. The system must retain stratum sizes, randomized selection records, audit labels, and estimation weights. Confidence scores should not substitute for verified correctness.
If an organization considers an automated judge, its procurement and evaluation process should establish class-conditional sensitivity and false-positive rates on representative examples. The resulting estimate should propagate calibration uncertainty and include actual inference and labeling expenditure. Raw agreement alone is insufficient to determine whether the judge improves precision per dollar.
The boundary matters. The study evaluates multiple-choice benchmarks with known answer keys: 8,000 MMLU-Pro questions and a fixed 1,000-question MMMU-Pro subset. Human auditing is simulated by revealing existing ground truth. The verifier-efficiency calculations use operating-rate estimates from the available judgment populations, without a separate held-out calibration experiment or implemented sequential class-enrichment process.
The principal economic comparator is direct random auditing, not an optimized confidence-stratified direct audit. Consequently, even a judge that clears the reported break-even threshold has not established cost superiority over the stronger direct-sampling alternative.
Open-ended evaluations, human disagreement, changing task distributions, and calibration drift remain unresolved in the main empirical tests. Appendix D additionally shows analytically that unknown human verification error can prevent latent accuracy from being point identified without further calibration assumptions or information.
The relevant metric is accuracy per verification budget
The paper’s contribution is a statistical framework for deciding how scarce human verification should be spent. It shows why model confidence can remain valuable despite miscalibration, why a capable automated judge can still be an inefficient estimator, and why evaluation savings require more than inexpensive inference.
For an operator choosing an evaluation workflow, the benchmark should be the precision of the final population estimate relative to the full cost of obtaining it. Confidence-guided direct auditing is a credible starting point; calibrated automated verification becomes preferable only under measured operating characteristics and favorable economics.
That is a narrower conclusion than automated judges replacing human evaluation, but it is one an evaluation system can test quantitatively.
Cognaptus: Automate the Present, Incubate the Future.
-
Harine Choi and Chenchen Kuai and Zihao Li and Jiwan Jiang and Yunlong Zhang and Zhiwen Fan and Zhengzhong Tu and Yang Zhou (2026). From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing. arXiv:2610.35698. https://arxiv.org/abs/2610.35698 ↩︎