Item-Response-Theory

TL;DR for operators Repository-level agent benchmarks are expensive enough that teams have a strong incentive to run only part of them. The risk is not merely estimating the wrong average score; a poorly chosen subset can also change which agent appears better. PTA-IRT uses historical execution traces to make that subset more informative.1 At a 10% calibration budget, it records the best reported MAE, Kendall’s tau, and Spearman’s rho in every metric column across the four evaluated SWE-bench variants. Its averages are 0.041 MAE, 0.888 tau, and 0.973 rho. ...

TL;DR for operators A workflow that uses model confidence to auto-accept an answer, escalate it, or send it to human review depends on more than whether the underlying model is accurate. The confidence signal itself has to distinguish cases the model should find easy from cases it should find difficult. Chen et al. test this distinction in Latent Confidence Alignment for LLM Self-Assessment.1 Across 20 LLMs and 100 text-only MedXpertQA questions, supplying an external difficulty signal significantly improved the alignment between models’ stated error probabilities and their expected error probabilities. Structured reflection alone did not significantly improve that alignment. At the same time, latent task ability showed no significant differences across the four evaluated conditions. ...