TL;DR for operators When a model answer is mostly right but fails at one point, rewriting the entire response may waste both human effort and information about what the model already does well. onPanda turns that situation into a different annotation workflow: find the first inappropriate token, replace it, then let the same model continue from the corrected prefix. ...
TL;DR for operators Repository-level agent benchmarks are expensive enough that teams have a strong incentive to run only part of them. The risk is not merely estimating the wrong average score; a poorly chosen subset can also change which agent appears better. PTA-IRT uses historical execution traces to make that subset more informative.1 At a 10% calibration budget, it records the best reported MAE, Kendall’s tau, and Spearman’s rho in every metric column across the four evaluated SWE-bench variants. Its averages are 0.041 MAE, 0.888 tau, and 0.973 rho. ...