The Benchmark Is in the Trace: Reusing Agent Trajectories to Shrink SWE Evaluation

TL;DR for operators

Repository-level agent benchmarks are expensive enough that teams have a strong incentive to run only part of them. The risk is not merely estimating the wrong average score; a poorly chosen subset can also change which agent appears better.

PTA-IRT uses historical execution traces to make that subset more informative.¹ At a 10% calibration budget, it records the best reported MAE, Kendall’s tau, and Spearman’s rho in every metric column across the four evaluated SWE-bench variants. Its averages are 0.041 MAE, 0.888 tau, and 0.973 rho.

The operational point is easy to misread. A new agent does not need to produce trajectories across the full benchmark. Historical trajectories are used offline to learn which tasks carry useful measurement signal. The new agent is then executed only on the selected calibration subset, and its full-benchmark performance is estimated from those outcomes.

For teams that repeatedly compare or regression-test coding agents against a stable benchmark, this makes old execution traces potentially valuable beyond debugging. They can help decide which future benchmark runs are worth paying for. The boundary is equally concrete: the paper tests four SWE-bench variants, depends on parseable historical trajectories, and does not show that 10% is a safe universal evaluation budget.

A cheaper benchmark is only useful if it preserves the decision

Suppose an evaluation team has several coding agents to compare. Running each one over hundreds or thousands of repository-level issues means paying repeatedly for exploration, tool calls, edits, tests, and long execution paths.

Running fewer tasks solves the cost problem but creates a measurement problem. Benchmark tasks are not interchangeable. Some are easy enough that nearly every agent passes; others may be so difficult that almost none do. Neither type necessarily tells you much about the difference between two agents near the middle of the capability range.

This is the setting in which Item Response Theory becomes relevant. Instead of treating every task as one equal vote, IRT models tasks by properties such as difficulty and discriminability: how hard a task is and how strongly success on it separates agents of different estimated ability.

PTA-IRT extends that idea with information most benchmark-compression methods discard: what historical agents actually did while attempting the task.

The trace changes how informative a task looks

The method converts historical execution logs into four structured fields: Task Goal, Context Explored, Edits Executed, and Path Overview. Those summaries are then used to adjust the modeled difficulty and discriminability of an item for particular historical agent-task interactions.

That matters because identical pass/fail outcomes can conceal different processes. Two agents may both fail while exploring very different parts of a repository, or both pass through substantially different edit paths. PTA-IRT treats that process variation as evidence about the measurement characteristics of the task rather than as incidental logging data.

The selection step then combines two considerations. First, a task should provide high Fisher information—roughly, it should be useful for locating an agent on the ability scale. Second, that estimate should be backed by enough usable trajectory evidence. The method therefore incorporates an effective-sample-size term so that an apparently informative task supported by sparse or unreliable summaries receives less weight.

Tasks are not simply ranked globally and the top few chosen. PTA-IRT allocates the evaluation budget across historical pass-rate difficulty strata, then selects high-information tasks within those strata. This distinction becomes material in the ablations.

There is one more architectural step. The trajectory-rich model acts as a teacher during training, while a student learns to predict from the information that will actually be available when evaluating a new agent. This is Learning Using Privileged Information: historical traces can improve the evaluator without requiring equivalent full-benchmark traces at test time.

At 10% of the benchmark, ranking recovery remains strong

The paper evaluates the method with four-fold model-level cross-validation on SWE-bench Lite, Verified, Full, and Pro. Historical models are split into training and held-out groups; the evaluator sees only a fixed calibration subset for a held-out model and attempts to recover its full-benchmark score and ranking.

At a 10% calibration budget, PTA-IRT outperforms all seven reported baselines—MLE, MCMC, VI, VIBO, Deep-IRT, PSN-IRT, and AutoJudger—in every reported MAE, Kendall’s tau, and Spearman’s rho column.

For an operator choosing among agents, the ranking metrics are particularly consequential. A smaller benchmark that estimates the mean accurately but reverses model order can still produce the wrong procurement, release, or regression decision.

The budget-sensitivity evidence is more restrained. On SWE-bench Lite, PTA-IRT remains the strongest reported method from 5% through 25% calibration. Kendall’s tau rises from 0.768 at 5% to 0.886 at 20%, while MAE generally declines as more tasks are added. This supports a tunable cost-versus-fidelity tradeoff; it does not identify one universally sufficient subset size.

The ablations point to process information, not complexity alone

A more complicated evaluator can outperform a simpler one for many reasons. The paper therefore removes components and corrupts its privileged information to test where the gain comes from.

One detail prevents an overly tidy interpretation: Verified’s MAE is marginally lower without LUPI, 0.042 versus 0.043. The stronger claim is therefore the aggregate one. Across benchmarks and ranking metrics, removing LUPI worsens recovery; the component is not uniformly superior on every individual cell.

The representation analysis adds supporting context rather than another performance claim. Task Goal embeddings align most closely with issue text, Edits Executed aligns most closely with submitted patches, while Context Explored and Path Overview retain more process-specific variation. Submitted patches exhibit stronger outcome-related geometry than full trajectory summaries, but the process fields preserve more diversity among solution paths.

That pattern is consistent with the mechanism PTA-IRT assumes: traces are useful because they contain information different from both the problem statement and the final patch.

Historical traces can become measurement infrastructure

For an evaluation team, the business inference is not simply “run fewer tasks.” It is more specific.

If the organization repeatedly evaluates new versions of coding agents against a relatively stable benchmark, historical tool-use and editing traces can be retained as inputs to future benchmark design. Instead of treating those records solely as debugging artifacts, the evaluator can use them to estimate which tasks best distinguish agent ability and which estimates are supported by sufficient historical evidence.

The affected decisions include model selection, release gates, regression testing, and recurring comparative evaluation. The likely economic benefit comes from reducing repeated repository-level execution while preserving more of the ranking signal needed for those decisions.

The paper does not quantify monetary savings or establish an ROI threshold. Those depend on an organization’s per-task execution cost, benchmark frequency, trajectory-storage pipeline, and tolerance for ranking error. PTA-IRT supplies evidence about measurement fidelity under reduced calibration budgets, not a complete business case.

The 10% result is a benchmark-specific finding, not a default policy

Three boundaries matter before adopting the method.

First, the evidence comes from four SWE-bench variants. Lite and Verified are curated subsets of Full, so they should not be interpreted as fully independent benchmark families. External performance on other software-engineering evaluations remains untested.

Second, model coverage is uneven. Lite contains 35 evaluated models and Verified 70, while Full and Pro contain only 14 each. Strong recovery on Full and Pro is encouraging, but the historical agent population underlying those settings is comparatively sparse.

Third, the pipeline depends on historical trajectories that can be parsed and summarized. The paper uses a particular summarization model and sentence-embedding model and does not establish robustness across alternative choices. Effective-sample-size weighting helps when trajectory information is missing or degraded, but both weighted and unweighted variants weaken when usable process evidence becomes scarce.

Those limits define where the result is most actionable: repeated evaluation against a benchmark with substantial historical interaction data, not a newly created benchmark with little trajectory history or an agent population that differs sharply from the one used to build the evaluator.

Keep the traces if you expect to evaluate again

The most consequential idea in PTA-IRT is not that 10% of a benchmark can replace 100%. The paper does not support that rule outside its tested settings.

It is that benchmark history contains more than a matrix of passes and failures. Exploration paths, edits, and execution sequences can reveal which tasks are informative measurements of agent ability. When those traces already exist, discarding them after debugging may also discard information that could make the next evaluation cheaper and more faithful.

For teams operating coding-agent evaluation as a recurring process rather than a one-time leaderboard exercise, that changes what should be retained. The benchmark tasks are assets. The outcomes are assets. The trajectories may be assets too.

Cognaptus: Automate the Present, Incubate the Future.