The Agent Benchmark Without the Agent Bill

TL;DR for operators

Agent evaluations are expensive for a fairly obvious reason: the agent has to do something. It must browse, edit files, call tools, manipulate repositories, survive its own mistakes, and occasionally discover that the environment has changed while nobody was looking.

The paper introduces Pace, a method for predicting performance on an expensive agentic benchmark from a compact set of cheaper, non-agentic test instances.¹ Across 14 frontier models and four agentic benchmarks, a 100-instance Pace proxy produces an average mean absolute error of 3.80 percentage points, a 0.81 Spearman rank correlation, and 84.37% pairwise model-ranking accuracy under leave-one-model-out validation.

The operational interpretation is not “cancel the agent benchmark.” It is:

1. Run a cheap proxy across every candidate model or training checkpoint.
2. eliminate candidates that are clearly uncompetitive;
3. reserve full agentic evaluation for finalists, unusual models, and deployment gates;
4. periodically recalibrate the proxy as model families and agent harnesses change.

The strongest evidence is not merely that Pace correlates with agent scores. Ordinary Lasso and Ridge models fit the calibration data almost perfectly and then deteriorate on held-out models. Pace performs worse in-sample but better out-of-sample. In benchmark prediction, apparently, modesty remains an underrated regularizer.

The principal boundary is distribution shift. Pace was tested on 14 models and four benchmarks evaluated through a common OpenHands agent framework. A new architecture, scaffold, tool environment, or missing capability can break the relationship between static proxy scores and actual agent performance. Use the proxy to allocate evaluation resources, not to certify production reliability.

Full agent evaluation is becoming an accounting problem

A static model benchmark is usually simple to operate. Present an instance, collect an answer, score it, repeat.

An agentic benchmark adds an environment around that exchange. The model may need to inspect a repository, decide which files matter, invoke tools, execute code, observe failures, revise a plan, and continue over a long trajectory. The result is more representative of practical work, but it is also slower, noisier, and operationally heavier.

This creates an awkward development loop.

A team may have dozens of candidate checkpoints, prompt configurations, inference settings, scaffolds, or third-party models. Running a full agent benchmark for each candidate can consume days of execution and substantial API expenditure. The economical response is often to evaluate less frequently, use a small target subset, or test only the models somebody already expects to win.

None of those choices is especially scientific. They are merely affordable.

Pace starts from a useful reframing: perhaps the purpose of a cheap evaluation is not to reproduce the agent task. Perhaps it only needs to predict which models would perform well on it.

That is a narrower objective. It is also considerably more tractable.

The headline result survives the held-out-model test

The paper evaluates Pace on four agentic targets:

• GAIA, involving tool-assisted information seeking and reasoning;
• SWE-Bench Verified, involving repository-level issue resolution;
• SWE-Bench Multimodal, adding visual information to software-engineering tasks;
• SWT-Bench, involving generation of tests that expose real software bugs.

The source pool contains instances from 19 non-agentic benchmarks spanning 11 capability categories, including instruction following, planning, retrieval, code generation, tool calling, multimodal understanding, and verification.

The central evaluation uses strict leave-one-out cross-validation. For each fold, Pace selects proxy instances and fits its predictor using 13 models, then predicts the fourteenth model, which was absent from both selection and regression. The procedure repeats until every model has been held out.

That protocol matters. A model that predicts benchmarks already included in its calibration set would be an attractive spreadsheet, not persuasive evidence.

At a budget of 100 source instances, the held-out results are:

These metrics describe three different kinds of usefulness.

Mean absolute error asks whether the predicted score is numerically close to the observed agent-benchmark score. An average error of 3.80 percentage points is meaningful for coarse screening, although whether it is adequate for a particular decision depends on the distance between candidates.

Spearman correlation asks whether models are ordered correctly, without requiring exact numerical calibration. The average correlation of 0.81 suggests that Pace is stronger as a ranking instrument than as a precise score oracle.

Pairwise accuracy asks a more direct procurement question: given two models, which one is likely to perform better? Pace answers correctly in 84.37% of held-out comparisons on average.

That last result is easy to overstate. An 84% pairwise accuracy is useful when filtering a broad field. It is not sufficient when two finalists are close, the consequences of a wrong choice are material, or the production task differs substantially from the benchmark. A screening test can be economically valuable without becoming a final authority. Corporate life contains several precedents.

The differences among targets reveal what the proxy actually knows

The aggregate result conceals an important distinction.

On SWE-Bench Multimodal, Pace achieves both low absolute error and strong ranking performance: 2.23% MAE and 0.89 Spearman correlation. For SWT-Bench, its absolute error is larger at 5.12%, but its ranking correlation remains 0.89 and pairwise accuracy reaches 90.11%.

GAIA shows a similar pattern in weaker form: 5.77% absolute error but a respectable 0.79 rank correlation.

This suggests that the proxy often learns relative capability order more reliably than exact target-score calibration. It can identify that Model A is probably stronger than Model B even when it is less certain whether Model A will score 41% or 46%.

For many business workflows, relative ordering is enough at the first stage. A platform team screening 30 model candidates does not need courtroom-grade estimates for all 30. It needs a defensible way to identify the five worth testing properly.

The result is less reassuring when decisions depend on narrow numerical thresholds. A proxy prediction of 48%, for example, should not automatically satisfy a release policy requiring 50% on the actual agent benchmark. The paper reports aggregate predictive performance, not a universal tolerance band around every individual prediction.

The budget sweep shows a funnel, not a magic sample size

The paper evaluates proxy budgets from 25 to 500 source instances. This is a robustness and sensitivity test: its purpose is to show how prediction changes as the proxy becomes larger, not to establish a second thesis about one sacred benchmark size.

The broad trend is favorable, but it is not perfectly monotonic. Fifty instances perform slightly worse than 25 on all three metrics. Pairwise accuracy dips at 400 before reaching its best value at 500. Absolute prediction improves through 400 instances and then weakens at 500.

That irregularity is not especially surprising. There are only 14 models, leaving 13 calibration observations in each fold. Adding more proxy information does not create more independent model-level examples. Eventually the method is attempting to extract increasingly fine structure from a rather small panel.

The practical lesson is not that every organization should standardize on 100 instances. It is that different evaluation budgets support different operating modes:

The paper identifies 100 instances as a practical headline configuration because it remains competitive while keeping evaluation light. Businesses should choose the budget from the cost of false elimination, not from the aesthetic appeal of a round number.

Weak in-sample fit is one of the paper’s strongest results

Given thousands of candidate benchmark instances and only a small number of calibration models, the obvious statistical approach is to fit a regularized model, inspect the coefficients, and retain the most influential features.

The paper tests this approach using Lasso and Ridge baselines.

They fit the available models extremely well.

They also generalize worse.

This comparison is main supporting evidence for the design of Pace.

Lasso nearly reconstructs the calibration targets, with an average in-sample MAE of 0.51%. Its held-out error then rises to 5.69%, while Spearman correlation falls from 0.99 to 0.46. Ridge exhibits the same pattern less dramatically.

Pace accepts a weaker fit to the known models and obtains better predictions for the unknown one.

The reason is dimensionality. The candidate pool contains many source instances, but the target labels come from a tiny number of models. A joint feature-selection and prediction model can discover combinations that describe those particular models almost perfectly. The fit looks impressive because it has memorized the available cross-model geometry.

Pace separates instance selection from final regression. Instead of asking one model to discover the features and exploit them simultaneously, it selects instances using simpler signals designed to remain stable under limited calibration data.

This is the technical contribution with the clearest business analogue: when evaluation history is shallow, optimizing the proxy to explain every historical result is often the quickest route to building a highly accurate model of last quarter.

Pace combines target relevance with global informativeness

Once the empirical case is established, the mechanism is comparatively straightforward.

For every candidate non-agentic instance, Pace considers two signals.

Local selection asks whether an instance tracks the target

The local signal measures rank consistency between model performance on one source instance and model performance on the target agentic benchmark.

Suppose models that perform well on a particular planning item also tend to perform well on SWT-Bench. That item receives a high target-relevance score for SWT-Bench.

Local selection therefore specializes the proxy to the target. A GAIA proxy and a SWE-Bench proxy need not contain the same instances because they are predicting different outcomes.

Its weakness is equally clear: an instance can correlate with a target among the small calibration set for accidental reasons.

Global selection asks whether an instance distinguishes models meaningfully

Pace also applies singular value decomposition to the source-score matrix and calculates leverage-based geometric importance.

Conceptually, this asks whether an instance contributes information about the broader structure separating the evaluated models. An item on which every model behaves identically is not useful merely because it belongs to a relevant benchmark category. An item that exposes a major axis of model variation may be valuable even when its connection to the target is less direct.

The global selection score combines this leverage signal with target relevance. It therefore favors source instances that are both structurally informative and connected to the target outcome.

The two views are combined rather than forced into one compromise

Local selection emphasizes task specialization. Global selection emphasizes stable cross-model structure. Pace creates both representations and combines their predictions using a target-specific learned weight.

That architecture is less glamorous than declaring a new foundation model for benchmarks. It is also well matched to the data regime. With only a small number of calibration models, the method uses restrained filters to reduce the chance that selection becomes an elaborate description of noise.

Bootstrapping addresses noisy target labels, not model drift

Agentic benchmark scores are themselves estimates. Each benchmark contains a finite collection of tasks, and a model’s observed average depends on which instances happen to be included.

Pace resamples the target instances with replacement, creating bootstrap versions of each calibration model’s target score. The predictor is trained across these resampled outcomes rather than treating one observed mean as an exact label.

The paper isolates this component in an ablation. Removing bootstrapping worsens average MAE from 3.80% to 4.57% and lowers average Spearman correlation from 0.81 to 0.66. Every target deteriorates on both measures.

This ablation supports a specific conclusion: accounting for target-instance sampling noise improves held-out prediction.

It does not show that Pace is robust to future model families, new tool environments, different agent scaffolds, or production distribution shift. Bootstrap resampling can represent uncertainty within the observed target tasks. It cannot manufacture evidence about systems that were never represented in the calibration set.

The distinction matters because “bootstrap” occasionally causes otherwise sensible people to assume uncertainty has been generally defeated.

The selected instances are capability signatures, not causal anatomy

Pace also analyzes which source benchmarks and capability categories appear in each target-specific proxy.

The selections differ in intuitively plausible ways:

• GAIA draws heavily from instruction-following and planning-oriented sources.
• SWE-Bench Multimodal allocates substantial capacity to long-context aggregation, verification, planning, multimodal understanding, and code generation.
• SWT-Bench emphasizes verification, testing, and planning.
• SWE-Bench Verified unexpectedly draws strongly from visual and retrieval-oriented benchmarks, despite being a text-only software-engineering target.

The last case is the most informative.

A selected instance does not necessarily identify a capability that is literally exercised in the target workflow. It may instead be a particularly effective discriminator among the available models. Visual reasoning performance, for example, could track broader representation quality that also helps distinguish software-engineering performance.

Accordingly, Pace’s allocations are best read as predictive capability signatures:

These are the inexpensive signals that best separate models for this target within the observed calibration population.

That is not equivalent to:

These are the causal ingredients of success on this target.

The difference limits how the analysis should influence training investment. A company should not redirect its software-agent roadmap toward image puzzles simply because visual benchmarks were selected for SWE-Bench Verified. The allocation is useful for diagnosis and hypothesis generation, but it is not an intervention study.

The business value is evaluation triage

The paper directly shows that selected static instances can predict held-out agent-benchmark results within its experimental setting.

Cognaptus infers a broader operating model: use proxy evaluation to create a multi-stage evidence funnel.

Stage 1: screen every candidate cheaply

Run the target-specific proxy on:

• training checkpoints;
• inference configurations;
• quantized or distilled variants;
• external model candidates;
• prompt or system-policy variants, where the proxy format remains applicable.

The objective is high recall for promising candidates. Eliminate obvious underperformers and detect material regressions early.

Stage 2: test finalists on representative agent tasks

Candidates with similar proxy predictions should proceed to a direct target subset or a controlled internal agent suite.

This stage resolves close calls and checks whether the static-to-agent relationship still holds. It also provides fresh calibration observations.

Stage 3: reserve full evaluation for consequential gates

Full agent benchmarks and production-specific tests remain appropriate for:

• deployment approval;
• major model-family changes;
• new scaffolds or tool policies;
• security-sensitive capabilities;
• cases where failure costs exceed evaluation costs.

The economic benefit comes from reducing how many candidates reach this stage, not from abolishing it.

A useful policy is to tie evaluation depth to both prediction margin and novelty.

A model predicted to outperform the incumbent by 15 points and belonging to a familiar family may require less preliminary investigation than a model predicted to win by one point using a new architecture, unusual reasoning mode, or different tool interface.

Four controls are needed before operationalizing a proxy

A production proxy should be managed like a model, not stored like a questionnaire.

The calibration refresh deserves special emphasis. Leave-one-out validation demonstrates interpolation across the observed collection of frontier models. It is not a time-forward test in which a proxy trained on one generation predicts architectures arriving two years later.

A robust implementation should therefore record:

• predicted and observed target scores;
• residuals by model family and scaffold;
• ranking reversals;
• proxy-instance exposure;
• changes to the target benchmark or execution environment.

Once residuals begin widening systematically, the proxy is not becoming “more conservative.” It is expiring.

What remains uncertain

The paper’s limitations are substantive enough to determine how Pace should be used.

Four targets share one agent framework

All four target results come from evaluations using the OpenHands framework. The tasks differ, but the shared harness may create common performance structure that helps proxy prediction.

Generalization to browser agents, enterprise workflow agents, embodied systems, proprietary toolchains, or substantially different scaffolds remains untested.

Fourteen models are not a deep calibration market

The models are strong and diverse by current commercial standards, but 14 observations remain 14 observations. In each leave-one-out fold, selection and prediction are learned from only 13 models.

The strong comparison against Lasso and Ridge shows that Pace handles this scarcity better than the tested alternatives. It does not make the scarcity disappear.

Public proxies invite optimization

A fixed public proxy can become a training target. Model developers may improve proxy performance without improving the agent behavior it is intended to predict.

Rotating or private evaluation sets reduce this risk, although they also complicate transparency and reproducibility.

Missing capabilities cannot be selected

Pace chooses among the signals present in its 19 source benchmarks. When an agentic target depends on a capability absent from that pool, no selection algorithm can recover it.

A proxy for a database agent, for example, may fail if the source pool lacks meaningful tests of schema discovery, transactional reasoning, or structured execution. The regression cannot infer an unmeasured dimension through professional enthusiasm.

Static tests do not observe trajectories

Agent failures frequently emerge from interaction effects: early mistakes, tool errors, context accumulation, recovery policies, latency, permission handling, and state corruption.

Static items may correlate with those outcomes across familiar models. They do not directly exercise them. This is the fundamental reason Pace is a proxy rather than an agent benchmark with better branding.

A cheaper answer to “what should we test next?”

Pace does not make agent evaluation unnecessary. It makes evaluation sequencing more intelligent.

Its empirical contribution is credible because the paper tests the question that matters: whether the proxy predicts models excluded from its own construction. The average results—3.80% absolute error, 0.81 rank correlation, and 84.37% pairwise accuracy—are strong enough to support screening and prioritization, but not strong enough to erase the distinction between predicted performance and observed behavior.

Its methodological contribution is restraint. Pace avoids jointly optimizing thousands of candidate instances against a tiny model panel. It combines target relevance, global geometric informativeness, and bootstrap treatment of target noise. The resulting predictor fits the past less perfectly and the held-out model more accurately.

For businesses, the appropriate outcome is an evaluation funnel:

cheap proxy for breadth, direct agent tests for discrimination, and full operational evidence for deployment.

That is less dramatic than replacing agent benchmarks. It is also more useful. Expensive evidence should be spent where it can still change the decision.

Cognaptus: Automate the Present, Incubate the Future.