Machine Learning Operations

TL;DR for operators Agent evaluations are expensive for a fairly obvious reason: the agent has to do something. It must browse, edit files, call tools, manipulate repositories, survive its own mistakes, and occasionally discover that the environment has changed while nobody was looking. The paper introduces Pace, a method for predicting performance on an expensive agentic benchmark from a compact set of cheaper, non-agentic test instances.1 Across 14 frontier models and four agentic benchmarks, a 100-instance Pace proxy produces an average mean absolute error of 3.80 percentage points, a 0.81 Spearman rank correlation, and 84.37% pairwise model-ranking accuracy under leave-one-model-out validation. ...

TL;DR for operators Many tabular foundation models behave like very competent consultants with a mildly expensive habit: they want the entire labelled training set placed in front of them at inference time. That works neatly on small datasets. It becomes rather less charming when the table grows to tens or hundreds of thousands of rows and the model’s attention cost starts behaving like it has discovered compound interest. ...