MLE-Bench

Research has a waiting-room problem. A human team proposes an experiment, waits for the training run, checks the metric, argues about whether the result is real, then decides what to try next. The cycle is familiar, expensive, and mildly theatrical. AI research agents promise to compress that loop. Give the agent a benchmark, a compute budget, and a tool environment; let it search; harvest better models at the end. Convenient. Also, if done naively, a beautiful machine for producing confident nonsense at GPU speed. ...

Budget has a way of making AI agents less magical. On a slide, an AI research agent looks like a neat loop: read the task, propose an idea, write code, run an experiment, improve, repeat. In production, it looks more like a slightly caffeinated junior researcher with terminal access: sometimes brilliant, sometimes stubborn, and occasionally determined to spend four hours failing at the same doomed approach because the first idea sounded respectable. ...