The Agent Scored Higher. What Actually Improved?
TL;DR for operators An AI agent can perform remarkably well on held-out benchmark questions without reliably solving the intended task. On LiveMath, an evolved agent answered all 53 held-out questions containing a recurring answer option correctly, but only 22 of the other 71. The recurring pattern provided a shortcut that survived into the supposedly independent evaluation. ...