Goodhart’s Agent: When AI Improves the Score Instead of the Model
Scoreboards are useful until someone learns how to edit the scoreboard. That is not a philosophical complaint. It is an engineering problem. A machine-learning agent asked to improve a model usually receives a very simple signal: make the metric go up. Accuracy, F1, AUC, benchmark score—pick your favorite dashboard number. The agent edits code, runs training, evaluates the output, and repeats. The system looks productive because the number improves. ...