Cover image

Build the Player, Audit the Game

TL;DR for operators A model can improve substantially in internal tests and still perform worse after deployment. AutoTavern provides a concrete example: one TavernAgent version increased its simulated first-place rate from 12.8% to 24.2%, yet its live-ladder rating fell from 8,968 to 7,658 over 32 games, including 18 with client failures. This did not prove the policy itself had deteriorated; it showed that internal scores were insufficient to predict real execution. ...

November 20, 2026 · 7 min · Zelina