Commit First, Fail First: Qualify the Judge Before It Steers Optimization
TL;DR for operators If an LLM judge controls which prompts, agents, models, or generated outputs survive repeated optimisation, test the judge independently before trusting its scores. In Idil Gozel’s experiments, ordinary best-of-N search found many Python solutions that scored highly under a shipped LLM-judge configuration while failing hidden correctness checks.1 Requiring the judge to solve the problem before seeing the candidate fixed that failure completely on one task: divergence fell from 90/96 and 93/96 candidates across two seeds to 0/96 in both. ...