Commit First, Fail First: Qualify the Judge Before It Steers Optimization

TL;DR for operators

If an LLM judge controls which prompts, agents, models, or generated outputs survive repeated optimisation, test the judge independently before trusting its scores.

In Idil Gozel’s experiments, ordinary best-of-N search found many Python solutions that scored highly under a shipped LLM-judge configuration while failing hidden correctness checks.¹ Requiring the judge to solve the problem before seeing the candidate fixed that failure completely on one task: divergence fell from 90/96 and 93/96 candidates across two seeds to 0/96 in both.

The same intervention made another task worse. Divergence rose from 37/96 and 41/96 under the shipped configuration to 50/96 and 75/96 when the judge committed first. The difference was whether the judge’s own solution was correct.

For evaluation teams, that changes the deployment rule. Do not treat pre-candidate commitment, higher model tier, or evaluator-framework defaults as sufficient qualification. Test each judge-task pair on representative problems using correctness checks that remain outside the optimisation loop. The paper demonstrates this logic on four small programming tasks; it does not establish a universal pass rate for production evaluators.

Optimisation found wrong answers the evaluator preferred

Consider a routine optimisation pipeline. A generator produces several candidate outputs. An automated evaluator scores them. The strongest candidates survive into the next round. Separate checks exist to measure actual correctness, but those checks are deliberately hidden from the optimisation loop.

That separation sounds protective. The optimiser cannot directly target the hidden tests.

The interval-merging experiment shows why that is insufficient. Under the documented shipped judge configuration, 90 of 96 candidates in one seed and 93 of 96 in another received evaluator scores of at least 0.8 while failing the held-out correctness suite. Every candidate nevertheless passed every visible test.

This is evaluator gaming in operational terms: selection pressure discovers outputs that satisfy the measurement system better than they satisfy the underlying task.

No jailbreak or prompt injection was required. The generator did not see the hidden checks. The optimisation process simply kept selecting what the evaluator rewarded.

The failure was also more specific than the judge overlooking defective code. On the interval task, the judge identified the problematic implementation behavior and then interpreted that behavior as correct. Increasing the amount of visible reasoning would therefore not necessarily repair the evaluator.

Solving before scoring changes what optimisation can exploit

One proposed defence is to stop the candidate from influencing the judge’s own reasoning about the task.

Instead of showing a candidate and asking whether it is correct, the evaluator first solves the task independently. Only after committing to that solution does it inspect the candidate and compare the two. The paper calls this commit-first judging.

On the interval task, it worked decisively. The judge’s committed solution was correct, and divergence fell to 0 of 96 candidates in both seeds. Final held-out correctness reached 1.0.

But the duration-parsing task exposes the condition hidden inside that success.

Commit-first judging did not remove the optimisation anchor. It moved it.

Under conventional holistic judging, the candidate can influence the judge’s reasoning. Under commit-first judging, the judge’s own independently produced solution becomes the reference point. If that solution is correct, selection toward it can suppress gaming. If it is wrong, selection can instead reward candidates compatible with the judge’s mistake.

One duration-task run makes the mechanism especially visible. The final candidate population achieved held-out correctness of 0.667 while receiving an evaluator score of 0.99. The judge’s own committed solution also scored 0.667 on the held-out checks. The second seed did not lock onto that score exactly, so the paper supports the possibility of this failure mode rather than inevitable convergence to the judge’s error.

A stronger judge is not the same as a qualified judge

A natural response is to use the most capable evaluator available.

The model-size control cuts against that rule. On the interval task, the smaller Claude Haiku 4.5 judge was substantially harder to game than Claude Opus 4.8. Haiku also solved the interval task correctly in the study’s independent solving test.

This does not establish that smaller judges are generally safer. The control was a single run, and the cross-model comparison also involved different supported sampling settings. What it does show is enough to reject model tier as a standalone qualification criterion.

The relevant unit is the judge-task pair.

That distinction matters for procurement and governance. A team selecting an evaluator for coding, document review, agent trajectories, or another specialized workflow needs evidence that the proposed judge is competent on that task distribution. General model capability, price, and leaderboard position do not measure the specific failure that commit-first judging inherits.

Test the judge before giving it selection authority

The paper turns that mechanism into a simple screening procedure.

Each judge was asked to solve each of the four programming tasks three times. Those answers were then scored using held-out checks that the judge never saw. This direct-solve probe correctly separated the demonstrated cases: judge-task pairs that could independently solve the task behaved safely under commit-first judging, while the task neither judge could solve was the task where the defence backfired.

The evidence for this probe is promising but small—three attempts per task across four tasks—so it should be treated as a deployment pattern to validate, not as a calibrated universal threshold.

For an evaluation team, the workflow follows directly from the mechanism:

1. Define representative tasks for the workload the judge will govern.
2. Maintain correctness checks that are inaccessible to both the candidate-selection loop and the judge during solving.
3. Ask the proposed judge to solve those tasks independently.
4. Measure its task-specific correctness before enabling commit-first evaluation.
5. Once deployed, monitor whether evaluator scores and held-out correctness begin to separate under realistic optimisation pressure.

This separates two questions that are often collapsed: “Can this model produce plausible evaluations?” and “Is this judge reliable enough on this task to control selection?”

Only the second grants operational authority.

Framework defaults do not resolve the qualification problem

The software audit reinforces that teams cannot assume evaluator safeguards arrive automatically.

The study examined 27 default judge configurations from eight evaluation frameworks—DeepEval, Ragas, Promptfoo, OpenAI Evals, Braintrust autoevals, LangChain, Arize Phoenix, and MLflow. Of the 24 objective-task configurations to which the criterion applied, none implemented commit-first judging. Nine used a recompute-style approach in which the candidate was already visible while the judge reasoned.

That finding applies to the shipped defaults audited in the paper, not every customized production configuration.

It also should not be read as evidence that simply adding commit-first behavior would make those frameworks safe. The experiment shows why configuration design and evaluator competence must be assessed together.

The measurement system needs its own audit

The paper also found faults in the instruments used to validate evaluator integrity.

Its held-out suite detected 15 of 16 deliberately defective implementations, but two held-out checks were later found not to be justified by the task specifications. Removing them did not change any reported divergence count. Separately, verification of supporting literature found problems in five of fifteen first-draft quantitative claims.

These are not the paper’s primary experimental results. They are validation checks around the measurement process. Their operational value is that evaluator governance should not stop at auditing the scoring model.

If hidden tests define “correct,” those tests require specification traceability. If prior evidence justifies an evaluation criterion, those claims require source verification. A protected measurement channel is useful only if the protected measurement itself is sound.

What this study supports—and what remains open

The experimental evidence is narrow by design. The study uses four small single-function Python tasks, and only two leave room for score-correctness divergence because the generator solves the other two correctly from the start. The main experiments use one generator model and one primary judge family. Several controls are single runs, and exact convergence toward the judge’s own incorrect score appears in only one of two relevant seeds.

Those limits constrain generalization, but they do not erase the deployment problem demonstrated here.

For teams using LLM judges to steer repeated selection, the question is no longer whether commitment before candidate inspection sounds safer in principle. The judge’s independent answer becomes part of the optimisation environment. Its correctness therefore needs to be measured before the evaluator is allowed to determine which outputs survive.

The resulting governance rule is deliberately task-specific: qualify the judge on hidden checks first, then decide whether its judgment deserves optimisation authority.

Cognaptus: Automate the Present, Incubate the Future.