The Right Answer Was in the Pool

TL;DR for operators

An AI system can generate the correct answer and still return an incorrect one. The distinction matters when teams spend additional inference resources producing multiple candidate solutions but rely on an imperfect mechanism to choose among them.

Xiao Yue and Guangzhi Qu introduce Coverage, Realization, and Validity Evidence (CRV), a diagnostic protocol for separating correct-answer availability, answer-selection performance, and evidence that a correct-answer derivation is supported by its premises.¹

On their difficult formal-geometry benchmark, sixteen generated candidates contain a correct numerical answer in 68.93% of evaluated cases, yet verifier-weighted self-consistency selects the correct answer only 37.99% of the time. A separate process audit raises another concern: reaching the right numerical result does not establish that the accompanying reasoning is justified.

For AI teams, the immediate lesson is to measure how much available answer quality actually reaches users before purchasing more inference capacity. Candidate selection and reasoning assurance require their own tests. These findings diagnose a specific research setting, not a measured failure rate for business AI applications.

More candidates, almost no improvement in the delivered answer

Suppose a reasoning service generates eight alternative solutions, scores them, and returns one answer. Its developers double the generation budget to sixteen, expecting that additional candidates will make the final response more reliable.

Yue and Qu test this expectation on HardShift441, a collection of 441 formal-geometry problems derived from FormalGeo7K. Their principal generator is a LoRA-adapted Qwen2.5-7B model, with results averaged across three generator seeds and three verifier seeds.

The key distinction is between finding a correct answer somewhere in the candidate pool and actually selecting it. The first quantity is called pass@K: the percentage of problems for which at least one of K generated numerical answers matches the reference target.

Doubling the number of samples increases coverage by 8.61 percentage points, but selected-answer accuracy rises by just 0.12 percentage points. The coverage-to-selection gap consequently increases by 8.49 points, with a reported 95% bootstrap confidence interval of 6.60 to 10.43 points.

The additional generation has discovered more correct answers. The selection mechanism has captured almost none of that improvement.

This is a consequential distinction for inference-budget allocation. If the objective is a better answer delivered to the user, improvements in candidate availability are valuable only when the downstream readout can recover them.

CRV measures where reasoning performance is lost

CRV is primarily an evaluation protocol, not a replacement verifier or a new selection algorithm. Its contribution is to distinguish three questions that a single accuracy statistic cannot answer.

Coverage asks whether the correct numerical answer appears among generated candidates. Realization asks whether the system’s actual readout recovers that answer. Validity evidence asks whether a separate, label-blinded process critic judges a constructed derivation to be supported by its stated premises.

To isolate selection performance, the researchers freeze generated candidate pools before comparing readout mechanisms. Answers are parsed, numerically matched, and grouped into clusters sharing the same result.

Three readouts are compared:

• Majority voting selects the most frequent numerical answer.
• Best-of-K selects the answer associated with the highest-scoring individual candidate.
• Verifier-weighted self-consistency (WSC) adds transformed verifier scores within each numerical-answer cluster, making both the scores and the number of agreeing candidates relevant.

At sixteen candidates, the primary HardShift441 results are:

CRV expresses the difference between coverage and realized accuracy as a generation-to-selection gap:

For WSC, that gap is 30.94 percentage points.

The authors also calculate a realization ratio: how much of the improvement available between Majority accuracy and perfect selection from the sampled pool is captured by a readout. WSC realizes only 12.54% of that headroom on HardShift441.

These are answer-level measurements. Even perfect recovery of the reference numerical answer would not independently establish that the accompanying reasoning is mathematically valid.

Rare correct answers are particularly difficult to select

Why does the verifier struggle when a correct answer is already available?

Part of the explanation lies in correct-answer multiplicity: how many of the sixteen candidates produce the right result.

WSC aggregates scores across an answer cluster. If several candidates repeat an incorrect answer, their combined weight may exceed that of a single correct candidate. The problem is not necessarily that every incorrect trace scores highly; frequency-sensitive aggregation can amplify a competing answer group.

The observed performance changes sharply with multiplicity.

Best-of-K performs substantially better in the first two bands, while WSC performs better when correct answers are more numerous.

The researchers also examine hardest-negative discrimination: whether the verifier’s highest-scoring correct candidate outranks its highest-scoring incorrect candidate on the same problem.

Its win rates are 14.57%, 26.59%, and 66.82% across the one-correct, two-correct, and three-or-more bands. These exceed multiplicity-adjusted random-ranking baselines, but reveal substantial difficulty distinguishing a rare correct candidate from convincing incorrect alternatives.

This helps explain why an aggregate verifier AUC of approximately 0.819 on HardShift441 does not translate into reliable final selection. Overall candidate discrimination and winning the decisive comparison within a particular problem are different requirements.

Multiplicity is a diagnostic signal, however, not an experimentally established cause of accuracy differences. Changing generation seeds affects candidate quality as well as frequency. A within-problem comparison supports the association, but does not isolate multiplicity’s causal contribution.

For verifier developers, the actionable test is whether the correct candidate survives competition with the strongest incorrect alternatives, particularly when it appears only once.

A correct result still needs a defensible derivation

Even successful answer selection leaves a separate question: does the generated reasoning follow from the supplied premises?

The researchers investigate this through a blinded process audit. They construct one correct-answer representative and one difficult incorrect-answer representative for each of 195 covered problems from the 300-problem HardShift441 adaptation partition.

GPT-5.4, acting as a process critic without access to answer-correctness labels, evaluates whether each derivation is supported by the formal geometry state.

Among the 195 constructed correct-answer representatives, the critic reports:

The critic-labeled supported proportion is 6.15%, with a Wilson 95% confidence interval of 3.55% to 10.45%.

The result illustrates a serious measurement distinction, but its interpretation must remain narrow. These are constructed representative traces, not an exhaustive audit of generated candidates or the outputs selected by the deployed readouts. The verdicts are model judgments rather than independently established mathematical proof results.

There are reasons to examine the critic itself. Approximately 12% of its refutations involve geometric configuration information not sufficiently determined by the textual state, creating potential over-refutation. Secondary agreement checks cover only small selected samples: human reviewers agree on 20 of 20 reviewed verdicts, while another model agrees on 33 of 34.

Neither agreement result establishes the critic’s overall accuracy as a proof checker.

For reasoning-intensive applications, the research therefore supports separating answer correctness from evidence about intermediate reasoning. It does not justify interpreting the 181 critic refutations as 181 independently proven invalid derivations.

Improve the bottleneck before expanding the budget

The researchers screen several possible interventions, including stronger generators, larger verifiers, groupwise ranking, set-level selection, and verifier ensembles.

The improvements are conditional. For example, a larger Qwen3-32B verifier improves WSC accuracy by 4.17 percentage points over the Qwen2.5-7B verifier in the legacy-hard setting. Groupwise rank training improves accuracy by 4.93 points on the 81-problem HardShift441 development split but reduces it by 9.24 points on legacy hard.

These experiments explore possible remedies rather than establishing a universal ranking. Several interventions use different splits, candidate pools, or configurations, and none consistently eliminates the gap.

For a team deploying multi-candidate reasoning, Cognaptus draws a measurement-first workflow from these findings.

First, preserve candidate traces, parsed answers, scores, and final selections in a reproducible evaluation set. Where reference answers are available, report candidate coverage alongside actual selected-answer accuracy. Reuse frozen pools when comparing alternative selectors so changes in generation do not obscure readout improvements.

Next, examine selection failures at the problem level. Compare Majority, Best-of-K, and weighted aggregation; measure how often the strongest incorrect alternative defeats a correct candidate; and investigate whether answer-frequency patterns help explain failures. Multiplicity bands are particularly valuable for offline diagnosis, even though true correct-answer multiplicity is ordinarily unknown during production inference.

Finally, evaluate reasoning support separately wherever the justification matters to the user or downstream decision. Model critics can provide diagnostic evidence, but assurance claims require validation appropriate to the task. Abstention or escalation for weakly supported outputs is a possible design response, not a production policy validated by this study.

What the evidence does not transfer automatically

HardShift441 concerns numerical answers and textual reasoning over formal geometry states, not complete symbolic theorem proving. Its difficulty profile is partly defined by failures of a reference solver; those failures do not imply invalid reference targets or guarantee that every problem is globally unseen.

The primary matched-pool readout comparisons are comparatively strong evidence about selection behavior within the evaluated setting. They do not establish a causal effect of multiplicity, and the legacy-hard and shifted benchmark panels use different generator-training configurations. Reported bootstrap intervals also condition on already-trained model seeds rather than incorporating retraining uncertainty.

Most importantly, neither the measured accuracy gap nor the critic’s support judgments can be directly projected onto financial analysis, coding agents, operational decision support, or other production workflows.

The transferable contribution is the diagnostic separation itself.

A system’s ability to produce an answer, its ability to choose that answer, and its ability to justify the reasoning are different capabilities. Optimizing one without measuring the others can produce improvements that remain invisible to users.

For AI operators, the more informative question is not simply how many correct answers additional computation generates. It is how much of that improvement survives selection—and what evidence supports the answer ultimately delivered.

Cognaptus: Automate the Present, Incubate the Future.