More Shots, More Coverage? Measure the Set, Not the Sample

TL;DR for operators

If your system produces five patches, ten evidence candidates, or twenty molecular designs, choosing its model and inference settings from single-answer benchmarks can select the wrong configuration for the workflow you actually run.

The central measurement problem is redundancy. Several outputs can each be valid and individually strong while repeatedly landing on the same task-relevant result. Conversely, outputs that look different in wording or structure may add no genuinely new option.

Florian Le Bronnec and Rio Yokota’s Evaluating Multiple LLM Generations with Validated Task Coverage¹ evaluates the candidate set itself. Its Validated Task Coverage, or VTC, counts or measures the distinct useful outcomes that survive deterministic task validation and accumulate across a finite number of attempts.

Across five domains, the configuration with the highest coverage after one attempt was not the leader at the task’s headline multi-attempt budget in four cases. Selecting configurations by one-draw quality produced 12.1% task-macro VTC regret; selecting by surface diversity produced 11.0%.

For production teams, this suggests evaluating model, temperature, reasoning mode, and candidate-generation strategy at the number of candidates the workflow will actually generate and review. VTC does not yet solve the cost side of that decision: an attempt is not a normalized unit of tokens, latency, compute, or human attention.

The best first answer can build the narrower candidate set

Suppose a repair system generates ten candidate patches. Ordinary evaluation makes it natural to favor the configuration that produces the strongest patch on one draw. But the deployment objective is different: after ten attempts, the team wants a set containing several distinct passing interventions rather than ten variations on the same one.

That distinction changes rankings.

In four of VTC-Bench’s five tasks—molecule design, repository repair, differential diagnosis, and evidence search—the configuration with the highest coverage after one attempt was not the configuration with the highest coverage at the task’s headline attempt budget. Bug finding was the exception.

The mismatch is largest where repeated generation changes what the system can discover. Selecting the Top-5 configurations using one-attempt performance instead of coverage at the headline budget produced a 9.0% task-macro relative VTC regret and only 11 of 25 Top-5 positions in common.

The evaluation target therefore cannot simply be “How good is one sample?” when the product buys and exposes several samples.

VTC measures what survives validation and adds something new

VTC formalizes the set-level objective in three steps.

Each generated attempt is first validated according to the task. A valid output is then mapped to one or more task-specific useful outcomes. Finally, the outcomes accumulated across all attempts are merged before the set is scored:

The union is the critical operation. Producing an outcome already represented in the candidate set adds no new coverage.

The benchmark makes “different” operational rather than linguistic. For molecule design, useful outcomes are distinct Bemis-Murcko scaffolds among molecules satisfying requested properties. Repository repairs are grouped by the set of modified functions among passing patches. Bug-finding candidates receive credit for distinct behavioral bug classes exposed. Differential diagnoses are mapped to reference UMLS concepts, while evidence search is scored through covered reasoning aspects.

These mappings deliberately sit between exact-output identity and unrestricted semantic judgment. Two patches with different textual diffs may collapse to the same modified-function set; two differently worded diagnoses may map to the same UMLS concept.

That choice is also a boundary of the metric. VTC can only measure useful coverage after the task has a defensible definition of what counts as the same outcome.

Visible diversity is not the same as useful coverage

A natural substitute for set-level evaluation is to measure how varied the outputs look. The benchmark tests that shortcut and finds substantial disagreement.

Selecting the five configurations with the highest task-matched surface diversity produced 11.0% task-macro VTC regret and only 11 of 25 Top-5 positions in common with VTC selection. One-draw quality did little better, with 12.1% regret and 10 of 25 positions in common.

The distinction is functional. Surface diversity measures non-repetition in observable outputs. VTC asks whether those differences survive validation and correspond to additional task-relevant outcomes.

The two can diverge in either direction. A generator may rephrase or restructure essentially the same solution, inflating visible diversity without expanding the useful set. Another configuration may produce superficially similar outputs that nevertheless cross task-relevant boundaries.

For operators, a diversity dashboard is therefore only meaningful if the diversity unit matches something users can actually act on.

Inference settings become portfolio decisions

Once the evaluation target changes from one response to a finite set, familiar inference choices behave differently.

Raising temperature from 0.6 to 1.2 increased headline-budget VTC in 38 of 40 matched comparisons. In 25 of those comparisons, one-draw quality moved downward at the same time. Paired instance resampling left 36 of the 40 VTC differences entirely above zero.

Thinking showed a weaker alignment between single-output improvement and set coverage. It improved one-draw quality in 44 of 60 matched comparisons but improved headline-budget VTC in 36; quality and coverage moved in opposite directions in 23 comparisons.

Explicitly telling later generations to differ from earlier ones was even more task-dependent:

These experiments do not establish a universally better temperature, reasoning mode, or prompting strategy. They show that settings optimized for individual answers do not automatically optimize the portfolio of outcomes accumulated over several attempts.

Coverage curves can inform how much generation to buy

The business application is less about adopting one benchmark score than changing the unit of evaluation.

For a workflow that returns multiple patches, hypotheses, diagnoses, designs, or evidence items, Cognaptus would treat the model-inference configuration and candidate budget as a joint decision. Evaluate coverage at five attempts if users inspect five candidates. Evaluate it at ten if the system actually generates ten.

The full coverage curve adds another operational signal: diminishing returns. If attempts six through ten mostly repeat already covered outcomes, additional inference is buying little new option value. If new validated outcomes continue accumulating, a larger budget may be justified.

A production extension would go further and measure coverage per dollar, token, second, or review-hour. The paper itself does not provide that normalization. Its controlled budget axis is number of attempts, and attempts can differ substantially in compute use, output length, latency, and downstream human effort.

The metric inherits the task’s definition of “new”

The paper’s robustness checks strengthen the benchmark comparison without removing this measurement dependence.

Paired instance bootstrap resampling preserves the four one-attempt-to-headline-budget leader reversals and the proxy-selection mismatches. That is a robustness test against reweighting the frozen benchmark instances, not an estimate of uncertainty over future generations or an unseen population of real-world tasks.

The molecule-design sensitivity analysis also checks whether one leader reversal disappears when the useful-outcome representation changes from Bemis-Murcko scaffolds to coarser ring systems. The endpoint reversal persists, although finer configuration rankings remain sensitive to outcome granularity.

Other mappings have similar limits. Modified-function sets identify where a patch intervenes, not whether two patches embody genuinely different repair strategies. Diagnosis and evidence-search coverage depend on fixed human-curated reference targets that may omit valid alternatives.

So VTC is strongest where validation and outcome identity can be defined reproducibly and meaningfully. It should not be interpreted as a task-independent measure of “diversity.”

Evaluate the workflow you deploy

Repeated generation changes the object being optimized. Once a system retains several outputs, the relevant question is no longer only whether each draw is good. It is whether the finite set accumulates additional validated possibilities that matter to the task.

VTC gives that question a reproducible measurement structure, and VTC-Bench shows why the distinction affects configuration choice. Single-response quality, visible diversity, reasoning mode, and novelty prompting can all point in different directions once useful coverage across several attempts becomes the objective.

For operators, the next evaluation step is concrete: define what counts as a distinct useful outcome, measure how coverage grows at the candidate budget users can actually consume, and then attach the missing economics—compute, latency, and review cost—to that curve.

Cognaptus: Automate the Present, Incubate the Future.