Same Questions, Harder Test

TL;DR for operators

When several models all score near the top of an internal multiple-choice suite, the benchmark may still contain useful information. The remaining problem can be the evaluation format rather than the questions themselves.

The paper introduces answer pooling: combine the answer options from several related questions, then require the model to assign one candidate to each question without reusing candidates. The questions and correct labels do not change. Across eight benchmarks and eighteen models, every evaluated model scores lower under pooling, and model rankings move enough that the format can affect comparative decisions.

For evaluation teams, the method has three practical uses. It can recover benchmark headroom without new question authoring, estimate how much ordinary multiple choice rewards elimination of local distractors, and create exact abstention tests by deliberately removing known correct answers. The strongest use case is a paired diagnostic alongside existing scores, not a claim that every pooled error reveals a general reasoning failure.

High benchmark scores can be partly format-specific

Consider a model-selection review in which three systems all score above 90% on the same multiple-choice suite. If the gaps are small, the natural conclusion is that the benchmark has little room left to separate them.

But suppose the questions and correct answers remain identical while the candidate structure changes. All three models lose accuracy, and their ordering changes. The measured capability was therefore not independent of the evaluation wrapper.

That is the central result of Eltahir et al.’s Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling.¹ Across the reported text benchmarks, rank correlations between ordinary multiple choice and the harder format range from 0.87 to 0.97. Across image and video benchmarks, they fall to 0.65–0.82. Every evaluated model loses accuracy under pooling on every reported benchmark.

The implication is narrower than “multiple choice is invalid.” Ordinary accuracy remains informative. But when procurement, release gates, or safety reviews depend on small score differences, the format can contribute enough advantage to affect the decision.

Pooling replaces local distractors with broader competition

Ordinary multiple choice gives each question a small, specially written set of alternatives. A model does not always need to identify the answer positively. It can sometimes succeed by rejecting three candidates that are locally implausible.

Answer pooling changes that competition. Several related questions are grouped, all of their options are placed into one candidate set, and the model must assign one answer to each question while using each candidate at most once.

For five four-option questions, ordinary independent guessing gets the entire group right with probability about $10^{-3}$. With twenty pooled candidates and five non-repeating assignments, the corresponding floor is approximately $5.4 \times 10^{-7}$. The same labels therefore support substantially more group-level headroom.

The more interesting quantity is the accuracy gap. The authors call the difference between ordinary multiple-choice accuracy and pooled accuracy elimination credit when the correct answer remains recoverable and no newly pooled candidate is also a valid answer.

On QuALITY, for example, Gemini 3.8 Flash falls from 0.955 to 0.923, a difference of about 0.03. Qwen2.5-7B falls from 0.704 to 0.467, a difference of about 0.24. The weaker model appears to receive substantially more benefit from the original local candidate set.

That interpretation has a condition. A pooled error does not automatically demonstrate a reasoning failure. If another candidate introduced by pooling also answers the question, the original answer is no longer uniquely recoverable. The paper screens groups for this ambiguity using three models, but does not replace that process with exhaustive human adjudication.

The control points to plausible alternatives, not just longer prompts

Pooling changes more than the number of distractors. It also creates a longer prompt and a joint assignment problem. Either could in principle explain the accuracy loss.

The paper’s unrelated-pool control is therefore important as a mechanism check rather than a second headline result. It preserves the pooled assignment format but replaces wrong options with unrelated answers from other groups.

For the evaluated local models on QuALITY, mean accuracy is 0.758 under ordinary multiple choice and 0.596 under normal pooling. With the unrelated pool, it rises to 0.842. On MMMU-Pro, the corresponding figures are 0.395, 0.277, and 0.798.

If joint assignment or prompt length were the primary problem, replacing plausible alternatives with unrelated candidates should not produce that recovery. The control instead supports the interpretation that candidate plausibility is doing much of the work.

Cross-domain tests add a boundary. Mixing questions from different domains partly recovers accuracy on RACE and GPQA, but makes QuALITY substantially harder because several long passages must be included together. Pool construction can therefore introduce a separate context-length burden if related questions are not grouped carefully.

Removing answers turns the benchmark into an abstention test

The same construction produces a diagnostic that ordinary accuracy cannot provide.

Because every correct answer is already known, evaluators can deliberately delete selected correct candidates. They now know exactly which questions are unanswerable without creating new items or asking another model to judge the response.

On QuALITY, pooled false-answer rates range from 0.167 to 0.998 while answerable-question accuracy occupies a much narrower range of 0.427 to 0.758. Seven of eight evaluated open-weight text models answer 87–100% of questions whose correct answer has been removed. On MedXpertQA-MM, evaluated vision models answer roughly 96–100% of such cases.

Adding an explicit “none of these” option improves false-answer rates for every QuALITY model, but generally increases false abstention. Abstention is therefore not a single model property revealed by one score; it depends materially on how the evaluator permits the model to decline.

For safety and reliability teams, this is arguably more actionable than simply making the benchmark harder. Two models with similar answerable accuracy can have very different tendencies to invent an answer when the correct one is unavailable.

Existing suites can become better model-selection instruments

Cognaptus inference: organizations with grouped internal evaluations can use pooling as a second measurement layer before paying for a replacement benchmark.

For procurement, the relevant question becomes whether a vendor ranking survives stronger candidate competition. For release reviews, a pooled score can show whether a high ordinary score depends heavily on eliminating weak distractors. For reliability testing, removed-answer variants expose false-answer behavior with exact ground truth. Context-removal variants can also test whether performance actually depends on the passage, image, or video that the benchmark is intended to measure.

This is especially attractive when new item creation is expensive or contamination concerns make every new benchmark release costly. The method reuses existing labels rather than pretending those labels have become worthless once ordinary accuracy approaches a ceiling.

It should remain a paired evaluation rather than a replacement number. Ordinary multiple-choice accuracy answers one question; pooled performance, elimination credit, abstention behavior, and context dependence answer different ones.

The method works only when the grouping remains defensible

Answer pooling requires several questions that can be grouped by shared context, topic, subject, or analogous structure. It is less natural for isolated items with no meaningful neighboring candidates.

Its interpretation also depends on answer uniqueness. The authors use model-based ambiguity screening, which reduces but does not eliminate the risk that an additional pooled answer is also correct.

Larger pools create longer prompts, and multimodal pools may require several images or other contexts at once. At some point, the test can begin measuring long-context handling in addition to competition among plausible answers.

Finally, the reported evaluations use direct answering, greedy decoding, and disable explicit thinking where supported. The paired comparisons are therefore the evidence to emphasize; the absolute scores should not be read as replacements for reasoning-mode model-card results.

A saturated score is not necessarily a saturated benchmark

The paper’s most useful move is methodological rather than numerical. It separates the questions a benchmark contains from the candidate structure through which those questions are measured.

Once those are treated separately, an apparently exhausted evaluation can still reveal whether models recognize answers under stronger competition, whether rankings survive a different candidate structure, whether systems abstain when the answer is absent, and whether they continue to depend on the intended context.

For evaluation teams, that means benchmark maintenance need not always begin with writing the next benchmark. Sometimes the existing questions still have work left to do.

Cognaptus: Automate the Present, Incubate the Future.