Judge, Jury, and Benchmark: The Metanym Game Grades the Graders

TL;DR for operators

The Metanym Game asks models to invent structured analogies across unrelated domains, grade one another’s submissions, and reveal which graders deserve to be trusted.¹ Because the test material is produced during the run, there is no fixed question bank waiting to appear in tomorrow’s training corpus.

The clever part is not merely letting models vote. The paper separates two problems that most automated evaluation systems casually blend together:

1. Can a model detect factual errors? The benchmark estimates this from the dominant pattern in a matrix of peer ratings, using a graded singular value decomposition rather than a gold answer key.
2. Can a model apply a stable subjective standard? The benchmark changes the numerical calibration anchor and checks whether the evaluator preserves its ranking of the same submissions.

In the twelve-model experiment, factual judging is scarcer than factual generation. Some models produce strong work while contributing almost no useful error-detection signal. The strongest factual judge, meanwhile, is not the strongest generator. Apparently, being eloquent and being qualified to mark the examination remain different occupations, even for machines.

For businesses, the immediate value is evaluator qualification. Before using an LLM to score support conversations, review compliance reports, rank generated analyses, or supervise another agent, an organization could test whether that model is informative, stable, and robust to calibration changes. The method is promising, but it is not yet a self-validating truth engine: its scores remain relative to the participating panel, its central factual assumption can fail under shared bias, and its proposed promotion-and-relegation system has not been run end to end.

Every vendor bake-off has a grader problem

Consider a familiar procurement exercise. Several vendors perform the same task, and each is invited to review the others’ work. One vendor writes beautifully. Another finds subtle errors. A third assigns nearly identical scores to everything while producing a very professional explanation of why.

A simple average treats all three opinions as equally useful.

A popularity vote is not much better. Agreement may indicate expertise, but it may also indicate common training data, shared stylistic preferences, duplicated blind spots, or a collective unwillingness to use the lower half of a ten-point scale. Peer evaluation becomes especially awkward when the work is novel enough that no authoritative answer key exists.

The Metanym Game is an attempt to turn that awkwardness into the test itself. Models do not merely answer questions. They generate the questions, produce the candidate material, evaluate their competitors, and supply the data used to infer which evaluators are competent.

That makes the paper more interesting than its final leaderboard. The leaderboard is the residue. The mechanism is the contribution.

The benchmark begins with production, not a question bank

A metanym task starts with an abstract relational structure that can be instantiated in multiple, distant domains.

The paper’s introductory example begins with a passage about cell signalling. Selected domain terms are replaced while the surrounding relational language remains mostly fixed, producing a parallel description of human language. Cells become humans, tissues become communities, organisms become societies, and the claims must remain factually defensible sentence by sentence.

The terminology is slightly elaborate:

• A context template is the shared relational paragraph with slots.
• A metanym set supplies the domain-specific terms for those slots.
• A parallel context is one instantiated domain.
• An archetypal context is the abstract structure supposedly shared by all the parallel contexts.

In the canonical run, each model produces a portfolio containing five archetypal contexts. Each context has a template and five domain instantiations, giving 25 parallel contexts per standard portfolio.

This matters because most analogy tests ask a model to recognize a relationship that someone else has already prepared. The Metanym Game asks the model to construct a multi-part mapping from scratch, make it travel across distant domains, and preserve factual validity after every substitution.

Recognition can sometimes be solved through elimination, familiar surface patterns, or test-taking heuristics. Production removes much of that scaffolding. There is no list of candidate mappings and no convenient option C.

The task also creates its own error-bearing material. A model that invents an ambitious template may expose itself to factual mistakes in biology, finance, software, ecology, or whichever domains it has recruited into the analogy. Those errors become the substrate on which the other models’ judging competence can be tested.

Fresh generation reduces contamination without abolishing it

Because the portfolios are generated during the run, the benchmark has no permanent bank of fixed items. A future model cannot simply have memorized the exact test questions, because the exact questions did not exist before the run.

That is a meaningful improvement over static benchmarks whose questions, explanations, and answer keys circulate freely online.

It is more accurate, however, to call the design contamination-resistant than contamination-proof. Future models could learn the prompt format, the published anchor portfolio, earlier submissions, or common strategies for constructing persuasive metanym tables. The paper proposes screening new portfolios against archived submissions and treating an unusually large gap between generation and evaluation scores as a possible warning sign. Those are reasonable safeguards, not demonstrated guarantees.

The benchmark refreshes the content automatically. It does not repeal imitation.

Six scoring axes create two different epistemic problems

Every portfolio is evaluated on six axes.

The length criterion may look peculiar until the incentive problem becomes visible. If factual correctness were rewarded without any counterweight, the safest strategy would be to write the shortest possible template. Fewer sentences mean fewer opportunities to be wrong. Rewarding sustained length makes minimalism less attractive, while factual scoring ensures that padding is not free.

More important is the division between the first axis and the other five.

Factual correctness has an external concept of truth, even when the benchmark does not possess the labels in advance. Beauty and intelligence do not. A majority can be wrong about a factual claim; it can also be unanimous about an aesthetic convention without that convention becoming objectively correct.

The paper therefore refuses to use one aggregation rule for everything. That decision is central. Many peer-evaluation systems apply consensus broadly and quietly rename conformity as competence.

One matrix separates sharp judges from agreeable ones

For factual scoring, the benchmark constructs a matrix whose rows are evaluators and whose columns are generated parallel contexts. Each entry is the evaluator’s factual score from 1 to 10.

The matrix is row-centred first, removing each evaluator’s average leniency. A model that gives everyone an 8 should not be considered insightful merely because it is reliably cheerful.

The paper then takes the leading singular component:

Here:

• $u$ represents the evaluators’ loading on the dominant shared rating pattern.
• $v$ represents the relative factual standing of the generated contexts.
• $\sigma_1$ captures the strength of that shared pattern.

The intended interpretation is that competent evaluators converge on which contexts are relatively strong or weak after their individual scoring baselines have been removed. An evaluator with a large loading on $u$ is contributing to that common factual signal. An evaluator whose centred scores are flat or idiosyncratic approaches zero.

This is more useful than simply measuring whether an evaluator is internally consistent. A model can apply the same mistaken rule with exquisite regularity. The benchmark calls that precision without information.

The use of graded scores also avoids imposing an arbitrary true-or-false threshold. In the paper’s sensitivity check, binarizing the scores at thresholds of 4, 5, or 6 changes the evaluator ordering and flips the marginal council seat. The graded decomposition retains distinctions that thresholding discards.

The mechanism has one large assumption: the dominant pattern shared by competent evaluators is factual truth.

That assumption becomes more plausible with independent evaluators and less plausible when the panel shares systematic biases. If every model has inherited the same misconception, spectral decomposition will identify a beautifully coherent misconception. Linear algebra is many things, but an exorcist is not among them.

The generator score is not a second independent discovery

The right singular vector can be aggregated by generator to produce a factual-generation score. This lets the same decomposition characterize both sides of the interaction:

• Which judges detect weaker factual claims?
• Which generators produce stronger factual claims?

That two-sided construction is elegant, but its interpretation needs discipline. The generator-factuality score is not independent confirmation of the evaluator score. It is effectively the panel’s factual ratings, reweighted by the inferred competence of the evaluators. In the reported data, the two equivalent constructions agree almost exactly, with correlation rounded to 1.00 and differences no greater than 0.14.

The decomposition earns most of its keep by estimating evaluator competence. On the generation side, it reorganizes the same evidence rather than manufacturing a second witness.

The unanchored leaderboard chooses the ruler; it is not the final result

The benchmark cannot begin with a standing council and fixed reference because neither exists yet. It therefore uses a three-stage bootstrap.

First, all twelve models grade one another without a calibration anchor. This produces an initial generator leaderboard and selects the strongest submission as the future reference portfolio.

Claude Opus 4.5 leads this initial round with a mean score of 9.28. Claude Opus 4.1, Opus 4.0, and Sonnet 4 follow, creating a tightly compressed top group. The model families form three broad tiers in this roster, with family differences larger than the observed within-family size gradients.

Those results are interesting, but the first leaderboard is mainly infrastructure. Its job is to choose the anchor. It is not the official final ranking.

The experiment covers twelve models from Anthropic, Google, and OpenAI, with temperature set to zero and reasoning and tools disabled. It is therefore not a general parameter-scaling study, nor a verdict on every contemporary deployment configuration. It is one deliberately heterogeneous panel under a reproducible direct-response protocol.

Anchoring improves resolution, then becomes a diagnostic

Claude Opus 4.5’s winning portfolio is fixed as the reference and assigned a score of 7 on every criterion. Evaluators then score each target relative to it.

This substantially improves discrimination. Compared with unanchored scoring:

• Between-target variance increases by 2.5 times.
• Mean within-target standard deviation increases by only 1.14 times.
• The resolution F-statistic rises from 0.33 to 0.66.

The statistic remains below 1, meaning evaluator disagreement within a target is still larger than the average difference between targets. Anchoring improves the signal; it does not suddenly turn the leaderboard into atomic spectroscopy. The paper appropriately relies on bootstrap intervals for tier-level conclusions.

The researchers then rerun the evaluation while declaring the same reference portfolio to be worth 5, 6, 7, and 8. Only the calibration instruction changes.

This sweep has two functions.

First, it tests whether the factual SVD result is sensitive to the anchor. Row-centering should remove much of the numerical offset introduced by calibration, and the appendix reports that the factual competence ordering remains stable enough to preserve its relationship with GPQA across anchor values.

Second, the sweep becomes the estimator for subjective judging reliability.

A stable subjective judge should survive a moving yardstick

For each evaluator and each non-factual criterion, the benchmark compares the model’s score vectors across all pairs of anchor values. The mean Pearson correlation becomes its anchor-shift consistency.

A reliable evaluator may raise or lower all its numbers when the anchor moves. Pearson correlation permits that. What should not happen is a wholesale reordering of which submissions are beautiful, intelligent, structurally diverse, or distinct.

This is an invariance test:

Change something that should not alter the underlying judgement, then check whether the judgement survives.

The method is preferable to peer agreement on subjective criteria because it does not require an evaluator to share the panel’s taste. A model can disagree with every other judge and still be internally stable. Consensus is used only where the paper believes consensus can reveal truth; stability is used where no singular truth is available.

The factual axis illustrates why consistency alone is insufficient. GPT-4.1 Mini shows relatively strong anchor-shift consistency across the rubric, yet its factual SVD loading is near the inert band. It applies a stable standard, but its factual scores contribute little to the shared error-detection signal.

That distinction is operationally important. A compliance reviewer that consistently misses the same violation is easy to reproduce and unwise to employ.

The council requires both information and stability

A model earns a council seat only by clearing two gates:

Five evaluators qualify:

• Gemini 3.1 Pro
• Claude Opus 4.5
• Gemini 2.5 Flash
• Claude Opus 4.0
• Claude Opus 4.1

The selection is not based on generation rank. Claude Sonnet 4 generates near the top but fails the factual-evaluator gate. Gemini 2.5 Flash obtains the weakest council seat: it clears the factual band, while its criterion-reliability score sits exactly at the 0.78 threshold.

That marginal case deserves attention because it shows how procedural choices enter the supposedly self-contained system. The council emerges from the data, but the reliability floor, the definition of the inert band, the anchor values, and the seat protocol remain designed choices. Self-contained does not mean assumption-free.

The main finding is a factual maker–judge split

The final score averages four anchored components:

where:

• $G_F$ is factual generation quality.
• $G_C$ is non-factual generation quality.
• $E_F$ is factual evaluator competence.
• $E_C$ is subjective criterion reliability.

Selected results make the separation visible:

The strongest factual judge is Gemini 3.1 Pro, whose generator scores are merely mid-pack relative to the leading Claude models. Claude Sonnet 4 almost matches the anchor’s factual-generation score but contributes little factual-judging signal. GPT-4.1 generates reasonably strong factual material while its inferred factual-evaluator competence is close to zero.

The divide is specifically sharp on factual judgement. On the five non-factual criteria, generator quality and evaluator reliability move together much more closely, with anchored cosine alignment ranging from 0.84 to 0.91. Models that produce attractive or structurally sophisticated work generally have more stable standards for recognizing it.

The paper is therefore not claiming that making and judging are universally unrelated. Its more precise result is that factual error detection can dissociate sharply from factual production, while aesthetic and structural production align more closely with stable evaluation.

That is a more useful conclusion than the usual “models are inconsistent” shrug. It tells system designers what must be measured separately.

The vendor-removal test checks one obvious failure mode

The SVD estimator could mistake a large same-vendor bloc for competence. The paper therefore recomputes the factual generator ordering after removing each provider’s evaluators.

The full-panel ordering remains highly stable:

• Removing Anthropic judges: Spearman correlation 0.96
• Removing Google judges: 0.98
• Removing OpenAI judges: 0.99

A panel containing only Google and OpenAI evaluators still places the Claude generators at the top, while the GPT-4o family remains near the bottom.

This is a robustness test, not a second thesis. It weakens the narrow explanation that Claude models rank highly only because other Claude models favour them. It does not eliminate the broader possibility of shared misconceptions across vendors, especially when commercial foundation models train on overlapping public corpora and inherited conventions.

Cross-vendor is useful. It is not the same thing as independent in the experimental-science sense.

GPQA agreement supports relevance, not oraclehood

The paper compares its key-free factual scores with GPQA Diamond, using the same twelve models under the same temperature-zero, no-tools, no-reasoning protocol.

The correlations are:

By contrast, factual judging and factual generation correlate with each other at Pearson 0.59.

The GPQA comparison is the paper’s main external validation evidence. It shows that the internally recovered factual signal tracks a well-known expert-level science benchmark remarkably closely in this sample. The reported Pearson correlation remains 0.92 even when the calibrated anchor point is excluded.

It does not establish that either benchmark is an oracle. GPQA has its own finite question set, answer-key assumptions, and measurement boundaries. The comparison involves only twelve models, and the same restricted inference configuration is used for both tests.

The appropriate conclusion is convergent validity: two very different instruments produce similar cross-model factual rankings. That makes it less likely that the Metanym score is merely spectral theatre. It does not prove that the council has discovered truth in the philosophical, universal, or quarterly-earnings-call sense.

The experiments play different roles

The paper contains several numerical exercises, and they should not all be treated as equivalent evidence.

This distinction matters because the paper sometimes moves quickly from a demonstrated bootstrap to a proposed self-improving institution. The former has data. The latter has architecture.

What the paper directly shows

The experiment directly supports four practical findings.

First, models can generate a sufficiently rich body of fresh, falsifiable material to serve as one another’s evaluation substrate. The worked portfolios are imperfect, but that is the point: errors create opportunities to measure detection.

Second, evaluator competence cannot be inferred safely from generator performance. The observed factual maker–judge gap is too large to dismiss as minor ranking noise.

Third, consistency and factual informativeness are separate properties. A stable judge can still be wrong or uninformative, which means evaluator qualification needs at least two tests.

Fourth, a key-free factual signal derived from peer ratings can align closely with an external factual benchmark, at least for this panel and task.

Those findings are already useful without accepting the paper’s larger vision of a self-governing model council.

Cognaptus inference: qualify the reviewer before automating review

The most immediate business application is not replacing every benchmark with metanym poetry. It is adopting the paper’s evaluator-governance logic.

Organizations increasingly use models to:

• score customer-service conversations;
• review generated reports;
• check policy compliance;
• rank search or retrieval outputs;
• evaluate agent trajectories;
• supervise other models;
• decide whether an answer should be escalated to a human.

In many deployments, the evaluator is selected because it is the largest available model, performs well on generation benchmarks, or sounds persuasive when explaining its score. The paper suggests a stricter process.

1. Generate fresh, organization-specific cases

Create tasks from current workflows rather than relying entirely on public benchmarks. Let candidate systems produce analyses, classifications, summaries, plans, or decisions with enough complexity that genuine errors occur.

2. Cross-score the outputs

Have several candidate evaluators score the same atomic units. Preserve graded judgements rather than reducing everything prematurely to pass or fail.

3. Remove evaluator leniency

Separate whether a model uses high or low numbers from whether it detects the same relative problems as other informative judges.

4. Stress the scoring interface

Change calibration anchors, rubric ordering, irrelevant formatting, or other non-semantic conditions. A production evaluator should not reverse its ranking because “acceptable” was labelled 6 rather than 7.

5. Separate reviewer qualification from author qualification

A model may be excellent at drafting a financial narrative and poor at identifying unsupported assumptions in someone else’s narrative. Another may be a mediocre writer but a valuable verifier. Route work accordingly.

This is a governance pattern rather than a single benchmark. The return on investment comes from reducing silent evaluator failure—the comfortable situation in which automated quality assurance produces stable numbers, attractive dashboards, and no useful error signal whatsoever.

A standing council could reduce repeated labelling costs

The paper’s longer-term proposal is a fixed-size council whose seats are contestable.

A new model would:

1. Produce a portfolio.
2. Be scored against the frozen anchor.
3. Evaluate incumbent portfolios.
4. Demonstrate both generation quality and evaluator reliability.
5. Replace the lowest council member only when its advantage is statistically resolvable.
6. Trigger a recalculation of the council’s ratings and a new versioned record.

For an enterprise, an analogous system could maintain a panel of approved evaluator models. New releases would not be admitted because their vendor announced an upgrade. They would have to outperform an incumbent and prove they can judge reliably.

This could reduce the need to rebuild a fully human-labelled benchmark for every model cycle. Human experts would still be needed to design the operating boundaries, audit high-risk failures, and periodically check the panel against reality. But they would not necessarily need to label every routine comparison forever.

That is the plausible economic pathway: use expensive external validation to bootstrap and audit a cheaper recurring peer-evaluation process.

The paper demonstrates the initial council selection. It does not demonstrate an actual promotion round, anchor replacement, rule revision, or long-term resistance to institutional drift.

The benchmark is self-contained, not self-authenticating

The phrase “self-contained” can invite a stronger interpretation than the evidence supports.

The benchmark does not require a standing gold key to calculate its scores. Every scored object is generated within the participating system, and the panel’s own structure supplies the estimators.

But the system cannot certify the assumptions that make those estimators meaningful.

It cannot determine internally whether:

• all participating models share the same factual misconception;
• the task rewards a narrow style of analogy construction;
• the chosen anchor is strategically distorting later scores;
• the panel roster excludes an important model family or viewpoint;
• the reliability threshold is appropriate for a high-risk use case;
• deterministic direct responses predict tool-using agent performance;
• the council’s future procedural changes preserve construct validity.

The GPQA comparison is important precisely because internal consistency is not enough. A closed system needs occasional contact with an independent instrument, or it risks becoming a very well-calibrated club.

Boundaries for operational use

Four limitations materially constrain business interpretation.

One prompt, one roster, one experimental regime

The reported bootstrap intervals capture variation across generated items, not robustness across prompts, alternative model rosters, different council sizes, reasoning settings, tool access, or stochastic sampling.

Temperature zero improves reproducibility. It also tests a narrower behavior than many deployed systems, where models reason, retrieve documents, call tools, and operate over multiple turns.

Ratings are relative to a constructed anchor

Claude Opus 4.5 is pinned at 7. The scale therefore supports comparisons against that reference, not an absolute claim that a total score of 6.69 represents a universal quantity of intelligence.

Anchor replacement and historical recalibration are proposed, but not empirically tested.

Agreement can reveal shared bias as easily as truth

The vendor-removal analysis is reassuring, yet all twelve models may still share data sources, evaluation conventions, or factual blind spots. The SVD method needs diversity of error, not merely diversity of logos.

In high-stakes domains, the panel should be checked against domain experts, trusted datasets, adversarial cases, or independently constructed evaluation instruments.

The governance loop remains a specification

The canonical experiment creates council version 0. It does not show that the council can promote a challenger, replace its anchor, revise its rules, or improve itself without accumulating bias.

The paper describes a constitutional design. It has not yet governed a succession.

The real contribution is separating competence from confidence

The Metanym Game is an ambitious paper, occasionally ambitious enough to peer over the edge of its evidence. Its strongest contribution is nevertheless concrete.

It constructs a fresh production task that exposes cross-domain structural reasoning to sentence-level factual scrutiny. It turns the resulting peer scores into two different evaluator tests: shared factual signal for objective claims and calibration invariance for subjective criteria. It then demonstrates that the models best at producing material are not necessarily those best qualified to judge its truth.

That separation should travel well beyond this particular word game.

Automated evaluation systems often assume that a capable model is a capable reviewer, that a consistent score is an informative score, or that a panel average becomes trustworthy through arithmetic. The paper provides mechanisms for challenging all three assumptions.

The business lesson is therefore not that models can finally be left alone to determine reality. It is that model evaluators should be treated as instruments: calibrated, stress-tested, compared with independent measures, and removed from service when they provide confidence without information.

Peer review is not automatically reliable because the peers are artificial. At least now there is a proposal for reviewing the reviewers.

Cognaptus: Automate the Present, Incubate the Future.