When AI Becomes the Reviewer: Pairwise Judgment at Scale

Opening — Why this matters now

Large scientific user facilities run on scarcity. Beam time, telescope hours, clean-room slots—there are never enough to go around. Every cycle, hundreds of proposals compete for a fixed, immovable resource. The uncomfortable truth is that proposal selection is not about identifying absolute excellence; it is about ranking relative merit under pressure, time constraints, and human fatigue.

For decades, this ranking problem has been handled by individual scoring: experts read a handful of proposals, assign scores, and hope the aggregation resembles something coherent. Hope, as it turns out, is doing a lot of work.

Background — Context and prior art

The paper examines proposal selection at large user facilities, using the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory as a real-world testbed. The traditional method—individual scoring (IS)—has three structural weaknesses:

1. Weak inter-proposal comparability: reviewers see only small subsets of proposals.
2. Human inconsistency: mood, fatigue, and order effects quietly distort scores.
3. Limited analytical depth: humans struggle to assess similarity or redundancy across hundreds of documents.

There is a known alternative: pairwise preference (PP). Instead of asking “How good is this proposal?”, reviewers ask a simpler, cognitively cleaner question: “Which of these two is better?”

In theory, PP produces more consistent rankings. In practice, it explodes into an (O(N^2)) workload and is therefore unusable—unless the reviewers are machines.

Analysis — What the paper does

The authors introduce an LLM-driven pairwise preference system. Every proposal pair within a cycle is compared by an LLM acting under a carefully constrained reviewer prompt. The model must summarize, compare, reason, and only then declare a winner (or tie).

The resulting win–loss matrix is converted into a global ranking using the Bradley–Terry model, a probabilistic framework that estimates latent “strength” scores from head-to-head outcomes:

$$ P(i > j) = \frac{s_i}{s_i + s_j} $$

This turns thousands of micro-judgments into a single, internally consistent ordering.

The workflow also exploits LLM embeddings, mapping each proposal into a high-dimensional semantic space. This enables similarity analysis—detecting resubmissions, overlaps, and near-duplicates—at a scale no human panel could reasonably manage.

Findings — Results with visualization

1. Ranking agreement

Across 20 historical run cycles and three representative beamlines, LLM rankings correlate positively with human rankings. Spearman correlations typically fall between 0.2 and 0.8, improving to ≥ 0.5 once extreme outliers are excluded.

This is not perfect alignment—and that is precisely the point. The divergences highlight proposals where human judgment may be noisy or inconsistent, offering review committees something they rarely get: diagnostic insight.

2. Publication outcomes

When evaluated against downstream publication output, LLM rankings perform no worse than human rankings in identifying proposals with high publication potential. Mean performance metrics are statistically indistinguishable across beamlines.

In other words, replacing human scoring with LLM-assisted ranking does not degrade scientific productivity.

3. Cost structure

Here the numbers stop being subtle.

For typical proposal pools (30–70 submissions), the LLM approach is hundreds of times cheaper. Even with quadratic scaling, the crossover point where LLMs become more expensive than humans is purely theoretical.

4. Similarity analysis

Embedding-based similarity maps reveal resubmissions and topic overlaps instantly. What would require weeks of human cross-reading becomes a linear-time operation followed by fast vector math.

This is not a marginal improvement; it is a qualitative shift in what review committees can realistically know.

Implications — Next steps and significance

This paper quietly dismantles a long-standing assumption: that human review is the gold standard by default. When the task is relative ranking under scale, LLM-assisted pairwise judgment is not merely cheaper—it is structurally better.

For funding agencies and large facilities, the implications are immediate:

• Human reviewers can focus on outliers and edge cases, not routine comparisons.
• Review processes gain auditability and consistency.
• Similarity detection reduces redundancy and gaming.

The framework is also portable. Any system forced to rank many text-heavy submissions—grants, fellowships, conference papers—faces the same underlying math.

Conclusion — Automating judgment without pretending it’s wisdom

The paper does not argue that LLMs are better scientists. It argues something more precise: that machines are exceptionally good at doing the kind of repetitive, pairwise reasoning humans are uniquely bad at sustaining.

Used correctly, LLMs do not replace reviewers; they reshape the review surface, turning an opaque, exhausting process into one that is scalable, inspectable, and—perhaps for the first time—honest about its limitations.

Cognaptus: Automate the Present, Incubate the Future.