Before You Ask the Judge, Read the Logits

TL;DR for operators

A scientific agent that generates 16 plausible hypotheses does not need every candidate to receive the same expensive treatment. The operational problem is deciding which hypotheses deserve simulation, experiment, or expert attention first.

Rajwal, Das, and Ghosal show that, on a benchmark built from 1,323 published papers, simply measuring how strongly open-weight models internally support each candidate can rank the published hypothesis substantially better than asking GPT-5 to compare all candidates explicitly.¹ Pooled first-place accuracy was 33.1% using normalized token likelihood and 32.8% using raw model logits, versus 16.6% for the tested zero-shot GPT-5 judge.

That does not establish raw logits as the superior evaluator. The best single configuration reached 53.1% Hit@1, but raw-logit performance varied dramatically by model and showed very low cross-model agreement. Normalized likelihood produced almost the same pooled accuracy with considerably greater consistency.

For operators building scientific-agent workflows, the defensible use is therefore a first-pass ranking layer: exploit inexpensive intrinsic confidence to reduce the candidate set, validate the scorer on the specific model being deployed, and reserve experimental or expert validation for the hypotheses that survive. Model confidence remains a prioritization signal, not scientific validation.

The expensive decision comes after generation

Generating candidate hypotheses is only the beginning of an automated research workflow. Once an agent has produced several plausible directions, someone—or something—must decide which one receives the next costly unit of attention.

The conventional solution is easy to understand: give the candidates to a strong model, ask it to compare them, and use its ranking as the decision. The paper tests whether that explicit act of judging is necessary.

Its benchmark contains 1,323 ResearchBench papers across 12 disciplines. Each task provides a scientific background, a research question, and 16 candidate hypotheses: one hypothesis from the published paper and 15 plausible distractors. The evaluation asks whether a scoring method can place the published hypothesis near the top.

For the seven open-weight models, candidates are not shown together. Each hypothesis is evaluated independently under the same background and research question. The model is effectively asked, through its token predictions rather than through an instruction, how expected that hypothesis is in context.

That distinction matters operationally. Explicit judging adds another task: understand the ranking instruction, jointly manage 16 candidates, compare them, and return the requested structure. Intrinsic scoring bypasses that interface and extracts a signal the model already produces while processing the candidate.

Internal confidence beats the tested explicit judge

The main comparison is unusually clear.

Hit@1 is simply the fraction of papers for which the method ranks the published hypothesis first. On that measure, both intrinsic approaches roughly double the prompted GPT-5 baseline.

The first intrinsic measure is negative log-likelihood, or NLL. In practical terms, a lower NLL means that the candidate’s actual words were less surprising to the model given the supplied scientific context. The paper averages that signal over the tokens belonging only to the hypothesis.

The second removes the softmax normalization step. Raw target-logit energy uses the model’s pre-softmax preference for each observed token: a larger target logit becomes a lower, better energy score.

The benchmark therefore supplies evidence for a narrower but operationally meaningful proposition: a model’s internal predictive state can carry ranking information that is degraded when the evaluation is reformulated as an explicit listwise judgment.

It does not identify why GPT-5 underperforms. Only one proprietary model and one zero-shot prompting strategy are tested. The study therefore cannot establish that intrinsic scoring dominates every possible LLM-as-judge design.

The 53% result is not the result to deploy

The most eye-catching number belongs to Llama 3.2 1B using raw-logit scoring: 53.1% Hit@1 and 77.4% Hit@5. Its pairwise result is also strong—the published hypothesis outranks an individual distractor 82.2% of the time.

But that configuration is the maximum observed after testing 14 model-by-scorer combinations. The authors explicitly treat it as a post-hoc upper bound rather than an unbiased estimate of expected performance.

More importantly, raw scoring behaves very differently across models.

Llama 3.2 3B reaches 79.1% pairwise accuracy with Raw, and Mistral 7B reaches 74.7%. Gemma 2 2B falls to 45.8%, below the 50% level expected from random pairwise ordering. Its Hit@1 drops from 35.2% under NLL to 13.5% under Raw.

NLL is far less volatile: pairwise performance across the seven open-weight models stays within roughly 68.8% to 74.3%.

This changes the engineering interpretation. Raw logits are not a portable scoring rule that can be attached to any model exposing logits. Their usefulness depends on the model’s output scale, calibration, and transformations applied to its logits. The source package specifically notes that model-specific operations such as Gemma 2’s final-logit soft-capping can compress the information Raw scoring attempts to exploit.

For a production system, the unit that must be validated is therefore the model-scorer pairing, not the scorer in isolation.

Similar average accuracy hides a robustness difference

The pooled numbers make NLL and Raw look nearly interchangeable: 33.15% versus 32.76% Hit@1, with MRR of 0.474 versus 0.462. Variation between models is much larger than the average difference between scoring rules.

The cross-model agreement analysis, used here as a robustness check rather than a second performance headline, exposes a different property.

Under NLL, all seven open-weight models independently rank the published hypothesis first on 167 of the 1,323 papers—12.6% of the benchmark. Under Raw, unanimous agreement occurs on only three papers, or 0.2%.

That does not automatically make NLL the better evaluator. Agreement can arise from shared biases as well as shared signal. But for an operator who wants a ranking system to survive a model upgrade or provide comparable governance behavior across deployments, this difference is consequential.

A Raw-based filter that works exceptionally well with one model may change substantially when the underlying model is replaced. That creates a requirement for calibration, regression testing, and preferably comparison against an independently validated scorer before raw logits are allowed to influence expensive downstream decisions.

Use intrinsic confidence to allocate validation, not replace it

What the paper directly shows: within ResearchBench, intrinsic likelihood and logit signals from the tested open-weight models rank the published hypothesis better in aggregate than the tested zero-shot GPT-5 listwise judge.

Cognaptus inference: scientific-agent platforms could exploit this signal as a low-cost triage mechanism. If an agent generates many candidate hypotheses, local intrinsic scoring could reduce the set passed to proprietary judges, simulations, laboratory workflows, or domain experts. The affected decision is resource allocation: which candidate receives the next expensive validation step.

That inference has a firm boundary.

ResearchBench does not contain genuinely unseen hypotheses awaiting scientific validation. Its gold hypotheses come from already-published papers, and the distractors were curated after publication. Better ranking may therefore reflect scientific compatibility, but it may also capture fluency, familiarity, writing quality, or other textual cues. The strongest Llama models have December 2023 knowledge cutoffs while the benchmark papers were published in 2024 or later, which weakens a simple memorization account without eliminating familiarity or leakage concerns.

The benchmark also does not test whether the top-ranked hypothesis is true, novel, experimentally reproducible, or worth pursuing.

The useful architecture is consequently layered: generate candidates, use a validated intrinsic scorer to prioritize them, and let evidence-producing processes—experiments, simulations, retrieval, or expert review—decide whether the surviving hypothesis deserves belief.

The paper makes internal model confidence more interesting as an operational signal. It does not turn confidence into scientific evidence.

Cognaptus: Automate the Present, Incubate the Future.