The Leaderboard Can Pick the Wrong Model

TL;DR for operators

A visible development metric can be a poor basis for choosing an AI system when it measures only part of the outcome that ultimately matters. In this CBCT report-generation study, the variant tuned for lexical overlap produced the highest BLEU-4 among the deployable systems but scored only 0.2909 on the reconstructed composite objective. An objective-aware decoder with geometry gating reached 0.4122 despite lower BLEU-4.

The deeper diagnosis is more consequential than the score difference. A 29M-parameter image encoder produced prevalence-weighted out-of-fold AUC of 0.486 across evaluable report statements, while simple acquisition geometry predicted several field-of-view statements strongly. Acquisition centre also predicted sentence choice better than the image-derived model, indicating that the benchmark rewarded institutional dictation style as well as anatomy.

For teams selecting generative systems, the practical sequence is: audit the actual objective, test cheap controls to identify what signal is genuinely recoverable, and add output constraints where local predictions do not guarantee global consistency. Only then does additional model capacity become an informed investment rather than a default response.

A better visible score can still select the worse system

A familiar model-selection problem appears whenever the development dashboard exposes only part of what ultimately determines success. Teams tune against what they can measure, compare systems on that number, and naturally prefer the variant that improves it.

That logic fails when the observable metric and the decision objective reward different behavior.

George and colleagues examine exactly this problem in cone beam CT report generation.¹ In their benchmark, the final score gives 80% weight to factual entailment and 20% to the average of BLEU-4 and METEOR:

During development, however, only the lexical component was visible.

The resulting model-selection error is measurable. The lexically tuned report reaches BLEU-4 of 0.1663, higher than the objective-aware geometry-gated system’s 0.1268. Yet its reconstructed composite score is 0.2909, compared with 0.4122 for the geometry-gated system. Surrogate entailment precision rises from 0.266 to 0.522.

Higher BLEU-4 therefore does not mean greater factual correctness in this comparison. The lexical metric rewards matching wording; the dominant benchmark component is intended to reward whether the generated claims are supported.

The model-capacity problem was partly a signal problem

Once lexical optimization is shown to be misaligned, the obvious response might be to improve the image model. The paper tests that route and gets an inconvenient result.

A 29M-parameter multi-planar image encoder achieves prevalence-weighted out-of-fold AUC of 0.486 over 985 evaluable statements. Its full report system also scores 0.3403 on the reconstructed composite objective, below the image-independent corpus-prior system at 0.3575.

The diagnostic controls help explain why.

Nine acquisition-header geometry features strongly predict certain field-of-view statements. Reported AUC reaches 0.945 for mandible coverage and 0.872 for condyle coverage. Across 37 statements with adequate support, header geometry averages AUC 0.627, compared with 0.542 for resampled intensity statistics.

A compact 122-feature global image descriptor performs better than the neural encoder as well, reaching prevalence-weighted discrimination of 0.593 while occupying roughly 52 kB rather than hundreds of megabytes of neural checkpoints.

These comparisons are best read as diagnostic controls, not as evidence that metadata can replace image interpretation. They identify which report content can be predicted reliably from simple acquisition information under this dataset. They do not establish pathology recognition from metadata.

The reports also contain a centre-specific wording signal

The second diagnostic is more uncomfortable for lexical evaluation.

Acquisition centre predicts usage of the 60 most frequent prototype sentences at mean AUC 0.718, compared with 0.663 for the image-derived predictor. On average, 72.2% of each sentence’s use is concentrated in the centre where it appears most often.

That makes institutional dictation convention part of the prediction target.

The paper gives a useful illustration: even perfect knowledge of absent teeth adds only 0.0041 BLEU-4. Much of the rewarded four-gram overlap lies in sentence framing rather than in the patient-specific identifiers that carry the anatomical information.

This also helps interpret the unseen-centre evaluation. On 50 cases from a centre absent from training, the delivered system achieves BLEU-4 of 0.0943 ± 0.0354 and METEOR of 0.3542 ± 0.0599. The weaker transfer of exact four-gram overlap is consistent with BLEU-4 being more sensitive to site-specific phrasing.

For evaluation and procurement teams, this is a reason to add site or subgroup diagnostics whenever generated language may encode institutional convention. A model can become better at reproducing local prose without becoming correspondingly better at representing the underlying facts.

Reconstructing the objective was not enough

The paper then encounters a second failure mode: an automatic objective can be better aligned and still produce unreasonable output when optimized without structural constraints.

Unconstrained composite optimization continues improving until 53 statements are emitted. That may increase the automatic score, but it creates an implausibly long report and can admit redundant or mutually incompatible assertions.

The delivered decoder therefore does not simply maximize the reconstructed score. It retains eight unconditional statements, five statements conditioned on demonstrably predictive acquisition geometry, and explicit constraints covering polarity, laterality, incompatible anatomical coverage, tooth status, and redundant assertion signatures. The resulting reports average 11.2 sentences.

This is where the system design becomes operationally relevant. Local statement probabilities do not guarantee that the assembled document is globally coherent. Constraint-based generation provides a separate control layer for properties that a decomposed scoring function may fail to enforce.

What changes for AI system selection

For a product, evaluation, or model-governance team, the affected decision is not whether this particular CBCT decoder should be copied. It is whether the next unit of engineering effort should go into a larger model or into the measurement and decision system surrounding it.

When the objective is only partly observable, three tests deserve to precede scaling.

First, reproduce the measurable objective closely enough to understand what system selection is actually rewarding. Second, compare the main model against inexpensive priors, metadata-only models, subgroup predictors, and simple feature baselines. Their value is diagnostic: they reveal whether the expensive model is extracting unique signal at all. Third, inspect whether independently rewarded outputs can combine into contradictions, duplication, or other invalid global states.

The paper demonstrates all three within one benchmark. That does not make them universal prescriptions, but it does make model capacity a later question rather than the first one.

The clinical boundary remains substantial

The most important limitation concerns the reconstructed composite score itself. RadFact, which contributes 80% of the official objective, was unavailable during development. The reported factual precision, recall, and composite improvements therefore depend on an offline entailment surrogate rather than the official judge.

The delivered system also derives most of its patient-specific adaptation from acquisition geometry, particularly field-of-view information. The experiments do not establish reliable recognition of low-prevalence pathology or tooth-specific abnormalities, nor do they show that CBCT anatomy is intrinsically difficult for learned image models. The authors restrict that conclusion to the available dataset, labels, and supervision.

Accordingly, the 0.4122 result is evidence about benchmark-oriented system design under a partially observed objective. It is not evidence that the system is a comprehensive diagnostic reporter.

Improve the decision system before scaling the model

The paper’s strongest contribution is methodological rather than architectural. It treats the scoring function, dataset signals, and decoder constraints as parts of the engineering problem instead of assuming that model improvement alone will resolve weak benchmark performance.

That shift changes where teams look first when progress stalls. A disappointing large model may indicate insufficient capacity. It may also indicate that the visible metric rewards the wrong behavior, the dataset contains stronger nuisance signals than task signals, or the decoder is permitted to assemble locally attractive predictions into globally invalid output.

Those possibilities can be tested cheaply. Finding them before another scaling cycle can change both system quality and where engineering resources are spent.

Cognaptus: Automate the Present, Incubate the Future.