When AUROC Agrees and the Decision Still Changes

TL;DR for operators

CRS-Bench¹ tests a decision that clean-test AUROC does not fully answer: which pretrained medical image encoder deserves the next round of engineering, labeling, adaptation, and validation investment. Across 15 encoder families, AUROC and the benchmark’s broader reliability score are positively associated, yet 21 of 105 pairwise model choices reverse. The disagreement is not evidence that AUROC is useless. It shows that discrimination can preserve the broad ordering while missing differences in calibration, label efficiency, and robustness that change actual selection decisions.

For medical-imaging teams, the operational change is to treat encoder evaluation as a multi-axis gate rather than a one-number leaderboard. CRS-Bench also solves a benchmark-management problem by fixing its reference anchors, so adding a new candidate does not retroactively change previously reported scores. But the benchmark does not identify a universal clinical winner: its resampling supports a leading tier of PanDerm, MedSigLIP, and MedGemma, while post-LoRA rankings only partially preserve the frozen ordering.

Similar AUROC does not imply the same selection decision

Consider a model-selection meeting with several encoders showing strong validation AUROC. If discrimination is the only comparison criterion, the ordering appears straightforward: advance the model with the best score unless there is already evidence of a distribution-shift problem.

CRS-Bench shows where that rule becomes incomplete. Across its 15 encoders, clean AUROC and its multi-axis reliability ranking have Kendall $\tau=0.600$ and Spearman $\rho=0.811$. Those values indicate substantial agreement. Yet 21 of the 105 possible pairwise choices reverse, with a mean absolute rank displacement of 1.87.

That distinction matters operationally. A benchmark does not need to produce a completely different ranking to change resource allocation. It only needs to reverse the comparison between two models a team is seriously considering.

The top results make the point concrete. MedGemma has the highest aggregate AUROC at 0.909, followed by MedSigLIP at 0.908. PanDerm is lower at 0.871. Once calibration, label efficiency, and robustness enter the comparison, aggregate CRS instead places PanDerm at 0.703, MedSigLIP at 0.701, and MedGemma at 0.645.

The paper therefore challenges a specific inference: the encoder with the highest clean-test discrimination is not necessarily the encoder with the strongest reliability profile under the benchmark’s declared conditions.

CRS turns four reliability properties into a selection gate

CRS-Bench evaluates frozen encoder features under a matched downstream setup: standardized representations followed primarily by L2-regularized logistic regression. The same evaluation framework is applied across dermatology, ophthalmology, and radiology, with CheXpert-to-MIMIC-CXR transfer providing one observed institutional shift.

The benchmark tracks four dimensions.

Discrimination is measured with macro-AUROC. Calibration is measured primarily through expected calibration error. Label efficiency asks how much of an encoder’s own full-label performance it retains when labels fall to 25%, 10%, 5%, and 1%. Robustness combines average and worst corruption losses and, for CheXpert, the observed institutional-shift loss.

These axes are not interchangeable. One of the paper’s calibration analyses finds that post-hoc calibration reduces ECE by 54–77% across six primary tests while leaving AUROC essentially unchanged. Separately, corruption-induced discrimination loss and calibration loss correlate only moderately across 43 cells ($r=0.63$). A model can therefore retain much of its ranking ability while becoming less trustworthy in the probabilities it emits.

The Clinical Reliability Score aggregates the four-axis profile in three ways. It rewards models that Pareto-dominate more reference peers, rewards proximity to a balanced ideal profile, and penalizes a weak worst-performing axis. The default component weights are explicit design choices rather than learned clinical utilities.

That makes CRS a selection summary, not a clinical outcome measure.

Fixed references prevent benchmark history from moving

Multi-objective scoring creates a second problem that matters once benchmarks become operational infrastructure. If scores are normalized against the current candidate cohort, adding a strong new model can change old models’ reported values even though nothing about those models has changed.

CRS-Bench instead normalizes each reliability axis against fixed lower and upper anchors. Its aggregate score is also evaluated relative to a declared reference panel.

The consequence is deliberate: in the paper’s fixed-reference insertion tests, adding new candidates produces exactly zero displacement of previously published scores. A cohort-renormalized alternative would move them.

For organizations maintaining internal model registries, this is more than statistical housekeeping. Historical evaluations can remain comparable as new encoders arrive. A newly screened candidate can be positioned against an existing standard without silently rewriting the score history used in earlier governance decisions.

The boundary is equally important: stable historical scores do not guarantee stable rankings on a new dataset. Fixed-reference normalization solves score-history comparability, not external validity.

The result is a leading tier, not one definitive winner

The aggregate CRS table puts PanDerm narrowly ahead of MedSigLIP, 0.703 versus 0.701. Reading those point estimates as a resolved first and second place would overstate the result.

The benchmark’s paired-seed bootstrap gives PanDerm a rank-1 probability of 0.664 and MedSigLIP 0.336, while their paired CRS difference interval overlaps zero. PanDerm, MedSigLIP, and MedGemma each have $P(\text{top-3})=1.000$.

Other stress tests reinforce the tier more than the exact internal ordering. Across 1,000 perturbations of the scalarization weights, mean Kendall agreement remains around 0.985–0.986 and the Top-3 never changes. Reference-panel perturbations also preserve the leading set well.

These are sensitivity and uncertainty tests, not evidence that the chosen weights are clinically optimal. Their narrower implication is that the benchmark’s headline shortlist is not an artifact of one precise coefficient choice or one seed realization.

For a screening decision, that distinction changes the action. The evidence justifies advancing a stable group of candidates for deeper evaluation more strongly than it justifies declaring a single universally superior encoder.

Frozen evaluation should end the screening stage, not the decision process

The most consequential boundary appears when the encoders are adapted.

In a matched LoRA arm covering 135 encoder–dataset–seed runs, discrimination improves, and the same Top-3 remains. But the full adapted-versus-frozen CRS ordering falls to Kendall $\tau=0.562$, with substantial reorganization through the middle ranks.

Frozen probing therefore measures something useful but narrower than post-adaptation performance. It compares representation quality under controlled downstream capacity. Once trainable adaptation is introduced, that ordering becomes only partially transferable.

The paper reaches a similar limit on unseen settings. Leave-one-dataset-out analyses and the MIMIC-CXR evaluation do not support treating CRS as a universal predictor of new-task AUROC. Only one observed institutional shift is included, the controlled corruptions represent acquisition and image-quality degradation rather than population or pathology change, and prospective deployment, subgroup analysis, operating-point selection, and human–AI interaction are outside the protocol.

The operational change is to make the leaderboard a gate

Cognaptus interprets CRS-Bench as a governance pattern for screening, not as a replacement for deployment validation.

For a team selecting a medical imaging foundation model, the first change is to preserve the four reliability dimensions rather than collapsing the process immediately into clean AUROC. A candidate that advances should clear explicit expectations for discrimination, confidence calibration, scarce-label retention, and degradation under relevant stress.

The second change is to treat the aggregate score as a shortlist mechanism. If several candidates occupy a statistically stable leading tier, downstream decisions can incorporate adaptation cost, task-specific validation, operating thresholds, subgroup performance, and deployment constraints rather than forcing the benchmark to resolve a distinction it cannot support.

The third change is benchmark lifecycle discipline. Fixed references allow future models to be inserted without changing historical scores, while repeated saturation of those anchors would signal that the reference standard itself needs revision.

CRS-Bench does not eliminate judgment from medical-model selection. It makes clearer which judgments belong in the screening benchmark and which still require evidence from the intended deployment setting.

Cognaptus: Automate the Present, Incubate the Future.