TL;DR for operators

A medical-image classifier can retain excellent benchmark accuracy even when its evaluation data contain serious integrity defects. In one public brain-tumor MRI benchmark, 28.75% of test images had near-duplicates in training. Removing them changed balanced accuracy by only -0.009 on average across nine architectures, meaning the cleaned scores were slightly higher.

The audit also uncovered substantial patient overlap and found that file metadata alone could predict tumor presence with 0.959 balanced accuracy. These results demonstrate contamination and non-anatomical label predictability, not that the neural networks necessarily ignore tumor anatomy.

For medical-AI developers, dataset buyers, and model evaluators, predictive performance and evaluation integrity should be separate acceptance criteria. Before approving a dataset or supporting a performance claim, check repeated images, patient independence, acquisition provenance, and whether information unrelated to anatomy predicts the labels. An unchanged score after cleaning establishes none of those conditions by itself.

A contaminated test set produced almost the same accuracy after cleaning

Suppose a medical-AI team discovers that part of its test set contains images closely resembling those used in training. It removes those images, evaluates the same trained models again, and finds that accuracy barely changes. The result appears reassuring: perhaps the repeated images were not materially affecting performance.

That interpretation is precisely what a new benchmark audit complicates.

In Scores That Hold, Benchmarks That Leak, Bhanu Prakash Vangala and colleagues examine widely redistributed brain-tumor MRI datasets and ask whether benchmark integrity can be established independently of the accuracy reported by trained classifiers.1

Their principal experiment concerns Nickparvar, a four-class MRI collection with an official partition of 5,600 training images and 1,600 test images.

Using normalized cross-correlation to identify repeated scans after resizing or re-encoding, the researchers find 460 test images with near-duplicates in training: 28.75% of the official test set (Section VI-A, Table I).

They then remove those 460 images and reevaluate nine trained neural-network architectures without retraining them.

The mean difference between official and duplicate-filtered balanced accuracy is -0.009 (Section VI-D, Table VI). Balanced accuracy averages recall across classes, reducing the influence of class imbalance. The negative difference means accuracy is marginally higher after removing duplicates.

ConvNeXt-T, for example, achieves 0.950 balanced accuracy under both evaluation conditions. VGG-16 moves from 0.945 to 0.946, and ViT-B/16 moves from 0.948 to 0.949.

The experiment measures deduplication sensitivity: how a fixed model’s measured accuracy changes when known repeated test images are excluded.

The small change is an empirical result about that intervention. It is not a finding that the original split was independent, or that other forms of dependence have disappeared.

The distinction matters because an evaluator can successfully remove a documented problem without producing a trustworthy test set.

Three contamination layers require different integrity checks

The authors organize benchmark contamination into three layers. Each concerns a different way training and evaluation data can fail to be independent.

Contamination layer What goes wrong Evidence from the audit
Image duplication The same underlying scan appears in training and testing, potentially after re-encoding or resizing. 28.75% near-duplicate leakage in Nickparvar; 65.74% in SARTAJ
Patient overlap Different MRI slices from the same patient appear on both sides of the partition. 192 of 201 traceable Nickparvar test images share a patient with training
Acquisition-source confounding Technical properties associated with the source of an image predict its diagnostic label. Anatomy-free file metadata achieves 0.959 binary balanced accuracy

The first layer is directly measurable through image comparison. The study converts images to normalized 64×64 grayscale representations and uses a correlation threshold of 0.98 for its main near-duplicate criterion.

A negative-control test on pediatric chest radiographs finds no duplicate matches under this criterion. By contrast, a conventional difference-hash configuration incorrectly flags 93% of that control’s test images (Section II-A and Appendix C). This robustness check illustrates why medical-image similarity thresholds need validation rather than assuming that ordinary perceptual hashes distinguish identical scans reliably.

The second layer requires patient identity. An MRI collection can contain multiple slices from one individual, and those slices need not be visually identical. The original Cheng figshare collection has 3,064 slices from only 233 patients.

By tracing redistributed images back to figshare patient identifiers, the authors establish that 192 of 201 traceable Nickparvar test images share patients with the training partition (Section VI-B, Table III).

That is 95.52% of the traceable subset, not 95.52% of the entire test set. Only 12.56% of Nickparvar test images could be linked to those original patient identifiers. Patient overlap across the remaining images remains unmeasured.

Dataset names provide no stronger guarantee of independence. Across the complete corpora, 2,705 of 3,264 SARTAJ images—82.87%—have near-duplicates in Nickparvar (Section VI-A, Table II).

Consequently, evaluating a model on a separately named public collection may still involve substantial overlap with its training source. External validation requires provenance checks, not merely a different dataset label.

File metadata can predict tumors without examining anatomy

The third layer raises a different question: can the dataset’s labels be predicted from information unrelated to the medical features the classifier is supposed to recognize?

Nickparvar combines images assembled from different public sources. If those sources have different diagnostic-label compositions, technical properties associated with image origin can become predictive.

To test that possibility, the authors train a gradient-boosted classifier using file properties such as image dimensions, aspect ratio, file size, bytes per pixel, image mode, and JPEG quantization tables.

No tumor anatomy is provided.

On the binary tumor-versus-no-tumor task, this metadata-only classifier achieves 0.959 balanced accuracy, compared with a chance baseline of 0.500. On the four-class task, it achieves 0.652 against a chance baseline of 0.250 (Section VI-C, Table V).

Adding global intensity summaries increases the respective scores to 0.966 and 0.830, although that extension uses pixel-derived information.

The binary metadata-only score is remarkably close to the 0.965 achieved by ResNet-18 and 0.967 achieved by ResNet-50 on the corresponding benchmark evaluation (Section VI-E, Table VIII).

This comparison identifies an alternative predictive signal, not the precise mechanism behind the neural networks’ decisions. The evaluated image networks receive decoded pixels rather than file headers. They could nevertheless encounter acquisition-related patterns preserved in those pixels, including framing and intensity characteristics.

The metadata experiment establishes that the dataset permits highly accurate prediction without anatomical evidence. It does not establish that the neural networks exploit exactly those signals, or that they cannot recognize tumors.

A further negative-control result reinforces the need for that distinction: metadata classification on chest radiographs reaches 0.992 balanced accuracy under five-fold cross-validation within the training pool, but only 0.496 on the official test set. Source-related predictability may disappear when the evaluation partition changes.

For model selection, the implication is that a high image-classification score cannot by itself identify which information supports prediction.

External testing and attribution do not isolate the cause

The study also evaluates the trained architectures on a cleaned external SARTAJ subset after removing detected image overlap with the training corpus.

Balanced accuracy declines by approximately 0.030 on average, although the changes vary by architecture. ResNet-18, for example, improves from 0.862 on the official Nickparvar test split to 0.924 externally (Section VI-D, Table VI).

Mean expected calibration error moves from 0.044 in-domain to 0.048 externally. This limited change in calibration does not independently establish clinical validity, particularly because the external SARTAJ data have documented label-quality concerns and are not proven patient- or acquisition-independent.

The authors also examine whether predictions rely on image regions outside the estimated brain boundary. Mean attribution mass outside that region is similar for leaked and clean images: 29.1% and 28.1%, respectively (Section VI-F, Figures 8–9).

That comparison provides descriptive evidence concerning non-anatomical signals, but attribution magnitude alone cannot separate instance memorization, anatomical recognition, and acquisition shortcuts.

An attempted difference-in-differences-style memorization analysis encounters an additional problem. Two zero-shot controls disagree about whether the leaked subset is inherently easier: BiomedCLIP shows a leaked-minus-clean accuracy gap of +0.035, while CLIP shows -0.101 (Section VI-F, Table IX).

Because the controls disagree, the proposed comparison cannot reliably identify the contribution of memorization.

The unresolved mechanism is consequential. The research firmly documents contamination and demonstrates score stability, but it does not establish a causal decomposition of why the trained models retain their performance.

Model approval needs a data-integrity decision as well as an accuracy decision

The operational application is to separate two questions that conventional benchmark reporting frequently combines.

The first is whether a model performs adequately on the specified evaluation set. That requires task-appropriate metrics, calibration, and comparisons across relevant evaluation conditions.

The second is whether the evaluation set provides credible evidence for the intended use. This requires documented image-level duplication checks, patient-disjoint partitions where identifiable, acquisition-source analysis, and tests of non-anatomical label predictability.

For a dataset procurement decision, a vendor’s high accuracy or claim of external validation should therefore be accompanied by source manifests, patient-mapping coverage, duplicate-detection procedures, and the identities of the underlying image collections.

For a model-development decision, inexpensive metadata-only probes can expose potential confounding before further investment in architecture optimization. A high metadata-only score should trigger an investigation of source composition and evaluation design, rather than being treated as a measured estimate of neural-network shortcut dependence.

For a product-performance claim, the stronger requirement is evaluation on independently collected, patient-disjoint, acquisition-diverse data appropriate to the intended clinical setting.

These are Cognaptus operational inferences from the audit, not additional interventions experimentally validated by the authors.

What the audit does not establish

The numerical contamination rates belong to this family of public brain-tumor MRI datasets. They should not be extrapolated to medical imaging generally.

Patient-overlap estimates are restricted to recoverably linked images. The duplicate detector does not identify every distinct slice belonging to the same individual. The nine-architecture comparison uses a common fixed training schedule rather than architecture-specific optimization.

There is also an internal reporting inconsistency: the paper describes Nickparvar as containing 7,023 images but gives an official partition totaling 7,200. The contamination calculations cited here use the explicitly reported official partitions.

Most importantly, the study does not independently evaluate clinical performance on newly collected, patient-disjoint and acquisition-independent cohorts.

Its strongest conclusion concerns evaluation validity, not demonstrated clinical failure.

A stable score is not an integrity certificate

The counterintuitive finding is that removing a substantial quantity of leaked test images can leave reported accuracy essentially unchanged.

That observation makes post-cleaning performance a poor substitute for provenance auditing. Duplicate removal addresses one measurable form of dependence, while overlapping patients and source-associated labels may continue to influence evaluation.

For organizations deciding whether to buy a medical dataset, approve a model, or make a performance claim, the appropriate response is neither automatic acceptance of stable scores nor automatic rejection of the evaluated architectures.

It is to require credible evidence that the evaluation measures the intended medical task under the conditions in which the model is expected to operate.

Accuracy remains necessary. Benchmark integrity determines what that accuracy can legitimately mean.


Cognaptus: Automate the Present, Incubate the Future.


  1. Bhanu Prakash Vangala and Sowmya Guda and Latha Peddi and Navya Vangala (2026). Scores That Hold, Benchmarks That Leak: Measuring Dataset Contamination in Public Brain-Tumor MRI Classification. arXiv:2610.00421. https://arxiv.org/abs/2610.00421 ↩︎