Noise Rewrote the CT Leaderboard

TL;DR for operators

A team using a clean CT leaderboard must decide which high-scoring models deserve costly validation. That ranking may be useful for screening, but it may not survive the operating conditions the model will actually encounter.

When the same 200 breast CT cases were exposed to mild, previously unseen Poisson noise, the clean and noisy rankings became essentially uncorrelated, with Spearman’s $\rho$ of about 0.04. The clean-data champion no longer reduced reconstruction error relative to the condition-matched baseline, giving it zero calibrated headroom, while a method ranked lower on clean inputs became the noisy-condition leader.

Retraining the models on matching noise restored much of the original ordering and returned the former winner to rank one. The collapse therefore reflected substantial distribution mismatch: the clean-trained models were being evaluated on a realistic input change they had not encountered.

For imaging R&D, procurement, and governance teams, clean benchmark rank should identify candidates rather than certify reliability. Condition-matched stress tests should come before scarce validation resources are committed, or a leaderboard-driven process may advance the wrong architecture and understate deployment risk.

A clean winner can become the wrong validation candidate

Most model-selection processes begin reasonably. A team compares methods under a common benchmark, advances the highest-scoring few, and spends scarce validation capacity on those candidates. The hidden assumption is not that the benchmark is perfect. It is that rank is stable enough to guide the next decision.

The study Agentic Autoresearch for CT Reconstruction tests that assumption under a tightly controlled change.¹ The researchers first evaluated reconstruction methods on noiseless, sparse-view breast CT. They then applied a Poisson photon-noise perturbation at an incident count of $I_0=100{,}000$ to the same 200 held-out cases, without retraining the clean-trained models.

The clean and noisy rankings were essentially unrelated: Spearman’s $\rho$ was about 0.04. Dual-domain-supervised reconstruction led the clean benchmark with calibrated headroom of 0.8948, then fell to 0 under noise. Learned primal-dual reconstruction moved in the opposite direction, rising from 0.7233 on clean data to 0.9349 under the noisy condition.

Calibrated headroom measures how much of the reconstruction-error gap relative to filtered back projection, or FBP, a method closes using the same condition-specific measurements. A score of zero means the method does not beat that baseline; values near one indicate that most of the error gap has been removed. Because each condition has its own matched FBP baseline, the metric supports ranking within a condition, not casual comparison across different datasets or acquisition regimes.

This is larger than a small performance drop. A clean leaderboard would have prioritized one architecture, while the changed condition favored another. Teams can therefore misallocate scanner studies, radiologist review, integration work, or regulatory documentation and understate the risk attached to the apparent winner.

Retraining shows mismatch drove much of the collapse

The first noise experiment is a robustness test: it asks whether clean-trained methods retain their relative standing when the input changes but the cases, architectures, and evaluation framework remain fixed. It does not by itself reveal why the order changed.

The matched-noise retraining experiment serves a different purpose. The researchers kept the architectures and clean-best hyperparameters fixed, changed training exposure to include the same noise condition, and re-evaluated the methods. This is a mechanism test for distribution mismatch rather than a fresh architecture search.

The result corrected a tempting misreading of the reversal. Dual-domain-supervised returned to rank one with headroom of 0.958. ITNet variants clustered near 0.938–0.939, learned primal-dual reached 0.936, and the clean-versus-retrained rank correlation rose to about 0.61.

The replacement belief should be more precise than “noise changes rankings.” A leader may be specialized to the benchmark distribution. When inputs move away from it, rank can change sharply; when training is realigned, much of the order may return.

A failed stress test should therefore trigger diagnosis rather than automatic disqualification. The next investment may be retraining, architecture replacement, or acquisition redesign, depending on whether the failure reflects missing exposure or a structural limitation.

Measurement checks can protect a model when inputs shift

The ranking reversal also offers a plausible architectural explanation. Some methods keep the CT forward operator active during inference. They repeatedly compare the current reconstruction with the scanner measurements and use the mismatch to update the image. Other methods mainly transform an initial reconstructed image using a learned image-domain prior.

Learned primal-dual reconstruction belongs to the first group. Its update alternates between measurement-domain and image-domain states, allowing the measured sinogram to constrain the reconstruction throughout inference. Under unseen noise, that repeated measurement check can limit how far the result drifts from what the scanner actually observed.

The paper does not establish a universal rule that more physics engagement always wins. The clean benchmark was led by a different method, and matched-noise training restored that method to first place. The narrower result is that in-loop measurement consistency was associated with graceful transfer in this specific shift, while a strongly clean-specialized mapping collapsed when its input moved outside the learned distribution.

The compact-solver experiments reinforce the same problem-specific view. Under noise-limited Mayo low-dose CT, the selected ultra-small architecture was a 969-parameter denoiser. Under sparse-view breast CT, where missing angular information was the dominant bottleneck, the selected solver used only 195 parameters but incorporated filtered data consistency and a primal-dual-style update. Parameter efficiency did not come from one portable miniature network. It came from matching the computational structure to the physical failure mode.

The agent scales the test bench, not the scientific judgment

Testing many methods across multiple conditions is usually expensive because each paper brings its own implementation assumptions, data interfaces, and tuning process. The study addresses that burden with a coding agent constrained by a fixed research loop: diagnose the previous result, change one item, state a hypothesis, run a short cluster job, read a frozen metric, and accept or discard the change.

Under a shared differentiable fan-beam projector, fixed per-iteration compute, validation-only selection, and immutable records, the agent implemented and tuned 26 established methods. The campaign ran about 1,100 iterations and consumed roughly 134 GPU-hours. Each run preserved its configuration, reconstruction, metric, figures, and version-control provenance.

This is the paper’s main system contribution: agentic automation can reduce the labor of reproduction, tuning, and repeated comparative testing when the evaluation contract is explicit. It also shows where autonomy stopped. Humans supervised the geometry pipeline, supplied the compact-recombination objective, suggested unexplored directions after the agent plateaued, decomposed tasks, and audited provenance. The agent did not originate a new reconstruction principle.

For imaging R&D, that division of labor is more credible than a fully autonomous research narrative. Machines can execute a repetitive experimental matrix and maintain records. Researchers still decide which shifts are clinically relevant, whether the metric matches intended use, and whether an implementation is faithful enough for procurement or safety decisions.

What changes for R&D, procurement, and governance

Imaging R&D teams should add shift matrices before selecting a winner. Noise level, view count, geometry, and expected acquisition differences should be planned evaluation conditions. The goal is to map where each candidate remains credible, not identify one universal champion.

Procurement teams should request evidence for the intended operating environment. A clean score can justify entry into a shortlist, but not robustness claims when scanner, protocol, dose, or patient population differs from the test distribution.

Model-governance teams should treat provenance as part of the comparison. A ranking is interpretable only when the projector, baseline, mask, split, compute budget, training exposure, and selection path are recoverable. The agentic workflow offers a pattern for maintaining that record at scale.

Engineering leaders can use compact reconstruction as a cost lever, but only after identifying the dominant inverse-problem bottleneck. The paper’s two compact optima were different because noise and angular incompleteness demanded different mechanisms. Parameter count alone is not an architecture strategy.

The boundary is narrower than clinical robustness

The within-benchmark evidence is strong because the study controlled the projector, metric, splits, test cases, condition-specific FBP baseline, and compute rules. The transfer experiment also isolates one clear change by applying the same noise process to the same breast test cases.

The practical boundary remains substantial. Only one photon level was tested. The breast benchmark used a synthetic internal split rather than the challenge’s withheld test set. Mayo evaluation covered five held-out patients, making its leading methods a statistical tier rather than a definitive winner. Agent-written implementations may differ from official versions, some compact results were seed-fragile, and two per-scene methods did not finish within budget.

Most importantly, the study evaluates reconstruction metrics, not diagnostic performance, workflow outcomes, or patient benefit. It does not show how the ranking behaves under scanner-vendor changes, motion, metal artifacts, protocol drift, pathology shifts, or combinations of factors. Cognaptus therefore infers a governance practice—stress-test candidates under expected operating conditions—from controlled evidence in two CT settings and one noise perturbation. The paper does not certify that this procedure is sufficient for clinical deployment.

A leaderboard should open validation, not close it

The central result is not that clean benchmarks are uninformative. They remain efficient tools for narrowing a field. The result is that their ordering can vanish under a small, realistic change that the models did not encounter during training.

Matched-noise retraining then adds the more constructive finding: collapse under shift may be recoverable. Teams should diagnose mismatch before declaring an architecture fundamentally weak, and they should design validation around the conditions that will govern real operation.

Agentic autoresearch makes this broader evaluation agenda more feasible. Its value is not an autonomous discovery claim. It is the ability to run more controlled comparisons, preserve their provenance, and expose where rankings depend on the benchmark rather than the deployment environment.

Cognaptus: Automate the Present, Incubate the Future.