A Full Distribution Is Not a Risk Certificate

TL;DR for operators

A team is deciding whether an AI risk dashboard should trigger a safer action, an operational alert, or a governance review. The system predicts a range of possible returns for each action rather than only an average, so its output may appear to provide stronger evidence about worst-case outcomes.

The audit shows why that appearance is not enough. For QR-DQN, 84% of the top-ranked risk claims were refuted in both Breakout and Seaquest, while none were confirmed after pooled false-discovery-rate control. The claims most likely to influence a decision were not merely noisy: their placement was close to a truth-blind control.

Treat learned return distributions as unverified model outputs, not risk certificates. Before using them for worst-outcome action selection, explanations, monitoring thresholds, procurement claims, or automated controls, independently test the specific states and action comparisons that would trigger intervention. Strong task performance does not establish trustworthy risk calibration.

Richer predictions can still support the wrong intervention

Suppose an operations team has two available actions. Their expected outcomes are similar, but one dashboard shows a worse lower tail for the first action. The team may choose the second action, trigger an alert, or require human review. That intervention assumes the displayed difference reflects the environment rather than the training process.

The difficulty is that a model can choose effective actions while assigning the wrong shape to their possible-return distributions. Hari Prasad’s Auditing the Risk Claims of Distributional Reinforcement Learning tests that distinction directly.¹ In the main QR-DQN results, 84% of the strongest claims were refuted in Breakout and Seaquest, and 66% were refuted in Asterix. Learned-to-ground-truth correlations were 0.02, −0.01, and −0.04. The model outputs were richer than a mean estimate, but the additional structure carried almost no state-level alignment with the audited risk signal.

Distributional reinforcement learning is still valuable as a control approach. The paper does not show that these agents cannot perform well. It shows that performance and literal risk calibration are separate properties. A head may help optimize the policy while remaining unreliable as an explanation of why one action is safer than another.

The audit targets the claims that would change a decision

The paper audits risk claims where a user would act: states showing the largest apparent difference between the two actions with the highest learned mean return. It removes the portion explained by a simple shift in average value. The resulting excess Wasserstein gap is

A positive gap indicates a shape difference that could support a genuine risk trade-off. It is zero when one return distribution first-order stochastically dominates the other, so the test isolates crossing or tail structure rather than a higher average alone.

The study collected 2,000 unique states per run and audited 49 from seven rank strata. Ground truth came from stored snapshots: force each candidate action, independently reseed each restored copy, then run 2,000 rollouts per action in MinAtar. Per-state permutation nulls tested confirmation; Monte Carlo bootstrap thresholds tested refutation; pooled results received false-discovery-rate control. The complete study covered 2,450 states and 9.6 million ground-truth rollouts.

This design changes the interpretation. The main result is not that the distributions contain some error. It is that the claims selected because they appear most decision-relevant are often the least defensible.

The strongest claims fail across three projection families

The main MinAtar study includes 33 trained agents across QR-DQN, C51, and IQN. These algorithms represent distributions differently, so replication across them tests whether the problem is confined to one projection scheme.

Lower-ranked claims had lower refutation rates, but that does not establish calibration. Small claims are harder to falsify under the paper’s asymmetric test. The top-strata result carries more weight because those are the claims most likely to be surfaced by a dashboard, selected for investigation, or used to justify a risk-sensitive action.

A truth-blind permutation control preserved claim magnitudes and ranks while randomizing their placement across states. Its top-strata refutation profile closely matched the learned heads. The likely interpretation is a winner’s-curse effect: ranking by the largest learned gap selects states where seed-specific approximation and optimization noise happened to create the most dramatic shape difference.

Checkpoint and sibling-seed probes support that mechanism. The artifact appeared by 500,000 training steps, persisted while score nearly doubled, and was largely idiosyncratic across seeds. These probes are not causal identification, but they fit a training-artifact account better than calibrated environment risk.

Positive controls show the audit is not merely too strict

A skeptical reader could argue that the statistical procedure is incapable of confirming distributional trade-offs. The known-answer controls address that concern.

In RISKYGRID, the audit confirmed 97% of pooled states. In the fainter-risk variant, it confirmed 96–100% of nonzero-risk states down to the designed detection floor. Learned-to-true correlations were roughly 0.89–0.92. These tests serve as positive controls: they show that the audit can recover real action-conditional risk when the environment contains it at detectable magnitude.

Near-zero confirmations in MinAtar do not prove that the games contain no risk under any policy. They show that the audited heads did not reliably identify the claimed trade-offs between their top two mean-ranked actions at the tested states.

CVaR guidance can help, fail, or invert

The practical test is not only whether a distribution looks wrong. It is whether acting on it selects a genuinely safer option.

Conditional Value at Risk, or CVaR, ranks actions by performance in the lower tail. At the most-flagged states with a bootstrap-supported safer action, QR-DQN’s CVaR advice selected the safer action 81% of the time in Breakout, compared with 44% for a mean-greedy chooser. In Seaquest, the same head selected the safer action only 34% of the time, versus 63% for the mean baseline. Asterix was close to chance.

C51 was beaten by the mean-greedy baseline in all three games. Its Seaquest head selected the safer action 28% of the time, compared with 56% for the baseline, and incurred substantially higher CVaR regret.

The same output can be useful in one environment and anti-predictive in another, with no validated indicator in the head showing which regime applies. Safety value cannot be inferred from architecture, confidence, or task score alone.

Common repairs reduce overclaiming more readily than they recover risk

The paper also tests several repair strategies. These experiments are best read as diagnostic extensions, not as a second main thesis.

CVaR-greedy training did not resolve the problem. Ensemble barycenters attenuated some refutation rates but left the strongest claims uncalibrated. Isotonic recalibration could pass on MinAtar by shrinking claim magnitudes toward zero and effectively asserting that no meaningful trade-offs existed.

A repair should preserve useful risk information while improving calibration and decision quality. Passing by suppressing nearly all claims may reduce false alarms, but it does not recover a trustworthy risk model. Repair evaluations should report both claim validity and retained decision usefulness.

Restart audits have two silent failure modes

The paper’s methodological contribution extends beyond distributional RL. Snapshot-restart evaluations can produce convincing but invalid conclusions in two ways.

First, cloning an environment can clone its random-number-generator state. Restored copies may then replay matching futures, creating artificial agreement or underestimating uncertainty. Each copy must be independently reseeded after restoration.

Second, empirical Wasserstein distance has a positive finite-sample noise floor. Two samples drawn from the same distribution will not have zero measured distance. A fixed threshold can therefore label sampling noise as a meaningful shape difference. The audit calibrates a per-state null at the deployed rollout count rather than assuming that observed distance is signal.

These controls affect any testing workflow that compares stochastic outcomes from restored simulations, digital twins, or reproducible agent states.

A practical validation gate for risk-sensitive deployment

What the paper directly shows: for the tested QR-DQN, C51, and IQN agents on three MinAtar games, the strongest relative risk claims were frequently refuted, almost never confirmed, and placed close to truth-blind chance. Positive controls demonstrate that the audit can detect known risk. Decision-level results show that unvalidated CVaR guidance can improve or worsen safer-action selection depending on the environment.

What Cognaptus infers: model-risk teams should validate the claims that would trigger intervention, not approve a distributional head because it produces plausible plots or strong task scores. A deployment gate should record top-claim confirmation and refutation rates, realized tail-risk regret, seed reproducibility, and performance against a mean-based fallback. Procurement claims about “risk-aware” control should specify which environments, states, policies, and tail metrics have been independently tested.

What remains uncertain: the study covers three canonical value-based methods, three MinAtar games, and limited full-Atari checks. The pretrained Atari agents were trained without sticky actions, so those results are consistency checks rather than fully matched stochastic-risk replications. Snapshot access may also be unavailable in real systems, and the aliasing control may miss shape-only effects. The findings therefore support a validation requirement, not a universal verdict on every distributional method.

Risk information must earn decision authority

A predicted distribution contains more numbers than a predicted mean. It does not follow that those numbers describe environment risk accurately enough to govern an intervention.

The paper supplies a concrete standard: identify the claims that would change an action, reconstruct their outcome distributions under controlled restarts, calibrate the statistical test against sampling noise, and measure whether following the head reduces realized tail-risk regret. Until that evidence exists, distributional outputs belong in the category of model hypotheses—not certified risk estimates.

Cognaptus: Automate the Present, Incubate the Future.