Selective Prediction

TL;DR for operators Many production AI systems do not need a more poetic answer. They need a cheaper way to decide whether the answer should be trusted at all. The paper introduces Calibrated Variance Propagation (CVP), a test-time method for Bayesian deep learning that estimates predictive uncertainty without repeatedly sampling model weights through many forward passes.1 It targets a practical bottleneck: recent variational training methods can now produce Gaussian weight posteriors for large neural networks at training costs comparable to standard optimizers, but using those posteriors at inference usually means Monte Carlo sampling. That is expensive, especially when the model must respond in real time. Apparently, reliability is still expected to fit inside latency budgets. Outrageous. ...

Confidence is cheap. A classifier can always give you a probability. The awkward question is whether that probability deserves to be believed. This is not a philosophical problem when the model is recommending a movie. It becomes expensive when the model is screening documents, triaging support tickets, flagging fraud, routing legal clauses, or deciding whether a case should be escalated to a human. In those settings, “92% confident” is not decoration. It is an operating instruction. ...

TL;DR for operators A model that says “I don’t know” is not automatically trustworthy. It may be cautious. It may be badly calibrated. It may be uncertain for the wrong reasons. It may also be using uncertainty as a very elegant trapdoor. Polite refusal, unfortunately, is still refusal. Stephan Rabanser’s thesis, Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning, is useful because it treats uncertainty not as a philosophical mood, but as an operational control layer.1 The key question is not whether a model can emit a confidence score. Most models can emit something confidence-shaped. The harder question is whether that score can decide which cases should be automated, deferred, reviewed, rejected, routed to a larger model, or audited. ...