Confidence Calibration

TL;DR for operators A post-trained classifier can disagree with its own-base pretrained model without that example becoming useless for calibration. The paper behind SupportCal finds that adding some disagreement examples back into label-free calibration improves Expected Calibration Error, but giving the entire disagreement set full weight eventually makes calibration worse. The useful decision is therefore not simply whether to retain disagreement, but how much influence each disagreement example receives. ...

TL;DR for operators A reasoning model can have a useful estimate of how likely its current answer is to be correct without having enough information to reliably identify the best answer among several alternatives. That distinction is the central result of this paper. A lightweight classifier reading the model’s hidden state immediately before its final answer produces substantially better calibrated confidence than the model’s own stated score. Yet when that same signal is used to pick one answer from many sampled traces, its advantage over majority voting is small and remains far below the oracle that knows whether any candidate is correct. ...

TL;DR for operators A model can answer correctly, be challenged by the user, and then talk itself into being wrong. Deployment gates that look only at first-turn accuracy or reported confidence can miss that failure. Saadat and Nemzer’s Certainty Robustness Benchmark1 tests 200 LiveBench math and reasoning questions with independent follow-ups: “Are you sure?”, “You are wrong!”, and a request for 1–100 confidence. GPT-5.2 and Claude Sonnet 4.5 began at almost the same accuracy, yet each collapsed under a different form of pushback. ...

TL;DR for operators A workflow that uses model confidence to auto-accept an answer, escalate it, or send it to human review depends on more than whether the underlying model is accurate. The confidence signal itself has to distinguish cases the model should find easy from cases it should find difficult. Chen et al. test this distinction in Latent Confidence Alignment for LLM Self-Assessment.1 Across 20 LLMs and 100 text-only MedXpertQA questions, supplying an external difficulty signal significantly improved the alignment between models’ stated error probabilities and their expected error probabilities. Structured reflection alone did not significantly improve that alignment. At the same time, latent task ability showed no significant differences across the four evaluated conditions. ...

TL;DR for operators A reasoning model can have useful confidence at one point in its workflow and misleading confidence at another. In a controlled comparison using the same Qwen2.5-7B-Instruct backbone and reasoning-data mixture, on-policy distillation produced the strongest average signal for estimating difficulty before reasoning, supervised fine-tuning supplied particularly useful confidence for stopping weak traces during generation, and reinforcement learning gained the most from confidence-based filtering after traces were complete.1 ...

Phishing emails do not need to be brilliant. They only need to be plausible at the wrong moment. A message about a failed payment, a suspended account, or an urgent verification request arrives while someone is clearing a crowded inbox. The user is not solving a formal classification task. They are deciding whether a sentence feels wrong enough to interrupt their day. That is why phishing defense is not only a machine-learning problem. It is a judgment problem disguised as an email problem. ...