Confidence Is Not a Reranker
TL;DR for operators A reasoning model can have a useful estimate of how likely its current answer is to be correct without having enough information to reliably identify the best answer among several alternatives. That distinction is the central result of this paper. A lightweight classifier reading the model’s hidden state immediately before its final answer produces substantially better calibrated confidence than the model’s own stated score. Yet when that same signal is used to pick one answer from many sampled traces, its advantage over majority voting is small and remains far below the oracle that knows whether any candidate is correct. ...