When Confidence Tracks Consistency Instead of Correctness
TL;DR for operators An AI assistant’s confidence score can become substantially better at predicting correctness without becoming reliable at recognizing its own mistakes. Across 10 open-weight LLMs and five dataset variants, the mean correlation between confidence and the probability assigned to the correct answer rose from 0.04 to 0.73. However, the comparison also changed confidence measurement from verbal reporting to a learned probe, so the improvement cannot be attributed to training alone. ...