Self-Consistency

TL;DR for operators When one model answer can trigger a consequential downstream step, spending more inference compute before trusting that answer can help—but the form of that spending matters. In the reported experiment, generating several reasoning attempts and aggregating the answer that repeatedly emerged increased verified acceptance from 56.2% to 64.9%. Asking the same model to critique and revise itself produced a smaller increase, from 47.2% to 50.6%. Adding a second model did not improve the reported acceptance rate: 47.4% of outputs survived cross-model verification versus a 48.7% single-model baseline. ...

TL;DR for operators A model giving the same answer five times is comforting in the same way that five interns copying the same spreadsheet error is comforting: technically consistent, operationally useless. The paper behind this article proposes structural uncertainty, a black-box method for evaluating whether an LLM can stably rank its own reasoning paths, not merely whether its final answers agree.1 The method samples multiple candidate solutions, asks the same model to compare pairs of its own outputs, turns those comparisons into ranking distributions using Bradley-Terry or TrueSkill plus PageRank, then measures two things: whether rankings fluctuate across comparison trials, and whether each trial remains ambiguous among candidates. ...