When Confidence Tracks Consistency Instead of Correctness

TL;DR for operators

An AI assistant’s confidence score can become substantially better at predicting correctness without becoming reliable at recognizing its own mistakes. Across 10 open-weight LLMs and five dataset variants, the mean correlation between confidence and the probability assigned to the correct answer rose from 0.04 to 0.73. However, the comparison also changed confidence measurement from verbal reporting to a learned probe, so the improvement cannot be attributed to training alone.

The crucial distinction is between accuracy tracking and output-consistency tracking. On general-knowledge and medical questions, learned confidence followed how strongly the model favored an answer more closely than whether that answer was correct (mean correlation difference: −0.12). Accuracy tracking was stronger on closely related arithmetic questions (+0.27), while consistency tracking emerged earlier and transferred more broadly. A model could therefore remain highly confident in an incorrect answer.

For organizations using confidence thresholds to accept answers automatically or escalate them for human review, the practical requirement is to evaluate confident errors, unfamiliar domains, and independently verified correctness. A stronger confidence benchmark alone does not establish reliable error detection.

A stronger confidence score can still miss the wrong answer

Consider an enterprise assistant that has performed well on a company’s established knowledge collection. Its confidence estimates appear reliable, so the organization introduces a threshold: high-confidence answers proceed automatically, while uncertain answers receive human review.

The assistant is then asked about an unfamiliar regulatory subject. It strongly favors one answer and reports high confidence. Unfortunately, the answer is incorrect.

The problem is not merely that an LLM can make mistakes. The organization has already attempted to manage that risk through confidence estimation. The question is whether the confidence score recognizes a mistake when the model itself strongly believes the incorrect answer.

Nicolas Yax, Stefano Palminteri, and Pierre-Yves Oudeyer investigate this problem in LLMs learn different forms of metacognition when trained to predict their own accuracy.¹

Their experiments cover 10 open-weight language models, approximately 3B–9B parameters, and five dataset variants constructed from MMLU-Pro, MedMCQA, and procedurally generated arithmetic questions. Models are trained using QLoRA adapters and linear probes to estimate their probability of answering correctly before producing an answer.

The initial performance improvement appears substantial. Across models and datasets, the mean Pearson correlation between confidence and probabilistic accuracy increases from 0.04 ± 0.01 to 0.73 ± 0.03 (Section 3.1).

One qualification is essential: the baseline uses verbally reported confidence, while the trained approach uses a learned probe. The comparison establishes the performance of the combined approach, not the isolated contribution of fine-tuning.

More fundamentally, a high correlation does not establish that the estimator recognizes every kind of error. The paper’s subsequent experiments show why.

Accuracy and consistency measure different properties

The central contribution is identifying two different relationships that can produce apparently successful confidence estimates.

Probabilistic accuracy measures the probability assigned to the correct multiple-choice option. It is not binary correctness from one generated answer.

Output consistency measures the concentration of probabilities across the available options, independently of which option is correct. The researchers calculate it using normalized Shannon entropy:

Here, $H(p)$ is the entropy of the answer distribution, and $M$ is the number of options. Consistency equals zero when all options receive equal probability and approaches one when probability is concentrated on a single option.

Suppose a model assigns almost all probability to an incorrect answer. Its consistency will be high because the distribution is concentrated, while its probabilistic accuracy will be low because the correct option receives little probability.

A confidence estimator that closely follows consistency may therefore report high confidence precisely when the model is strongly committed to an error.

The researchers compare correlations between learned confidence and these two quantities. They define $\Delta r$ as the confidence–accuracy correlation minus the confidence–consistency correlation. Positive values favor accuracy tracking; negative values favor consistency tracking.

Source: Sections 3.2–3.3. Reported uncertainty values are standard errors of the mean.

The findings do not imply that models learn only consistency. They show that learned confidence can reflect different information depending on the questions being evaluated.

For example, Qwen3.5-9B on MedMCQA has a stronger confidence relationship with consistency than accuracy (0.94 versus 0.81). On Math, the relationship reverses (0.99 with accuracy versus 0.84 with consistency).

The medical-question results also include cases where confidence remains above the four-option chance level of 25% despite near-zero probabilistic accuracy (Section 3.2, Figure 2).

These are the errors that matter most when high confidence is used to bypass further checking.

Familiar questions develop accuracy tracking; unfamiliar questions retain consistency

Why does arithmetic produce a different result?

The researchers divide training and evaluation data by question categories rather than randomly distributing individual questions. MMLU-Pro holds out computer science, history, and philosophy; MedMCQA holds out ophthalmology, physiology, and skin-related questions.

The arithmetic split is different. Training includes eight combinations of operators and difficulty, while testing uses easy multiplication. These questions remain relatively similar to the training examples.

Consequently, the arithmetic result supports a more specific interpretation: accuracy tracking can generalize effectively when evaluation questions closely resemble those used for confidence training.

Cross-dataset experiments strengthen the distinction. Math-trained confidence models track accuracy more strongly on either Math variant, but switch toward consistency tracking when tested on general or medical questions. Models trained on MMLU-Pro or MedMCQA generally continue tracking consistency more strongly across evaluation datasets.

The authors also measure semantic distance using embeddings from Qwen3-Embedding-0.6B. For each evaluation question, they calculate the average cosine distance to its 100 nearest training questions.

Across the held-out question variants, greater distance is associated with confidence being numerically closer to consistency than accuracy. The reported association is $r=0.49$ (Section 3.4, Figure 4).

This turns familiarity into a measurable evaluation variable rather than merely a label attached to different benchmarks. However, the relationship is correlational and depends on the chosen embedding representation. It does not establish that semantic distance itself causally determines which signal the model uses.

Training reveals two different learning trajectories

The temporal evidence helps explain why the two signals generalize differently.

Checkpoint analyses in Appendix E show that consistency tracking develops early during confidence training. Accuracy tracking strengthens later, particularly on training questions and closely related examples.

The authors interpret consistency as a broadly accessible signal about the model’s own answer distribution. Accuracy tracking, by comparison, requires learning how that distribution relates to external correctness feedback.

This is a plausible interpretation of the experimental pattern, not direct identification of the underlying computation.

A further robustness experiment moves the confidence probe from the end of the transformer to a middle layer (Appendix F). If consistency tracking were merely a consequence of reading the final output representation, moving the probe earlier might eliminate it.

It does not.

Substantial consistency tracking persists, although confidence–accuracy correlation declines from 0.73 ± 0.03 to 0.55 ± 0.04. The association between semantic distance and relative prediction error also weakens from 0.49 to 0.07.

This robustness test makes a simple final-layer-logit explanation insufficient. It does not prove how internal representations generate confidence, nor does it establish that accuracy and consistency are processed by separate computational modules.

Confidence-based automation needs domain-specific evidence

For an organization deploying an AI assistant, the paper changes what should count as evidence that a confidence threshold is trustworthy.

A model assurance team choosing whether to release high-confidence answers automatically should distinguish three evaluations:

1. Confidence–correctness alignment: Does confidence predict externally established correctness, rather than simply correlate with the concentration of output probabilities?
2. Confident-error behavior: How frequently do incorrect answers remain above the proposed automatic-acceptance threshold, particularly in consequential question categories?
3. Domain transfer: Does the threshold remain reliable on entirely held-out subjects or newly introduced knowledge collections?

These checks serve different purposes. Aggregate correlations describe broad associations; confident-error analysis examines cases where the acceptance rule fails; domain-disjoint evaluation investigates whether familiar-domain performance transfers.

Cognaptus infers an additional architectural implication: for workflows where factual errors have material consequences, internal confidence should be treated as one routing signal, while correctness should be assessed through independently grounded checks when required.

That could involve source validation, deterministic rules, specialized verification tools, or human review, depending on the decision and available evidence.

The paper does not experimentally demonstrate that adding such mechanisms improves production performance. Independent verification is a system-design response to the observed limitation, not a tested intervention in this study.

What these results do not establish

The evidence is reasonably broad within its controlled setting: 10 models, five dataset variants, repeated training runs, category-disjoint evaluation, cross-dataset transfer, and intermediate-layer probes.

However, the scope is specific. The models are four-bit quantized and relatively small, and the tasks are multiple-choice questions without chain-of-thought reasoning. Probabilistic accuracy concerns option probabilities, while consistency comes from entropy over those options. Neither directly measures the reliability of complete, open-ended responses.

The study therefore does not establish how confidence behaves in larger frontier models, long reasoning sequences, tool-using agents, or multi-stage business workflows. Nor does it directly measure selective prediction, abstention outcomes, or the costs of erroneous automated decisions.

Within those boundaries, the finding is consequential: improved confidence prediction can coexist with persistent inability to detect confidently held errors.

For organizations deciding when an AI system can act without review, the relevant evidence is not simply whether confidence has become more predictive on a benchmark. It is whether that relationship survives unfamiliar questions and, especially, whether high confidence continues to permit incorrect answers.

Reference

Cognaptus: Automate the Present, Incubate the Future.