When Better Calibration Means Worse Confidence Ranking

TL;DR for operators

A low calibration error does not establish that an LLM confidence score can reliably distinguish correct answers from incorrect ones. In a matched evaluation of 842 answers, behavioral consensus achieved 2.02 times the resolution of verbal confidence and a lower Brier score, despite having worse Expected Calibration Error (ECE).

The distinction matters for automated acceptance, abstention, and escalation. A confidence score can closely match aggregate correctness frequencies while offering limited discrimination between individual answers.

The study identifies three operational constraints:

• Verbal confidence is prompt-dependent. Changing the elicitation format substantially changes how questions are ranked by confidence.
• Behavioral consensus is more informative in the tested comparison, but expensive. It requires ten sampled answers, at approximately ten times the inference cost of a single response.
• Internal probes need stronger validation. Early predictions of sampling inconsistency often perform no better than simple features extracted from the question text.

These findings support more disciplined confidence evaluation, not a demonstrated improvement in production routing or safety.

The confidence gate has two jobs

Consider an AI assistant answering customer questions or preparing research for analysts. Its monitoring system reports a confidence probability for each answer. A routing rule must decide whether to release that answer automatically or send it through additional verification.

Two properties matter. The score should correspond reasonably well to observed correctness frequencies, and it should distinguish answers that are likely to be correct from those likely to be wrong.

Those properties are related but not identical.

In Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models, Debojeet Ghosh, Shreyansh Kumar, and Ayushman Garg examine whether three ways of measuring LLM confidence provide comparable and decision-relevant information.¹

The authors evaluate five Qwen2.5 models, ranging from 0.5B to 7B parameters, across three factual-retrieval tiers and three mathematical-reasoning tiers. The design covers 30 model-task cells, each containing 500 questions, for 15,000 model-question instances.

They compare three confidence channels: a model’s verbal assessment of its certainty, consistency among multiple sampled answers, and statistical probes trained on internal hidden states.

The central finding is that these measurements behave differently enough that selecting one by a single calibration metric can produce a misleading evaluation.

The better-calibrated signal separated answers less effectively

Expected Calibration Error measures how closely predicted probabilities align with observed correctness frequencies. If questions assigned approximately 70% confidence are correct approximately 70% of the time, that group is well calibrated.

But calibration alone does not tell us whether the score identifies which particular answers should receive additional scrutiny.

The authors examine this distinction using Murphy decomposition of the Brier score. Here, resolution measures how effectively confidence predictions separate groups with different correctness rates. The Brier score measures squared errors between predicted probabilities and binary correctness outcomes, with lower values indicating better overall probabilistic prediction.

Their matched-item comparison yields the following results.

Source: Section 6.2, Table 3. All three signals are evaluated on the same 842 test observations from ten experimental cells. ECE uses 15 bins; Murphy decomposition uses ten bins. Arrows indicate the preferred direction.

Verbal confidence wins on ECE but loses on resolution and Brier score. Behavioral consensus, derived from the consistency of ten sampled answers at the level of meaning, produces approximately twice the resolution of verbal confidence.

The apparent contradiction becomes clearer when considering the underlying probability distributions.

The matched sample has a correctness rate of 28.15%. In nine of ten cells, calibrated verbal confidence never exceeds 0.50. Predictions concentrated near a cell’s correctness base rate can achieve low calibration error without assigning meaningfully different probabilities to answers with different outcomes.

Such a score can describe aggregate uncertainty while providing relatively little separation for an answer-level routing decision.

The paper’s pre-registered ECE comparison also fails to establish its hypothesized ranking: pooled ECE values are 0.0356 for verbal confidence, 0.0427 for behavioral confidence, and 0.0368 for internal confidence. The worst-minus-best difference has a bootstrap confidence interval spanning zero.

That pooled analysis uses unequal signal coverage. The 842-item comparison is more informative for comparing the three signals on identical questions, although its restriction to ten cells limits generalization.

For evaluation teams, the lesson is specific: a confidence estimator selected primarily for low ECE may not be the strongest estimator for sorting answers by reliability. Resolution and proper scoring rules need to accompany calibration error.

Asking for confidence changes the ranking itself

One possible response to weak verbal confidence is to recalibrate its numerical output. That assumes different prompting formats preserve the same underlying ordering of questions, even if they produce different numerical values.

The paper tests that assumption directly.

The models express confidence through three formats: a numerical probability from 0 to 100, five ordinal certainty labels mapped to probabilities, and an answer-or-pass decision under a scoring rule that rewards correct answers and penalizes incorrect ones.

The resulting Spearman rank correlations are:

Source: Section 5, Table 2. All three comparisons fail the pre-registered rank-stability threshold of 0.60.

These are not merely differences in average confidence.

A rank correlation near zero means that knowing which questions appear relatively confident under one format provides little information about their ordering under another.

The numerical format also exhibits severe saturation. Of 15,000 responses, 13,121 assign confidence of at least 90. Yet 26.8% of those high-confidence cases produce a pass decision under the alternative payoff-based format.

This does not establish that the payoff format reveals the model’s authentic uncertainty. It establishes that the confidence elicitation procedure materially changes the measurement.

For an organization applying probability thresholds to customer-facing answers, a prompt revision can therefore alter which questions cross the threshold. A numerical recalibration cannot, by itself, repair rankings that are not stable across elicitation formats.

Confidence prompts should be treated as part of the evaluated system, not interchangeable presentation choices.

An early hidden-state signal is not necessarily internal knowledge of correctness

Internal probes appear to offer another option: extract confidence-related information from the model’s activations rather than asking the model to report its certainty.

The authors capture hidden states at the final prompt token, before answer generation, and train logistic-regression probes to predict whether ten stochastic answers to a question will exhibit relatively high semantic entropy.

Their target is a binary measure of sampling inconsistency, defined using the median semantic entropy within each experimental cell.

It is not a direct label for whether the model’s eventual answer is correct.

The representation-depth analysis asks how early this inconsistency information becomes predictively accessible within the transformer.

In the original pre-registered five-depth analysis, reasoning probes become informative 13.64 percentage points later than retrieval probes on average. Although the bootstrap confidence interval is positive, the exact one-sided permutation test yields p = 0.052, and the result is sensitive to two late-onset GSM8K cells.

The subsequent 21-depth follow-up produces a stronger separation:

• Factual retrieval: mean onset at 6.82% of transformer depth.
• Mathematical reasoning: mean onset at 28.08%.
• Difference: 21.26 percentage points, with a 95% bootstrap interval of 9.72–33.05 and exact permutation p = 0.004.

Source: Sections 4.1–4.3, Table 1. These are post-registration follow-up results using an AUROC threshold of 0.65 at two consecutive depth positions.

The follow-up is a useful sensitivity analysis with finer measurement resolution, but it was developed after the five-depth results were inspected. Its stronger statistical finding should not be presented as the original pre-registered result.

There is also substantial task heterogeneity. Several MATH cells have onset at 5% depth, while GSM8K cells account for much of the later reasoning onset. Excluding GSM8K reduces the average gap to 12.63 percentage points.

More consequentially, early probe performance often lacks a clear advantage over information available directly from the input.

In nine of eleven retrieval cells, the probe at its onset depth performs no better than a prompt-only TF-IDF baseline.

That comparison functions as a negative challenge to the mechanistic interpretation. If question-text features can predict sampling inconsistency just as well, an early hidden-state probe need not have discovered a distinct internal computation of confidence.

The study also finds no statistically supported association between model size and earlier probe onset (Spearman correlation -0.22, p = 0.30).

For teams developing activation-based uncertainty estimators, the relevant test is therefore not whether a probe exceeds chance. It is whether the probe provides additional predictive information beyond inexpensive text-based alternatives, on the deployment target that actually matters.

Choosing a confidence signal for answer routing

The results suggest a task-specific evaluation process rather than a universal confidence estimator.

For an organization operating an automated research or customer-service assistant, Cognaptus draws four design implications.

First, evaluate signals on matched answers. Compare candidate estimators against the same questions and correctness labels. Report coverage, calibration error, resolution, and Brier score together. Otherwise, a difference in apparent quality may reflect different sample composition rather than the estimator.

Second, test verbal confidence under prompt changes. If changing from numeric probabilities to categorical certainty substantially rearranges question rankings, threshold-based routing needs to be revalidated. Valid-looking probabilities are insufficient when measurement behavior changes with the interface.

Third, allocate repeated sampling according to its incremental value. Behavioral consensus achieves stronger measured resolution and Brier performance in the matched comparison, but the study’s ten-generation procedure costs approximately ten times a single response. This makes it a candidate for selective use on consequential or ambiguous requests, not automatically a default for every query. The study does not establish the cost-effectiveness of a particular selective-sampling policy.

Fourth, require internal probes to clear simple baselines. An activation probe that cannot outperform prompt-text features provides weak justification for the additional capture, training, integration, and maintenance infrastructure. Its evaluation should also target correctness or a relevant operational loss, rather than assuming that predicting sampling inconsistency is equivalent to predicting error.

The common principle is to align measurement with the decision. An estimator intended for answer routing should be tested for its ability to separate cases requiring different actions, under the costs and error rates of the actual workload.

What this evidence does not establish

The study is a comparative measurement audit, not a production routing experiment.

All five models belong to the Qwen2.5 family. The nominal experimental grid has 500 questions per cell, but the strongest three-signal comparison covers only 842 test observations across ten of the 30 cells. That complete-case restriction may select questions and models with particular measurement characteristics.

The internal-probe outcome concerns semantic-entropy-derived inconsistency rather than independently validated correctness. Its interpretation is further constrained by prompt-only baseline performance and the post-registration status of the finer depth sweep.

Some supporting comparisons cannot be completed under the study’s own validity rules. In particular, the Qwen2.5-7B-Base model produces effectively degenerate ordinal confidence outputs, preventing the planned base-versus-instruct alignment test. The grading audit also identifies mathematical answers that the automatic grader incorrectly rejects.

Finally, the paper does not demonstrate that behavioral consensus, verbal confidence, or hidden-state probes improve deployed abstention, escalation, or safety outcomes. Establishing such benefits would require testing explicit routing policies against operational error costs, latency, and workload distributions.

Confidence should be judged by the decision it supports

The paper’s strongest contribution is not a new confidence estimator. It is evidence that commonly compared confidence measurements do not preserve the same information, and that familiar evaluation metrics can reward different properties.

Verbal confidence changes with the format used to elicit it. Behavioral consistency provides stronger discrimination in the matched evaluation but consumes more computation. Internal probes can predict inconsistency, although early predictive success is often explainable by simpler features.

For teams setting confidence thresholds, the decision is not which score looks most calibrated in isolation. It is which validated signal most effectively separates answers requiring different treatment, at an acceptable inference cost.

That remains an empirical question for each deployment.

Cognaptus: Automate the Present, Incubate the Future.