Confidence Needs a Difficulty Check Before It Routes Work

TL;DR for operators

A workflow that uses model confidence to auto-accept an answer, escalate it, or send it to human review depends on more than whether the underlying model is accurate. The confidence signal itself has to distinguish cases the model should find easy from cases it should find difficult.

Chen et al. test this distinction in Latent Confidence Alignment for LLM Self-Assessment.¹ Across 20 LLMs and 100 text-only MedXpertQA questions, supplying an external difficulty signal significantly improved the alignment between models’ stated error probabilities and their expected error probabilities. Structured reflection alone did not significantly improve that alignment. At the same time, latent task ability showed no significant differences across the four evaluated conditions.

For operators, the paper suggests a specific governance rule: do not let confidence control routing merely because the model is strong or because it has been prompted to reflect. Evaluate self-assessment separately, and test whether difficulty information improves the signal before using it for workflow decisions.

A capable model can still misjudge when it is likely to fail

The paper starts from an operational distinction that conventional accuracy metrics do not resolve. A model can answer many questions correctly while still being poor at estimating its likelihood of error on a particular question.

The experiment makes that separation visible. Among the seven highest-ability models reported in the paper, baseline self-assessment alignment varies substantially. GPT-5 has a baseline LCAE of 1.058, while Gemini 2.5 Pro has 4.129. The relationship between latent ability and alignment is not monotonic.

That matters when confidence becomes an input to policy. A model that performs well overall but expresses poorly aligned uncertainty may still route difficult cases incorrectly. Higher benchmark ability is evidence about task competence; it is not, by itself, evidence that the model knows when its answer deserves scrutiny.

The paper formalizes this distinction by putting model ability and item difficulty on a shared Rasch scale. For model $i$ and item $q$, expected correctness is

where $\theta_i$ represents model ability and $b_q$ represents item difficulty. The corresponding latent error probability is $1-P_{iq}$.

This gives the evaluation something ordinary outcome-based calibration does not: an expected error reference specific to both the model and the item.

LCAE tests confidence against expected difficulty

The paper calls its resulting measure Latent Confidence Alignment Error, or LCAE. It compares the model’s self-assessed error probability with the error probability implied by its latent ability and the item’s estimated difficulty.

Conceptually, the metric asks: given what this model can generally do and how difficult this item appears on the shared scale, does the model express an appropriate probability of being wrong?

LCAE is calculated as binary cross-entropy between the model’s stated error probability and the Rasch-derived latent error probability. Lower LCAE means closer alignment.

This distinction is easy to miss. LCAE is not conventional calibration error against the observed binary outcome of each question. Its reference is a latent model-item error probability. The metric is designed to separate the quality of the answer from the quality of the model’s judgment about that answer.

The paper first estimates the Rasch model using 20 LLMs under five prompting strategies, treating each model-prompt combination as an examinee. It then focuses its mechanism comparison on the Standard Prompting subset and evaluates four conditions: baseline, a difficulty signal, structured reflection, and both together.

Difficulty information helped; reflection alone did not

The main experimental result is not that additional reasoning improves confidence. It is that an external difficulty reference improves self-assessment more reliably than reflection by itself.

The four-condition comparison produces a significant overall difference in LCAE: Friedman $\chi^2(3)=35.15$, $p<0.001$. Pairwise tests show that the IRT-based Difficulty Signal, or IDS, significantly reduces LCAE relative to baseline, with an adjusted $p=0.001$. The combined IDS+DPR condition is also significantly better than baseline, again with adjusted $p=0.001$.

Dual-Process Routing with structured reflection, or DPR, does not significantly differ from baseline: adjusted $p=0.330$.

The last comparison is particularly useful for interpretation. Although IDS+DPR is descriptively favorable for several models, the reported test does not establish that adding reflection produces an improvement beyond IDS alone.

The individual-model results show why the aggregate finding deserves attention. Gemini 2.5 Pro has an LCAE of 4.129 at baseline. Under DPR alone it rises to 5.870. With IDS it falls to 1.588, and with IDS+DPR it is 1.717.

This is an illustrative case, not proof that reflection generally makes confidence worse. Its role is narrower: additional reflective reasoning does not guarantee better self-assessment, while an external difficulty reference can materially change the confidence behavior of some models.

Better self-assessment did not require higher measured ability

The paper also tests whether the mechanisms changed the models’ latent task ability. They did not do so significantly.

Across baseline, DPR, IDS, and IDS+DPR, the ability comparison gives Friedman $\chi^2(3)=0.52$, with $p>0.05$. Every pairwise Holm-Bonferroni-adjusted $p$-value is 1.00.

That result strengthens the interpretation of the LCAE finding. The improvement associated with IDS is not explained by a statistically detected increase in task ability across conditions. The intervention is acting on the model’s assessment of its expected error rather than demonstrating an improvement in underlying competence.

For a deployment team, that creates a useful separation of controls. Improving the answer generator and improving the trust signal do not have to be the same engineering task. A system may need both, but they should be evaluated independently.

Model selection becomes a three-dimensional decision

The paper extends the analysis from ability and LCAE to a simplified measure of cost efficiency. These dimensions do not move together consistently.

For example, gpt-oss-120B reports LCAE 1.061 under IDS+DPR and the highest cost-efficiency value among the seven models in the paper’s comparison table, at 8.73 million tokens per USD. Its reported ability under that condition is 0.462, well below several higher-ability models.

The point is not that this model dominates the alternatives. It illustrates why a single leaderboard position cannot answer a deployment question involving competence, confidence reliability, and cost simultaneously.

Cognaptus infers a more appropriate procurement or routing process: define minimum ability first, separately test whether uncertainty is trustworthy enough for the intended decision right, and then compare cost among models that satisfy those requirements. Which trade-off is acceptable depends on the user, workflow, and consequence of incorrect routing.

Production use requires a stable difficulty reference

The evidence is meaningful but bounded.

The experiment uses 100 text-only questions from one medical benchmark. The scale is modest, and applicability outside medical reasoning remains untested. More importantly for deployment, IDS is derived from the same evaluated model population. Its behavior for unseen models or a substantially different model population is not established.

That makes the reference population part of the system design. If an organization uses difficulty-aware confidence in production, the evaluation set must represent the tasks and models to which the signal will be applied. Changes in model mix, domain, or case distribution may require re-estimating and revalidating the difficulty reference.

The paper’s cost-efficiency measure also has a narrower meaning than operational cost. It is based on a simplified average of input and output token prices. It does not capture latency, infrastructure, retries, review labor, or the downstream cost of routing errors.

Confidence should earn its decision right

The paper’s strongest contribution is not another way to rank model accuracy. It is a way to ask whether a model’s expressed uncertainty is credible relative to the difficulty of the case it is handling.

Within the studied medical benchmark, supplying difficulty information improves that alignment while reflection alone does not reliably do so, and the change occurs without a statistically significant shift in measured ability.

For workflows in which confidence determines whether an output passes automatically, reaches a human, or is withheld, that distinction changes the evaluation target. Confidence should be treated as a separately tested operational signal. A stronger model, or a more reflective prompt, is not sufficient evidence that the signal is ready to control the workflow.

Cognaptus: Automate the Present, Incubate the Future.