Confidence Is Not a Stop Signal: Test Whether the Model Knows When Information Is Missing

TL;DR for operators

A model can be given an explicit way to say “the available information is insufficient” and still choose an unsupported answer most of the time.

Tahermazandarani, Mahmood, Islam, and Sheng test this directly across five LLMs.¹ They remove the correct answer from medical multiple-choice questions, replace it with an insufficient-information option, and observe abstention rates ranging from just 0.156 to 0.382. Reported unsafe rates range from 0.186 to 0.828. In a separate experiment, progressively stronger warnings that the clinical information may be incomplete or ambiguous also produce little reduction in model confidence.

For operators, the lesson is narrower than “LLMs are overconfident.” Baseline accuracy does not establish that a model recognizes when the evidence supporting an answer has deteriorated. Neither does the presence of an abstention option.

Before a confidence field, an answer-versus-abstain decision, or a confidence threshold controls automated acceptance, escalation, or human review, validate that signal under missing-information conditions. Measure not only whether the model answers correctly, but whether its confidence tracks correctness, whether it withholds answers when evidence is insufficient, and how often wrong answers remain highly confident.

Giving the model an exit does not mean it will use it

A deployed workflow eventually has to make a routing decision: accept the answer, send it for review, or withhold it because the available evidence is inadequate.

It is tempting to assume that a capable model will make this distinction naturally. The answer-removal experiment tests that assumption under a deliberately simple condition. The researchers take each sampled question, remove its known correct option, and insert an explicit insufficient-information choice. The abstention option is therefore visible, and the correct benchmark answer is unavailable.

Every evaluated model still abstains less than half the time.

This is the paper’s strongest operational result. The condition is constructed so that selecting an ordinary answer cannot recover the removed ground truth. Yet the models frequently continue answering rather than selecting the explicit refusal path.

That behavior should not be read as a clinical harm rate. It is a controlled diagnostic test on multiple-choice questions. What it does establish is that answer production and uncertainty recognition are separate behaviors. A model that knows many medical answers under normal benchmark conditions may still fail to recognize that the information needed for a justified answer is no longer present.

Warning language changes the prompt more than the confidence

The second experiment asks a different question: does telling the model that the scenario may be uncertain cause its confidence to adjust?

The underlying question, answer choices, and correct label remain fixed. Only the framing changes, from no warning to increasingly explicit statements that details may be incomplete, ambiguous, or insufficient for full certainty.

Accuracy remains relatively stable. More strikingly, confidence changes little.

GPT-4o, for example, moves from 0.909 mean reported confidence under the baseline framing to 0.895 under the strongest uncertainty cue. LLaMA-3 moves from 0.908 to 0.895. Mistral-7B declines from 0.987 to 0.971 while its accuracy stays around 0.50.

GPT-5 behaves differently from the other evaluated models in this setting. Its reported confidence sits slightly below its accuracy, with accuracy between 0.820 and 0.836 and confidence between 0.798 and 0.809. The paper therefore does not support treating all five models as uniformly overconfident.

The broader finding is about adaptation: stronger linguistic signals of uncertainty do not reliably produce a commensurate confidence response. Telling a model to “answer cautiously if uncertain” is not evidence that its confidence score has become uncertainty-aware.

Accuracy and reliability answer different operational questions

Ordinary benchmark accuracy asks whether a model selects the correct answer when a valid answer is available.

Routing systems need additional information. If a model says it is 90% confident, operators need evidence that similarly confident outputs are correct at roughly the expected rate. That relationship between expressed confidence and observed correctness is what calibration measures.

The paper adds two other quantities that matter when errors have asymmetric consequences. Abstention measures whether the model declines to answer when information has been deliberately made insufficient. Unsafe Confident Error Rate, or UCER, measures the share of evaluated cases that are both incorrect and above a designated high-confidence threshold.

These metrics expose differences that a single accuracy ranking cannot.

Mistral-7B provides the clearest example in the uncertainty-framing experiment: accuracy is around 0.50 while reported confidence remains near 1.0, producing confidence-accuracy gaps of roughly 0.48. GPT-5, by contrast, combines higher accuracy with much smaller gaps in that experiment.

The relevant procurement or validation question is therefore not merely, “Which model answers more questions correctly?” It is also, “Which model produces a confidence and abstention behavior that can safely support the routing rule we intend to automate?”

A confidence field is another model output to validate

The paper also compares verbalized confidence with confidence derived from answer-option logits where those logits are available.

For LLaMA-3 in the answer-removal setting, self-reported confidence is 0.970 while logit-based confidence is 0.280. For Mistral-7B, the corresponding values are 0.998 and 0.657.

That discrepancy does not prove that logits capture true epistemic uncertainty. The study treats option-token probabilities as an informative internal signal, not a complete theory of model uncertainty.

It does undermine a simpler assumption: that a model’s stated confidence can be treated as a transparent probability field without validation.

For a workflow designer deciding whether “confidence ≥ 0.9” should bypass human review, this distinction is consequential. The routing variable itself needs evaluation under the conditions where the workflow expects it to discriminate between supported and unsupported answers.

GPT-5 cannot be included in this comparison because its answer-option logits were unavailable through the evaluated interface.

What model validation should add

Cognaptus infers a concrete validation change from these experiments.

Before allowing confidence or non-abstention to control acceptance, triage, or escalation, clinical AI validation teams can add a degraded-information test set alongside ordinary task evaluation. The degradation should preserve the workflow’s surface form while removing or weakening information required for a justified answer.

Then measure several behaviors separately:

• whether accuracy changes under ordinary uncertainty framing;
• whether confidence falls when evidence quality deteriorates;
• whether the model uses an available abstention path;
• whether high-confidence errors remain frequent;
• whether alternative confidence signals agree closely enough for the intended routing rule.

The affected decision is specific: whether a model output may pass automatically or requires review. The relevant condition is incomplete or insufficient evidence. The result needed before deployment is not perfect uncertainty estimation, but empirical evidence that the chosen routing signal behaves predictably when information degrades.

Those tests should also be rerun after model changes. A confidence threshold validated for one model or version should not be presumed to transfer to another solely because their benchmark accuracies are similar.

The evidence diagnoses behavior, not clinical outcomes

The study uses 500 questions sampled from MedMCQA, evaluates five models at temperature 0, and constructs uncertainty through controlled prompt wording and correct-answer removal. That gives the experiments useful internal control: the researchers can compare the same questions while changing the information condition.

It also defines the limit of the result.

The study does not test open-ended clinical reasoning, interactive decision support, multimodal evidence, naturally occurring ambiguity, or patient outcomes. Its reported unsafe rates are properties of the constructed evaluation, not observed rates of clinical injury.

There is also an internal numerical inconsistency in the source tables for LLaMA-3. Table IV reports abstention of 0.156 and hallucination of 0.798 even though the paper defines hallucination as $1-\mathrm{Abstention}$. The same answer-removal setting also reports LLaMA-3 mean confidence as 0.963 in one table and 0.970 in another. Those values should be preserved as reported rather than reconciled by assumption.

The paper’s proposed explanation—that next-token objectives, answer-present training distributions, instruction-following tendencies, and limited abstention supervision may encourage forced answering—is plausible but not experimentally identified here.

Reliability begins where the valid answer disappears

The most consequential part of this study is not that some models are overconfident. It is that ordinary competence does not guarantee useful behavior when the information environment changes.

A workflow that depends on a model recognizing insufficient evidence needs to test that recognition directly. Warning language is not enough. An abstention option is not enough. A confidence number is not enough.

For operators deciding whether outputs can be accepted automatically or must be escalated, uncertainty recognition is a measured capability, not a property that follows automatically from benchmark accuracy.

Cognaptus: Automate the Present, Incubate the Future.