Correct, Consistent, Still Unverified: Testing Clinical AI Explanations

TL;DR for operators

A clinical AI assistant may answer examination questions correctly, repeat those answers consistently, and agree with other models while providing explanations whose cited clinical factors have little demonstrated influence on its decisions.

In a study of six language models and 150 medical examination questions, only 23.3% of cited clinical concepts satisfied a test in which removing the concept changed the model’s answer. Surprisingly, correct answers scored lower on this test than incorrect answers: 0.212 versus 0.398. Repeated-answer consistency was negatively associated with the score, and models agreed on answers far more often than on their cited clinical concepts.

For organizations evaluating clinical AI, the implication is to supplement accuracy and model consensus with three distinct checks: whether explanations remain stable, whether cited evidence affects answers when removed, and whether equivalent wording preserves recommendations. These tests can expose weaknesses that conventional benchmarks miss, but they do not prove how a model reasons internally or establish clinical safety.

The procurement problem behind a correct answer

Consider a healthcare organization comparing AI assistants for clinical decision support. One candidate achieves high examination accuracy, produces similar recommendations across repeated queries, and frequently agrees with competing models. Its explanations are fluent and clinically plausible.

These results establish several desirable properties. They do not establish whether the clinical factors cited in those explanations materially influence the model’s answers.

The distinction matters when an explanation becomes part of the evidence supporting a clinical recommendation. If a model cites a symptom, diagnostic finding, or treatment consideration, an evaluator needs to know whether that information affects the prediction or merely appears in its accompanying explanation.

A 2026 study, Right Answer, Wrong Reason: Accuracy, Consistency, and Consensus Are Misleading Indicators of LLM Faithfulness in Clinical Decision Support, examines this gap through black-box behavioral tests.¹

Its most counterintuitive finding is that correct answers exhibit lower, not higher, average sensitivity to removing cited clinical concepts. The result does not establish that incorrect answers contain better medical reasoning. It establishes that answer correctness and the tested relationship between an explanation and its cited evidence can diverge.

Three tests distinguish stability from evidence sensitivity

The researchers evaluate six models on 150 randomly sampled, five-option MedQA-USMLE questions, producing 900 model-question observations.

Each model supplies an answer, reasoning explanation, cited clinical factors, and confidence score. The evaluation then examines three different properties.

For ESI, each model-question combination is queried ten times at temperature 0.7. The researchers use Sentence-BERT embeddings to calculate average semantic similarity across the 45 distinct explanation pairs. This measures whether the model describes its reasoning consistently, independently of whether the description is faithful.

CFS introduces a different test. Each explanation identifies five clinical concepts. The researchers remove each concept’s explicit textual mention from the original question separately and query the model again.

If three of five removals change the answer, CFS equals 0.60. A concept whose removal leaves the answer unchanged does not pass this particular test.

Across the evaluation, the five concepts per model-question pair imply 4,500 individual concept-ablation tests.

PSS examines another source of instability. GPT-4o-mini generates three paraphrases intended to preserve each question’s clinical meaning. The evaluated model receives the rewritten questions, and the researchers measure how frequently its answer remains unchanged.

These measures are complementary. A model can consistently repeat an explanation that cites factors with little measured influence. It can also be sensitive to cited information while changing its answer under clinically equivalent wording.

Correct answers show lower concept-ablation sensitivity

Across all model-question observations, mean CFS is 0.233. Under the study’s criterion, fewer than one-quarter of cited concepts trigger an answer change when individually removed.

The correctness comparison is more striking:

Source: Section IV.B, Table II and Figure 1.

The 0.186-point difference means that explanations accompanying incorrect answers are, on average, more sensitive to the deletion of their cited concepts.

One possible interpretation is that a model can answer familiar questions through stable learned associations while generating an explanation containing clinically plausible factors that are not individually decisive under textual ablation. Less reliable predictions might instead be more easily disrupted by removing information.

That is an interpretation, not an identified internal mechanism.

Higher CFS does not mean an answer is medically preferable. A wrong answer that changes readily when information is removed remains wrong. Conversely, a correct answer may remain unchanged because other information in the question provides sufficient evidence.

For assurance teams, the result challenges using accuracy as a substitute for testing explanation behavior. It does not justify prioritizing CFS over clinical correctness.

Repetition and model consensus do not resolve the discrepancy

Repeated answers might seem to provide additional reassurance. The benchmark reports the opposite association under its faithfulness measure.

Across 900 observations, answer consistency correlates negatively with CFS:

• Spearman correlation: −0.466, p<0.001.
• Pearson correlation: −0.481, p<0.001.

A Kruskal-Wallis comparison across answer-consistency quartiles also identifies differences in CFS (H=195.2, p<0.001), with generally lower scores among more consistent observations.

These are observational associations within the evaluation. They do not show that making a model more consistent causes its explanations to become less faithful.

MiniMax illustrates the practical distinction. It achieves 0.83 accuracy, 0.88 explanation stability, and 0.91 answer consistency, yet its CFS is only 0.13 in Table I. Reliable repetition of an answer is therefore not equivalent to demonstrated sensitivity to the factors cited in its explanation.

Could an organization compensate by checking whether independent models agree?

Across all 15 model pairs, the researchers report:

Source: Section IV.D, Table III and Figure 3.

MiniMax and MedGemma agree on answers approximately 94% of the time while sharing only about 9% of their cited clinical concepts.

This does not prove that the models use different internal mechanisms. It shows that their reported justifications overlap much less than their final predictions.

For a multi-model verification system, agreement on an answer therefore provides limited corroboration of the particular evidence used to justify it. Comparing cited factors adds information that answer-level voting cannot supply.

Model rankings change with the property being measured

The six models also exhibit different strengths across accuracy, explanation stability, concept sensitivity, and paraphrase robustness.

Source: Section IV.A, Table I. Model names are abbreviated as in the source table; PSS is reported there as answer stability under paraphrasing. Values are reproduced from Table I rather than reconciled with differing figures elsewhere in the paper.

MedGemma leads on accuracy and paraphrase stability, while DeepSeek-R1 has the highest CFS. MiniMax combines relatively strong accuracy and consistency with the lowest CFS.

No model dominates every evaluation dimension.

Exploratory analyses suggest that evaluation slices also matter. Question difficulty is positively correlated with CFS (Spearman r=0.408, p<0.001), although only 23 questions fall into the HARD category. A rule-based concept analysis reports lower behavioral faithfulness for SYMPTOM concepts (19.2% among 156) than for ANATOMY concepts (31.8% among 170).

These comparisons can guide targeted testing, but they are insufficient to establish universal differences between clinical specialties, concept types, or model architectures.

Clinical AI assurance needs several independent checks

For a clinical AI procurement or quality-assurance team, Cognaptus draws a bounded design implication: retain accuracy as a necessary evaluation criterion, but assess explanation behavior separately.

A practical offline assessment can proceed through four connected activities.

1. Measure prediction quality and reproducibility. Record examination accuracy, repeated-answer consistency, and explanation similarity as separate outcomes rather than collapsing them into one reliability score.

2. Test cited evidence. For a representative set of cases, record the clinical concepts named by each explanation and assess the answer’s sensitivity to individually removing them. Examine whether the modified questions remain clinically coherent.

3. Test wording robustness. Present clinically equivalent formulations of each question and compare the answers. Check the validity of paraphrases before interpreting disagreement as a model failure.

4. Inspect ensemble justifications. When several models agree on a recommendation, compare the clinical factors they cite. Record disagreement about supporting evidence rather than treating answer consensus as comprehensive verification.

These checks can be segmented by difficulty, concept category, and case characteristics. Their immediate value is diagnostic: they provide different evidence about how a candidate system behaves under specified tests.

The study does not establish optimal acceptance thresholds, the cost-effectiveness of these checks, or whether adding them improves patient outcomes.

What concept ablation cannot establish

The central measurement boundary is that textual intervention is not internal mechanistic interpretability.

Deleting a clinical concept may alter the coherence of a question or remove correlated information. Alternatively, redundant clues may preserve the answer despite the removed concept having influenced the original prediction.

Consequently, both answer changes and unchanged answers admit multiple interpretations. CFS should be treated as a behavioral sensitivity measure, not proof that a particular concept was or was not used internally.

Other boundaries also matter. The sample contains 150 examination questions rather than patient encounters. Paraphrases are generated by GPT-4o-mini, itself one of the evaluated models, and are not all independently clinically validated. The concept taxonomy is rule-based.

There are also reporting discrepancies between model-specific tables and accompanying text, including differences in ESI and consistency figures and a significance footnote that conflicts with a reported p-value. These limit the precision of some model-level comparisons without eliminating the main reported descriptive patterns.

Most importantly, the researchers do not test prospective clinical deployment or patient outcomes. The findings support additional evaluation, not clinical safety certification.

The assurance standard is broader than agreement

A correct clinical answer is evidence of predictive performance. A repeated answer provides evidence of reproducibility. Agreement between models indicates consensus on an output.

None independently verifies the clinical concepts offered as justification.

The paper’s contribution is to make that distinction measurable through complementary behavioral tests. For organizations adopting clinical decision-support systems, the resulting discipline is to require evidence appropriate to each claim of reliability—and to avoid presenting stable, plausible explanations as proof of faithful reasoning.

Cognaptus: Automate the Present, Incubate the Future.