Correct, Consistent, Still Unverified: Testing Clinical AI Explanations
TL;DR for operators A clinical AI assistant may answer examination questions correctly, repeat those answers consistently, and agree with other models while providing explanations whose cited clinical factors have little demonstrated influence on its decisions. In a study of six language models and 150 medical examination questions, only 23.3% of cited clinical concepts satisfied a test in which removing the concept changed the model’s answer. Surprisingly, correct answers scored lower on this test than incorrect answers: 0.212 versus 0.398. Repeated-answer consistency was negatively associated with the score, and models agreed on answers far more often than on their cited clinical concepts. ...