The Answer Looked Clinical. The Critical Steps Were Missing.
TL;DR for operators A clinical assistant can follow instructions, produce a complete-looking answer, and read like expert work without performing the reasoning needed for a safe clinical decision. Those surface qualities may justify further evaluation, but they do not justify procurement or decision authority. Across five deliberately difficult tasks, the models satisfied 80–90% of the least consequential criteria but only 32.4–41.7% of the most clinically consequential criteria. All three models missed 56 of 108 critical requirements. The systems were therefore strongest where omissions mattered least and weakest on reasoning and safety steps most closely tied to harm. ...