AI Falls Short on Differential Dx

New PrIME-LLM benchmark shows strong diagnostic accuracy but persistent gaps in clinical reasoning across 21 large language models

1

AI models produced accurate final diagnoses but struggled significantly with differential diagnosis in clinical scenarios.
2

The study evaluated 21 large language models using a new metric, PrIME-LLM, to assess performance across the clinical workflow.
3

Differential diagnosis tasks had failure rates exceeding 80%, while final diagnosis tasks had failure rates below 40%.
4

Current evaluation methods may overestimate AI models' clinical readiness by focusing on final answers rather than reasoning processes.
5

Despite advancements, off-the-shelf AI models lack the intelligence for safe clinical deployment and should be supervised by physicians.

1