To evaluate the diagnostic accuracy of the DeepRare AI system for rare diseases, specifically comparing its performance against existing diagnostic tools and experienced physicians.
Key Findings:
DeepRare achieved 57% Recall@1 and 65% Recall@3 on phenotype-based tasks, outperforming the second-best method (Reasoning LLM) by 24% and 19%, respectively, across 6,401 cases.
In comparisons using human phenotype ontology and genetic data, DeepRare had a Recall@1 of 69.1% compared to Exomiser's 55.9% in 168 cases.
The system maintained performance across heterogeneous datasets, achieving 29% Recall@1 in the MIMIC-IV dataset.
DeepRare demonstrated higher diagnostic accuracy than five experienced physicians, with 64% Recall@1 vs 55% for physicians across 163 cases.
Failure analysis revealed reasoning weighting errors (41%) and phenotypic mimic diagnosis (39%) as common causes of incorrect diagnoses.
Interpretation:
DeepRare shows promise as a valuable decision support tool for non-specialist physicians in diagnosing rare diseases, potentially improving diagnostic accuracy and efficiency across various medical specialties.
Limitations:
Incomplete integration of available data sources.
Difficulty distinguishing conditions with similar clinical features.
Patient interaction features not fully validated.
Intended primarily for patients already suspected of having a rare disease.
Conclusion:
DeepRare's advanced diagnostic capabilities could significantly aid in the timely and accurate diagnosis of rare diseases, though further validation and integration are needed to enhance its clinical utility.
