Clinical Scorecard: Insights from Patients on Responses from Large Language Models Regarding Moyamoya Disease
At a Glance
Category
Detail
Condition
Moyamoya disease
Key Mechanisms
Progressive cerebrovascular occlusion leading to stroke risk and neurological deficits
Target Population
Non-expert general community seeking information on moyamoya disease
Care Setting
Outpatient and community settings with cerebrovascular disease management
Key Highlights
Large Language Models (ChatGPT-4o and Gemini 1.5 Flash) provide responses perceived by patients as similar or somewhat better in quality compared to physicians' answers.
LLM responses frequently omit discussion of potential risks associated with procedures and medications and fail to indicate when medical consultation is urgently needed.
Clinicians identified significant limitations in LLM responses, including lack of recent research updates and failure to address urgent symptoms requiring higher-level care referral.
Guideline-Based Recommendations
Diagnosis
Consult healthcare professionals for accurate diagnosis; LLMs do not replace clinical evaluation.
Management
Use LLM-generated information cautiously; verify with medical professionals especially regarding procedures and medication risks.
Do not rely solely on LLMs for self-care guidance; recognize when symptoms require urgent medical attention.
Monitoring & Follow-up
Regular clinical follow-up is essential; LLMs do not provide comprehensive monitoring guidance.
Risks
LLMs often omit potential risks of treatments and fail to highlight urgent symptoms, posing safety concerns.
Patients should be advised to seek professional care promptly if symptoms worsen or self-care is insufficient.
Patient & Prescribing Data
Patients and caregivers seeking information about moyamoya disease
LLM responses may lack critical safety information on medications and procedures; professional consultation remains necessary.
Clinical Best Practices
Use LLMs as supplementary informational tools rather than primary sources for clinical decision-making.
Ensure patients understand the limitations of AI-generated medical information and encourage direct communication with healthcare providers.
Update educational materials regularly to incorporate recent advances and urgent symptom recognition.
Clinicians should review and validate AI-generated content before dissemination to patients.
by Marcella R. Ruppert-Gomez, Joon Hyeok Choi, Steven J. Staffa, Katherine Holste, Jordan Xu, Catherine Stratton, Sophia D. Kocher, Edward R. Smith, Alfred Pokmeng See
