To evaluate the usefulness, safety, helpfulness, and accuracy of responses from publicly available Large Language Models (LLMs) regarding moyamoya disease, focusing on specific aspects such as clarity and comprehensiveness.
Key Findings:
Output length varied significantly between LLMs (p < 0.001).
ChatGPT and Gemini responses failed to address potential risks for procedures and medications (38% and 28.6%, respectively).
Omissions included when to consult a medical professional (27.2% for ChatGPT, 40.8% for Gemini).
Community respondents rated LLM answers similarly to physician responses, with 47.8% for ChatGPT and 49% for Gemini.
Physicians noted LLM responses did not address recent advances in the field (57.5% for ChatGPT, 62.5% for Gemini) and urgent symptoms (70% for both).
Interpretation:
While LLM responses are perceived as comparable to physician responses, significant limitations exist regarding safety, omission of critical information, and potential impacts on patient-physician relationships, which must be acknowledged.
Limitations:
LLMs did not adequately address risks associated with procedures and medications.
Responses lacked guidance on when to seek professional medical advice.
Physicians found LLM outputs lacking in current research and urgent care considerations.
Conclusion:
LLMs can provide information perceived as similar to that from physicians, but their limitations in safety and comprehensiveness must be acknowledged.
by Marcella R. Ruppert-Gomez, Joon Hyeok Choi, Steven J. Staffa, Katherine Holste, Jordan Xu, Catherine Stratton, Sophia D. Kocher, Edward R. Smith, Alfred Pokmeng See
These 10 states make it more practical for physicians to participate in hospital ownership by aligning statutory structure, corporate practice of medicine rules, and population trends.
