Clinical Report: Exploring the Capabilities of Large Language Models in Formulating Questions and Answers from Electronic Health Records
Overview
This study evaluates the ability of large language models (LLMs) to generate medical exam questions and answers from electronic health records (EHRs) and compares their performance to human experts.
Background
The integration of large language models (LLMs) in medical education presents an innovative approach to generating exam content from electronic health records (EHRs). This study addresses the gap in utilizing LLMs for open-ended question generation, which has been limited in previous research.
ERNIE 4 achieved the highest cumulative score for question generation (16.47).
Human experts scored higher in sufficiency of key information (3.67) compared to LLMs.
LLMs outperformed human experts in information correctness (3.63 vs. 4.03–4.57).
Doubao excelled in coherence, factual consistency, and professionalism among LLMs.
Human coherence scores were significantly better than those of LLMs, particularly compared to Llama and Mistral.
Clinical Implications
The findings indicate that LLMs can generate exam questions and answers from EHRs, but the reliance on LLMs should be carefully considered in the context of medical education.
Conclusion
This study highlights the capabilities of LLMs in medical education while acknowledging their limitations.
