To generate medical qualification exam questions and answers from real-world electronic health records (EHRs) using large language models (LLMs) and compare their output to that of human medical experts.
Approach:
Database Utilization: Used the China Elderly Comorbidity Medical Database (CECMed) to sample admission reports.
Model Evaluation: Eight LLMs were tasked with generating questions and answers through few-shot prompting, and their outputs were scored by an independent expert panel.
Key Findings:
ERNIE 4 achieved the highest cumulative score for question generation.
Human experts outperformed LLMs in sufficiency of key information but LLMs had higher information correctness.
Doubao outperformed other LLMs in answer generation metrics such as coherence and factual consistency.
Human coherence scores were significantly better than those of LLMs.
Interpretation:
Mainstream LLMs can generate questions and answers from EHRs, though they have limitations in certain aspects.
Limitations:
LLMs performed dissatisfactorily in some evaluation criteria.
The study does not provide a comprehensive performance ranking of different LLMs.
