Assessing multiple-choice question quality in internal medicine: a comparative analysis of three large language models against expert consensus - Report - MDSpire
Advertisement
Assessing multiple-choice question quality in internal medicine: a comparative analysis of three large language models against expert consensus
Clinical Report: Evaluating the Quality of Multiple-Choice Questions in Internal Medicine
Overview
This study evaluated the performance of three large language models (LLMs) in assessing the quality of multiple-choice questions (MCQs) in internal medicine. Findings indicate that while some LLMs show moderate to strong agreement with expert evaluations on cognitive levels and technical flaws, their alignment with learning outcomes remains weak.
Background
The quality of assessment instruments, particularly multiple-choice questions (MCQs), is crucial in medical education as they evaluate knowledge across various cognitive levels. Poorly constructed MCQs can lead to misrepresentation of examinee competence, threatening the integrity of high-stakes examinations. The National Board of Medical Examiners (NBME) provides guidelines to minimize technical flaws and enhance the quality of MCQs, yet inconsistencies in item writing persist.
Data Highlights
Model
Cognitive Level Agreement (κ)
Technical Flaw Detection (κ)
Alignment with Learning Outcomes (ICC)
Gemini
0.424
0.897 (negatively worded stems)
0.121–0.380
Claude
0.415
0.897 (negatively worded stems)
0.121–0.380
Llama
0.266
Poor performance
0.121–0.380
Key Findings
Gemini and Claude showed moderate agreement with experts for cognitive level classification (κ = 0.424 and κ = 0.415, respectively).
Claude and Gemini achieved almost perfect agreement on detecting negatively worded stems (κ = 0.897).
Weak-to-moderate agreement was observed for alignment with learning outcomes across all models (ICC 0.121–0.380).
Llama performed poorly across most technical criteria.
A hybrid approach combining LLM-assisted screening with human expertise is suggested for optimizing item quality assurance.
Clinical Implications
The findings indicate that LLMs like Claude and Gemini can assist in identifying certain technical flaws in MCQs, but their limitations in aligning with learning outcomes suggest that they should not replace expert judgment.
Conclusion
The study highlights the performance of LLMs in MCQ review. Further validation across multiple centers is required to generalize these findings.
Creatine's ATP-buffering trick isn't just for biceps. In mice, it juices up dendritic cells' energy supply, sharpening T-cell priming and shrinking tumors on daily injections. Human DCs got the memo too, at least in a dish. Early days, but a fun plot twist for gym-bag chemistry.
