Objective:

To compare the performance of three large language models (LLMs) with that of human experts in evaluating the quality of multiple-choice questions (MCQs) from an internal medicine clerkship examination.

Approach:

• Study Design: A comparative methodological study evaluating 85 MCQs assessed by three LLMs and three medical education experts.
• Evaluation Criteria: Assessment focused on cognitive level classification, alignment with intended learning outcomes, and presence of technical flaws based on NBME guidelines.
• Statistical Analysis: Agreement was calculated using Fleiss’ kappa for cognitive level classification, Cohen’s kappa for binary technical flaw criteria, and intraclass correlation coefficients for Likert-scale alignment ratings.

Key Findings:

• Gemini and Claude showed moderate agreement with experts for cognitive level classification (κ = 0.424, p < 0.001 and κ = 0.415, p < 0.001, respectively).
• Llama demonstrated lower agreement for cognitive level classification (κ = 0.266, p < 0.001).
• Weak-to-moderate agreement was observed for alignment with learning outcomes across all models (ICC 0.121–0.380).
• Claude and Gemini achieved almost perfect agreement on detecting negatively worded stems (κ = 0.897, p < 0.001) and substantial agreement on inconsistent numerical data (κ = 0.661, p < 0.001).
• Llama performed poorly across most technical criteria.

Interpretation:

Claude and Gemini demonstrate moderate to strong agreement with experts for cognitive level classification and detection of objective technical flaws. However, weak agreement on learning outcome alignment indicates that LLMs cannot replace expert judgment.

Limitations:

• Study conducted at a single institution with a limited item set (n = 85).
• Findings require multi-center validation before broader generalization.

Conclusion:

Claude and Gemini demonstrate moderate to strong agreement with experts for cognitive level classification and detection of objective technical flaws. However, weak agreement on learning outcome alignment indicates that LLMs cannot replace expert judgment.