Assessing multiple-choice question quality in internal medicine: a comparative analysis of three large language models against expert consensus - Takeaways - MDSpire

Assessing multiple-choice question quality in internal medicine: a comparative analysis of three large language models against expert consensus

  • By

  • Mevlüt Okan Aydin

  • Belkıs Nihan Coşkun

  • İbrahim Hamal

  • July 9, 2026

  • 0 min

Share

  • 1

    The study evaluated 85 multiple-choice questions from an internal medicine clerkship using three large language models and expert consensus.

  • 2

    Gemini and Claude showed moderate agreement with experts for cognitive level classification, while Llama demonstrated lower agreement.

  • 3

    Claude and Gemini achieved almost perfect agreement on detecting negatively worded stems and substantial agreement on inconsistent numerical data.

  • 4

    Weak agreement was observed for alignment with learning outcomes across all models, indicating limitations in LLM evaluations.

  • 5

    The findings suggest a hybrid approach combining LLM-assisted screening with human expertise may optimize item quality assurance.

Original Source(s)

Assessing multiple-choice question quality in internal medicine: a comparative analysis of three large language models against expert consensus

  • by Mevlüt Okan Aydin, Belkıs Nihan Coşkun, İbrahim Hamal

  • July 9, 2026

Related Content

Development and validation of an interpretable machine learning model for predicting atrial fibrillation risk in middle-aged and older patients with coronary heart disease

Same-Day COVID, Flu Vaccination Appears Safe

Study found no increased 90-day risk of serious adverse events with coadministration compared with vaccination alone.

Oral Microbiome Richness Tied to CRS

A cross-sectional metagenomic study found greater oral microbiome richness among adults with chronic rhinosinusitis, particularly nonallergic chronic rhinosinusitis, while associations with asthma, airway inflammation, and most lung-function measures were inconsistent.