To evaluate the alignment of newer LLMs with orthopaedic expert consensus and track changes in AI responses over time.
Approach:
Study Design: Re-tested 97 clinical cases using ChatGPT-5, Gemini 2.5 Flash, and Grok-3, comparing responses to pooled decisions from practicing clinicians.
AI Tools: Evaluated three updated LLMs: ChatGPT-5, Grok-3, and Gemini 2.5 Flash.
Clinical Cases: Sourced from OrthoBullets, covering various orthopaedic subspecialties with multiple-choice questions.
Outcome Measures: Primary outcome was the proportion of AI responses matching the most popular clinician response, with secondary analyses on response proximity and performance on controversial questions.
Key Findings:
The study provides a longitudinal evaluation of AI model performance in orthopaedic decision-making, tracking changes over time.
AI responses were compared against a benchmark of clinician consensus rather than absolute clinical accuracy.
The methodology allows for consistent evaluation of AI evolution over time.
Interpretation:
Limitations:
The study does not assess absolute clinical accuracy or guideline-derived ground truth.
Responses may still be regionally biased or dependent on training levels.
