To compare the accuracy, quality of information, safety, response length, and readability of responses from ChatGPT, Gemini, and Claude regarding periprosthetic fractures.
Approach:
Study Design: A question-based paired investigation was conducted using a set of 25 standardized patient questions related to periprosthetic fractures.
Question Development: Questions were generated by orthopaedic residents and refined through a two-step reduction process to ensure clarity and relevance.
Response Evaluation: Responses from each model were assessed by a blinded expert panel using predefined scoring systems for accuracy and quality.
Key Findings:
Large language models can produce fluent text but may also convey inaccurate information.
A total of 25 questions were developed to cover key patient concerns and decision points.
Limitations:
The single-query design may not fully capture variability in LLM-generated responses.
Responses were evaluated in a randomized order, which may affect scoring consistency.
Machine-learning models distinguished healthy knees from those with osteoarthritis-related cartilage damage, but age and BMI performed as well as or better than the vibration-based models.
