Objective:

To compare the multidimensional response quality and guideline concordance of DeepSeek-R1 and ChatGPT-5 in addressing clinical questions related to vertebral compression fractures (VCFs) based on the 2024 NASS guidelines.

Approach:

• Study Design: Cross-sectional observational evaluation using an adapted QUEST-aligned human evaluation framework.
• Evaluation Framework: Utilized the QUEST framework focusing on Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.

Key Findings:

• DeepSeek-R1 and ChatGPT-5 were evaluated for their responses to clinical questions derived from the updated 2024 NASS guidelines.
• Performance of the models was hypothesized to be higher for closed-ended questions and recommendations with stronger evidence grades.

Interpretation:

The study aims to fill the information gap regarding the performance of large language models on VCF management and their adherence to updated clinical guidelines.

Limitations:

• The study is limited to the evaluation of two specific LLMs and may not represent the performance of all LLMs in the medical domain.
• Potential biases in the evaluation framework and the selection of clinical questions may affect the results.

Conclusion:

This study provides insights into the capabilities of LLMs in generating clinical recommendations for VCF management based on authoritative guidelines.

Sources:

• North American Spine Society Guidelines