Performance of DeepSeek-R1 and ChatGPT-5 in the Generation of North American Spine Society Clinical Guidelines for Adult Vertebral Compression Fractures: Comparative Study - Summary - MDSpire
Advertisement
Performance of DeepSeek-R1 and ChatGPT-5 in the Generation of North American Spine Society Clinical Guidelines for Adult Vertebral Compression Fractures: Comparative Study
To compare the multidimensional response quality and guideline concordance of DeepSeek-R1 and ChatGPT-5 in addressing clinical questions related to vertebral compression fractures (VCFs) based on the 2024 NASS guidelines.
Approach:
Study Design: Cross-sectional observational evaluation using an adapted QUEST-aligned human evaluation framework.
Evaluation Framework: Utilized the QUEST framework focusing on Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.
Key Findings:
DeepSeek-R1 and ChatGPT-5 were evaluated for their responses to clinical questions derived from the updated 2024 NASS guidelines.
Performance of the models was hypothesized to be higher for closed-ended questions and recommendations with stronger evidence grades.
Interpretation:
The study aims to fill the information gap regarding the performance of large language models on VCF management and their adherence to updated clinical guidelines.
Limitations:
The study is limited to the evaluation of two specific LLMs and may not represent the performance of all LLMs in the medical domain.
Potential biases in the evaluation framework and the selection of clinical questions may affect the results.
Conclusion:
This study provides insights into the capabilities of LLMs in generating clinical recommendations for VCF management based on authoritative guidelines.
Teriparatide followed by zoledronic acid increased bone mineral density but did not reduce fracture risk compared with standard care in adults with osteogenesis imperfecta.
