To present a practical framework for determining the evidence and oversight needed when evaluating generative AI applications in rheumatology.
Approach:
Framework: A narrative review organizes evaluation by contextual complexity and influence on patient care.
Evaluation domains: The review considers workflow benefits, clinical effectiveness, safety, equity, privacy, governance, and implementation.
Key Findings:
Evidence on AI applications’ benefits, accuracy, and reliability is mixed.
Studies of ambient AI scribes indicate reduced documentation burden; evidence that clinical decision-support applications improve patient outcomes is more limited.
For tools with limited clinical influence, initial evaluation has focused on workflow, usability, and documentation quality.
Tools that synthesize longitudinal information warrant assessment of completeness and source traceability; tools affecting diagnosis or treatment require prospective evaluation of clinical benefit and safety.
Interpretation:
The authors propose combining relevant vendor evidence with proportionate local testing and ongoing monitoring. Safeguards should address hallucinations, automation bias, documentation-error propagation, and performance across patient groups. Changes in functionality or clinical use should prompt reassessment.
Limitations:
The article is a narrative review; the provided context does not describe a systematic search or formal evidence-grading method.
The authors report that evidence on benefits, accuracy, and reliability is mixed and that evidence for patient-outcome improvements from clinical decision-support applications is limited.
Conclusion:
The framework is intended to support adoption decisions and identify research priorities, including patient outcomes and the feasibility of oversight in routine rheumatology care.
