Clinical Scorecard: Evaluating and Monitoring Emerging Generative AI Tools in Rheumatology Practice

At a Glance

Key Highlights

• Evidence on AI application benefits, accuracy, and reliability is mixed.
• Studies of ambient AI scribes indicate a lower documentation burden.
• Evidence that clinical decision-support applications improve patient outcomes is more limited.
• Evaluation and oversight should be proportionate to contextual complexity and influence on patient care.
• Changes in tool functionality or clinical use should trigger reassessment.

Guideline-Based Recommendations

Diagnosis

• For tools affecting diagnosis, require prospective assessment of clinical benefit and safety.

Management

• Consider relevant vendor-provided evidence alongside proportionate local testing.
• For applications with limited clinical influence, initial evaluation has emphasized workflow, usability, and documentation quality.
• For tools synthesizing longitudinal information, further assess completeness and traceability to source information.

Monitoring & Follow-up

• Continue monitoring AI applications after implementation.
• Reassess when functionality or clinical use changes.

Risks

• Safeguards should address hallucinations, automation bias, and propagation of documentation errors.
• Consider whether performance differs across patient groups.
• As contextual complexity increases, errors may be more difficult to identify; as influence on care increases, undetected errors may have greater consequences.

Patient & Prescribing Data

No specific patient population or prescribing regimen is described.

The review reports more limited evidence for improved patient outcomes from AI applications assisting clinical decisions; tools affecting treatment require prospective assessment of clinical benefit and safety.

Clinical Best Practices

• Match the evidence and oversight required to each application’s contextual complexity and influence on patient care.
• Use local testing proportionate to the application, in addition to considering relevant vendor-provided evidence.
• Assess workflow, usability, and documentation quality for applications with limited clinical influence.
• Assess completeness and source traceability when tools synthesize longitudinal information.
• Establish safeguards for hallucinations, automation bias, and propagation of documentation errors, and consider performance across patient groups.

Related Resources & Content

• Article DOI: 10.1002/acr.80195