Explainable AI Challenges in Specialist AI for Glaucoma Referrals in Primary Care

Overview

Specialist AI models for glaucoma referrals outperform primary care providers but face challenges in collaboration due to knowledge disparities. Although AI explanations were introduced to improve trust and decision-making, they did not enhance referral accuracy and sometimes increased over-reliance on incorrect AI outputs.

Background

Glaucoma is a progressive optic neuropathy causing irreversible vision loss, requiring timely specialist referrals for effective management. Primary eye care providers, such as optometrists, play a crucial role in screening and referring patients but show variability in referral practices. Specialist AI models trained on clinical data can identify urgent glaucoma referrals with high accuracy, yet their assumptions may not fully align with clinical realities. Explainable AI (XAI) techniques aim to bridge this knowledge gap by making AI decisions more transparent to providers, potentially improving human-AI collaboration.

Data Highlights

Key Findings

• Human-AI teams achieved 60% accuracy in glaucoma referral decisions, outperforming humans alone at 51%.
• AI alone reached 80% accuracy, surpassing both humans alone and human-AI teams.
• Providing intrinsic or post-hoc AI explanations did not improve human-AI team performance.
• Post-hoc explanations increased over-reliance on incorrect AI recommendations, leading to errors.
• Both explanation types contributed to anchoring bias, causing participants to align more closely with AI referrals regardless of correctness.
• Challenges remain in designing AI support that enhances performance while preserving human clinical judgment and agency.

Clinical Implications

Clinicians should be cautious when integrating AI explanations into glaucoma referral workflows, as explanations may inadvertently increase bias and reduce critical evaluation of AI outputs. While AI can augment referral accuracy, reliance on AI without effective interpretability tools may compromise decision quality. Future AI tools must balance transparency with usability to support provider trust and optimal patient outcomes.

Conclusion

Specialist AI improves glaucoma referral accuracy but current explainable AI approaches do not effectively support primary care providers in surpassing AI performance. Addressing knowledge disparities and designing better explanation mechanisms are essential for successful human-AI collaboration in clinical decision-making.

References

1. Navigating the Explainable AI Challenge Amid Knowledge Disparities in Specialist AI for Glaucoma Referrals in Primary Care Settings