Clinical Scorecard: Determinants of Trust and Satisfaction in AI Medical Chatbots: Insights from a Mixed Methods Survey of Caregivers Seeking Advice on Pediatric Infectious Diseases

At a Glance

Key Highlights

• AI chatbots are transforming access to medical advice for caregivers.
• Quality of chatbot responses is critical to prevent misinformation.
• User trust significantly impacts the adoption of AI chatbots.
• Existing evaluation methods for chatbots include automatic metrics and human frameworks.
• Human evaluation frameworks are essential for assessing response quality.

Guideline-Based Recommendations

Diagnosis

• Ensure chatbot responses are factually correct and comprehensible.

Management

• Implement human evaluation frameworks alongside automatic metrics.

Monitoring & Follow-up

• Regularly assess the quality of chatbot interactions to maintain trust.

Risks

• Inaccurate or unclear responses may delay care or cause anxiety.

Patient & Prescribing Data

Caregivers of children experiencing early symptoms of illness.

AI chatbots can provide immediate guidance but must be evaluated for quality.

Clinical Best Practices

• Utilize both automatic and human evaluation methods for chatbot responses.
• Focus on response clarity and appropriateness to enhance user trust.
• Regularly update evaluation frameworks based on user feedback and clinical guidelines.

Related Resources & Content

• Meskó B, Topol EJ. The imperative for regulatory oversight of large language models in healthcare.
• Ayers JW, Poliak A, Dredze M, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum.
• Walker HL, Ghani S, Kuemmerli C, et al. Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument.