In-Hospital AI Systems for Trustworthy Clinical Decision Support
By
Li Zhang
Georg Wölflein
Dyke Ferber
Junhao Liang
Zunamys I. Carrero
Xuewei Wu
Julien Vibert
Jan Clusmann
Lino Möhrmann
Elena E. Möhrmann
Catharina Wichmann
Fabian Wolf
Tim Lenz
Jakob Nikolas Kather
September 15, 2026
Clinical Scorecard: In-Hospital AI Systems for Trustworthy Clinical Decision Support
At a Glance
Category Detail
Condition Clinical Decision Support Systems Key Mechanisms Operational and decisional trust in AI agents for clinical decision-making. Target Population Healthcare institutions utilizing AI for clinical decision support. Care Setting On-premise healthcare deployment.
Key Highlights
AI agents can engage in multi-turn clinical dialogues and generate diagnoses. Operational trust involves governance of data and models, while decisional trust relates to reliability of outputs. Confidence estimation and calibration are critical for safe AI deployment in clinical settings. A multi-perspective confidence framework was developed to assess reliability in AI outputs. On-premise AI models achieved competitive diagnostic performance compared to cloud-based models. Guideline-Based Recommendations
Diagnosis
Utilize AI agents to assist in generating diagnoses with accompanying rationales. Management
Implement governance frameworks to ensure safe deployment of AI systems. Monitoring & Follow-up
Continuously evaluate AI outputs for reliability and trustworthiness. Risks
Address potential biases and uncertainties in AI-generated recommendations. Patient & Prescribing Data
Patients in clinical settings where AI decision support is implemented.
AI systems can enhance diagnostic accuracy but require clinician oversight for uncertain outputs.
Clinical Best Practices
Establish clear governance protocols for AI deployment in healthcare. Differentiate between cases suitable for autonomous handling and those requiring clinician review. Integrate uncertainty estimation into AI evaluation processes. Related Resources & Content
