Clinical Scorecard: Bringing Machine Learning–Based Delirium Risk Prediction Into Clinical Practice: Beyond Discrimination Metrics

At a Glance

Key Highlights

• The fixed model was implemented without retraining across 9,020 admissions at a second hospital; AUROC was 0.91 there versus 0.94 at the development hospital.
• The second hospital changed the high-risk threshold from 0.50 to 0.55 to align with available resources.
• At the second hospital, the number classified as high risk and receiving CAM evaluation increased from 266 (9.2%) before implementation to 515 (19.4%) afterward.
• At both hospitals, model use was associated with increased delirium detection and lower antipsychotic and benzodiazepine use.
• The authors report within-health-system transportability; they state that broader external validation is needed and that AUROC alone does not assess calibration or individual risk estimates.

Guideline-Based Recommendations

Diagnosis

• For subsequent evaluations, the authors recommend conducting CAM assessments uniformly in every eligible patient, regardless of model-assigned risk.
• For prospective studies, the authors recommend that a study member blinded to model risk stratification conduct CAM evaluations to limit incorporation and verification bias.

Management

• The authors suggest using risk stratification to inform targeting of delirium detection and multicomponent prevention, while accounting for available resources and workflow.
• The authors recommend assessing the balance between prevention benefits, team capacity, assessment burden, and resources required for actionable prevention.

Monitoring & Follow-up

• The authors recommend continued monitoring of model performance and adjustment of preventive actions as appropriate.
• During implementation, monitor positive and negative predictive values, assessment burden, and resources required for prevention.
• Evaluate calibration as well as discrimination; AUROC alone does not establish the accuracy of individual risk probabilities.

Risks

• External validation is needed in hospitals with different EHRs, documentation practices, staffing models, and patient populations.
• The authors note that risk estimates may be overestimated or underestimated, potentially resulting in resource waste or adverse patient consequences.
• Risk-directed CAM assessment and reliance on routine coding or documentation may miss delirium in low-risk patients or hypoactive delirium and may bias accuracy estimates.
• Sensitivity, specificity, positive predictive value, negative predictive value, and F1 scores could not be compared across the two sites because their thresholds differed.

Patient & Prescribing Data

The study compared 9,020 admissions at the second hospital with 21,186 admissions at the development hospital. The populations differed in age, racial and ethnic composition, medical versus surgical case mix, dementia prevalence, and length of stay.

At both hospitals, model use was associated with increased delirium detection and lower use of antipsychotics and benzodiazepines. The article does not provide medication dosing or patient-level prescribing recommendations.

Clinical Best Practices

• The authors distinguish within-system transportability from broad external generalizability and call for validation in differing clinical environments.
• The authors caution against relying on AUROC alone when evaluating prognostic use and recommend assessing calibration and predictive values.
• The authors recommend uniform, model-blinded CAM assessment in prospective evaluations to reduce workflow-related verification and incorporation bias.
• The authors recommend monitoring assessment workload and prevention resources alongside model performance, and adapting thresholds to local resource availability during implementation.

Related Resources & Content