Assess whether a fixed multimodal machine-learning model for delirium risk could be implemented at a second hospital without retraining, and describe its performance and implementation-associated outcomes.
Approach:
Design and setting: The identical model was deployed across 9,020 admissions at Mount Sinai Morningside Hospital and compared with results from 21,186 admissions at the development hospital.
Implementation: The model was not retrained; the high-risk threshold was adjusted from 0.50 to 0.55 to fit local resources. Delirium was assessed using CAM evaluations or, when unavailable, diagnostic coding and documentation.
Key Findings:
AUROC was 0.94 (95% CI, 0.93–0.94) at the development hospital and 0.91 (95% CI, 0.90–0.92) at the second hospital.
At the second hospital, patients classified as high risk and receiving CAM evaluations increased from 266 (9.2%) before implementation to 515 (19.4%) afterward.
At both hospitals, model use was associated with increased delirium detection and lower antipsychotic and benzodiazepine use.
Interpretation:
The findings support transportability within one health system, but do not establish broad generalizability or the accuracy of individual risk probabilities. AUROC alone does not assess calibration or the balance between assessment burden, prevention resources, and potential consequences.
Limitations:
Both hospitals shared an EHR, data definitions, documentation practices, implementation tools, and a specialized delirium-service model; validation in different settings remains necessary.
CAM assessment was not uniform: at the second hospital, 2,649 of 9,020 admissions received CAM evaluations, while delirium status for the remainder relied on coding or documentation.
Because model risk influenced who received CAM assessment, differential verification and workflow-related incorporation bias may have affected observed accuracy.
Different risk thresholds across hospitals prevent direct comparison of sensitivity, specificity, predictive values, and F1 scores.
Conclusion:
The study advances evaluation of delirium risk prediction in routine practice. The article calls for independent validation, uniform CAM assessment in prospective studies, and monitoring of predictive values, assessment burden, and resources needed for prevention.
