Performance of an Open-Source AI Model for Identifying Chest Infections on After-Hours Emergency Department Radiographs in Singapore

By
Ying Jie Toh
Samantha Ying Ying Tan
Chin Wah Eng
Boon Kee Lee
Praveen M Yogendra
Min On Tan
October 13, 2026

Bmj Health & Care Informatics

Overview

In a retrospective evaluation of 1,006 after-hours emergency department chest radiographs in Singapore, CheXNeXt showed high specificity but limited sensitivity at published thresholds for consolidation, pneumonia, and pleural effusion. Youden index threshold optimisation increased sensitivity for all three findings while retaining specificity of 0.765–0.881; the authors state that external validation and real-world integration studies are needed.

Background

Chest radiographs are frequently used in emergency departments to assess thoracic conditions, including suspected pulmonary infection. At the study institution, limited night-shift and weekend radiologist coverage meant emergency clinicians interpreted images after hours, with formal reports delayed until later. Although AI systems have been studied for radiograph interpretation, the article notes that many are proprietary and costly, while open-source tools are less often evaluated in actual after-hours emergency settings. The American College of Radiology’s 2024 appropriateness criteria update includes acute respiratory illness and identifies chest radiography as usually appropriate in commonly encountered scenarios.

Data Highlights

Finding	Published-threshold sensitivity	Published-threshold specificity	PPV	NPV	AUC	Optimised sensitivity	Optimised specificity
Consolidation	0.234	0.990	0.846	0.849	0.900	0.894	0.765
Pneumonia	0.176	0.995	0.889	0.845	0.856	0.681	0.881
Pleural effusion	0.622	0.949	0.656	0.942	0.916	0.852	0.848

Study sample: 1,006 anonymised after-hours ED chest radiographs. Radiologists’ reports were the reference standard; optimised values used Youden index thresholds.

Key Findings

The retrospective study assessed CheXNeXt on 1,006 anonymised after-hours ED chest radiographs from a Singapore hospital, using radiologists’ reports as the reference standard.
At published thresholds, specificity was 0.990 for consolidation, 0.995 for pneumonia, and 0.949 for pleural effusion, while sensitivity was 0.234, 0.176, and 0.622, respectively.
At published thresholds, AUC was 0.900 for consolidation, 0.856 for pneumonia, and 0.916 for pleural effusion.
After Youden index threshold optimisation, sensitivity increased to 0.894 for consolidation, 0.681 for pneumonia, and 0.852 for pleural effusion; corresponding specificity was 0.765, 0.881, and 0.848.
The authors identify single-radiologist reporting and use of radiographic findings as a surrogate for infection as study limitations.
The article describes open-source AI as a potential support for triage and reporting workflows where radiologist coverage is limited, while calling for external validation and real-world integration studies to assess generalisability and clinical impact.

Clinical Implications

The study authors report limited sensitivity at published thresholds and higher sensitivity with lower specificity after Youden index threshold optimisation. They describe calibrated open-source AI as a potential support for triage and reporting workflows where radiologist coverage is limited, and call for external validation and real-world integration studies to assess generalisability and clinical impact.

Conclusion

At published thresholds, CheXNeXt showed high specificity and limited sensitivity for the evaluated radiographic findings. Youden index threshold optimisation increased sensitivity and reduced specificity for all three findings. The authors call for external validation and real-world integration studies to assess generalisability and clinical impact.