Assess whether an open-weight large language model deployed within a hospital network can identify documented pneumatic retinopexy (PnR) in ophthalmology electronic medical records.
Approach:
Design and records: Retrospective diagnostic accuracy study of records from two Australian hospitals, July 2023–June 2024. Candidate records were retrieved using “C3F8,” “SF6,” or “pneumatic.”
Reference standard and model: Two reviewers independently assessed longitudinal records, with specialist adjudication of equivocal cases. On-premises Qwen3-235B-A22B classified records using a two-stage zero-shot prompt.
Evaluation: Model performance was compared with the manual reference standard across 542 records.
Key Findings:
PnR was documented in 110 of 542 records (20.3%).
The model produced 108 true positives, 407 true negatives, 25 false positives, and 2 false negatives.
Sensitivity was 0.98 (95% CI, 0.94–0.99), specificity 0.94 (0.92–0.96), PPV 0.81 (0.74–0.87), and NPV 0.99 (0.98–0.99).
Among false positives, 14 involved pneumatic displacement for submacular haemorrhage and 11 involved gas tamponade associated with pars plana vitrectomy.
Interpretation:
The authors report that the model detected documented PnR with high sensitivity and NPV in this keyword-enriched cohort. They describe it as a potential case-finding tool for research and administrative workflows, with targeted human verification of positive classifications because PPV was 0.81.
Limitations:
Records were identified through keyword-enriched retrieval.
Only one model and prompt were assessed.
No rule-based comparator was included.
External multicentre validation and prospective workflow assessment were not performed.
Conclusion:
The authors favour human-supervised workflows rather than fully autonomous classification.
