The article examines the objective, findings, and clinical context described in Authors’ Response: Understanding Agreement in AI-Assisted American Society of Anesthesiologists (ASA) Classification.
Approach:
Agreement measures: The authors clarified that the original analysis used unweighted Cohen κ for exact agreement and added linearly weighted κ to account for the ordinal ASA categories.
Disagreement patterns: They examined patient-level cross-tabulations of AI and clinician classifications with and without NHI data.
Bootstrap analysis: They used 10,000 paired patient-level bootstrap replicates, sampling 600 patients with replacement and recalculating unweighted κ and its between-condition difference.
Key Findings:
Weighted κ was 0.893 with NHI data and 0.560 without NHI data.
Unweighted κ was 0.883 (95% CI 0.841–0.946) with NHI data and 0.518 (95% CI 0.412–0.691) without it.
The unweighted κ difference was 0.365, with a paired-bootstrap 95% CI of 0.292–0.440.
All 9 disagreements among 63 patients with NHI data were between adjacent categories. Without NHI data, 16 of 17 disagreements were adjacent; one clinician-assigned ASA IV patient was classified as ASA II.
Interpretation:
The supplemental paired-bootstrap interval excludes zero and supports higher overall agreement when NHI data were included. The authors note that adjacent-category disagreements may still be clinically relevant.
Limitations:
The large proportion of ASA II patients and small ASA III-to-IV subgroups limit interpretation by ASA category; neither κ measure establishes comparable performance across categories.
Comparisons with earlier interrater research require caution because rater configuration, clinical environment, case mix, and κ weighting vary.
Conclusion:
The authors provided supplemental analyses to clarify agreement, disagreement patterns, and uncertainty in the NHI-data comparison.
