Clinical Report: RadCoT may improve LLM detection of report errors

Overview

In a single-institution evaluation of 1,200 radiology reports, the six-step RadCoT framework increased mean micro-averaged F1 across seven large language models from 0.77 with standard prompting to 0.85. Gains were largest for interpretation and omission or insertion errors, while the authors cautioned that error-enriched testing, text-only inputs, and added processing time limit conclusions about routine clinical use.

Background

Radiology report proofreading is a time-consuming part of the workflow: preliminary reports are reviewed and finalized by attending radiologists, and workload, high-pressure environments, and speech-recognition errors can contribute to the burden. Prior work has examined LLMs as supplementary proofreading tools, but false-positive flags and low positive predictive value in settings with low error prevalence remain concerns. RadCoT was designed to structure review around radiology-specific checks, including consistency between report sections and clinical completeness. The study evaluated whether this framework improved error detection compared with standard prompting.

Data Highlights

Key Findings

• The evaluation used 1,200 reports from four modalities: 1,170 clinician-validated errors mapped to 900 error-containing reports, plus 300 error-free reports.
• Across seven commercial and open-source models, RadCoT increased mean micro-averaged F1 from 0.77 to 0.85, with mean precision rising from 0.80 to 0.88 and recall from 0.75 to 0.83.
• RadCoT’s largest category-level gains were for interpretation errors (F1 0.57 to 0.75), omission or insertion errors (0.64 to 0.82), and findings-impression discrepancies (0.69 to 0.81); gains were smaller for numerical and typographical errors.
• F1 gains were greater for CT and MRI than for radiography and ultrasound. The authors noted that cross-referencing and cross-sectional verification may have contributed, so the difference cannot be attributed to report complexity alone.
• With RadCoT, GPT-4o reached an F1 of 0.93 and Claude-3.5 Sonnet 0.92. Open-source models also improved, reaching 0.90 for Qwen-2.5-72B and 0.89 for Llama-3.3-70B.
• In comparisons with generic chain-of-thought prompting, RadCoT increased F1 from 0.90 to 0.93 for GPT-4o and from 0.86 to 0.89 for Llama-3.3-70B. The study was single-institutional and text-only; the authors called for prospective, multi-institutional evaluation.

Clinical Implications

The authors stated that RadCoT’s added processing time may be compatible with retrospective auditing or review before sign-off, but less suitable for real-time dictation. They also cautioned that the error-enriched evaluation may overestimate operational utility: at an assumed 2% error prevalence, modeled PPV was 0.15, with approximately 88 false-positive flags per 1,000 reports.

Conclusion

In this dataset, radiology-specific structured prompting improved LLM detection of report errors, particularly errors requiring higher-level reasoning. The authors noted that prospective, multi-institutional studies are needed to establish performance in real-world clinical settings.

Related Resources & Content

1. European Radiology Experimental, 2026 — RadCoT: a Radiological Chain-of-Thought framework for enhanced error detection in radiology reports
2. European Radiology, 2025 — Evaluating Large Language Models for Identifying Errors in Radiology Reports: A Comparison of Proprietary and Privacy-Conscious Open-Source Approaches
3. JMIR Medical Informatics, 2026 — Improving Radiology Report Error Detection Using a Multipass Large Language Model: Framework Development and Validation
4. Radiology, 2026 — Guidelines for Reporting Studies on Large Language Models in Radiology: An International Delphi Expert Survey
5. ASCO Publications — Combining Homologous Recombination-Deficient Testing and Functional RAD51 Analysis Enhances the Prediction of Poly(ADP-Ribose) Polymerase Inhibitor Sensitivity
6. European Radiology — GPT-4.1 and Llama 3.3 70 fail to detect clinically relevant errors in radiology reports in zero-shot evaluation
7. RadCoT: a Radiological Chain-of-Thought framework for enhanced error detection in radiology reports
8. Guidelines for Reporting Studies on Large Language Models in Radiology: An International Delphi Expert Survey