Evaluate whether a radiology-specific prompting framework improves large language models’ detection of errors in radiology reports.
Approach:
Dataset: Researchers assessed 1,200 reports from a single-hospital quality-assurance repository: 900 reports containing 1,170 clinician-validated errors and 300 error-free reports. Reports covered radiography, ultrasound, CT, and MRI.
Comparison: Seven commercial and open-source LLMs were tested with standard prompting and the six-step Radiological Chain-of-Thought (RadCoT) framework. Outcomes included error-level micro-averaged precision, recall, and F1, as well as performance by error type and modali…
Key Findings:
Across seven models, mean micro-averaged F1 increased from 0.77 with standard prompting to 0.85 with RadCoT; mean precision rose from 0.80 to 0.88 and recall from 0.75 to 0.83.
The largest gains were for interpretation errors (F1 0.57 to 0.75), omission or insertion errors (0.64 to 0.82), and findings-impression discrepancies (0.69 to 0.81). Gains were smaller for numerical and typographical errors.
F1 improved more for CT (0.68 to 0.80) and MRI (0.66 to 0.78) than for radiography (0.82 to 0.88) or ultrasound (0.77 to 0.83).
With RadCoT, GPT-4o reached an F1 of 0.93 and Claude-3.5 Sonnet 0.92. Open-source models also improved, reaching 0.90 for Qwen-2.5-72B and 0.89 for Llama-3.3-70B.
RadCoT increased inference time by 68%. GPT-4o processing time rose from about 15 to 22 seconds per report; open-source models took about 10 to 13 seconds per report with RadCoT.
At an assumed 2% error prevalence, modeled positive predictive value was 0.15, with about 88 false-positive flags per 1,000 reports; at 20% prevalence, estimated positive predictive value was 0.68.
Interpretation:
The authors describe RadCoT as a potential first-pass quality-assurance aid, not a replacement for human review. They note that the error-enriched evaluation dataset may overestimate utility in routine settings and that complex interpretation errors were still missed in approximately one in four cases.
Limitations:
Single-institution study with a relatively modest error sample, potentially limiting generalizability and representation of rare or nuanced errors.
Evaluation used report text only; the models could not detect perceptual misses on images or independently verify diagnostic correctness against the examination.
Explicit chain-of-thought output may not faithfully represent a model’s internal computation.
The authors called for prospective, multi-institutional studies to assess real-world performance.
Conclusion:
The study found improved report-error detection with RadCoT across evaluated models, alongside increased processing time. The authors positioned the framework as a complementary tool requiring human review.
A structured overview of recent FDA recalls, corrections, and alerts involving medications, ventilators, insulin delivery systems, cardiovascular devices, anesthesia products, and other equipment used in clinical practice.
