RadCoT may improve LLM detection of report errors

The radiology-specific framework produced its largest gains in detecting interpretation and omission or insertion errors across 7 language models.

1

In 1,200 radiology reports, RadCoT increased mean micro-averaged F1 across 7 LLMs from 0.77 with standard prompting to 0.85.
2

RadCoT produced the largest F1 gains for interpretation errors, from 0.57 to 0.75, and omission or insertion errors, from 0.64 to 0.82.
3

With RadCoT, F1 gains were greater for CT and MRI reports than for radiography and ultrasound reports.
4

RadCoT increased inference time by 68%, and at an assumed 2% error prevalence, estimated positive predictive value was 0.15 with about 88 false-positive flags per 1,000 reports.
5

The authors described RadCoT as a first-pass QA assistant requiring human review; it evaluates report text and missed complex interpretation errors in about 1 in 4 cases.

1