Clinical Scorecard: The Impact of Erroneous AI Outcomes on Radiologists: Insights from a Multi-Reader Pilot Study on Lung Cancer Detection via Chest Radiography

At a Glance

Key Highlights

• AI diagnostic accuracy for lung nodules/cancer on chest imaging shows ROC-AUC around 0.87 with sensitivity 0.87 and specificity 0.89, implying false positives and false negatives are expected.
• Incorrect AI feedback can negatively influence radiologist decision-making, potentially leading to missed or incorrect diagnoses.
• Human factors such as whether AI results are retained in patient files and whether suspicious areas are visually outlined affect radiologist performance and can be optimized to mitigate AI errors.

Guideline-Based Recommendations

Diagnosis

• Radiologists should anticipate AI false positives and false negatives when interpreting AI-assisted chest radiographs.
• AI feedback should be critically evaluated and not solely relied upon for final diagnosis.

Management

• Implement AI systems with ROC-AUC values above 0.85 to approximate clinical practice standards.
• Optimize AI feedback presentation, including decisions on retaining AI results in patient files and use of visual outlines to highlight suspicious areas.

Monitoring & Follow-up

• Monitor radiologist performance with and without AI assistance to identify potential negative impacts of erroneous AI outputs.
• Separate reading sessions and blind radiologists to patient identifiers to reduce bias in AI-assisted interpretation.

Risks

• False negative AI results may reduce radiologist cancer detection rates.
• False positive AI results may lead to unnecessary follow-up imaging or interventions.
• Inappropriate AI feedback presentation can exacerbate diagnostic errors.

Patient & Prescribing Data

Patients undergoing chest radiography for lung cancer screening or diagnosis

AI systems improve radiologist performance overall but require careful implementation to avoid adverse effects from erroneous AI outputs.

Clinical Best Practices

• Use AI as an adjunct tool, maintaining radiologist critical judgment.
• Design AI feedback to minimize negative human factors, such as by controlling whether AI results are kept or deleted in patient records.
• Incorporate visual cues (e.g., boxes outlining suspicious areas) judiciously to support accurate interpretation.
• Conduct ongoing training and evaluation of radiologists on AI-assisted diagnostic workflows.

References

• FDA AI Device Approvals
• 2020 ACR Member Survey on AI Use
• Review of AI Diagnostic Imaging Accuracy
• Studies Showing Improved Radiologist Performance with AI
• Alberdi et al. Mammography CAD Study
• ACR Data Science Institute Goals
• Human Factors Psychology in Radiology