Clinical Scorecard: Enhancing Accuracy and Reliability of a GPT-4 Model in Emergency Radiology Diagnosis and Classification Through Retrieval-Augmented Generation: A Proof-of-Concept Investigation

At a Glance

Key Highlights

• Trauma radiology workload is increasing due to faster CT scans shifting evaluation to imaging assessment.
• Correct classification and grading of traumatic injuries guide treatment decisions between conservative and operative approaches.
• Retrieval-augmented generation (RAG) improves GPT-4 Turbo performance by integrating task-specific expert knowledge from curated trauma radiology literature.

Guideline-Based Recommendations

Diagnosis

• Use radiologic imaging findings from modalities such as radiography, CT, and MRI for initial trauma assessment.
• Apply GPT-4 Turbo enhanced with RAG to assist in diagnosing traumatic injuries based on imaging reports.
• Incorporate expert knowledge from curated trauma radiology literature to improve diagnostic accuracy.

Management

• Base treatment decisions on accurate classification and grading of injuries as determined by radiologic evaluation.
• Utilize AI-assisted tools like TraumaCB to support radiologists in complex classification tasks.

Monitoring & Follow-up

• Continuously update and curate external knowledge bases to maintain AI tool relevance and accuracy.
• Monitor AI outputs for potential hallucinations or inaccuracies, especially in complex or unclassifiable cases.

Risks

• Potential for AI hallucinations and inaccuracies due to limited or outdated training data.
• Lack of transparency in AI source data may affect accountability.
• Overreliance on AI without expert oversight could lead to misclassification.

Patient & Prescribing Data

Patients with traumatic injuries undergoing radiologic imaging

Accurate radiologic classification supported by AI tools can guide appropriate conservative versus operative treatment decisions.

Clinical Best Practices

• Use a two-step AI prompting approach mimicking clinical workflow: first diagnosis, then classification and grading.
• Incorporate curated, peer-reviewed trauma radiology literature into AI prompts via embedding and similarity matching.
• Limit AI creativity (e.g., low temperature setting) to reduce hallucinations and improve answer precision.
• Ensure AI outputs are reviewed by experienced radiologists before clinical application.

References

• OpenAI GPT-4 Turbo
• RadioGraphics Top Ten Reading List for Trauma Radiology
• Text Embedding Model (text-embedding-ada-002-v2)
• LlamaIndex Framework