Clinical Scorecard: Advancing Real-Time Identification of Diagnostic Oversights in the Emergency Department Through the Use of Large Language Models

At a Glance

Key Highlights

• Overall prevalence of missed opportunities for diagnosis was 13.5%.
• Number needed to screen was 9.1 for 72-hour return and 5.4 for floor-to-ICU cohorts.
• Model discrimination AUCs ranged from 0.65 to 0.73 for 72-hour return.
• Physician interrater agreement was 81.9%, indicating variability in record reviews.
• LLMs can analyze unstructured clinical notes to detect missed diagnosis signals.

Guideline-Based Recommendations

Diagnosis

• Implement LLM-based screening tools for real-time diagnostic safety workflows.

Management

• Utilize a prescreening tool to triage eTriggers with high sensitivity and specificity.

Monitoring & Follow-up

• Monitor the sensitivity of LLMs as encounters progress and more information becomes available.

Risks

• Highly sensitive models may flag many non-missed opportunities, increasing review workload.

Patient & Prescribing Data

Patients presenting to the Emergency Department.

LLMs can assist in identifying high-risk patients for missed diagnoses.

Clinical Best Practices

• Incorporate LLMs into clinical workflows to enhance triage processes.
• Use LLMs to analyze vital signs, medical history, and clinical notes in real time.

Related Resources & Content

• Marks et al. Study