Large Language Models for World Health Organization–Uppsala Monitoring Centre Drug–Adverse Event Causality Assessment Using Food and Drug Administration Adverse Event Reporting System Cases: Comparative Performance Study - Summary - MDSpire
Advertisement
Large Language Models for World Health Organization–Uppsala Monitoring Centre Drug–Adverse Event Causality Assessment Using Food and Drug Administration Adverse Event Reporting System Cases: Comparative Performance Study
To systematically evaluate the performance of large language models (LLMs) in drug-adverse event (AE) causality assessment using the WHO-UMC framework based on structured FAERS cases.
Approach:
Data Source and Case Construction: Utilized data from the FDA Adverse Event Reporting System (FAERS) to construct structured analytic cases for drug-AE causality assessment, focusing on cases with specific inclusion criteria.
Sample Size Determination: Determined sample size based on agreement analysis using κ statistics, aiming for a minimum of 292 drug-level assessments to ensure statistical power.
Case Selection: Selected cases involving 2 to 11 suspected drugs, stratified by drug count, and ensured representation of clinically important categories, including 'Certain' classifications.
Key Findings:
The study constructed a final dataset comprising 55 cases and 337 drug-level assessments.
No 'Certain' drug-AE pairs were identified in the initial sample, necessitating additional case retrieval for this classification to ensure comprehensive evaluation.
Interpretation:
Limitations:
The study relies on the quality and completeness of FAERS data, which can be heterogeneous.
Expert interpretation introduces subjectivity and variability in causality assessment.
Conclusion:
The evaluation framework developed in this study aims to rigorously assess LLM performance in drug-AE causality assessment, providing a structured approach to enhance the reliability of pharmacovigilance.
