To conduct a comparative evaluation of DeepSeek-R1 and GPT-4o using pediatric licensing exam practice questions to assess accuracy, consistency, and reasoning fidelity.
Approach:
Pediatric Questions Tested: 280 single-answer multiple-choice questions spanning 11 pediatric subspecialties categorized into three levels of clinical complexity.
Study Design: Each question was processed twice by both models with a 4-week interval, ensuring isolated sessions to minimize contextual influence.
Statistical Analysis: Accuracy rates were reported as frequencies and percentages, with chi-square tests for categorical comparisons and statistical significance defined as P ≤ 0.05.
Key Findings:
DeepSeek-R1 achieved higher per-run accuracy (90.7% and 86.8%) compared to GPT-4o (81.8% and 79.3%) in both runs (P = 0.003 and P = 0.02, respectively).
DeepSeek-R1 outperformed GPT-4o in aggregate accuracy (92.5% vs. 85.7%, P = 0.01) and consistent accuracy (85% vs. 75.4%, P < 0.05).
Both models showed comparable inter-run consistency (91.4% vs. 87.9%, P > 0.05).
Performance declined with increasing question complexity, with both models showing lower accuracy on more complex questions.
Interpretation:
Limitations:
The study was limited to a specific set of pediatric licensing exam questions and may not generalize to other medical domains.
The evaluation was conducted in isolated sessions, which may not reflect real-world usage scenarios.
