Blinded two-phase evaluation of large language models in complex cardiac surgery: task-specific performance and human-AI collaboration - Takeaways - MDSpire

Blinded two-phase evaluation of large language models in complex cardiac surgery: task-specific performance and human-AI collaboration

  • By

  • Marc Leon

  • Ruibin Feng

  • Manuel Quiroz Flores

  • Glenn Pelletier

  • Daniel Bethencourt

  • Masafumi Shibata

  • Hao He

  • Chawannuch Ruaengsri

  • May 29, 2026

  • 0 min

Share

  • 1

    A two-phase evaluation framework was developed to assess large language models (LLMs) and human-LLM collaboration in complex cardiac surgery.

  • 2

    Fifteen high-fidelity cardiac surgery scenarios were created by senior surgeons, each paired with a reasoning task and expert-curated reference answers.

  • 3

    LLM performance varied, with median normalized scores highest for O1 (0.896) and lowest for Llama3-OpenBioLLM-70B (0.521) across scenarios.

  • 4

    Second-round evaluations showed a decline in scores for four LLMs, with a notable percentage of ratings revised from affirmative to negative.

  • 5

    All models exhibited clinical limitations, particularly in complex reasoning tasks, indicating they are not yet ready for safe use in surgical settings.

Original Source(s)

Blinded two-phase evaluation of large language models in complex cardiac surgery: task-specific performance and human-AI collaboration

  • by Marc Leon, Ruibin Feng, Manuel Quiroz Flores, Glenn Pelletier, Daniel Bethencourt, Masafumi Shibata, Hao He, Chawannuch Ruaengsri

  • May 29, 2026

Related Content

7 Days Noninferior in Bacteremia

BALANCE compared 7 vs 14 days of treatment among hospitalized patients with bloodstream infections.

EULAR Updates Behçet’s Syndrome Recommendations

EULAR releases updated recommendations supporting earlier biologic use in patients with eye, vascular, and nervous system involvement

AI Scribes Lag Clinicians on Note Quality

A VHA study across 11 vendors finds AI-generated primary care notes score lower than clinician-written notes, with the largest deficits in thoroughness, organization, and usefulness