Objective:

To evaluate the performance and decision behavior of inference-only large language models (LLMs) compared to an outcome-trained machine learning model in predicting in-hospital mortality in patients with spontaneous intracerebral hemorrhage (SICH).

Approach:

• Study Design: A retrospective benchmarking study comparing an outcome-optimized machine learning model with an inference-only LLM using identical clinical inputs.
• Ethical Considerations: The study was approved by the Institutional Review Board of Chi Mei Medical Center, and informed consent was waived due to the retrospective design and use of deidentified data.

Key Findings:

• Inference-only LLMs may exhibit different operational characteristics compared to outcome-trained machine learning models.
• LLMs are more sensitive to prompting conditions.
• There is weaker alignment between language-based feature prioritization and empirically derived predictor importance in LLMs.

Interpretation:

The study proposes a conceptual framework distinguishing outcome-optimized predictive models from language-based inference systems, highlighting the differences in their operational characteristics.

Limitations:

• The study design does not fully capture all aspects of conventional prediction model development or validation.
• The findings are based on a retrospective analysis, which may limit generalizability.

Conclusion:

The study aims to provide insights into the reliability and interpretability of LLMs in clinical prediction tasks, particularly in high-stakes environments like ICU settings.