Objective:

To conduct a comparative study of four configurations of AI systems for postoperative discharge support, specifically evaluating their performance, quality, and reliability in domain-specific question-answering.

Approach:

• Study Design: A controlled comparative evaluation framework was used to assess four LLM configurations: a baseline model, a fine-tuned model, an RAG-based model, and a hybrid fine-tuning+RAG system.
• Dataset and Split Strategy: Two disjoint datasets were utilized: one for model development with 600 expert-authored QA pairs and another for final benchmarking with 150 patient-style queries.

Key Findings:

• Hybrid models combining fine-tuning and retrieval mechanisms showed promising results in postoperative decision support.
• Fine-tuning improves performance on narrow tasks but risks overwriting general knowledge.
• RAG enhances model performance by integrating external knowledge during inference, providing more accurate responses.

Interpretation:

The study outlines the strengths and limitations of each AI configuration in delivering accurate postoperative information.

Limitations:

• The practical constraints and interactions among the techniques remain poorly understood, which may affect the applicability of the findings.
• The study focused on postoperative information support tools rather than diagnostic or triage systems, limiting its scope.

Conclusion:

The findings offer insights into model configurations for patient-facing postoperative decision support, particularly in generating safe and accurate discharge instructions.