To develop and systematically evaluate a retrieval-augmented generation (RAG) question-answering system tailored for stroke-related queries.
Approach:
Study Design: Comparative experimental design evaluating LLM performance and usability under zero-shot and RAG settings.
Knowledge Base Construction: Compiled a localized stroke knowledge base from authoritative sources, including both domestic and international guidelines.
Evaluation Metrics: Assessed technical performance, clinical validity, and end-user usability using various metrics including exam questions and clinician-led validation.
Key Findings:
RAG technology improves accuracy and robustness of responses by retrieving pertinent information from external knowledge bases.
The study highlights trade-offs between performance enhancement and output understandability.
End-user usability was assessed using System Usability Scale (SUS) and Net Promoter Score (NPS) scales.
Interpretation:
The integration of RAG with LLMs shows potential to enhance the reliability of AI-assisted stroke care, addressing gaps in user experience and information accuracy.
Limitations:
The study did not involve clinical intervention or treatment modification.
Limited empirical validation involving end users such as patients or caregivers.
Conclusion:
This research provides a feasible, evidence-based solution for stroke management and lays groundwork for AI integration in chronic disease care.
