Objective:

To systematically evaluate the performance of five large language models in common question and answer scenarios related to pulmonary tuberculosis, focusing on readability, quality, and reliability.

Approach:

• Research Procedure: Twenty frequently asked questions about pulmonary tuberculosis were compiled and entered into five large language models. Their answers were evaluated for readability, reliability, and overall quality.
• Readability Evaluation: Various readability formulas were used to assess the generated text, including the Automated Readability Index, SMOG, Coleman-Liau Readability Index, and others.
• Quality Assessment: The Patient-education suitability (C-PEMAT-P) scale and the Global Quality Score (GQS) were utilized to evaluate the reliability of the answers.

Key Findings:

• The performance of the five large language models varied significantly in terms of readability and quality.
• Models demonstrated differences in language style, comprehensibility, and accuracy of citations.
• The study emphasizes the need for evaluating AI-generated health information for patient education.

Interpretation:

The findings suggest that while large language models can provide health information, their variability in quality and readability may affect user understanding and trust.

Limitations:

• The study did not involve human or animal experiments, which may limit the applicability of findings to real-world scenarios.
• The absence of an internationally accepted standard for readability assessment may affect the reliability of the results.

Conclusion:

This study establishes a basis for enhancing large language models in health education concerning pulmonary tuberculosis.