Objective:

To evaluate the added value of customized large language models (LLMs) for data extraction and QUIPS-based appraisal in complex rheumatology systematic reviews.

Approach:

• Study Design: A retrospective, two-part comparative methodological study was conducted, comparing customized GPT-based LLMs against human reviewers.
• Data Extraction Component: Included 15 English-language full-text articles from a systematic review on metabolomics in systemic lupus erythematosus.
• QUIPS Component: Included 19 full-text studies from a prognostic review with QUIPS assessments completed by trained reviewers.

Key Findings:

• Customized GPTs demonstrated potential for streamlining systematic review workflows.
• Human reviewers served as the reference standard for evaluating AI-generated assessments.
• The study highlighted the need for domain-specific benchmarking of LLMs in rheumatology.

Interpretation:

The findings suggest that while customized LLMs can assist in systematic reviews, they may not fully replace human reviewers, emphasizing the importance of human oversight.

Limitations:

• The study did not include patient-level data.
• Only a limited number of articles were evaluated, which may affect generalizability.

Conclusion:

Customized GPT models may complement existing systematic review platforms by enhancing data extraction and risk-of-bias appraisal processes.