Clinical Scorecard: How Do Language and Context Influence the Accuracy of AI-Generated Health Advice?

At a Glance

Key Highlights

• Sensitivity to authorised health claims was highest in English and closely related European languages, and decreased as syntactic distance from English increased.
• In status-labelled UK and EU register extensions, models identified authorised claims but frequently classified non-authorised claims as supported.
• PUBHEALTH accuracy was lower and varied by topic and source; inaccuracy was greatest for general-health claims and claims associated with nature.com.
• English-language performance did not consistently generalise across languages or public-health contexts.
• The authors support multilingual, domain-specific evaluation before public-health deployment.

Guideline-Based Recommendations

Diagnosis

• Evaluate health-claim verification by language, topic, and information source rather than relying only on aggregate or English-language results.

Management

• Assess languages and information contexts relevant to intended use when evaluating health AI for public-health settings.

Monitoring & Follow-up

• Measure performance across languages and public-health contexts; the study reports variation by syntactic distance from English, topic, and source.

Risks

• In status-labelled UK and EU register extensions, models frequently classified non-authorised claims as supported.
• The source notes that falsehoods in prompts or upstream information sources may be replicated and amplified.
• English-language results did not consistently generalise across languages or public-health contexts.

Patient & Prescribing Data

No patient treatment cohort was studied; the analysis evaluated seven broadly accessible LLMs against health-claim registers and 9,088 journalist-vetted PUBHEALTH claims.

No treatment or prescribing intervention was evaluated. The study assessed binary classification of claims as supported or unsupported.

Clinical Best Practices

• The authors support multilingual, domain-specific evaluation before public-health deployment.
• Assess languages, topics, and information sources relevant to intended use.
• Distinguish authorised from non-authorised claims when evaluating performance.

Related Resources & Content