How Do Language and Context Influence the Accuracy of AI-Generated Health Advice?

1

The study tested 7 widely accessible large language models on UK and EU health-claim registers and 9088 journalist-vetted PUBHEALTH claims.
2

Sensitivity to authorised register claims was highest in English and closely related European languages and decreased as syntactic distance from English increased.
3

In status-labelled UK and EU register extensions, models often misclassified non-authorised claims as supported despite identifying authorised claims.
4

PUBHEALTH accuracy varied by topic and source, with the greatest inaccuracy for general-health claims and claims associated with nature.com.
5

The authors conclude that health-claim verification accuracy varies by language, topic and source, and recommend evaluating intended-use contexts rather than assuming English results generalise.

1