To assess whether seven widely accessible large language models (LLMs) vary in health-claim verification accuracy by language, topic, and information source.
Approach:
Datasets: Models were tested against UK and EU health-claim registers translated into 21 languages and 9,088 journalist-vetted PUBHEALTH claims.
Evaluation: Across repeated runs, modal model responses classified claims as supported or unsupported. PUBHEALTH analyses examined topic and selected source domains.
Key Findings:
Sensitivity to authorised register claims was highest in English and closely related European languages, and declined as syntactic distance from English increased.
In status-labelled UK and EU register extensions, models identified authorised claims but frequently classified non-authorised claims as supported.
PUBHEALTH accuracy was lower and varied by topic and source; inaccuracy was greatest for general-health claims and claims associated with nature.com.
Interpretation:
The authors report that strong performance on authorised English-language register claims may obscure weaker performance in other languages and public-health settings. They identify training-data exposure, editorial framing, and topic-specific tuning as plausible contributors to disparities.
Limitations:
The study used selected registers and a journalist-vetted claim corpus; the datasets were not presented as a general-purpose benchmark.
PUBHEALTH analyses were limited to English-language claims.
The supplied context does not provide detailed model-specific results or quantitative accuracy estimates.
Conclusion:
LLM health-claim verification accuracy differed by language, topic, and source. The authors conclude that English-language results do not consistently generalise across contexts and support evaluating intended-use languages and domains.
