Assess how accurately four large language models detect and interpret internal inconsistencies in published ophthalmic manuscripts.
Approach:
Study sample: Forty ophthalmic articles from journals with impact factors above 3.0 were randomly selected. Researchers inserted one prespecified inconsistency into 30 manuscripts and left 10 unchanged.
Evaluation: Four LLMs independently assessed all articles using the same prompt. Researchers recorded detection, localisation, explanation, and correction of planted errors, and manually adjudicated additional error flags.
Analysis: Between-model differences in planted-error detection were assessed using Cochran’s Q and pairwise McNemar tests with Holm correction.
Key Findings:
The models detected 98 of 120 planted errors (81.7%). Detection sensitivity ranged from 63.3% for GPT-5.5 Instant to 96.7% for GPT-5.6 High Effort; overall differences were significant (Cochran Q = 16.86, p < 0.001).
GPT-5.6 and Claude Sonnet 5 detected planted errors more often than GPT-5.5 after Holm correction (p = 0.012 and p = 0.039, respectively).
Detection was highest for sample-size/subgrouping inconsistencies (90.0%), followed by percentage/count discrepancies (82.5%) and statistical reporting inconsistencies (72.5%).
All detected planted errors were localised correctly; explanations and corrections were correct for 97 of 98. Complete end-to-end performance occurred in 97 of 120 assessments (80.8%).
Of 355 additional error flags, 337 (94.9%) were adjudicated as genuine, representing 158 unique pre-existing inconsistencies. Rounding discrepancies accounted for 99 of 134 additional genuine inconsistencies detected by GPT-5.6.
Interpretation:
The authors report that performance differences were mainly in detecting errors rather than interpreting them. Models sometimes missed planted errors while identifying other genuine inconsistencies in the same manuscript, indicating that systematic detection remained a limitation. The authors describe LLMs as a complementary quality-control layer, not a substitute for conventional review.
Limitations:
The evaluation used 40 ophthalmic manuscripts and a defined set of planted inconsistency types; the context does not report testing in other fields or with other error categories.
The context does not provide a source URL or full bibliographic details.
