Clinical Report: Evaluating LLMs for Ophthalmic Manuscript Consistency

Overview

In a study of 40 published ophthalmic manuscripts, four large language models detected 98 of 120 planted inconsistencies across model evaluations (81.7%). GPT-5.6 had the highest sensitivity (29/30, 96.7%). All detected errors were localised correctly, and 97/98 were explained and corrected accurately; systematic detection remained the principal limitation identified by the study authors.

Data Highlights

Key Findings

• Across 120 evaluations of manuscripts containing planted alterations, the models detected 98 errors (81.7%); between-model differences were significant (Cochran Q = 16.86, p < 0.001).
• GPT-5.6 detected 29/30 planted errors (96.7%) and Claude Sonnet 5 detected 27/30 (90.0%); both outperformed GPT-5.5 after Holm correction (p = 0.012 and p = 0.039, respectively).
• Detection was highest for sample-size/subgrouping inconsistencies (90.0%), followed by percentage/count discrepancies (82.5%) and statistical reporting discrepancies (72.5%).
• All 98 detected planted errors were localised correctly, and explanation and correction were correct for 97 (99.0%); complete end-to-end performance occurred in 97/120 assessments (80.8%).
• Of 355 additional error flags, 337 (94.9%) were adjudicated as genuine, corresponding to 158 unique pre-existing inconsistencies after duplicate detections were collapsed. Rounding discrepancies accounted for 99/134 additional genuine inconsistencies detected by GPT-5.6 (73.9%).
• Models sometimes missed planted errors while identifying other genuine inconsistencies in the same manuscript. The study authors identified systematic detection, rather than interpretation after detection, as the principal limitation.

Clinical Implications

The study authors proposed that LLMs could serve as an additional manuscript quality-control layer before submission and complement, rather than replace, conventional editorial assessment. The study evaluated manuscript consistency checking and did not assess clinical outcomes.

Conclusion

In this ophthalmic manuscript sample, LLMs frequently detected planted and pre-existing inconsistencies, and their explanations and corrections were usually accurate once an error was identified. Missed planted errors indicate that detection was not systematic.

Related Resources & Content

1. Study authors, supplied article context, year not provided — Evaluating Large Language Models for Detecting Internal Inconsistencies in Ophthalmic Manuscripts
2. ICMJE Recommendations: Artificial Intelligence
3. Benchmark evaluation of multi-modal large language models for ophthalmic diagnosis in real world
4. Performance of large language models for ophthalmic literature retrieval
5. Assessing the Use of Multimodal Large Language Models for Differentiating High Myopia from High Myopia Accompanied by Glaucoma
6. ICMJE | Recommendations | Manuscript Preparation and Submission
7. A large-scale randomized study of large language model feedback in peer review | Nature Machine Intelligence
8. CONSORT 2025 statement: updated guideline for reporting randomised trials