Clinical Report: Enhanced Knowledge Localization Improves Performance of Domestic Large Language Models in Chinese Radiation Oncology Assessments

Overview

This study benchmarks domestic and international large language models (LLMs) against a Chinese radiation oncologist using a national examination. Domestic models, particularly Qwen 3 Max, outperformed the human reference, highlighting the importance of localized knowledge in model performance.

Background

The integration of large language models in specialized medical fields like radiation oncology is critical as these models have shown promise in general medical assessments. However, their effectiveness in non-English contexts and specialized domains remains underexplored. Understanding how these models perform in localized settings can inform their development and application in clinical practice.

Data Highlights

Key Findings

• Qwen 3 Max achieved an accuracy of 86.30%, outperforming the single physician reference.
• International models like GPT-5 showed significant performance declines in localized knowledge retrieval.
• Translating the examination into English did not improve performance for international models.
• Majority of errors in international models stemmed from discrepancies between Western and Chinese clinical guidelines.
• Localized knowledge alignment is crucial for model performance in specialized medical assessments.

Clinical Implications

The findings suggest that domestic LLMs may be better suited for localized clinical assessments in radiation oncology. Clinicians should consider the limitations of international models when applying them in non-English contexts, particularly in specialized fields.

Conclusion

This study underscores the importance of localized knowledge in enhancing the performance of large language models in specialized medical assessments, particularly in radiation oncology.

Related Resources & Content

1. npj Digital Medicine, 2026 -- Assessment of Large Language Models for Generating Diagnostic Impressions from Brain MRI Reports
2. The ASCO Post, 2026 -- Large Language Models May Generate Concise, Coherent Pathology Summaries, Reducing Physician Burden
3. npj Digital Medicine, 2026 -- CancerLLM: a large language model in cancer domain
4. npj Digital Medicine, 2026 -- Collaboration Between Humans and Large Language Models in Clinical Practice: A Systematic Review and Meta-Analysis
5. ASTRO, 2025 -- ASTRO publishes first clinical guideline on radiation therapy for gastric cancer
6. ScienceDirect, 2025 -- Primary target volume delineation for radiotherapy in nasopharyngeal carcinoma: CSTRO, CACA, CSCO, HNCIG, ESTRO, and ASTRO guidelines and contouring atlas
7. ASTRO, 2025 -- Ten-year clinical trial report finds radiation comparable to surgery for early-stage non-small cell lung cancer
8. ASTRO publishes first clinical guideline on radiation therapy for gastric cancer - American Society for Radiation Oncology (ASTRO)
9. Primary target volume delineation for radiotherapy in nasopharyngeal carcinoma: CSTRO, CACA, CSCO, HNCIG, ESTRO, and ASTRO guidelines and contouring atlas - ScienceDirect
10. Ten-year clinical trial report finds radiation comparable to surgery for early-stage non-small cell lung cancer - American Society for Radiation Oncology (ASTRO)