AI Scribes Lag Clinicians on Note Quality

A VHA study across 11 vendors finds AI-generated primary care notes score lower than clinician-written notes, with the largest deficits in thoroughness, organization, and usefulness

By
Kerri Miller
April 17, 2026
6 min

Conexiant

Overview

A study evaluating AI scribe tools found that notes generated by these tools received lower quality scores compared to those written by clinicians across five standardized primary care scenarios. The findings highlight significant gaps in documentation quality, particularly in thoroughness and organization.

Background

The rapid adoption of ambient AI scribes in clinical settings aims to alleviate documentation burdens and improve efficiency. However, the quality of notes produced by these tools has not been thoroughly evaluated, raising concerns about their impact on clinical documentation and patient care. Understanding the limitations of AI-generated notes is crucial for ensuring effective integration into healthcare workflows.

Data Highlights

{'table': {'rows': [{'scenario': 'New Patient Visit', 'human_notes_avg_score': 'N/A', 'ai_notes_avg_score': 'N/A', 'statistical_significance': 'Not Significant'}, {'scenario': 'Pharmacy Consultation', 'human_notes_avg_score': 'N/A', 'ai_notes_avg_score': 'N/A', 'statistical_significance': 'Not Significant'}]}}

Key Findings

AI-generated notes scored lower than human-generated notes in all 10 quality domains assessed.
The largest deficits in AI notes were in thoroughness, organization, and usefulness.
Statistically significant differences were observed in three out of five scenarios evaluated.
Human notes averaged 43.8 points in the acute low back pain scenario, while AI notes averaged 20.3 points.
Differences in freedom from hallucination and bias were also statistically significant.

Clinical Implications

Clinicians should be aware of the limitations of AI-generated notes, particularly in terms of quality and thoroughness. As AI scribes become more integrated into clinical workflows, ongoing evaluation and oversight will be essential to ensure that documentation meets the necessary standards for patient care.

Conclusion

The findings underscore the need for careful consideration of AI scribe tools in clinical practice, emphasizing the importance of documentation quality alongside efficiency gains.