AI Takes on More Independent Responsibilities in Health Care — The Monitor

By
Joel Luther
Hagere Yilma
Irving Washington
October 8, 2026

Kaiser Family Foundation

Overview

Federal programs are testing AI systems that interact directly with patients and perform tasks such as therapy, diagnosis, and treatment adjustment with less direct physician involvement. Accuracy concerns, device recalls, changing system outputs, and limited public trust complicate evaluation of these applications. Research described in the article also suggests that conversational presentation and perceived source authority can affect trust in health claims, including inaccurate claims.

Background

AI has been used in narrower clinical functions, including interpreting scans and test results, while federal pilots are exploring systems with broader responsibilities. These systems raise questions about how to assess safety and accuracy when they may make or influence clinical decisions with reduced direct physician involvement. The article reports that the FDA has not established evaluation rules for this newer category and is accepting public comments through October 19, 2026. Separately, research on health claims indicates that how information is presented or attributed may influence trust independently of its accuracy.

Data Highlights

Measure	Finding
Companies approved by CMS to test comparable AI tools	More than 200
AI tools permitted by FDA for use in Medicare patients outside its usual approval pathway	4
FDA-cleared AI devices reviewed through late 2024	950
Devices recalled and total recalls	60 devices; 182 recalls
Timing of recalls	Almost half occurred during the devices’ first year on the market
Respondents who said they would trust an AI-enabled app, website, or patient portal providing personalized health information	32%
Participants offered the option to verify chatbot responses who used it	Half

Key Findings

Federal pilots are evaluating AI for direct patient interaction, therapy delivery, diagnosis, and treatment recommendation or adjustment with reduced direct physician involvement; examples include a cognitive behavioral therapy chatbot and a federally funded heart-failure medication initiative subject to legally required physician oversight.
The New York Times reported that more than 200 companies had obtained CMS approval to test comparable tools, while the FDA had permitted four for use in Medicare patients outside its usual approval pathway.
In an August 2025 research letter in JAMA Health Forum, researchers reported that 60 of 950 FDA-cleared AI devices reviewed through late 2024 had been recalled 182 times; diagnostic or measurement errors accounted for most recalls, and almost half occurred in the first year on the market.
The FDA has said newer AI applications may be harder to assess because outputs can be unpredictable and change over time; the agency had not established evaluation rules for this category and was accepting public comments until October 19, 2026.
A September 2025 KFF poll found that 32% of respondents said they would trust an AI-enabled app, website, or patient portal to access medical records and provide personalized health information.
In an experiment described in the article, participants expressed greater trust in a more conversational chatbot, including for claims researchers characterized as “ridiculously wrong”; offering verification did not significantly change trust overall, and only half of those offered the option used it.

Clinical Implications

The article describes ongoing uncertainty about standards for evaluating AI systems that make or influence clinical decisions with reduced direct physician involvement, particularly when outputs may change over time. It also reports that conversational presentation and perceived authority can affect trust in health claims, including inaccurate information; clinicians and health systems should account for these reported concerns when considering such tools.

Conclusion

Federal pilots are extending AI evaluation from narrower support functions toward more autonomous clinical activities, while questions about accuracy, oversight, and public trust remain. The article reports that regulatory evaluation standards for this newer category have not yet been established.