A study evaluated the appropriateness of ChatGPT-5 responses to acne-related questions, finding that only 65% of responses were classified as appropriate. The appropriateness varied significantly by question category, with the lowest ratings for diagnosis and general information.
Background
The use of artificial intelligence in healthcare, particularly in dermatology, is growing, yet its reliability for diagnosis and treatment remains uncertain. This study highlights the challenges faced by AI models like ChatGPT-5 in providing accurate information for acne management, a common dermatological condition.
Data Highlights
Category
Appropriateness (%)
Skin Care
83
Nutrition and Lifestyle
78
Treatment and Medications
50
Diagnosis and General Information
33
Key Findings
65% of ChatGPT-5 responses were classified as appropriate.
Appropriateness was highest for skin care (83%) and lowest for diagnosis (33%).
Misleading information was the most common reason for inappropriate ratings.
68% of questions met the success criteria of at least 2 of 3 appropriate responses.
Low inter-rater agreement among dermatologists indicated variability in response classification.
The study's findings may not generalize beyond acne due to its specific focus.
Clinical Implications
The findings suggest that AI responses should be validated by a qualified dermatologist.
Conclusion
The study highlights the limitations of AI in providing reliable dermatological advice, particularly in complex areas like diagnosis and treatment.
A large audit of biomedical publications suggests fabricated references are increasingly appearing in peer-reviewed papers — often in ways that are difficult for reviewers and readers to detect.
