Objective:

To evaluate a modular human-AI collaboration framework for thematic analysis across qualitative health studies.

Approach:

• Framework Development: The study decomposes thematic analysis into three discrete subtasks to facilitate collaboration between human researchers and AI.
• Comparative Assessment: The framework is assessed across three qualitative health studies involving patients with interstitial lung disease, postural orthostatic tachycardia syndrome, and chronic obstructive pulmonary disease.

Key Findings:

• LLMs can support coding, theme generation, and synthesis in qualitative analysis when prompts are structured and outputs are reviewed by human researchers.
• Human oversight is crucial for interpreting AI-generated themes, especially for deeper cultural and emotional interpretations.
• There are significant limitations in LLM performance for context-dependent interpretation, with lower agreement rates for nuanced themes.

Interpretation:

The findings suggest that while LLMs can enhance qualitative workflows, their effectiveness is contingent on the dataset, task, and workflow structure.

Limitations:

• Many studies focus on a single model or dataset, complicating the understanding of model-specific versus workflow-specific effects.
• Heterogeneous evaluation practices make it difficult to draw consistent conclusions across studies.
• Critical concerns remain regarding the potential loss of clinically meaningful insights due to overcompression of data by models.

Conclusion:

The study highlights the potential of a human-AI collaborative framework in thematic analysis while emphasizing the need for careful oversight and methodological rigor.