Background/Purpose: While qualitative research techniques provide valuable insights into patient experiences and perceptions, analyzing textual data from interviews can be time consuming and require multiple analysts for concept triangulation. Triangulation in qualitative data analysis minimizes potential biases that stem from relying on a single investigator. This approach enhances the credibility and validity of the findings by corroborating information from various perspectives and investigators. The capability of Large Language Model (LLM) technology to identify patterns and themes in textual data offers potential for conducting independent analysis of patients perceptions of care. This study explored a novel approach to triangulation in mixed-methods research (MMR) by employing ChatGPT-4, an LLM, for analyzing data from interviews of patients with advanced arthritis facing barriers to joint replacement.

Methods: We compared the thematic analysis and survey generation between investigators and ChatGPT-4, an AI-powered natural language processing tool that utilizes Generative Pre-trained Transformer (GPT), a type of large language model (LLM), as its core model. We used qualitative data from an existing qualitative study that explored patient perceptions of barriers to joint replacement utilization. Textual data from six semi-structured interviews were analyzed by human investigators and ChatGPT-4 independently. Human investigators used NVivo software to analyze the data and generate the codes and themes that emerged. Once the analysis was completed and it was determined that thematic saturation had been achieved, we proceeded to utilize ChatGPT-4 to directly generate themes from the interview transcripts. Both, human investigators and ChatGPT-4 independently, used the themes to create a survey. We compared the investigator-generated themes and survey with those generated by ChatGPT-4.

Results: ChatGPT-4 generated analogous dominant themes and a comprehensive corresponding survey as the human investigators but in significantly less time. (Table 1) Human-led thematic analysis produced 6 themes and 30 survey questions, taking significant time. ChatGPT-4 generated the same themes in under 45 minutes and created 17 survey questions in less than 1 minute. Themes generated by ChatGPT-4 covered the same topics as human-generated themes. However, ChatGPT-4’s survey questions were shorter and less specific compared to human-generated questions. The Figure 1 mixed-methods flowchart proposes integrating LLMs alongside human investigators as a supplementary tool for the preliminary thematic analysis of qualitative data and survey generation.

Conclusion: The potential of ChatGPT-4 as a tool for assisting in triangulation was evident in both the qualitative portion of mixed methods research (MMR) by assisting with thematic analysis, and in the quantitative portion of MMR by facilitating survey development. This integration may improve the overall workflow efficiency of MMR. Given the novelty of LLM technology and the limited research on its embedded biases, investigators must consider the ethical and qualitative implications when using LLMs for research purposes.

« Back to ACR Convergence 2023

ACR Meeting Abstracts - https://acrabstracts.org/abstract/a-novel-approach-for-mixed-methods-research-using-language-learning-models-a-report-using-patients-perspectives-on-barriers-to-hip-and-knee-replacement/