Large Language Models to Optimize Data Abstraction for CIBMTR Reporting: A Pilot Study

Le résumé fourni par la source

Introduction Rigorous data reporting to the Center for International Blood and Marrow Transplant Research (CIBMTR) is mandated for all US transplant centers to ensure high quality standards and facilitate outcomes research. Currently, data abstraction largely relies on human efforts. At our own center, data abstraction for 2023 alone totaled 22,858 work hours. Large language models (LLMs) may offer a novel solution to ease this substantial burden. Methods We developed a pipeline to source raw data from the electronic medical record for downstream expert-guided data engineering, organize, mapping in a secure Google Cloud Platform environment. We assessed the “zero-shot” (i.e. without specific task training) extraction capability of 2 LLMs in completing the CIBMTR Post-Transplant Essential Data day 100 (TED-100) forms: (1) Gemma 2 and (2) secureGPT, a Stanford-developed and protected health information (PHI)-compliant GPT 4.0 interface. Using a cohort of 10 similar patients transplanted for acute myeloid leukemia , we compared LLM vs human completed forms. Results Across the cohort, we abstracted a total 1,665,806 data entries, including all lab values, pathology reports, and clinical notes from initial pre-HCT visit through day 100. Of 23 questions in the TED-100 form, Gemma 2 answered 61% and secureGPT 76% correctly. Of those, 50%, 3%, 4% for Gemma 2 and 70%, 5%, and 0% for secureGPT were fully correct, correct with excess correct information, and correct with missing information, respectively; 4% and 1% provided correct information with extra incorrect information. secureGPT was particularly effective in assessing neutrophil (90% accuracy) and platelet (100%) engraftment as well as development of aGVHD (90%), prophylaxis for liver toxicity (100%), development of post-HCT relapse (100%), and best response to infusion (100%). Gemma 2 and secureGPT answered 39% and 24% of questions incorrectly. Of these, 6 and 0% provided incorrect information taken from clinical documentation, 2 and 0% fabricated information, 10 and 0% stated ‘unknown’ when the correct answer was available, and 6 and 24% answered ‘no’ instead of ‘yes’ or vice versa. Common errors included engraftment dates, specific drugs, and use of post-HCT therapies. Both Gemma 2 and secureGPT were rapid, at 80 and 10 minutes per patient, compared to 2.5 hours for a trained data abstractor. Conclusions As transplant volume continues to grow and transplant itself becomes increasingly sophisticated, the burden of high-quality data abstraction will increase in tandem. To our knowledge, this is the first report on the zero-shot extraction capability of LLMs in abstracting transplant data for mandated reporting. While LLMs offer an imperfect solution as-is, our pipeline offers a PHI-compliant platform for rapid iteration, and our data a baseline for comparison of future prompt engineering, data mapping, and subsequent development.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.