Closing the screening gap but not the writing gap: a two-topic evaluation of LLMs for systematic reviews and meta-analyses in hepatology

Le résumé fourni par la source

Systematic reviews are essential but labor-intensive. We evaluated LLM-assisted literature screening and drafting in two hepatology topics: carvedilol in compensated cirrhosis and anticoagulation in portal vein thrombosis. For each topic, we searched PubMed, Cochrane, and EMBASE. A few-shot prompt with explicit inclusion/exclusion criteria was used to screen titles and abstracts, with results compared to manual review. Included studies were then processed using a retrieval-augmented LLM to generate ten automated systematic review and meta-analysis drafts per topic, which were evaluated by a separate judge LLM for PRISMA 2020 compliance against human reviews. Screening performance: After deduplication (703 and 370 records), LLM-assisted screening showed high agreement with manual review (sensitivity 86-93%, specificity 96-99%) while reducing screening time to 3 and 2 h versus 62 and 30 h manually. Drafting performance: RAG-enabled LLMs generated structured manuscripts with variable PRISMA 2020 compliance: 100% for titles, 91% for introductions, 75-80% for methods, and 68-75% for results, with downstream weaknesses in abstracts and discussions (<65%). LLM-based PRISMA scoring closely matched human review (ICC ≈ 0.90). LLM-assisted screening was highly accurate, reducing workload by >90%, but automated drafting was reliable mainly for titles and introductions, requiring human oversight to prevent errors and hallucinations.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.