Graph-based extractive versus zero-shot abstractive summarization of long clinical records: A sepsis case study

Le résumé fourni par la source

Objective Compare two methods, graph-based extractive and zero-shot abstractive, for summarizing long clinical records in Brazilian Portuguese, investigate the performance of multilingual abstractive large language model with an extractive method, use automatic and manual methods to evaluate clinical information retention and factual consistency. Methods Manually selected discharge summaries from 387 sepsis hospitalizations provided the physician-authored sepsis summaries used as reference texts against which generated summaries were evaluated. A graph-based extractive method used monolingual Brazilian Portuguese and multilingual embeddings to construct sentence similarity graphs. Summaries were generated by selecting the longest sentence in each Louvain community . The extractive method was compared with zero-shot abstractive approaches using Mistral-Small, Gemini, and Claude models. The evaluation employed ROUGE, BERTScore, BLEU, a clinical retention information entity recognition-based metric, and a manual review (n=90) of abstractive summaries for hallucinations. Results The zero-shot abstractive models generated shorter summaries, with median lengths ranging from 270 words for Mistral-Small to 629 words for Gemini but achieved lower automated metric scores than the extractive methods (median BERTScore F1: 0.755–0.779; median Clinical Information Retention: 13.6%–25.4%) and produced at least one hallucination in 11.1%–41.1% of the 90 summaries manually evaluated for each model. The graph-based extractive method (monolingual) produced longer summaries (median 706 words) with higher automated metric scores (BERTScore F1: 0.825, CIR: 39.3%) and direct traceability to the source text. Conclusion In this study, the evaluated zero-shot abstractive methods showed important reliability limitations when summarizing long clinical records. A monolingual graph-based extractive approach achieved higher retention of key medical entities.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.