Certified large language model-based diagnostic decision support in rheumatology: the ALLIANCE multicentre randomised controlled trial

Le résumé fourni par la source

Abstract Objectives To evaluate whether access to a certified large language model (LLM)-based clinical decision support system improves physician diagnostic performance in rheumatology compared with conventional diagnostic resources alone. Methods In this multicentre, open-label, randomised controlled trial, 82 physicians from seven hospitals in two countries were randomised 1:1 to conventional diagnostic resources plus Prof. Valmed or conventional resources alone. Participants assessed three rheumatology vignettes before and after assistance. The primary outcome was top-1 diagnostic accuracy. Secondary outcomes included top-3 accuracy, diagnostic reasoning, confidence, case-processing time and perceived support quality. Results Top-1 accuracy increased from 22.2% to 33.3% in the intervention group and from 23.3% to 35.0% in the control group, with no between-group difference in improvement (adjusted OR 0.99, 95% CI 0.45 to 2.19; p=0.979). Differences in top-3 accuracy, diagnostic reasoning and confidence were also not significant. Assisted case-processing time was substantially shorter with LLM support (94 vs 206 s; adjusted mean difference −112 s, 95% CI −141 to −83; p<0.001). Information timeliness and perceived diagnostic support quality were rated significantly higher in the intervention group. Exploratory analyses showed persistent overconfidence and substantial AI over-reliance. Conclusions Certified LLM-based diagnostic support did not improve diagnostic accuracy compared with conventional resources, but substantially reduced case-processing time and improved perceived support quality. These findings suggest potential workflow benefits while highlighting overconfidence and over-reliance as important safety considerations. Trial registration number NCT07166692 WHAT IS ALREADY KNOWN ON THIS TOPIC Accurate and timely diagnosis in rheumatology is challenging because workforce shortages coincide with non-specific, complex and often rare presentations. Large language models are increasingly used for diagnostic decision support and have shown strong performance in benchmarking studies. Randomised evidence for physician-facing diagnostic decision support, particularly in rheumatology and for certified LLM-based systems, is lacking. WHAT THIS STUDY ADDS The ALLIANCE trial is the first randomised controlled trial to evaluate physician-facing diagnostic decision support in rheumatology and the first to assess a medically certified LLM-based clinical decision support system in any clinical specialty. A certified LLM-based clinical decision support system did not improve top-1 diagnostic accuracy compared with conventional resources alone, as assistance led to similar gains in both groups. Use of a certified LLM-based clinical decision support system was associated with approximately half the assisted case-processing time of conventional resources alone, as well as higher perceived support quality. HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY LLM-based diagnostic decision support may be most useful for improving efficiency, support quality, and broadening differential diagnoses rather than increasing top-1 diagnostic accuracy. Safe clinical implementation will require careful attention to overconfidence, over-reliance, transparency, and trust. Further robust trials, ideally under real-world clinical conditions, are needed to define the role of clinical decision support in routine care.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.