External validation of existing dementia prediction models on observational health data

Le résumé fourni par la source

Abstract BackgroundMany dementia prediction models have been developed, but only few have been externally validated, which hinders clinical uptake and may pose a risk if models are applied to actual patients regardless. Replicating and externally validating a prediction model is a difficult task, where we mostly rely on the completeness of model reporting in a published article. In this study, we aim to externally validate existing dementia prediction models. To that end, we define replicability criteria, review published models, and externally validate three selected models using routinely collected health data from administrative claims and electronic health records.MethodsWe identified dementia prediction models that were developed between 2011 – 2020 and assessed if they could be replicated given a set of external validation criteria. In addition, we replicated three of these models (Walters’ Dementia Risk Score, Mehta’s RxDx-Dementia Risk Index, and Nori’s ADRD dementia prediction model) and externally validated them on a network of six observational health databases from the United States, United Kingdom, Germany and the Netherlands, including the original development databases of the models.ResultsWe reviewed 59 dementia prediction models. All models reported the prediction method, development database, and target and outcome definitions. Less frequently reported by these 59 prediction models were predictor definitions (46 models) including the time window in which a predictor is assessed (21 models), predictor coefficients (19 models), and the time-at-risk (39 models). The replicated model by Walters (development c-statistic: 0.84) showed moderate transportability (0.67 – 0.76 c-statistic). The Mehta model (development c-statistic: 0.81 ) transported well to some of the external databases (0.69 – 0.79 c-statistic). The Nori model (development AUROC: 0.69) transported well (0.62 – 0.68 AUROC), but performed modestly overall. Recalibration showed improvements for the Walters and Nori models, while recalibration could not be assessed for the Mehta model due to unreported baseline hazard.ConclusionWe observed that reporting is mostly insufficient to fully replicate and externally validate published dementia prediction models, and therefore, it is uncertain how well these models would work in other clinical settings. We emphasise the importance of following established guidelines for reporting clinical prediction model. We recommend that reporting should be more explicit and have external validation in mind if the model is meant to be applied in different settings.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.