Assumptions and statistical inference

Le résumé fourni par la source

To the Editors: Epilepsia has published two comparative effectiveness articles in the past 12 months that utilize systematic review and meta-analysis methodology to assess long-term seizure frequency reduction in focal and generalized onset epilepsies treated with neuromodulation.1, 2 The statistical assumptions in these analyses inadvertently bias the results. Using the mean from Kawai et al.3 and assuming the population follows a normal distribution illustrates a prime example where the meta-analytic assumptions are not well satisfied. Haneef and Skrehot extracted a mean seizure frequency reduction from Kawai et al. of 43.4% (SD = 140.6%) for generalized seizures.1 For all focal and generalized seizures, Touma et al. extracted a mean seizure frequency reduction from Kawai et al. of 27.2% (SD = 141.8%).2 The wide SDs of seizure frequency change in these populations should prompt consideration of potential skewness, given seizure frequency reduction has a lower limit of 100%. A 40-point difference in percentage between the median and mean of these distributions deepens our concerns of a violation of meta-analytic assumptions of normality (in Kawai et al.,3 median reduction in generalized seizures = 83.3%, median reduction in all seizures = 66.24%). Although it is challenging to assess normality without patient-level data, the skewness in the data can be evaluated by comparing the mean (−43.4%) and SD (140.6%) against the minimum (−100%, −.40 SDs from the mean) and maximum (+1700%, +12.4 SDs from the mean). Having a “12-sigma” event in such a small sample size is statistically implausible if the data follow a normal distribution. Mean and median are not interchangeable measures of the central tendency of a distribution at the sample or population level, and should not be used as such except in specific situations where data symmetry is first verified (i.e., no skewness as in a normal distribution). Meta-analytic methods assume normality. Violation of the normality assumption has serious implications for the accuracy of the resulting statistical inference4 and should be acknowledged as a major limitation within the conclusion of these meta-analyses. Use of the mean and SD to represent individual patient outcomes following intervention underrepresents the benefit of the therapy where outcomes are heavily skewed by frequent, fluctuating, or difficult to count seizures. Differences in seizure types assessed, methods for recording seizures, and demographic differences between populations with specific epilepsy etiologies can be a key source of heterogeneity between clinical trials. This is also why it is important to compare trials of like designs during meta-analysis, whenever possible. Given the high heterogeneity of both meta-analyses, it seems that the included evidence or outcome measures do not meet rigorous meta-analysis guidelines from Cochrane. The relevance of clinical research relates to distilling the results for patient care and education. The most critical questions a patient may have are (1) will this work for me? and (2) how much benefit will I receive from therapy? We must ensure that the statistics we use to compare therapies are appropriate to answer these questions, and that the generalizability of a study's outcome is tempered when clear evidence of heterogeneity exists. Ultimately, we are each (readers, writers, and editors) responsible for critical analysis of literature before, during, and after peer review. We ask the authors of these and future meta-analyses to carefully consider the assumptions inherent to any statistical method, and minimally to address those limitations within their Discussion section or ideally to test the impact of violated assumptions with appropriate sensitivity analyses. None. All authors are employees of LivaNova, the manufacturer of the VNS Therapy System, and hold stock or stock options with the company.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.