The Tip of the Iceberg? Insights into the Prevalence of Publication Bias in Two Probability-Based Academic Panels

Le résumé fourni par la source

Publication bias is the prioritized and selective reporting of significant results and often stems from the assumption that statistically significant research findings are rated as more important, credible, valuable, and publishable than nonsignificant or null findings. Previous work shows that this assumption makes researchers less likely to publish nonsignificant or null findings, a phenomenon termed file drawer bias. While prior research investigated the file drawer bias primarily at the study level (e.g., what share of studies has been published and which has not), we examine the prevalence of unpublished research on both the study and the hypothesis level (e.g. which hypotheses are published and which remain unpublished). We did so by comparing the submissions of 178 studies conducted in two probability-based German panel infrastructures (SOEP-IS and GESIS Panel) with their subsequent publications. We found substantial evidence for publication bias: after an average of more than six years of data availability, 43.8% of studies remained unpublished, 44.4% published at least one peer-reviewed article, and 11.8% published gray literature exclusively. Less than half of the published studies contained any submitted hypotheses, while the vast majority (>95%) included newly added (ad-hoc) hypotheses, with more than half of these studies publishing exclusively ad-hoc hypotheses. Taken together, over 73% of published hypotheses were supported and over 80% were ad-hoc, whereas more than two-thirds of submitted hypotheses remained unpublished. These findings highlight persistent incentives toward selective reporting and ad-hoc hypothesis formulation. Our results raise concerns about ongoing publication bias in the social, behavioral, and economic sciences.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.