The social and behavioural sciences have long grappled with replicability, but a recent large-scale crowd initiative including reveals that even when using identical data, the path to a conclusion is rarely singular. This study investigated 100 papers published between 2009 and 2018 across fields such as economics, political science, and psychology, tasking over 450 independent researchers with reanalysing the original datasets to test a specific claim from each article. The findings suggest that scientific results are frequently contingent on specific analytical choices, such as how variables are operationalised, how missing values are handled, or which statistical models and inference criteria are selected.
The results highlight a significant divergence between original reports and independent reanalyses. When comparing numerical outcomes, only 34% of reanalyses yielded a result within a narrow tolerance region (±0.05 Cohen’s d) of the original effect size. Even when expanding this tolerance region fourfold to ±0.20 Cohen’s d, only 57% of reanalyses matched the original result. While qualitative agreement was higher, with 74% of reanalyses reaching the same conclusion as the original authors, nearly a quarter of these independent attempts resulted in inconclusive findings, and 2% reported an effect in the opposite direction. Furthermore, the reanalyses showed a systemic trend toward smaller effect sizes than those originally published, with the mean effect size dropping from 0.73 in original studies to 0.49 in the reanalyses.
Notably, the study found that common assumptions about what makes research robust do not always hold. For instance, the self-reported expertise of the analysts did not correlate with the likelihood of producing a result that matched the original study, nor did the sample size of the underlying data. However, study design proved to be a critical factor. Experimental designs demonstrated 15–20% higher analytical robustness than observational ones, likely due to more constrained data collection and fewer researcher degrees of freedom.
These findings indicate that the standard practice of “single-shot” analyses, where a single team presents one analytical path, may offer a fragile basis for scientific claims. The researchers advocate for a shift toward systematic robustness testing, including multi-analyst and multiverse approaches, to better communicate the uncertainty inherent in data analysis. For both the academic community and journalists covering these fields, the study serves as a reminder that the evidence provided by a single analytical procedure is rarely the final word on a research question.
Link: Investigating the analytical robustness of the social and behavioural sciences
