Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication |
| |
Authors: | Robbie C M van Aert Marcel A L M van Assen |
| |
Institution: | 1.Department of Methodology and Statistics,Tilburg University,Tilburg,The Netherlands;2.Department of Sociology,Utrecht University,Utrecht,The Netherlands |
| |
Abstract: | The unrealistically high rate of positive results within psychology has increased the attention to replication research. However, researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study’s effect size is most probably overestimated because it is statistically significant, and this bias is not taken into consideration in traditional meta-analysis. We have developed a hybrid method that does take the statistical significance of an original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its statistical properties. By applying the hybrid method to data from the Reproducibility Project: Psychology (Open Science Collaboration, 2015), we demonstrate that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than those reported in the original study, and that some effects may even be absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and have developed a Web-based application (https://rvanaert.shinyapps.io/hybrid) for applying the hybrid method. |
| |
Keywords: | |
本文献已被 SpringerLink 等数据库收录! |
|