Clarifications on the application and interpretation of the test for excess significance and its extensions

This commentary discusses challenges in the application of the test for excess significance (Ioannidis & Trikalinos, 2007) including the definition of the body of evidence, the plausible effect size for power calculations and the threshold of statistical significance. Interpretation should be cautious, given that it is not possible to separate different mechanisms of bias (classic publication bias, selective analysis, and fabrication) that lead to an excess of significance and in some fields significance-related biases may follow a complex pattern (e.g. Proteus phenomenon and occasional preference for “negative” results). Likelihood ratio estimates can be used to generate the post-test probability of bias, and correcting effect estimates for bias is possible in theory, but may not necessarily be reliable.