Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses

The issue of publication bias in psychological science is one that has remained difficult to address despite decades of discussion and debate. The current article examines a sample of 91 recent meta-analyses published in American Psychological Association and Association for Psychological Science journals and the methods used in these analyses to identify and control for publication bias. Of the 91 studies analyzed, 64 (70%) made some effort to analyze publication bias, and 26 (41%) reported finding evidence of bias. Approaches to controlling publication bias were heterogeneous among studies. Of these studies, 57 (63%) attempted to find unpublished studies to control for publication bias. Nonetheless, those studies that included unpublished studies were just as likely to find evidence for publication bias as those that did not. Furthermore, authors of meta-analyses themselves were overrepresented in unpublished studies acquired, as compared with published studies, suggesting that searches for unpublished studies may increase rather than decrease some sources of bias. A subset of 48 meta-analyses for which study sample sizes and effect sizes were available was further analyzed with a conservative and newly developed tandem procedure of assessing publication bias. Results indicated that publication bias was worrisome in about 25% of meta-analyses. Meta-analyses that included unpublished studies were more likely to show bias than those that did not, likely due to selection bias in unpublished literature searches. Sources of publication bias and implications for the use of meta-analysis are discussed.     

Publication bias in psychological science: comment on Ferguson and Brannick (2012)

It is well documented that studies reporting statistically significant results are more likely to be published than are studies reporting nonsignificant results--a phenomenon called publication bias. Publication bias in meta-analytic reviews should be identified and reduced when possible. Ferguson and Brannick (2012) argued that the inclusion of unpublished articles is ineffective and possibly counterproductive as a means of reducing publication bias in meta-analyses. We show how idiosyncratic choices on the part of Ferguson and Brannick led to an erroneous conclusion. We demonstrate that their key finding--that publication bias was more likely when unpublished studies were included--may be an artifact of the way they assessed publication bias. We also point out how the lack of transparency about key choices and the absence of information about critical features of Ferguson and Brannick's sample and procedures might have obscured readers' ability to assess the validity of their claims. Furthermore, we demonstrate that many of the claims they made are without empirical support, even though they could have tested these claims empirically, and that these claims may be misleading. With their claim that addressing publication bias introduces subjectivity and bias into meta-analysis, they ignored a large body of evidence showing that including unpublished studies that meet the inclusion criteria of a meta-analysis decreases (rather than increases) publication bias. Rather than exclude unpublished studies, we recommend that meta-analysts code study characteristics related to methodological quality (e.g., experimental vs. nonexperimental design) and test whether these factors influence the meta-analytic results.     

A replication of the RIRD strategy to decrease vocal stereotypy in a student with autism

This study was conducted to decrease vocal stereotypy in a student with autism. Results of functional analysis indicated that vocal stereotypy was maintained by automatic reinforcement. We used the Response Interruption and Redirection (RIRD; Ahearn, Clark, & McDonald, 2007 ) strategy to decrease the vocal behavior. An ABAB design was used. Results indicated that the RIRD intervention successfully reduced the target behavior. Results are discussed and implications for practice are provided. Copyright © 2009 John Wiley & Sons, Ltd.     

Assessing and treating vocal stereotypy in children with autism

Previous research implies that stereotypic behavior tends to be maintained by the sensory consequences produced by engaging in the response. Few investigations, however, have focused on vocal stereotypy. The current study examined the noncommunicative vocalizations of 4 children with an autism spectrum disorder. First, functional analyses were conducted in an attempt to identify the function of each child's behavior. For each of the participants, it was found that vocal stereotypy was likely not maintained by the social consequences. Following assessment, response interruption and redirection (RIRD) was implemented in an ABAB design to determine whether vocal stereotypy could be successfully redirected. RIRD involved a teacher issuing a series of vocal demands the child readily complied with during regular academic programming. Vocal demands were presented contingent on the occurrence of vocal stereotypy and were continuously presented until the child complied with three consecutively issued demands without emitting vocal stereotypy. For each child, RIRD produced levels of vocal stereotypy substantially lower than those observed in baseline. For 3 of the children, an increase in appropriate communication was also observed. The children's teachers were trained to implement RIRD. Brief follow-up probes and anecdotal information implied that the treatment had a positive impact in the natural environment.     

A Meta-Analysis of Research on Hindsight Bias

We conducted a meta-analysis of research on hindsight bias to gain an up-to-date summary of the overall strength of hindsight effects and to test hypotheses about potential moderators of hindsight distortion. A total of 95 studies (83 published and 12 unpublished) were included, and 252 independent effect sizes were coded for moderator variables in 3 broad categories involving characteristics of the study, of measurement, and of the experimental manipulation. When excluding missing effect sizes, the overall mean effect size was Md = .39 with a 95% confidence interval of .36 to .42. Five main findings emerged: (a) effect sizes calculated from objective probability estimates were larger than effect sizes calculated from subjective probability estimates; (b) effect sizes of studies that used almanac questions were larger than effect sizes of studies that used real-world events or case histories; (c) studies that included neutral outcomes resulted in larger effect sizes than studies that used positive or negative outcomes; (d) studies that included manipulations to increase hindsight bias resulted in significantly larger effect sizes than studies in which there were no manipulations to reduce or increase hindsight bias; and (e) studies that included manipulations to reduce hindsight bias did not produce lower effect sizes. These findings contribute to our understanding of hindsight bias by updating the state of knowledge, widening the range of known moderator variables, identifying factors that may activate different mediating processes, and highlighting critical gaps in the research literature.     

Decreasing motor stereotypy in leisure contexts: Analysis of stimulus control with response interruption and redirection

Noncontingent access to preferred stimuli and activities may decrease stereotypic behavior in individuals with autism or other developmental disorders. However, prompted engagement, response interruption and redirection (RIRD), and/or contingent access to stereotypy may be required to achieve clinically significant reductions in stereotypy. Antecedent stimuli may also be manipulated during intervention to establish stimulus control by signaling when stereotypy will receive programmed consequences. The purpose of the current project was to replicate prior research in this area by a) evaluating the effectiveness of RIRD on motor stereotypy in an adolescent with autism and b) providing contingent access to stereotypy as a function of successively longer intervals without stereotypic behavior. We also assessed the extent that these procedures gained acceptable stimulus control. Results from this project suggest RIRD and contingent access to stereotypy may effectively decrease motor stereotypy in leisure contexts, though consistent inhibitory stimulus control may be difficult to establish.     

Owning leads to valuing: Meta-analysis of the mere ownership effect

Mere ownership effect is the phenomenon that people tend to value what they own more than what they do not own. This classic effect is considered robust, yet effect sizes vary across studies, and the effect is often confused for or confounded with other classic phenomena, such as endowment or mere exposure effects. We conducted a pre-registered meta-analysis of 26 samples published before 2019 (N = 3024), which resulted in psychological ownership on valuing effect of g ∼ 0.57 [0.46, 0.69]. Suggestive moderator analyses supported the use of replica as the strongest moderators. Mere ownership effects were different from the null across all moderator categories and in most publication bias adjustments. We consider this as suggestive evidence that psychological owning leads to valuing, yet caution that much more research is needed. All materials, data, and codes are available on https://osf.io/fdyqw/ .     

Methods for addressing publication bias in school psychology journals: A descriptive review of meta-analyses from 1980 to 2019

Although meta-analyses are often used to inform practitioners and researchers, the resulting effect sizes can be artificially inflated due to publication bias. There are a number of methods to protect against, detect, and correct for publication bias. Currently, it is unknown to what extent scholars publishing meta-analyses within school psychology journals use these methods to address publication bias and whether more recently published meta-analyses more frequently utilize these methods. A historical review of every meta-analysis published to date within the most prominent school psychology journals (N = 10) revealed that 88 meta-analyses were published from 1980 to early 2019. Exactly half of them included grey literature, and 60% utilized methods to detect and correct for publication bias. The most common methods were visual analysis of a funnel plot, Orwin's failsafe N, Egger's regression, and the trim and fill procedure. None of these methods were used in more than 20% of the studies. About half of the studies incorporated one method, 20% incorporated two methods, 7% incorporated three methods, and none incorporated all four methods. These methods were most evident in studies published recently. Similar to other fields, the true estimates of effects from meta-analyses published in school psychology journals may not be available, and practitioners may be utilizing interventions that are, in fact, not as strong as believed. Practitioners, researchers employing meta-analysis techniques, education programs, and editors and peer reviewers in school psychology should continue to guard against publication bias using these methods.     

GENDER DIFFERENCES IN SELF-REPORTED POSTTRAUMATIC GROWTH: A META-ANALYSIS

A meta-analysis was conducted to examine the direction and magnitude of gender differences in self-reported posttraumatic growth. Results from 70 studies ( N  = 16,076) revealed a small to moderate gender difference ( g  = .27, 95% CI = .21 −.32), with women reporting more posttraumatic growth than men. Moderator analyses were then conducted to identify possible sources of these differences. The following moderators were examined: mean age of sample, measure used, nature of the stressful event, language of the measure, and type of sample (i.e., community samples, college students, or mixed). The only significant moderator was age, with women reporting incrementally more posttraumatic growth as the mean age of the sample increased ( B  = .004,  p  < .01,  SE  = .001,  Q  = 9.13). To check for publication bias, effect sizes were compared across published and unpublished research. The size of the gender difference was not significantly different between published ( g  = .30, 95% CI = .23 − .38) and unpublished ( g  = .22, 95% CI = .12 −.31) studies. The present findings indicate that modest, but reliable gender differences exist in posttraumatic growth even when unpublished data are included in the analyses. Possible explanations for these findings and suggestions for future research are discussed.     

Preliminary Evidence for the Effectiveness of Dialectical Behavior Therapy for Adolescents

This meta‐analysis evaluated the effectiveness of dialectical behavior therapy for adolescents in reducing symptoms of depression, anxiety, self‐injury, and suicide risk. Twelve between‐group studies (N = 834) were chosen that met the inclusion criteria. Results revealed small‐to‐medium Hedges's g effect sizes for all 4 symptoms compared with control and alternative treatment groups. However, the small number of effect sizes available for each analysis limited the generalizability of the findings. Implications and suggestions for future research are discussed.     

Intervention,stimulus control,and generalization effects of response interruption and redirection on motor stereotypy

This study evaluated the effects of response interruption and redirection (RIRD) on motor stereotypy (hand and body movements) in a child with neurodevelopmental disorders. We also assessed whether a stimulus paired with RIRD could acquire inhibitory control when tested during nonintervention conditions. Compared with baseline phases in a reversal design, RIRD decreased both hand and motor stereotypies, and there was evidence of stimulus control. Extending RIRD to a second setting was also effective. The care providers responsible for intervention implemented RIRD with fidelity, and they rated the procedure positively. Clinical and research implications from these findings are discussed.     

Five direct behavior rating multi-item scales: Sensitivity to the effects of classroom interventions

Direct Behavior Rating (DBR) is a tool designed for the assessment of behavioral changes over time. Unlike methods for summative evaluations, the development of progress monitoring tools requires evaluation of sensitivity to change. The present study aimed to evaluate this psychometric feature of five newly developed DBR Multi-Item Scales (DBR-MIS). Teachers identified students with behaviors interfering with their learning or the learning of others and implemented a Daily Report Card (DRC) intervention in the classroom settings for two months. The analyses were performed on 31 AB single case studies. Change metrics were calculated at an individual level by using Tau-U_{A vs. B + trend B} and Hedges' g and at a scale-level by using Mixed Effect Meta-Analysis, Hierarchical Linear Models (HLMs), and Between-Case Standardized Mean Difference (BC-SMD). HLMs were estimated considering both fixed and random effects of intervention and linear trend within the intervention phase. The results supported sensitivity to change for three DBR-MIS (i.e., Academic Engagement, Organizational Skills, and Disruptive Behavior), and the relative magnitudes were consistent across the metrics. Sensitivity to change of DBR-MIS Interpersonal Skills received moderate support. Conversely, empirical evidence was not provided for sensitivity to change of DBR-MIS Oppositional Behavior. Particular emphasis was placed on the intervention trend in that responses to behavioral interventions might occur gradually or require consistency over time in order to be observed by raters. Implications for the use of the new DBR-MIS in the context of progress monitoring of social-emotional behaviors are discussed.     

The Efficacy of Interventions Aimed at Reducing Procrastination: A Meta‐Analysis of Randomized Controlled Trials

This meta‐analysis investigated the efficacy of interventions aimed at reducing procrastination in randomized controlled trials. Twelve studies, with a total of 646 participants, met inclusion criteria. The significant meta‐analytic effect size, Hedges's g = 1.18, indicates that the interventions had a large positive effect. Three variables significantly moderated effect size: Higher effects were associated with interventions delivered in person, student samples, and a no‐treatment control condition. The results lay a foundation for procrastination treatment and future research.     

A general linear model for estimating effect size in the presence of publication bias

When the process of publication favors studies with smallp-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailedp-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.Authors' note: The contributions of the authors are considered equal, and the order of authorship was chosen to be reverse-alphabetical.     

Meta‐Analysis of Counseling Outcomes for the Treatment of Posttraumatic Stress Disorder

This meta‐analysis of 152 published posttraumatic stress disorder (PTSD) clinical trials from 1990 to 2012 concluded that counseling generally produced a small to large effect of treatment across all comparison conditions at termination (d+ = 0.30 to 0.89). These gains were maintained at longest follow‐up (d+ = 0.58 to 0.86) for the wait‐list, treatment‐as‐usual, and single‐group comparisons, but not for the follow‐up placebo comparison (d+ = 0.15), probably because of the low power (j = 3 placebo studies). Clinical trial findings were synthesized using a random‐effects model. No effects of publication bias or moderating variables were evident. No difference was found between trauma‐focused and non‐trauma‐focused approaches. Implications for counseling practice and future PTSD outcome research are addressed.     

A Bayesian approach to mitigation of publication bias

The reliability of published research findings in psychology has been a topic of rising concern. Publication bias, or treating positive findings differently from negative findings, is a contributing factor to this “crisis of confidence,” in that it likely inflates the number of false-positive effects in the literature. We demonstrate a Bayesian model averaging approach that takes into account the possibility of publication bias and allows for a better estimate of true underlying effect size. Accounting for the possibility of bias leads to a more conservative interpretation of published studies as well as meta-analyses. We provide mathematical details of the method and examples.     

Workaholism and work engagement: how are they similar? How are they different? A systematic review and meta-analysis

Workaholism and work engagement can be depicted, respectively, as the pathological and the healthy form of heavy work investment. In spite of their different definitions and outcomes on individual and organizational life, workaholism and work engagement are not clearly and adequately distinguished by scholars and researchers as they appear to show some overlapping features. The aim of this investigation was to meta-analyze available studies, selected by systematic review, on the relations between subdimensions of workaholism and work engagement. Thirty-five studies were eligible for analysis. Associations emerged between Working Excessively and Absorption (g = .34), Working Compulsively and Absorption (g = .28), and Working Excessively and Dedication (g = .14). Moreover, the results were not influenced by publication bias and showed nationality to be a significant moderator. Overall, these findings suggest that further research is necessary to extend our knowledge of workaholism, work engagement, and the relationship between the two, in order to disentangle commonalities and differences between them.     

The Interdisciplinarity of Collaborations in Cognitive Science

We introduce a new metric for interdisciplinarity, based on co‐author publication history. A published article that has co‐authors with quite different publication histories can be deemed relatively “interdisciplinary,” in that the article reflects a convergence of previous research in distinct sets of publication outlets. In recent work, we have shown that this interdisciplinarity metric can predict citations. Here, we show that the journal Cognitive Science tends to contain collaborations that are relatively high on this interdisciplinarity metric, at about the 80th percentile of all journals across both social and natural sciences. Following on Goldstone and Leydesdorff (2006), we describe how scientometric tools provide a valuable means of assessing the role of cognitive science in broader scientific work, and also as a tool to investigate teamwork and distributed cognition. We describe how data‐driven metrics of this kind may facilitate this exploration without relying upon rapidly changing discipline and topic keywords associated with publications.     

Guidelines Versus Practices in Cross-Lingual Assessment: A Disconcerting Disconnect

The International Test Commission's Guidelines for Translating and Adapting Tests (2010) provide important guidance on developing and evaluating tests for use across languages. These guidelines are widely applauded, but the degree to which they are followed in practice is unknown. The objective of this study was to perform a systematic review of published articles to determine whether the methodologies related to test adaptation and evaluation of cross-lingual measurement bias have improved since the initial publication of the Guidelines. Results demonstrated that although reporting reliability estimates and statistically analyzing construct equivalence improved since publication of the Guidelines, the majority of test development and validation practices in the published literature have not embraced the recommendations put forth by the International Test Commission. Overall, these findings call into question many of the inferences derived from cross-linguistic comparisons and clearly signal a need for better dissemination mechanisms of the Guidelines.