Publication bias and the failure of replication in experimental psychology

Replication of empirical findings plays a fundamental role in science. Among experimental psychologists, successful replication enhances belief in a finding, while a failure to replicate is often interpreted to mean that one of the experiments is flawed. This view is wrong. Because experimental psychology uses statistics, empirical findings should appear with predictable probabilities. In a misguided effort to demonstrate successful replication of empirical findings and avoid failures to replicate, experimental psychologists sometimes report too many positive results. Rather than strengthen confidence in an effect, too much successful replication actually indicates publication bias, which invalidates entire sets of experimental findings. Researchers cannot judge the validity of a set of biased experiments because the experiment set may consist entirely of type I errors. This article shows how an investigation of the effect sizes from reported experiments can test for publication bias by looking for too much successful replication. Simulated experiments demonstrate that the publication bias test is able to discriminate biased experiment sets from unbiased experiment sets, but it is conservative about reporting bias. The test is then applied to several studies of prominent phenomena that highlight how publication bias contaminates some findings in experimental psychology. Additional simulated experiments demonstrate that using Bayesian methods of data analysis can reduce (and in some cases, eliminate) the occurrence of publication bias. Such methods should be part of a systematic process to remove publication bias from experimental psychology and reinstate the important role of replication as a final arbiter of scientific findings.     

Trying to be happier really can work: Two experimental studies

Does the explicit attempt to be happier facilitate or obstruct the actual experience of happiness? Two experiments investigated this question using listening to positive music as a happiness-inducing activity. Study 1 showed that participants assigned to try to boost their mood while listening to 12?min of music reported higher positive mood compared to participants who simply listened to music without attempting to alter mood. However, this effect was qualified by the predicted interaction: the music had to be positively valenced (i.e. Copland, not Stravinsky). In Study 2, participants who were instructed to intentionally try to become happier (vs. not trying) reported higher increases in subjective happiness after listening to positively valenced music during five separate lab visits over a two-week period. These studies demonstrate that listening to positive music may be an effective way to improve happiness, particularly when it is combined with an intention to become happier.     

Publication patterns in developmental psychology: Trends and social networks

Interest in publication patterns has been steady. Journals have instituted policies in an effort to curb bias and provide globally representative research. This study aimed to examine if publication patterns were present in two developmental psychology journals. It also explored the social networks of prominent authors and the prevalence of informal author–editor relationships, searching for any potential power groups. Data were taken from empirical articles published between 2005 and 2014 in Child Development (CD) and The International Journal of Early Childhood (IJEC) data points were geographical authorship affiliation, informal author relationships as established by co‐publishing, and connections to journal editors via identical affiliation. Results confirmed the previously established North American dominance in published research. In CD a strongly interlinked social network was identified between authors over the 10 years, with 15 chief influentialists binding groups of authors together. Results suggest that patterns are still present in published research in the realm of developmental psychology. To conclude, the potential implications of these patterns within developmental psychology are presented.     

Too special to be duped: Need for uniqueness motivates conspiracy beliefs

Adding to the growing literature on the antecedents of conspiracy beliefs, this paper argues that a small part in motivating the endorsement of such seemingly irrational beliefs is the desire to stick out from the crowd, the need for uniqueness. Across three studies, we establish a modest but robust association between the self‐attributed need for uniqueness and a general conspirational mindset (conspiracy mentality) as well as the endorsement of specific conspiracy beliefs. Following up on previous findings that people high in need for uniqueness resist majority and yield to minority influence, Study 3 experimentally shows that a fictitious conspiracy theory received more support by people high in conspiracy mentality when this theory was said to be supported by only a minority (vs. majority) of survey respondents. Together, these findings support the notion that conspiracy beliefs can be adopted as a means to attain a sense of uniqueness.     

Sad, thus true: Negativity bias in judgments of truth

An effect observable across many different domains is that negative instances tend to be more influential than comparably positive ones. This phenomenon has been termed the negativity bias. In the current work, it was investigated whether this effect pertains to judgments of truth. That is, it was hypothesized that information valence and perceived validity should be associated such that more negative information is deemed more true. This claim was derived from the findings that negative instances tend to demand more attentional resources and that more elaborate processing can render messages more persuasive. In three experiments, manipulating information valence through framing - and assessing judgments of truth - the hypothesized negativity bias was corroborated. Potential explanations and implications for further research are discussed.     

Publication trends in school psychology: 1963–1973

Articles appearing in three school psychology journals from 1963–1973 were classified and organized into 16 topic areas. Trends of published interests over the years 1963–1966, 1967–1969, and 1970–1973 were identified. Articles concerning Professional Identity were most prominent in the early years of the decade covered; Instrument Development and Evaluation, an area which was consistently popular over the decade, ranked highest during the last period evaluated. The most noteworthy trend in publication has been the growing diversification of published interests in the field of school psychology.     

Methods for addressing publication bias in school psychology journals: A descriptive review of meta-analyses from 1980 to 2019

Although meta-analyses are often used to inform practitioners and researchers, the resulting effect sizes can be artificially inflated due to publication bias. There are a number of methods to protect against, detect, and correct for publication bias. Currently, it is unknown to what extent scholars publishing meta-analyses within school psychology journals use these methods to address publication bias and whether more recently published meta-analyses more frequently utilize these methods. A historical review of every meta-analysis published to date within the most prominent school psychology journals (N = 10) revealed that 88 meta-analyses were published from 1980 to early 2019. Exactly half of them included grey literature, and 60% utilized methods to detect and correct for publication bias. The most common methods were visual analysis of a funnel plot, Orwin's failsafe N, Egger's regression, and the trim and fill procedure. None of these methods were used in more than 20% of the studies. About half of the studies incorporated one method, 20% incorporated two methods, 7% incorporated three methods, and none incorporated all four methods. These methods were most evident in studies published recently. Similar to other fields, the true estimates of effects from meta-analyses published in school psychology journals may not be available, and practitioners may be utilizing interventions that are, in fact, not as strong as believed. Practitioners, researchers employing meta-analysis techniques, education programs, and editors and peer reviewers in school psychology should continue to guard against publication bias using these methods.     

Relevant dimensions of cyberbullying — Results from two experimental studies

Cyberbullying is a prevalent problem of adolescents. However, several conceptual and measurement questions, regarding its defining characteristics and relevant dimensions in comparison to conventional bullying, remain unanswered. To this end we conducted two studies with experimental methods. Study I shows that power imbalance in terms of perceived popularity is relevant for the affective, cognitive, and behavioral experience of cyberbullying. Cyberbullying by a popular bully is more distressing than cyberbullying by an unpopular bully. Study II shows that factors unique to cyberbullying are also relevant for the experience of cyberbullying, namely the media and the type of cyberbullying. For example, different types of cyberbullying are related to different patterns of relevant coping strategies. Therefore, cyberbullying seems both a unique phenomenon and closely related to conventional bullying.     

Pervasiveness of effects in sample-based experimental psychology: A Re-examination of replication data from nine famous psychology experiments

Data previously reported from successful replications of 9 famous experiments (Zwaan et al., 2018) were re-analysed. Rather than the null hypothesis significance tests performed in the original study, this study performed a pervasiveness analysis, which examined the number of people in the original samples that demonstrated the effect each experiment is renowned for. Seven of the 9 experiments were shown to exceed the pervasiveness threshold adopted in this study. That is, in each of these 7 experiments, over 80% of participants demonstrated the target effect, indicating support for Zwaan et al.’s claim that these effects are robust. The remaining 2 experiments (Repetition Priming and Shape Simulation) did not meet this pervasiveness criterion, casting doubt on the reproducibility of these effects. The pervasiveness analysis was demonstrated to be a useful adjunct to traditional forms of analysis because it provides information directly relevant to claims about the reproducibility of psychological effects.     

Collective memory: a perspective from (experimental) clinical psychology

This paper considers the concept of collective memory from an experimental clinical psychology perspective. Exploration of the term collective reveals a broad distinction between literatures that view collective memories as a property of groups (collectivistic memory) and those that regard these memories as a property of individuals who are, to a greater or lesser extent, an integral part of their social environment (social memory). First, we argue that the understanding of collectivistic memory phenomena may benefit from drawing parallels with current psychological models such as the self-memory system theory of individualistic autobiographical memory. Second, we suggest that the social memory literature may inform the study of trauma-related disorders. We argue that a factual focus induced by collaborative remembering may be beneficial to natural recovery in the immediate aftermath of trauma, and propose that shared remembering techniques may provide a useful addition to the treatment of post-traumatic stress disorder.     

The rhetoric of experimental social psychology, 1930-1960: from caution to enthusiasm

Between 1930 and 1960, experimentation became the premier form of knowledge generation in social psychology. In journals, texts, and handbooks, experiment was now conceived as the active manipulation of an independent variable, and the sole method for the discovery of "causes." Understanding this change requires further investigation of the fine-grained discursive strategies used to promote experimentation during the 1930s and 1940s. In this paper we use discourse analysis to contrast the cautious rhetoric used by Gardner Murphy and Lois Murphy and the more enthusiastic, unhedged arguments for experimentation employed by Kurt Lewin. We argue that analysis of changes in discourse justifying experimentation can illuminate the processes by which methodological consensus was constructed.     

Assessing good qualitative research in the work psychology field: A response to the commentaries

This article is a response to three commentaries (Halbesleben, 2011; Hickson, 2011 & Locke, 2011) on ‘Assessing “good” qualitative research in the work psychology field: A narrative analysis’ (Cassell & Symon, 2011).     

Effect of cognitive bias modification-memory on depressive symptoms and autobiographical memory bias: two independent studies in high-ruminating and dysphoric samples

Memory bias is a risk factor for depression. In two independent studies, the efficacy of one CBM-Memory session on negative memory bias and depressive symptoms was tested in vulnerable samples. We compared positive to neutral (control) CBM-Memory trainings in highly-ruminating individuals (N?=?101) and individuals with elevated depressive symptoms (N?=?100). In both studies, participants studied positive, neutral, and negative Swahili words paired with their translations. In five study–test blocks, they were then prompted to retrieve either only the positive or neutral translations. Immediately following the training and one week later, we tested cued recall of all translations and autobiographical memory bias; and also measured mood, depressive symptoms, and rumination. Retrieval practice resulted in training-congruent recall both immediately after and one week after the training. Overall, there was no differential decrease in symptoms or difference in autobiographical memory bias between the training conditions. In the dysphoric but not in the high-ruminating sample, the positive training resulted in positive autobiographical bias only in dysphoric individuals with positive pre-existing bias.

We conclude that one session of positive retrieval-based CBM-Memory may not be enough to yield symptom change and affect autobiographical memory bias in vulnerable individuals.     

Longitudinal studies of personality disorders: four lessons from personality psychology

The three longitudinal projects described in this special section of the Journal of Personality Disorders raise a number of intriguing questions concerning the natural history of personality disorders and offer more than their share of surprises. In addition, they underscore several valuable lessons derived from the literature on normal-range personality traits. Drawing in part from the writings of the American trait psychologist Gordon Allport, I describe four such lessons: (1) change and continuity of personality traits and disorders can and do coexist, (2) the covariation among personality traits helps to account for the "comorbidity" among personality disorders, (3) personality traits and disorders influence how individuals interpret life events, and (4) personality traits must be distinguished from behavioral adaptations to these traits. These lessons remind us that the science of personality disorders must be informed by the basic science of personality.