Too good to be true: Publication bias in two prominent studies from experimental psychology

Empirical replication has long been considered the final arbiter of phenomena in science, but replication is undermined when there is evidence for publication bias. Evidence for publication bias in a set of experiments can be found when the observed number of rejections of the null hypothesis exceeds the expected number of rejections. Application of this test reveals evidence of publication bias in two prominent investigations from experimental psychology that have purported to reveal evidence of extrasensory perception and to indicate severe limitations of the scientific method. The presence of publication bias suggests that those investigations cannot be taken as proper scientific studies of such phenomena, because critical data are not available to the field. Publication bias could partly be avoided if experimental psychologists started using Bayesian data analysis techniques.     

What Kind of Empirical Research Should We Publish, Fund, and Reward?

ABSTRACT— When evaluating empirical papers for publication, grant proposals, or individual contributions (e.g., awarding tenure), the basic question one should ask is how much the contribution adds to understanding in psychology and not whether the contribution takes a particular form or represents one particular model of how to do empirical studies. Academic psychology has flourished with its mastery of the hypothesis–experiment model of science and its expertise in generating and eliminating alternative hypotheses and isolating causation. These accomplishments are a critical part of psychology, and they are well and appropriately taught by psychologists. However, they are only a part of science and should not comprise the almost exclusive criteria for evaluating research. In particular, discovery of fundamental phenomena, such as functional relations that apply to the real world and have generality, should have a higher priority in psychology. Such findings have been the basis for theoretical advances in other natural sciences.     

Why positive psychology is necessary

The authors provide a definition of positive psychology and suggest that psychologists should try to cultivate a more appreciative perspective on human nature. Examples are given of a negative bias that seems to pervade much of theoretical psychology, which may limit psychologists' understanding of typical and successful human functioning. Finally, a preview of the articles in the special section is given.     

The consistency test does not–and cannot–deliver what is advertised: A comment on Francis (2013)

The statistical consistency test of  Ioannidis and Trikalinos (2007) has been used recently by  Francis, 2012a, Francis, 2012c, Francis, 2012d, Francis, 2012e, Francis, 2013, Francis, in press, to argue that specific sets of experiments show evidence of publication bias. I argue that the test is unnecessary because publication bias exists almost everywhere as property of the research process, not individual studies. Furthermore, for several reasons, the test does not support the claims made on its behalf. Instead of focusing on testing sets of experiments for publication bias, we should focus on changes to scientific culture to reduce the bias.     

PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS

This article has 2 goals. First, we discuss publication bias and explain why it presents a potential problem for industrial and organizational psychology. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (trim and fill), which has been developed in the medical literature but is largely unfamiliar to industrial and organizational psychology researchers. Second, we demonstrate trim and fill by applying it to validity information reported in the technical manuals of 4 test vendors. In doing so, we assess the likelihood that criterion-related validity information provided by test publishers may overestimate test validity. In our analysis of 18 validity distributions, we found evidence of either no or minimal bias for 2 of the vendors' distributions and evidence of moderate-to-severe bias in at least 1 distribution from each of the other 2 vendors. In both cases in which publication bias was found, we noted instances in which the publishers tended to report only statistically significant correlations and that this practice was detected using publication bias methodology.     

A Cultural Analysis of the Experiment and an Experimental Analysis of Culture

What is it that one can and cannot learn from experiments? The present paper addresses this question from the perspective of cultural psychology. The first half of the paper examines implicit constructions of reality – like individualism, assumptions about the purity of the experimental context and the nature of mind, and the meaning of empirical– that constitute the culture of experimental social psychology. The second half of the paper illustrates an experimental approach to the study of culture with an example from a program of research on enemyship. Overall, the paper suggests that experimental methods may be best suited for arguments about proximal cause, but less appropriate for arguments about the distal causes of events in the everyday world. Rather than advocate that social psychologists abandon the experiment, a cultural psychology perspective suggests the reflexive use of experimental methods in conjunction with multiple techniques of empirical observation.     

Socio-Economic Class Bias in Rorschach Diagnosis: Contributing Psychosocial Attributes of the Clinician

The study attempted a replication of William Haase's finding (1956) of psychodiagnostic bias detrimental to the lower socio-economic class among clinical psychologists, while positing that psycho-social characteristics of the clinician, in particular Authoritarianism and Status Anxiety, would be significant correlates of this biased behavior. Sixty clinical psychologists were classified as High or Low Authoritarian, with level of status anxiety experimentally induced. Ss then interpreted paired (equated) Rorschach protocols with appended social histories of alleged lower or middle class origin. Social-class-related diagnostic bias was demonstrated, but was unrelated to the psychosocial characteristics of the clinician.     

Publication bias in research synthesis: sensitivity analysis using a priori weight functions

Publication bias, sometimes known as the "file-drawer problem" or "funnel-plot asymmetry," is common in empirical research. The authors review the implications of publication bias for quantitative research synthesis (meta-analysis) and describe existing techniques for detecting and correcting it. A new approach is proposed that is suitable for application to meta-analytic data sets that are too small for the application of existing methods. The model estimates parameters relevant to fixed-effects, mixed-effects or random-effects meta-analysis contingent on a hypothetical pattern of bias that is fixed independently of the data. The authors illustrate this approach for sensitivity analysis using 3 data sets adapted from a commonly cited reference work on research synthesis (H. M. Cooper & L. V. Hedges, 1994).     

On science and the discriminative law of effect

This article considers the process of the dissemination of scientific findings from the point of view of the discriminative law of effect. We assume that the purpose of science is to describe the state of the world in an unbiased and accurate manner. We then consider a number of challenges to the unbiased consensual development of science that arise from differences between science that is done, submitted for publication, and published. These challenges arise from the differential reinforcers for both research and publication delivered by journals and editors for novel results, the undervaluation of systematic replication and findings of invariance, and general lack of reinforcers for failed replications. All these challenges bias science toward searching for, reporting, and valuing novel results and consequently lead to a biased and erroneous view of the world. We suggest that science should be approached more conservatively, and that a reevaluation of the value of replication, and especially failed replication, is in order.     

A Bayesian approach to mitigation of publication bias

The reliability of published research findings in psychology has been a topic of rising concern. Publication bias, or treating positive findings differently from negative findings, is a contributing factor to this “crisis of confidence,” in that it likely inflates the number of false-positive effects in the literature. We demonstrate a Bayesian model averaging approach that takes into account the possibility of publication bias and allows for a better estimate of true underlying effect size. Accounting for the possibility of bias leads to a more conservative interpretation of published studies as well as meta-analyses. We provide mathematical details of the method and examples.     

Publication bias in psychological science: comment on Ferguson and Brannick (2012)

It is well documented that studies reporting statistically significant results are more likely to be published than are studies reporting nonsignificant results--a phenomenon called publication bias. Publication bias in meta-analytic reviews should be identified and reduced when possible. Ferguson and Brannick (2012) argued that the inclusion of unpublished articles is ineffective and possibly counterproductive as a means of reducing publication bias in meta-analyses. We show how idiosyncratic choices on the part of Ferguson and Brannick led to an erroneous conclusion. We demonstrate that their key finding--that publication bias was more likely when unpublished studies were included--may be an artifact of the way they assessed publication bias. We also point out how the lack of transparency about key choices and the absence of information about critical features of Ferguson and Brannick's sample and procedures might have obscured readers' ability to assess the validity of their claims. Furthermore, we demonstrate that many of the claims they made are without empirical support, even though they could have tested these claims empirically, and that these claims may be misleading. With their claim that addressing publication bias introduces subjectivity and bias into meta-analysis, they ignored a large body of evidence showing that including unpublished studies that meet the inclusion criteria of a meta-analysis decreases (rather than increases) publication bias. Rather than exclude unpublished studies, we recommend that meta-analysts code study characteristics related to methodological quality (e.g., experimental vs. nonexperimental design) and test whether these factors influence the meta-analytic results.     

Case for Non‐Biased Intelligence Testing Against Black Africans Has Not Been Made: A Comment on Rushton,Skuy, and Bons (2004)

This reply reviews the conceptual, methodological, and statistical foundations of Rushton, Skuy and Bons' article in this journal that compared Black Africans, Whites and East Indians on the Raven's Advanced Progressive Matrices, and concluded that the Raven's is an unbiased test. Through a technical re‐analysis of both the internal and external validity criteria for test bias using data reported in the Rushton et al. paper, we demonstrate that the Raven's Matrices test is in fact biased against Black Africans. We take issue with several additional elements of Rushton et al.'s study, including the use of non‐equivalent groups in test samples. We briefly review Rushton's racial‐realist research agenda and show that the assumption of test bias is central to advancing that agenda. Industrial/organizational and occupational psychologists should critically analyze and re‐evaluate the science employed in Rushton's racial‐realist research and also should better understand the ethical and social implications of accepting his reports of research findings on test bias and White–Black IQ differences as established scientific facts.     

The experimental psychology of attitude change and the tradition of classical rhetoric

Social psychologists might be surprised to learn that their discipline has been cut off from a vast and ancient family tree. The study of attitude change in the context of experimental social psychology began around 1918. It developed into a defined discipline in the 1930s and 1940s, particularly through the work of Carl Hovland and his associates. Unlike earlier specialties in experimental psychology, social psychology emerged well after the 19th-century split between psychology and philosophy in college curricula. Before this period of growth in empirical teaching and practice, the study of persuasion in classical rhetoric was a bedrock of higher education for more than 2000 years. Because of social psychology's late development in empirical science, there is a historical disconnect between experimental social psychology and its ancient philosophical counterpart, classical rhetoric. This article demonstrates similarities and differences between Hovland's findings and the theoretical groundings of classical rhetoric. We suggest areas where modern social psychology might benefit from a look at the older, more holistic theories of the art of rhetoric.     

The Authoritarian Personality, 50 Years Later: What Questions Are There for Political Psychology?

Adorno, Frenkel-Brunswik, Levinson, and Sanford's The Authoritarian Personality is probably the most deeply flawed work of prominence in political psychology. The methodological, procedural, and substantive errors of this study are well known, but they are frequently simply attributed to poor methodological judgments, issues of scaling (such as response set), or Freudian theories that legitimated circular interpretations. But a more fundamental bias arose from the attempt to empirically verify the existence of a "type" of person whom the researchers thought dangerous and with whom they did not empathize. This attempt involved two dangerous procedures: (1) the fusion of nominalist research procedures (in which empirical results were used to type respondents) with a realist interpretation of types (in which some people "just were" authoritarians and others not), and (2) a theoretically rich critique of the authoritarians and a lack of interest in the psychodynamics of liberals. This combination led to an intrinsically biased interpretive project that could not help but accumulate damning evidence about authoritarians. These subtler problems have haunted contemporary work in political psychology that avoids the methodological problems of Adorno et al.; Altemeyer's work on authoritarianism, which not only is free from the defects of the Adorno et al. study but also involves some methodologically exemplary experiments, is similarly distorted by asymmetries. The same fundamental problems seem to be at the heart of the weaknesses of the theory of symbolic racism to which critics have pointed. Political psychologists should regard The Authoritarian Personality as a cautionary example of bias arising from the choice of methodological assumptions.     

Strength of hindsight bias as a consequence of meta-cognitions

Hindsight bias is the tendency of people to falsely believe that they would have correctly predicted the outcome of an event once it is known. The present paper addresses the ongoing debate as to whether the hindsight bias is due to memory impairment or biased reconstruction. The memory impairment approach maintains that outcome information alters the memory trace of the initial judgement, whereas the biased reconstruction approach assumes that people who have forgotten their initial judgements are forced to guess and, in the presence of outcome information, are likely to use this information as an anchor. Whereas the latter approach emphasises the role of meta-cognitive considerations, meta-cognitions are not included in the memory impairment explanation. Two experiments show that the biased reconstruction approach provides a better explanation for empirical findings in hindsight bias research than does the memory impairment explanation.     

Strength of hindsight bias as a consequence of meta-cognitions

Hindsight bias is the tendency of people to falsely believe that they would have correctly predicted the outcome of an event once it is known. The present paper addresses the ongoing debate as to whether the hindsight bias is due to memory impairment or biased reconstruction. The memory impairment approach maintains that outcome information alters the memory trace of the initial judgement, whereas the biased reconstruction approach assumes that people who have forgotten their initial judgements are forced to guess and, in the presence of outcome information, are likely to use this information as an anchor. Whereas the latter approach emphasises the role of meta-cognitive considerations, meta-cognitions are not included in the memory impairment explanation. Two experiments show that the biased reconstruction approach provides a better explanation for empirical findings in hindsight bias research than does the memory impairment explanation.     

Unconscious manipulation of free choice by novel primes

The extent to which non-conscious perception can influence behaviour has been a topic of considerable controversy in psychology for decades. Although a challenging task, convincing empirical demonstrations have emerged suggesting that non-consciously perceived ‘prime’ stimuli can influence motor responses to subsequent targets. Interestingly, recent studies have shown that the influence of masked primes is not restricted to target-elicited responses, but can also bias free-choices between alternative behaviours. The present experiment extends these findings by showing that free-choices could also be biased by novel primes that never appeared as targets and therefore could not trigger acquired stimulus–response (S–R) mappings. This new evidence suggests that free-choice behaviour can be influenced by non-consciously triggered semantic representations. Furthermore, the results reported here support accounts of masked priming that posit an automatic semantic categorisation of non-consciously perceived visual stimuli.     

Happiness,excellence, and optimal human functioning revisited: Examining the peer-reviewed literature linked to positive psychology

Since the original call by Seligman and Csikszentmihalyi (2000) for a new science of happiness, excellence, and optimal human functioning, there has been an explosion of activity in, acclaim for, and criticism of positive psychology. The purpose of this study was to identify and examine the peer-reviewed literature linked to the positive psychology movement. An extensive systematic review identified 1336 articles published between 1999 and 2013. More than 750 of these articles included empirical tests of positive psychology theories, principles, and interventions. The results show a fairly consistent increase in the rate of publication, and that the number of empirical studies has grown steadily over the time period. The findings demonstrate that positive psychology is a growing and vibrant sub-area within the broader discipline of psychology, committed to using the same rigorous scientific methods as other sub-areas, in pursuit of understanding well-being, excellence, and optimal human functioning.