PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS

This article has 2 goals. First, we discuss publication bias and explain why it presents a potential problem for industrial and organizational psychology. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (trim and fill), which has been developed in the medical literature but is largely unfamiliar to industrial and organizational psychology researchers. Second, we demonstrate trim and fill by applying it to validity information reported in the technical manuals of 4 test vendors. In doing so, we assess the likelihood that criterion-related validity information provided by test publishers may overestimate test validity. In our analysis of 18 validity distributions, we found evidence of either no or minimal bias for 2 of the vendors' distributions and evidence of moderate-to-severe bias in at least 1 distribution from each of the other 2 vendors. In both cases in which publication bias was found, we noted instances in which the publishers tended to report only statistically significant correlations and that this practice was detected using publication bias methodology.     

A Call for an Unbiased Search for Moderators in Disfluency Research: Reply to Oppenheimer and Alter (2014)

In a commentary to our article, Oppenheimer and Alter stressed the importance of searching for moderators in disfluency research. We agree, but we would like to clarify two issues. First, unlike Oppenheimer and Alter, we assume that learner characteristics might indeed moderate the effects of disfluency. Second, conclusions concerning the impact of disfluency in educational settings might be biased because of the file‐drawer problem. To counteract this possibility, we want to encourage other researchers to publish their studies––regardless of whether they found (positive) effects of disfluency or not. Until then, any recommendation concerning the use of disfluency as an educational intervention seems premature.     

Combining independent p values: extensions of the Stouffer and binomial methods

The Stouffer z method, and other popular methods for combining p values from independent significance tests, suffer from three problems: vulnerability to criticisms of the individual studies being pooled, difficulty in handling the "file drawer problem," and vague conclusions. These problems can be reduced or eliminated by supplementing a test of combined probability with a variety of new analyses described here. Along with other advantages, these analyses provide a way to address the file drawer problem without making limiting assumptions about the nature of studies not included in the pooled analysis. These analyses can supplement a traditional meta-analysis, yielding conclusions not provided by widely used meta-analytic procedures.     

ORMA: A strategy to reduce Psychology's replication problems

Because of their historical reliance upon null hypothesis statistical tests (NHST), the human sciences have developed a number of potentially problematic research literatures. While aware of the file drawer effect since the 1970s, scientists have been largely unsuccessful at addressing its pernicious effects. Because significant results have a greater likelihood of being published than do nonsignificant effects, many of our research literatures might currently be constructed upon a series of Type I errors and inflated effect sizes. A method (called Original Replication of Meta-Analyses or ORMA) has recently been developed for identifying problematic research literatures and offering a method to address the problems due to publication bias. Philosophers of science have long argued that a chief reason for science's preeminence as a source of knowledge rests in its ability to self-correct. Researchers in the human sciences are now able to empirically test their research literatures to ascertain which are in need of repair. The use of ORMA serves to lessen the problems that led to the recent calls for bans on significant/nonsignificant statistics in human science research. ORMA will also improve psychology's ability to successfully replicate its research findings.     

Attitudes of academic and clinical researchers toward financial ties in research: A systematic review

Involvement of industry in academic research is widespread and associated with favorable outcomes for industry. The objective of this study was to review empirical data on the attitudes of researchers toward industry involvement and financial ties in research. A review of the literature for quantitative data from surveys on the attitudes of researchers to financial ties in research, reported in English, resulted in the 17 studies included. Review of these studies revealed that investigators are concerned about the impact of financial ties on choice of research topic, research conduct and publication, but this concern is less among investigators already involved with industry. Researchers approve of industry collaboration and financial ties when the ties are indirectly related to the research, disclosure is upfront, and results and ideas are freely publicized. However, their trust in disclosure as a way to manage conflicts may reveal a lack of awareness of the actual impact of financial incentives on themselves and other researchers. This project was supported by grant number 5 T32 HS00086 from the Agency for Healthcare Research and Quality, US Department of Health and Human Services. Dr. Glaser, Ph.D., Postgraduate Researcher, is currently Associate Specialist, Institute of Governmental Studies, UC Berkeley.     

Questionable research practices among Brazilian psychological researchers: Results from a replication study and an international comparison

Research on scientific integrity is growing in psychology, and questionable research practices (QRPs) have received more attention due to its harmful effect on science. By replicating the procedures of previous research, the present study aimed at describing the use of QRPs among Brazilian psychological researchers and to make an international comparison with previous studies in other countries—the US and Italy. Two hundred and thirty-two Brazilian researchers in the field of psychology answered questions related to 10 different QRPs. Brazilian researchers indicated a lower tendency to engage in two QRPs (failing to report all of a study's dependent measures; deciding whether to collect more data after looking to see whether the results were significant) when compared to their Italian and North American counterparts, but indicated a higher tendency to engage in two other QRPs (selectively reporting studies that “worked”; not reporting all of a study's conditions). Most of the sample did not admit integrity conflicts in their own research but indicated that others have integrity problems, as observed in previous studies. Those discrepancies could be attributed to contextual and systemic factors regarding different publication demands among the different nations. Further studies should focus on identifying the antecedents of QRPs.     

Methods for addressing publication bias in school psychology journals: A descriptive review of meta-analyses from 1980 to 2019

Although meta-analyses are often used to inform practitioners and researchers, the resulting effect sizes can be artificially inflated due to publication bias. There are a number of methods to protect against, detect, and correct for publication bias. Currently, it is unknown to what extent scholars publishing meta-analyses within school psychology journals use these methods to address publication bias and whether more recently published meta-analyses more frequently utilize these methods. A historical review of every meta-analysis published to date within the most prominent school psychology journals (N = 10) revealed that 88 meta-analyses were published from 1980 to early 2019. Exactly half of them included grey literature, and 60% utilized methods to detect and correct for publication bias. The most common methods were visual analysis of a funnel plot, Orwin's failsafe N, Egger's regression, and the trim and fill procedure. None of these methods were used in more than 20% of the studies. About half of the studies incorporated one method, 20% incorporated two methods, 7% incorporated three methods, and none incorporated all four methods. These methods were most evident in studies published recently. Similar to other fields, the true estimates of effects from meta-analyses published in school psychology journals may not be available, and practitioners may be utilizing interventions that are, in fact, not as strong as believed. Practitioners, researchers employing meta-analysis techniques, education programs, and editors and peer reviewers in school psychology should continue to guard against publication bias using these methods.     

The Academic Productivist Regime: Affective Dynamics in the Moral-Political Economy of Publishing

With proliferating neoliberal modes of science governance, publishing has become more important. Recent studies point to researchers’ feeling of exhaustion and anxiety as responses to academic performance regimes. Yet how affects underpin publishing in scientific cultures and communities, and what this implies for STS scholarship has remained underexplored. Drawing on insights from ethnographic fieldwork and the cultural studies of affect, this article traces the role of emotions, including hope, contempt, and excitement for understanding the new academic productivist regime at a Czech research institution. While junior researchers’ orientations are fostered through rendering publications objects of hope, a moral-political economy intersects with geopolitical history and values of research organization to shape the publication practices of many senior scientists. An affective labor of combat and equanimity is necessary for managing these orientations that are corporeally energized by a dynamic of thrill. This four-pronged approach makes palpable how emotions render scientists’ bodies hopeful, combative and excited while intersecting with ideals of meritocratic research organization and assessment. Frustration and failure are never entirely absent but serve as an immanent driving force for a publishing culture that thrives on adrenaline, combativeness, and hope. This makes it difficult to leverage failures towards criticism of the academic productivist regime—both for the scientists differently affected within the institution and STS researchers. Different engagements with this regime require a more capacious accounting for the pleasure and thrill generated by the uncertainty of publication outcome as well as by unacknowledged practices of care.     

Sales Training and Education: Tracking Academic Research in Selling and Sales Management: Authors,Authorships, Academic Institutions,and Journals

This study examines and tracks the extent of selling and sales management research from 1993–1997. Two hundred and sixty- six published journal articles representing sixteen journals are examined to identify where and when selling and sales management researchers received their doctoral training, their employment location at time of publication, regional trends in the location of these researchers, and a count by-journal of selling and sales management articles that have appeared during the time period. A total of 280 individual authors from 175 different colleges and universities are included in the study. Overall, the general trends for scholarly activity in selling and sales management appear to be robust when compared to prior studies of sales research productivity.     

The Relevance of Counseling Outcome Research: Current Trends and Practical Implications

This systematic review investigated current trends in the publication sources, research focus, and methods of 365 counseling outcome studies. The results suggest that outcome research can provide a considerable knowledge base regarding numerous client concerns and various clinical interventions in individual, career, and school counseling. Counseling practitioners with a thorough knowledge of outcome research methods who use the findings appropriately will find this literature a useful adjunct to clinical decision making. Suggestions for improving the use of counseling research are offered for both practitioners and researchers.     

A meta-analysis of the published research on the effects of child sexual abuse

A meta-analysis of the published research on the effects of child sexual abuse (CSA) was undertaken for 6 outcomes: posttraumatic stress disorder (PTSD), depression, suicide, sexual promiscuity, victim-perpetrator cycle, and poor academic performance. Thirty-seven studies published between 1981 and 1995 involving 25,367 people were included. Many of the studies were published in 1994 (24; 65%), and most were done in the United States (22; 59%). All six dependent variables were coded, and effect sizes (d) were computed for each outcome. Average unweighted and weighted ds for each of the respective outcome variables were .50 and .40 for PTSD, .63 and .44 for depression, .64 and .44 for suicide, .59 and .29 for sexual promiscuity, .41 and .16 for victim-perpetrator cycle, and .24 and .19 for academic performance. A file drawer analysis indicated that 277 studies with null ds would be required to negate the present findings. The analyses provide clear evidence confirming the link between CSA and subsequent negative short- and long-term effects on development. There were no statistically significant differences on ds when various potentially mediating variables such as gender, socioeconomic status, type of abuse, age when abused, relationship to perpetrator, and number of abuse incidents were assessed. The results of the present meta-analysis support the multifaceted model of traumatization rather than a specific sexual abuse syndrome of CSA.     

Evaluating the gold standard: a review and meta-analysis of the Structured Interview of Reported Symptoms

The Structured Interview of Reported Symptoms (SIRS; Rogers, Bagby, & Dickens, 1992) is often touted as the gold standard of measures of feigning. This label likely arises in part out of the impressive accuracy rates reported in the extensive validation research that preceded its publication. However, since its publication, researchers not only have continued to investigate the measure's utility but have expanded the study of the SIRS to include novel populations, different study methodologies, and abbreviated versions. The current review examines 26 studies using the SIRS to identify feigning, evaluating both its effectiveness at differentiating feigners from genuine responders and the potential impact of moderating variables. Meta-analyses revealed that research published since the initial validation studies demonstrate higher sensitivity but lower specificity rates than those reported in the SIRS manual. Studies in which feigners were composed of simulators yielded higher classification rates than studies sampling actual suspected malingerers. Furthermore, genuine patient samples were significantly more likely than nonclinical samples to be misclassified as feigning. Abbreviated versions of the SIRS also demonstrated equivalent accuracy with the standard measure. The implications of these findings for clinical practice are discussed.     

The item structure of the Eysenck Personality Questionnaire across method and culture

Since its publication, the Eysenck Personality Questionnaire (EPQ) has endured much criticism of its psychometric properties. As the most stringent of these attacks has derived from hierarchical factor analytic studies it is reasonable to ask whether the inadequacies reported accrue to the questionnaire itself or to the problematic features of factor analysis as a method. A further feature of empirical studies that are critical of the EPQ is that they are, by-and-large, based upon non-British samples. As the EPQ was constructed in Britain it is quite conceivable that cultural artifacts are responsible for the inconsistent findings of Canadian and American researchers. This paper offers a brief review of some of the critical research and reports a study in which the item structure of the EPQ is explored by non-metric multidimensional scaling techniques upon a sample of Irish adolescents. The MDS configuration was then compared with the higher order factor analytic results of American researchers to examine invariance across method. A clear item structure emerges supporting the Eysencks' theoretical model although items of the P scale are less homogenous than would be ideal for a psychometric tool.     

The meaning of the h-index

The h-index originates from the assumption that the number of citations received by a scientist is a better indicator of the relevance of his or her work than the number of papers he or she publishes or the journals where they are published. It takes into account the number of papers published and the citations to those papers in a balanced way, and thus is useful to make comparisons between scientists. The present paper addresses the most frequent questions about the h-index. Specifically, it explains its origin, its advantages compared to other indices, the factors that can influence it (e.g. age, field of knowledge, topic of research and language of publication), its variants, and the injustices it may lead to. In short, this paper provides a clear exposition of the hoped-for role of the h-index in the evaluation of scientists: that it serves as a useful complement to other indicators that are more subjective, and that it contributes to the progress of science by aiding decision-making on allocation of research resources in a more effective way, and on rewarding researchers who contribute to scientific progress in a more fair way.     

The relationship between qualitative and quantitative research: Lessons from feminist psychology

The dominant methodological approach in psychological research has involved the use of quantitative methods within a positivist framework. In this article we argue that both qualitative and quantitative methods have their strengths and limitations, depending on the research question under investigation. We examine some of the advantages of qualitative methods, paying particular attention to the value of such methods for feminist researchers. We challenge the positivist assumption that all research should be apolitical and value-free, arguing that the political context in which all research studies take place plays an important role in decisions about the appropriate research methods to use. Despite the value attached to qualitative methods by feminist researchers, there may be projects for which quantitative methods, or a combination of qualitative and quantitative techniques, are more suitable. We draw on examples from our research on the transition from school to the job market for young people, and a study of 16- to 19-year-old first time mothers to illustrate these points, examining the practical implications of our arguments for applied social psychology research.