The life of p: “Just significant” results are on the rise

Null hypothesis significance testing uses the seemingly arbitrary probability of .05 as a means of objectively determining whether a tested effect is reliable. Within recent psychological articles, research has found an overrepresentation of p values around this cut-off. The present study examined whether this overrepresentation is a product of recent pressure to publish or whether it has existed throughout psychological research. Articles published in 1965 and 2005 from two prominent psychology journals were examined. Like previous research, the frequency of p values at and just below .05 was greater than expected compared to p frequencies in other ranges. While this overrepresentation was found for values published in both 1965 and 2005, it was much greater in 2005. Additionally, p values close to but over .05 were more likely to be rounded down to, or incorrectly reported as, significant in 2005 than in 1965. Modern statistical software and an increased pressure to publish may explain this pattern. The problem may be alleviated by reduced reliance on p values and increased reporting of confidence intervals and effect sizes.     

我国近十年来心理学研究中HLM方法的应用述评

以2002-2011年中国期刊网收录的50例应用多层线性模型(HLM)的心理学期刊论文为研究对象,从样本描述、模型发展与规范、数据准备、估计方法与假设检验4个角度进行文献计量和内容分析,对我国心理学研究中HLM方法的使用现状进行评估。结果表明,HLM方法主要用于管理、发展和教育心理学,绝大多数应用都是两层模型且层2样本量较大。HLM方法在广泛应用的同时仍存在忽略前提假设检验、分析过程中的重要信息和结果报告不完整等问题,随后提供了4条建议。     

The accuracy of abstracts in psychology journals

This article provides an empirically supported reminder of the importance of accuracy in scientific communication. The authors identify common types of inaccuracies in research abstracts and offer suggestions to improve abstract-article agreement. Abstracts accompanying 13% of a random sample of 400 research articles published in 8 American Psychological Association journals during 1997 and 1998 contained data or claims inconsistent with or missing from the body of the article. Error rates ranged from 8% to 18%, although between-journal differences were not significant. Many errors (63%) were unlikely to cause substantive misinterpretations. Unfortunately, 37% of errors found could be seriously misleading with respect to the data or claims presented in the associated article. Although deficient abstracts may be less common in psychology journals than in major medical journals (R. M. Pitkin, M. A. Branagan, & L. F. Burmeister, 1999), there is still cause for concern and need for improvement.     

Age and scholarly impact

The claim that older scientists generate research of lower quality than do younger scientists was tested through two analyses in which the age distribution of authors of frequently cited articles in psychology journals was compared with the age distribution of authors of low-impact articles published in the same journals. Most high-impact articles were published by relatively young psychologists, but so were most low-impact articles. When allowance was made for relative numerical representation, there was no evidence that publications from older scientists have less impact. Results are discussed in the context of methodological issues in evaluation of relations between age and scientific achievement.     

International student-focused counseling research: A 34-year content analysis

With the rapid growth of the international student population in the United States and internationalization of the counseling field, there is an increasing need to understand how this population is represented in counseling psychology research. To this end, this study provides a content and methodological analysis of 85 international student-focused empirical articles published in journals related to counseling psychology from 1980–2014. Publication trends revealed that of the total number of published empirical articles (N = 6191, 1980–2014), only 1.37% of empirical articles focused on international students. And yet, it appears that the percentage of empirical articles increased from .77% (1980) to 1.75% (2014), indicating a growth of 127% in the number of published empirical articles on international students. Results revealed 10 content categories, of which cultural adjustments, psychological health, and help utilization were the most common topics of inquiry. Methodological trends suggested that quantitative methodologies and convenience sampling were used most frequently and the majority of the sample consisted of international students from Asian countries. Limitations of and recommendations for future research are addressed.     

THE PERCEPTIONS AND USAGE OF STATISTICAL POWER IN APPLIED PSYCHOLOGY AND MANAGEMENT RESEARCH

We first assess the current level of statistical power across articles in seven leading journals that represent a broad sample of applied psychology and management research. We next survey the authors of these articles to examine their perceptions and usage of statistical power analysis. Finally, we examine the perceptions and usage of power analysis in a survey of authors of regression-based research appearing in leading journals. Findings from the assessment of power and surveys of researchers indicate that power analyses are not typically conducted, researchers perceive little need for statistical power, and power in published research is low. We conclude by discussing implications of low power for the field and recommending avenues for improving researchers' awareness and usage of statistical power.     

Why researchers don't study women: The responses of 62 researchers

Sixty-two researchers accounting for 64 single-sex studies that appeared in 1970 1971 issues of the Journal of Personality and Social Psychology reported why they had limited their studies to a single sex and why they had preferred either males or females for their studies. Also, articles and abstracts were read as a way of checking researchers' reporting practices. The replies were analyzed for thematic content and formed three major types: scientific (56%), practical (28%), and extrascientific (15%). The responses helped to explain why there has been an imbalance in psychology which led to the study of men rather than women instead of the exploration of both sexes. Sggestions are offered to help counteract this trend in future psychological research and reporting.     

A Systematic Review of Propensity Score Methods in the Social Sciences

The use of propensity scores in psychological and educational research has been steadily increasing in the last 2 to 3 years. However, there are some common misconceptions about the use of different estimation techniques and conditioning choices in the context of propensity score analysis. In addition, reporting practices for propensity score analyses often lack important details that allow other researchers to confidently judge the appropriateness of reported analyses and potentially to replicate published findings. In this article we conduct a systematic literature review of a large number of published articles in major areas of social science that used propensity scores up until the fall of 2009. We identify common errors in estimation, conditioning, and reporting of propensity score analyses and suggest possible solutions.     

Reference List Inaccuracies: A Four-Decade Comparison

Errors in the reference lists of journal articles have been reported in studies conducted on journals issued in the 1970s and 1980s. On the average, faults and errors were found to occur in about 40% of the entries. The Personnel and Guidance Journal and a journal related to information science were the data sources for a four-decade study. Results indicated that error rates were lower in the mid-1950s in this sample but were very similar in the last three decades and similar to rates found in other studies. One or more errors occurred in about 40 to 45 of any 100 entries sampled from reference lists. Implications are explored.     

Count data in psychological applied research

As some authors have noticed in fields other than psychology, level of measurement and distributional characteristics of count data are commonly not taken into account, so that they are analysed as normally distributed continuous variables, and therefore some general linear model is applied. In this work, we review a random sample of 457 articles published in the last four years in journals with the highest impact factor in the Journal Citation Reports (JCR Social Sciences Edition) of the Institute for Scientific Information. The goals are to know how often count variables appear in psychological applied research and which data analyses are used when dealing with count response variables. Examination showed that there is a notable presence of count variables, especially in some topics in psychology and a remarkable misapplication of the general linear model. One deals with causes and consequences of this approach to data analysis and suggests the use of specific models to analyse count data.     

Integrating Cultural Community Psychology: Activity Settings and the Shared Meanings of Intersubjectivity

Cultural and community psychology share a common emphasis on context, yet their leading journals rarely cite each other’s articles. Greater integration of the concepts of culture and community within and across their disciplines would enrich and facilitate the viability of cultural community psychology. The contextual theory of activity settings is proposed as one means to integrate the concepts of culture and community in cultural community psychology. Through shared activities, participants develop common experiences that affect their psychological being, including their cognitions, emotions, and behavioral development. The psychological result of these experiences is intersubjectivity. Culture is defined as the shared meanings that people develop through their common historic, linguistic, social, economic, and political experiences. The shared meanings of culture arise through the intersubjectivity developed in activity settings. Cultural community psychology presents formidable epistemological challenges, but overcoming these challenges could contribute to the transformation and advancement of community psychology.     

The value of RCT evidence depends on the quality of statistical analysis

The authors examined statistical practices in 193 randomized controlled trials (RCTs) of psychological therapies published in prominent psychology and psychiatry journals during 1999-2003. Statistical significance tests were used in 99% of RCTs, 84% discussed clinical significance, but only 46% considered-even minimally-statistical power, 31% interpreted effect size and only 2% interpreted confidence intervals. In a second study, 42 respondents to an email survey of the authors of RCTs analyzed in the first study indicated they consider it very important to know the magnitude and clinical importance of the effect, in addition to whether a treatment effect exists. The present authors conclude that published RCTs focus on statistical significance tests ("Is there an effect or difference?"), and neglect other important questions: "How large is the effect?" and "Is the effect clinically important?" They advocate improved statistical reporting of RCTs especially by reporting and interpreting clinical significance, effect sizes and confidence intervals.     

Beyond inference by eye: Statistical and graphing practices in JEAB, 1992‐2017

Debates about the utility of p values and correct ways to analyze data have inspired new guidelines on statistical inference by the American Psychological Association (APA) and changes in the way results are reported in other scientific journals, but their impact on the Journal of the Experimental Analysis of Behavior (JEAB) has not previously been evaluated. A content analysis of empirical articles published in JEAB between 1992 and 2017 investigated whether statistical and graphing practices changed during that time period. The likelihood that a JEAB article reported a null hypothesis significance test, included a confidence interval, or depicted at least one figure with error bars has increased over time. Features of graphs in JEAB, including the proportion depicting single‐subject data, have not changed systematically during the same period. Statistics and graphing trends in JEAB largely paralleled those in mainstream psychology journals, but there was no evidence that changes to APA style had any direct impact on JEAB. In the future, the onus will continue to be on authors, reviewers and editors to ensure that statistical and graphing practices in JEAB continue to evolve without interfering with characteristics that set the journal apart from other scientific journals.     

A Meta-View of Multivariate Statistical Inference Methods in European Psychology Journals

We investigated the extent and nature of multivariate statistical inferential procedures used in eight European psychology journals covering a range of content (i.e., clinical, social, health, personality, organizational, developmental, educational, and cognitive). Multivariate methods included those found in popular texts that focused on prediction, group difference, and advanced modeling: multiple regression, logistic regression, analysis of covariance, multivariate analysis of variance, factor or principal component analysis, structural equation modeling, multilevel modeling, and other methods. Results revealed that an average of 57% of the articles from these eight journals involved multivariate analyses with a third using multiple regression, 17% using structural modeling, and the remaining methods collectively comprising about 50% of the analyses. The most frequently occurring inferential procedures involved prediction weights, dichotomous p values, figures with data, and significance tests with very few articles involving confidence intervals, statistical mediation, longitudinal analyses, power analysis, or meta-analysis. Contributions, limitations and future directions are discussed.     

近五年西方心理学重要期刊发表论文简评

摘要：以《心理学公报》、《心理学评论》和《心理学年度评论》三种心理学国际权威学术期刊从2005年到2009年发表的论文为基础,从研究领域、研究主题、引用率等方面对国际心理学的研究进行了文献计量分析。探讨了心理学发展趋势,指出基础/实验心理、社会心理和健康心理领域的研究仍然是心理学研究的主流,但研究方法、技术发生了根本改变;认知、学习与记忆、决策与判断、情绪健康等主题是心理学研究的热点,从而启发中国的心理学研究者应加强研究方法和统计的训练,关注研究的本土化,进行跨学科、多层次整合的研究以及拓展研究主题。     

Beyond the Schools of Psychology 1: A Digital Analysis of Psychological Review, 1894–1903

Traditionally, American psychology at the turn of the twentieth century has been framed as a competition among a number of “schools”: structuralism, functionalism, behaviorism, etc. But this is only one way in which the “structure” of the discipline can be conceived. Most psychologists did not belong to a particular school, but they still worked within loose intellectual communities, and so their work was part of an implicit psychological “genre,” if not a formalized “school.” In this study, we began the process of discovering the underlying genres of American psychology at the turn of the twentieth century by taking the complete corpus of articles from the journal Psychological Review during the first decade of its publication and conducting a statistical analysis of the vocabularies they employed to see what clusters of articles naturally emerged. Although the traditional functionalist school was among the clusters we found, we also found distinct research traditions around the topics of color vision, spatial vision, philosophy/metatheory, and emotion. In addition, momentary clusters corresponding to important debates (e.g., the variability hypothesis) appeared during certain years, but not others.     

Editors can lead researchers to confidence intervals, but can't make them think: statistical reform lessons from medicine

Since the mid-1980s, confidence intervals (CIs) have been standard in medical journals. We sought lessons for psychology from medicine's experience with statistical reform by investigating two attempts by Kenneth Rothman to change statistical practices. We examined 594 American Journal of Public Health (AJPH) articles published between 1982 and 2000 and 110 Epidemiology articles published in 1990 and 2000. Rothman's editorial instruction to report CIs and not p values was largely effective: In AJPH, sole reliance on p values dropped from 63% to 5%, and CI reporting rose from 10% to 54%; Epidemiology showed even stronger compliance. However, compliance was superficial: Very few authors referred to CIs when discussing results. The results of our survey support what other research has indicated: Editorial policy alone is not a sufficient mechanism for statistical reform. Achieving substantial, desirable change will require further guidance regarding use and interpretation of CIs and appropriate effect size measures. Necessary steps will include studying researchers' understanding of CIs, improving education, and developing empirically justified recommendations for improved statistical practice.     

20年来我国心理学研究方法的回顾与反思

对近20年来我国心理学研究所采用的研究方法及分析资料的方法进行了客观的回顾与评价,概述了我国心理学研究所采用的主要研究方法和分析资料的方法,探索研究方法的变化趋势,指出存在的主要问题,并且为今后研究科学水平的提高提出建设性的意见。