An overview of issues in infant and developmental research for the creation of robust and replicable science

In the last few years, the field of psychology has been challenged with a crisis in the rigor and reproducibility of the science. The focus of these issues has primarily been in social, cognitive, and cognitive neuroscience psychology, however, the area of developmental research is not immune to these issues. This paper provides an overview of the “replication crisis” and the choices made by researchers that are often not noted in methods, thus making the replication of studies more difficult. In this review we discuss issues of researcher flexibility in the data design and selection of sample size, collection, and analysis stages of research. In each of these areas we address examples of bias and how developmental researchers can address these issues in their own research.     

Education or Indoctrination? The Accuracy of Introductory Psychology Textbooks in Covering Controversial Topics and Urban Legends About Psychology

The introductory psychology class represents the first opportunity for the field to present new students with a comprehensive overview of psychological research. Writing introductory psychology textbooks is challenging given that authors need to cover many areas they themselves may not be intimately familiar with. This challenge is compounded by problems within the scholarly community in which controversial topics may be communicated in ideological terms within scholarly discourse. Psychological science has historically seen concerns raised about the mismatch between claims and data made about certain fields of knowledge, apprehensions that continue in the present “replication crisis.” The concern is that, although acting in good faith, introductory psychology textbook authors may unwittingly communicate information to readers that is factually untrue. Twenty-four leading introductory psychology textbooks were surveyed for their coverage of a number of controversial topics (e.g., media violence, narcissism epidemic, multiple intelligences) and scientific urban legends (e.g., Kitty Genovese, Mozart Effect) for their factual accuracy. Results indicated numerous errors of factual reporting across textbooks, particularly related to failing to inform students of the controversial nature of some research fields and repeating some scientific urban legends as if true. Recommendations are made for improving the accuracy of introductory textbooks.     

An a priori solution to the replication crisis

Possibly, the replication crisis constitutes the most important problem in psychology. It calls into question whether psychology is a science. Existing conceptualizations of replicability depend on effect sizes; the larger the population effect size, the greater the probability of replication. This is problematic and contributes to the replication crisis. A different conceptualization, not dependent on population effect sizes, is desirable. The proposed solution features the closeness of sample means to their corresponding population means, in both the original and replication experiments. If the researcher has specified the sampling precision desired, it is possible to calculate the probability of replication, prior to data collection, and without dependence on the population effect size or expected population effect size. In addition, it is not necessary to know population means or standard deviations, nor sample means or standard deviations, to employ the proposed a priori way of thinking about replicability.     

Small is beautiful: In defense of the small-<Emphasis Type="Italic">N</Emphasis> design

The dominant paradigm for inference in psychology is a null-hypothesis significance testing one. Recently, the foundations of this paradigm have been shaken by several notable replication failures. One recommendation to remedy the replication crisis is to collect larger samples of participants. We argue that this recommendation misses a critical point, which is that increasing sample size will not remedy psychology’s lack of strong measurement, lack of strong theories and models, and lack of effective experimental control over error variance. In contrast, there is a long history of research in psychology employing small-N designs that treats the individual participant as the replication unit, which addresses each of these failings, and which produces results that are robust and readily replicated. We illustrate the properties of small-N and large-N designs using a simulated paradigm investigating the stage structure of response times. Our simulations highlight the high power and inferential validity of the small-N design, in contrast to the lower power and inferential indeterminacy of the large-N design. We argue that, if psychology is to be a mature quantitative science, then its primary theoretical aim should be to investigate systematic, functional relationships as they are manifested at the individual participant level and that, wherever possible, it should use methods that are optimized to identify relationships of this kind.     

Publication bias and the failure of replication in experimental psychology

Replication of empirical findings plays a fundamental role in science. Among experimental psychologists, successful replication enhances belief in a finding, while a failure to replicate is often interpreted to mean that one of the experiments is flawed. This view is wrong. Because experimental psychology uses statistics, empirical findings should appear with predictable probabilities. In a misguided effort to demonstrate successful replication of empirical findings and avoid failures to replicate, experimental psychologists sometimes report too many positive results. Rather than strengthen confidence in an effect, too much successful replication actually indicates publication bias, which invalidates entire sets of experimental findings. Researchers cannot judge the validity of a set of biased experiments because the experiment set may consist entirely of type I errors. This article shows how an investigation of the effect sizes from reported experiments can test for publication bias by looking for too much successful replication. Simulated experiments demonstrate that the publication bias test is able to discriminate biased experiment sets from unbiased experiment sets, but it is conservative about reporting bias. The test is then applied to several studies of prominent phenomena that highlight how publication bias contaminates some findings in experimental psychology. Additional simulated experiments demonstrate that using Bayesian methods of data analysis can reduce (and in some cases, eliminate) the occurrence of publication bias. Such methods should be part of a systematic process to remove publication bias from experimental psychology and reinstate the important role of replication as a final arbiter of scientific findings.     

From crisis of evidence to a “crisis” of relevance? Incentive-based answers for social psychology’s perennial relevance worries

Current controversies in social psychology have sparked the promotion of new rules for evidence in the field. This “crisis of evidence” echoes prior concerns from the 1970s about a so-called “crisis of social psychology”, with such issues as replication and statistical significance once more under examination. I argue that parallel concerns about the relevance of our research, raised but not completely resolved in the 1970s crisis, also deserve a fresh look. In particular, the advances made in the current crisis of evidence came about because of changes in academic career incentives, particularly publishing. Today, many voices in psychology urge greater respect for relevance in topics, methods and communication, but the lack of clear and concrete incentives to do so has stood in the way of answers. I diagnose the current incentive structures, propose partial solutions that are within the reach of journal editors and professional societies, and conclude by discussing the links between relevance and evidence, as well as special challenges to the relevance of social psychology post-2016.     

Cumulating Evidence about the Social Animal: Meta‐Analysis in Social‐Personality Psychology

Like most scientific fields, social‐personality psychology has experienced an explosion of research related to such central topics as aggression, attraction, gender, group processes, motivation, personality, and persuasion, to name a few. The proliferation of research can be a monster unless it is tamed with the scientific review strategy of meta‐analysis, literally analyses of past analyses that produce a quantitative and empirical history of research on a particular phenomenon. The purpose of this article is to outline the basic process and statistics of meta‐analysis, as they pertain to social‐personality psychology. Meta‐analysis involves: (i) defining the problem under review; (ii) gathering qualified reports and putting their findings and methods into a database, (iii) analyzing that database, and (iv) interpreting the results and reporting them. Use of meta‐analytic strategies has paralleled the knowledge explosion in social‐personality psychology, but must be used and consumed with careful discernment if the cumulated evidence about the social animal, Homo sapiens, is to have maximal value.     

信息加工阶段与情绪刺激加工

情绪刺激的特定加工路径使其能够有效捕获注意并获得加工资源,在一定程度上反映了其加工优势。然而,由于信息加工资源的有限性,由情绪信息引发的、不受自上而下认知因素影响的加工优势并非一直存在。本文根据情绪刺激在信息加工过程不同阶段的加工特点,阐述了情绪注意效应与自上而下注意效应在知觉阶段的注意选择过程中的交互影响,以及在执行控制阶段,情绪刺激加工与执行控制功能间的相互作用。     

犯罪心理学研究中运用量化方法引起的理论思考

量化与统计分析,是一项重要的研究方法。这一方法在犯罪心理学研究中得到了广泛运用,同时也存在着使用的局限性。犯罪人有很强的心理掩饰性,对包括犯罪人在内的人的心理的测量实际上是一种间接测量,测得的数据是心理学家标定的数据,兼有客观性与主观性。量化研究如果不与科学思维相结合,有可能得出浅薄平庸甚至错误的结论。心理学研究应当以"人性"为基点,多种方法综合运用,实现量化方法与非量化方法的有机结合。     

Dr. Matataro Matsumoto: His career and achievements

Dr. M. Matsumoto (1865–1943) is the most eminent figure in the history of psychology in Japan. Before the close of the last century, he acquired the scientific methods and technics of experimental psychology in America and Germany and introduced all of them into Japanese universities. Establishing psychological laboratories in both Tokyo and Kyoto, he trained many famous psychologists. He advocated ‘Psychocynematics’ as early as 1910. Together with his followers he applied psychology to every aspect of our daily life. He organized the Japanese Psychological Association and several periodicals were started under his leadership. He also made some important artistic contributions to the fields of picture and caligraphy.     

As Good as GOLD: Gram–Schmidt Orthogonalization by Another Name

Generalized orthogonal linear derivative (GOLD) estimates were proposed to correct a problem of correlated estimation errors in generalized local linear approximation (GLLA). This paper shows that GOLD estimates are related to GLLA estimates by the Gram–Schmidt orthogonalization process. Analytical work suggests that GLLA estimates are derivatives of an approximating polynomial and GOLD estimates are linear combinations of these derivatives. A series of simulation studies then further investigates and tests the analytical properties derived. The first study shows that when approximating or smoothing noisy data, GLLA outperforms GOLD, but when interpolating noisy data GOLD outperforms GLLA. The second study shows that when data are not noisy, GLLA always outperforms GOLD in terms of derivative estimation. Thus, when data can be smoothed or are not noisy, GLLA is preferred whereas when they cannot then GOLD is preferred. The last studies show situations where GOLD can produce biased estimates. In spite of these possible shortcomings of GOLD to produce accurate and unbiased estimates, GOLD may still provide adequate or improved model estimation because of its orthogonal error structure. However, GOLD should not be used purely for derivative estimation because the error covariance structure is irrelevant in this case. Future research should attempt to find orthogonal polynomial derivative estimators that produce accurate and unbiased derivatives with an orthogonal error structure.     

Getting satisfied with “satisfaction of search”: How to measure errors during multiple-target visual search

Visual search studies are common in cognitive psychology, and the results generally focus upon accuracy, response times, or both. Most research has focused upon search scenarios where no more than 1 target will be present for any single trial. However, if multiple targets can be present on a single trial, it introduces an additional source of error because the found target can interfere with subsequent search performance. These errors have been studied thoroughly in radiology for decades, although their emphasis in cognitive psychology studies has been more recent. One particular issue with multiple-target search is that these subsequent search errors (i.e., specific errors which occur following a found target) are measured differently by different studies. There is currently no guidance as to which measurement method is best or what impact different measurement methods could have upon various results and conclusions. The current investigation provides two efforts to address these issues. First, the existing literature is reviewed to clarify the appropriate scenarios where subsequent search errors could be observed. Second, several different measurement methods are used with several existing datasets to contrast and compare how each method would have affected the results and conclusions of those studies. The evidence is then used to provide appropriate guidelines for measuring multiple-target search errors in future studies.     

Too good to be true: Publication bias in two prominent studies from experimental psychology

Empirical replication has long been considered the final arbiter of phenomena in science, but replication is undermined when there is evidence for publication bias. Evidence for publication bias in a set of experiments can be found when the observed number of rejections of the null hypothesis exceeds the expected number of rejections. Application of this test reveals evidence of publication bias in two prominent investigations from experimental psychology that have purported to reveal evidence of extrasensory perception and to indicate severe limitations of the scientific method. The presence of publication bias suggests that those investigations cannot be taken as proper scientific studies of such phenomena, because critical data are not available to the field. Publication bias could partly be avoided if experimental psychologists started using Bayesian data analysis techniques.     

On genes,environments and human development: A commentary to Goossens

Using the example of adolescent loneliness, Goossens (this issue) presented an outline of integral research programmes in European developmental psychology. Integral research programmes should combine longitudinal analysis with state-of-the-art methods from the neurosciences and use the gene×environment approach to track differential susceptibility to the environment. In my commentary, I suggest using control variables in gene–environment studies to rule out spurious associations, to make the studies more developmentally sensitive, and to stress the need for replication of gene–environment findings. I offer two hypothetical models for developmentally sensitive gene–environment studies on adolescent loneliness, namely a social cognition model and a social rejection model.     

《科学》和《自然》杂志2015年的两篇心理学文章

本文评述了《科学》和《自然》杂志2015年的两篇文章。《科学》上的文章“心理科学的重现性”对100项心理学实验进行了重复验证。结果表明,只有大约多于1/3的原始研究能够重复。该文章对出现这样结果的原因作了分析,并指出这不是心理学的挫折。我们强调科学精神在于它的不确定性,因而科学才能处在不停的探索中。《自然》上的文章“P值仅仅是冰山的一角”,批评了个别心理学杂志禁用P值的做法,指出研究者应该在实验设计、数据处理等实验的前期阶段下功夫,才能防止心理科学的坍塌。我们结合国内研究生培养的现状,就改进心理学研究生的实验训练提出了建议。     

To Think Human out of the Machine Paradigm: <Emphasis Type="Italic">Homo Ex Machina</Emphasis>

This article attempts to show that the metaphorical conception of human being as a machine takes a very specific epistemological standpoint. To make short the complex task of considering the implication of this paradigm for psychological and behavioral sciences, three important mismatches between the machine and the living human will be considered. Experience, agency and plasticity of human being are excluded in the scientific models and research activities when they are situated in the machine paradigm. For this reason, I claim that the machine paradigm does not offer the relevant frame for integrating results from various domains or approaches within human sciences, even if it can sometimes produce relevant scientific knowledge in certain domain at the scale of detailed investigation. Due to the importance of overcoming the fragmentation of scientific knowledge to solve the crisis in psychology, an “organic paradigm” should be elaborated which provides a new epistemological framework.