Null hypothesis significance testing and Type I error: The domain problem

Although many common uses of p-values for making statistical inferences in contemporary scientific research have been shown to be invalid, no one, to our knowledge, has adequately assessed the main original justification for their use, which is that they can help to control the Type I error rate (Neyman & Pearson, 1928, 1933). We address this issue head-on by asking a specific question: Across what domain, specifically, do we wish to control the Type I error rate? For example, do we wish to control it across all of science, across all of a specific discipline such as psychology, across a researcher's active lifetime, across a substantive research area, across an experiment, or across a set of hypotheses? In attempting to answer these questions, we show that each one leads to troubling dilemmas wherein controlling the Type I error rate turns out to be inconsistent with other scientific desiderata. This inconsistency implies that we must make a choice. In our view, the other scientific desiderata are much more valuable than controlling the Type I error rate and so it is the latter, rather than the former, with which we must dispense. But by doing so—that is, by eliminating the Type I error justification for computing and using p-values—there is even less reason to believe that p is useful for validly rejecting null hypotheses than previous critics have suggested.     

The error statistical philosopher as normative naturalist

We argue for a naturalistic account for appraising scientific methods that carries non-trivial normative force. We develop our approach by comparison with Laudan’s (American Philosophical Quarterly 24:19–31, 1987, Philosophy of Science 57:20–33, 1990) “normative naturalism” based on correlating means (various scientific methods) with ends (e.g., reliability). We argue that such a meta-methodology based on means–ends correlations is unreliable and cannot achieve its normative goals. We suggest another approach for meta-methodology based on a conglomeration of tools and strategies (from statistical modeling, experimental design, and related fields) that affords forward looking procedures for learning from error and for controlling error. The resulting “error statistical” appraisal is empirical—methods are appraised by examining their capacities to control error. At the same time, this account is normative, in that the strategies that pass muster are claims about how actually to proceed in given contexts to reach reliable inferences from limited data.     

Relative fluency and illusions of recognition memory

A hallmark of the experience of perceptual fluency is the sense that a familiar stimulus seems to pop out from its background, such as when one notices the face of a friend in a crowd of strangers. This experience suggests that fluency-based illusions of recognition memory may be more powerful when there are only a few fluent stimuli presented in a recognition context. The results of the present study were consistent with this prediction. The magnitude of fluency-based illusions of recognition memory was inversely related to the percentage of fluent stimuli on a recognition test. Furthermore, standard fluency manipulations did not impact recognition responses in between-participants designs. The results suggest that illusions of recognition memory may be more powerful when fluency is encountered in a context in which the surrounding stimuli are less fluent.     

The effect of working memory load on semantic illusions: What the phonological loop and central executive have to contribute

When asked how many animals of each kind Moses took on the Ark, most people respond with “two” despite the substituted name (Moses for Noah) in the question. Possible explanations for semantic illusions appear to be related to processing limitations such as those of working memory. Indeed, individual working memory capacity has an impact upon how sentences containing substitutions are processed. This experiment examined further the role of working memory in the occurrence of semantic illusions using a dual-task working memory load approach. Participants verified statements while engaging in either articulatory suppression or random number generation. Secondary task type had a significant effect on semantic illusion rate, but only when comparing the control condition to the two dual-task conditions. Furthermore, secondary task performance in the random number generation condition declined, suggesting a tradeoff between tasks. Response time analyses also showed a different pattern of processing across the conditions. The findings suggest that the phonological loop plays a role in representing semantic illusion sentences coherently and in monitoring for details, while the role of the central executive is to assist gist-processing of sentences. This usually efficient strategy leads to error in the case of semantic illusions.     

Visual illusions affect planning but not control

Much debate has arisen over how to account for the pattern of effects of visual illusions on action - that is, the findings that illusions affect actions in some circumstances but not others. I propose that this pattern can best be explained by postulating that visual illusions affect the planning of actions but do not affect the on-line control of actions. Strong evidence for this viewpoint comes from recent studies that show 'dynamic illusion effects': a large illusion effect early in a movement, but a decreasing effect as the hand approaches the target. These findings pose difficulties for other models of illusion effects on action.     

Trading on illusions: Unrealistic perceptions of control and trading performance

This paper examines the impact of illusory control beliefs on the performance of traders in financial instruments. The authors argue that the task and environment faced by traders are conducive to the development of illusions of control and that individual propensity to illusion of control will be (inversely) related to trader performance. Using an innovative computer task, designed to assess illusion of control in the field, data from 107 traders in four organizations showed individual differences in this bias to have a significant, inverse, association with performance, as measured by managers' ratings of trader performance and by total remuneration. The authors conclude that, at least in this context, illusion of control is maladaptive and that it is productive to take an individual difference approach to the study of such illusions. Implications for debates about the costs and benefits of positive illusions are discussed.     

预期错误在恐惧记忆更新中的作用与机制

依据错误驱动的学习理论,行为预期结果与实际结果之间的不匹配即预期错误(Predictionerror,PE)是学习产生的驱动因素。作为显著性信息中的一种,预期错误和物理显著性、惊讶、新异性等存在信息加工阶段的不同,与记忆更新的关系也有差异。近年来,记忆再巩固干预范式(reconsolidation interference)被证明可用于人类条件性恐惧记忆的更新,其中记忆提取激活阶段所包含的预期错误起到了引发记忆“去稳定”、开启记忆再巩固的关键作用。在促进恐惧记忆更新的行为机制上,PE被认为是记忆去稳定的必要非充分条件。记忆提取必须包含适量的PE,但其引发的是记忆去稳定、消退还是中间状态,还需结合记忆本身性质确定。在促进恐惧记忆更新的神经机制上,杏仁核、导水管周围灰质(PAG)、海马均在PE探测和计算过程中具有重要作用;前额叶皮层(PFC)及其亚区在PE开启记忆再巩固过程中扮演了重要角色。上述过程又受到神经系统中特定神经递质的重要调节,尤其是多巴胺能和谷氨酸能。未来研究应进一步探索基于PE计算模型的量化研究,整合PE与其他边界条件的交互作用,考察不同类型显著性在记忆再巩固中的作用等;并亟...     

Priors in perception: Top-down modulation,Bayesian perceptual learning rate,and prediction error minimization

I discuss top-down modulation of perception in terms of a variable Bayesian learning rate, revealing a wide range of prior hierarchical expectations that can modulate perception. I then switch to the prediction error minimization framework and seek to conceive cognitive penetration specifically as prediction error minimization deviations from a variable Bayesian learning rate. This approach retains cognitive penetration as a category somewhat distinct from other top-down effects, and carves a reasonable route between penetrability and impenetrability. It prevents rampant, relativistic cognitive penetration of perception and yet is consistent with the continuity of cognition and perception.     

老年人汉语阅读中预测误差成本的产生机制

阅读的认知加工机制随年龄变老只发生了量变还是也产生了质变,是毕生发展研究领域的重要问题。要进行有效的语言处理,读者必须结合已有的经验知识和当前语境对即将出现的信息进行预测加工。因此,要解决上述问题就要回答老年人阅读中预测加工机制发生了怎样变化,即预测误差成本是如何产生的。研究拟采用眼动脑电同步记录技术,在自然阅读过程中实时获取语言处理的行为和神经指标,聚焦老年人预测误差成本的产生机制,主要从副中央凹视觉、工作记忆负荷及语言能力三方面解释预测误差成本产生的原因。研究成果对构建汉语阅读的毕生发展模型有重要意义。     

Inserting mastered targets during error correction when teaching skills to children with autism

Research has identified a variety of effective approaches for responding to errors during discrete‐trial training. In one commonly used method, the therapist delivers a prompt contingent on the occurrence of an incorrect response and then re‐presents the trial so that the learner has an opportunity to perform the correct response independently. Some authors recommend inserting trials with previously mastered targets between the prompted response and opportunities to respond independently, but no studies have directly examined the benefits of this approach. In this study, we manipulated the placement of trials with mastered targets during discrete‐trial training to compare the effectiveness of error correction with and without this recommended insertion procedure. Four children with autism participated, and each was taught 18 targets across 3 target sets. Results indicated that embedding trials with mastered targets into error correction may not confer benefits for most children and that doing so may lead to less efficient instruction.     

The nature and outcomes of work: a replication and extension of interdisciplinary work-design research

Research into the changing nature of work requires comprehensive models of work design. One such model is the interdisciplinary framework (M. A. Campion, 1988), which integrates 4 work-design approaches (motivational, mechanistic, biological, perceptual-motor) and links each approach to specific outcomes. Unfortunately, studies of this framework have used methods that disregard measurement error, overlook dimensions within each work-design approach, and treat each approach and outcome separately. This study reanalyzes data from M. A. Campion (1988), using structural equation models that incorporate measurement error, specify multiple dimensions for each work-design approach, and examine the work-design approaches and outcomes jointly. Results show that previous studies underestimate relationships between work-design approaches and outcomes and that dimensions within each approach exhibit relationships with outcomes that differ in magnitude and direction.     

Measurement error and ANCOVA: Functional and structural relationship approaches

This article discusses alternative procedures to the standardF-test for ANCOVA in case the covariate is measured with error. Both a functional and a structural relationship approach are described. Examples of both types of analysis are given for the simple two-group design. Several cases are discussed and special attention is given to issues of model identifiability. An approximate statistical test based on the functional relationship approach is described. On the basis of Monte Carlo simulation results it is concluded that this testing procedure is to be preferred to the conventionalF-test of the ANCOVA null hypothesis. It is shown how the standard null hypothesis may be tested in a structural relationship approach. It is concluded that some knowledge of the reliability of the covariate is necessary in order to obtain meaningful results.     

Sample size determination for a matched-pairs study with incomplete data using exact approach

This research was motivated by a clinical trial design for a cognitive study. The pilot study was a matched-pairs design where some data are missing, specifically the missing data coming at the end of the study. Existing approaches to determine sample size are all based on asymptotic approaches (e.g., the generalized estimating equation (GEE) approach). When the sample size in a clinical trial is small to medium, these asymptotic approaches may not be appropriate for use due to the unsatisfactory Type I and II error rates. For this reason, we consider the exact unconditional approach to compute the sample size for a matched-pairs study with incomplete data. Recommendations are made for each possible missingness pattern by comparing the exact sample sizes based on three commonly used test statistics, with the existing sample size calculation based on the GEE approach. An example from a real surgeon-reviewers study is used to illustrate the application of the exact sample size calculation in study designs.     

Researchers misunderstand confidence intervals and standard error bars

Little is known about researchers' understanding of confidence intervals (CIs) and standard error (SE) bars. Authors of journal articles in psychology, behavioral neuroscience, and medicine were invited to visit a Web site where they adjusted a figure until they judged 2 means, with error bars, to be just statistically significantly different (p < .05). Results from 473 respondents suggest that many leading researchers have severe misconceptions about how error bars relate to statistical significance, do not adequately distinguish CIs and SE bars, and do not appreciate the importance of whether the 2 means are independent or come from a repeated measures design. Better guidelines for researchers and less ambiguous graphical conventions are needed before the advantages of CIs for research communication can be realized.     

Synchronizing movements with the metronome: nonlinear error correction and unstable periodic orbits.

The control of human hand movements is investigated in a simple synchronization task. We propose and analyze a stochastic model based on nonlinear error correction; a mechanism which implies the existence of unstable periodic orbits. This prediction is tested in an experiment with human subjects. We find that our experimental data are in good agreement with numerical simulations of our theoretical model. These results suggest that feedback control of the human motor systems shows nonlinear behavior.     

The affective costs of overconfidence

Positive illusions, though often beneficial (Taylor & Brown, 1988 ), can diminish the pleasure of outcomes. This prediction follows from decision affect theory. We investigated this prediction by measuring the confidence that recreational basketball players felt while making shots and the pleasure they felt with subsequent outcomes. Results showed that most players were overconfident. Those who were more overconfident tended to experience less enjoyment with their outcomes. Using individual parameter estimates from decision affect theory, we estimated how each player would have felt if their self assessments had been accurate. For the vast majority, better calibration would have led to greater pleasure. In a second study, we randomly assigned players to a debiasing treatment condition or a control condition. Relative to the control players, debiased players were better calibrated and derived greater average pleasure from the task. Copyright © 2004 John Wiley & Sons, Ltd.     

Negative cognitive errors and positive illusions for negative divorce events: Predictors of children's psychological adjustment

This study examined the relations among negative cognitive errors regarding hypothetical negative divorce events, positive illusions about those same events, actual divorce events, and psychological adjustment in 38 8- to 12-year-old children whose parents had divorced within the previous 2 years. Children's scores on a scale of negative cognitive errors (catastmphizing, overgeneralizing, and personalizing) correlated significantly with self-reported symptoms of anxiety and self-esteem, and with maternal reports of behavior problems. Children's scores on a scale measuring positive illusions (high self-regard, illusion of personal control, and optimism for the future) correlated significantly with less self-reported aggression. Both appraisal types accounted for variance in some measures of symptomatology beyond that explained by actual events. There was no significant association between children's negative cognitive errors and positive illusions. The implications of these results for theories of negative cognitive errors and of positive illusions, as well as for future research, are discussed.     

Validation and error in multiplicative utility functions

An approach to the concept of error in utility assessment is proposed. Four kinds of errors are considered and each kind is related to four separate elicitation methods - all in the context of a general multiplicative multiattribute utility model. The methods are a Keeney-Raiffa (1976) procedure; SMART, for Simple Multi-Attribute Rating Technique (Edwards 1977); SJT, for a Social Judgment Theory based regression model (Hammond et al. 1975); and HOPE, for Holistic Orthogonal Parameter Estimation (Barron and Person 1979).The individual judgments elicited are either holistic - in which the entity to be evaluated is considered as a whole - or decomposed - in which attention is directed to one or two aspects at a time.If a general multiplicative model can be assumed to be an appropriate representation of the decision maker's basic preference structure, error can occur in the direct estimation of the scaling constants and univariate utility functions for decomposition methods (Keeney-Raiffa and SMART), or in the holistic assessments for holistic methods (SJT and HOPE). Individual estimates may be merely subject to random noise or may be substantially incorrect. The utility model may be incorrectly specified; finally, all four methods may be subject to systematic error. The four assessment methods are considered in conjunction with errors of each kind.     

Bayesian modeling of measurement error in predictor variables using item response theory

It is shown that measurement error in predictor variables can be modeled using item response theory (IRT). The predictor variables, that may be defined at any level of an hierarchical regression model, are treated as latent variables. The normal ogive model is used to describe the relation between the latent variables and dichotomous observed variables, which may be responses to tests or questionnaires. It will be shown that the multilevel model with measurement error in the observed predictor variables can be estimated in a Bayesian framework using Gibbs sampling. In this article, handling measurement error via the normal ogive model is compared with alternative approaches using the classical true score model. Examples using real data are given.This paper is part of the dissertation by Fox (2001) that won the 2002 Psychometric Society Dissertation Award.     

Prediction and clinical inference: forty years later

The earlier controversy between clinical and statistical prediction has faded into the shadows. Few people today challenge the necessity for both approaches in seeking to develop and apply knowledge. In more contemporary terms, the issue of which prediction method should be preferred involves choosing between a quest for historical truth (i.e., correspondence demonstrated by statistical methods) and a quest for narrative truth (i.e., coherence achieved by clinical formulations).