Null hypothesis significance testing and Type I error: The domain problem

Although many common uses of p-values for making statistical inferences in contemporary scientific research have been shown to be invalid, no one, to our knowledge, has adequately assessed the main original justification for their use, which is that they can help to control the Type I error rate (Neyman & Pearson, 1928, 1933). We address this issue head-on by asking a specific question: Across what domain, specifically, do we wish to control the Type I error rate? For example, do we wish to control it across all of science, across all of a specific discipline such as psychology, across a researcher's active lifetime, across a substantive research area, across an experiment, or across a set of hypotheses? In attempting to answer these questions, we show that each one leads to troubling dilemmas wherein controlling the Type I error rate turns out to be inconsistent with other scientific desiderata. This inconsistency implies that we must make a choice. In our view, the other scientific desiderata are much more valuable than controlling the Type I error rate and so it is the latter, rather than the former, with which we must dispense. But by doing so—that is, by eliminating the Type I error justification for computing and using p-values—there is even less reason to believe that p is useful for validly rejecting null hypotheses than previous critics have suggested.     

Null hypothesis significance testing. On the survival of a flawed method

Null hypothesis significance testing (NHST) is the researcher's workhorse for making inductive inferences. This method has often been challenged, has occasionally been defended, and has persistently been used through most of the history of scientific psychology. This article reviews both the criticisms of NHST and the arguments brought to its defense. The review shows that the criticisms address the logical validity of inferences arising from NHST, whereas the defenses stress the pragmatic value of these inferences. The author suggests that both critics and apologists implicitly rely on Bayesian assumptions. When these assumptions are made explicit, the primary challenge for NHST--and any system of induction--can be confronted. The challenge is to find a solution to the question of replicability.     

Maintenance of mother-child aversive relationships: A fresh look at an old controversy

Data were obtained from 46 mother-child pairs from the Child Behavior Institute (CBI) at the University of Tennessee. These mother-child dyads were referred for psychological help because of the child’s aversive behavior. Child-parent interactions were quantified using a multi-code, observational system. Correlational and multiple regression analyses (stepwise) were conducted to find the best predictor of the child’s aversive behavior. Results showed that the child’s aversive behavior was significantly correlated with the mother’s aversive, indiscriminate, and compliance behaviors. The regression analysis showed mother’s aversive behavior to be the best predictor of the child’s aversive behavior. Moreover, our results seem to point out that the compliance hypothesis is more effective in predicting child aversive behavior than the predictability hypotheses. It appears that since the two hypotheses are dealing with similar processes, they are more complementary than oppositional. This study used data collected by the Child Behavior Institute (CBI) at the University of Tennessee, Knoxville. The authors are indebted to Robert Wahler, Michael Smith, and Sue Dunn for their helpful comments on this paper.     

Political identification and the defining issues test: reevaluating an old hypothesis

Research on the association between the development of moral judgment (as measured by the Defining Issues Test [DIT]; J. R. Rest, 1979) and political attitudes has demonstrated that these factors are often reliably related. N. Emler (1987, 1990) and colleagues have asserted that DIT scores actually measure test-takers' political identity rather than their developmental level. To test this claim, these researchers have designed "faking studies" in which respondents are asked to complete the DIT as if they were of a particular political orientation, regardless of their real political views. These faking studies have yielded contradictory conclusions, whereas tests of the incremental validity of the DIT have provided some evidence for its empirical distinctiveness. In the present study, the authors reexamined this issue by pitting scores on the DIT, Version 2 (DIT-2; J. R. Rest, D. Narvaez, S. J. Thoma, & M. J. Bebeau, 1999) against several more concrete measures of political identification in several predictive models of attitudes toward human rights and civil liberties. DIT-2 scores and political identification emerged as significant predictors in nearly all regression analyses.     

A new test of an old hypothesis: a quantification of sequence in the Rorschach

Rorschach interpretation often assumes that successive responses are not independent of one another but rather that they are part of a series of interconnected events. In this study, methods that have been used to analyze event sequence data were applied to Rorschach protocols. Results from a nonclinical group of 102 university students showed that location scores on successive responses were repeated more frequently than was predicted by chance. There was also a tendency for subjects to make transitions from larger to smaller or more detailed areas of the inkblot on successive responses. In addition, we found that subjects tended to make transitions from more adequate to less adequate use of form, and that the unusual and minus form categories tended to be repeated. A modest association between transition frequencies and individual differences in anxiety, but not between transition frequencies and depression or overall symptomatology, was demonstrated.     

Answering two criticisms of hypothesis testing: a comment

In a recent article, Leventhal (1999) responds to two criticisms of hypothesis testing by showing that the one-tailed test and the directional two-tailed test are valid, even if all point null hypotheses are false and that hypothesis tests can provide the probability of decisions being correct which are based on the tests. Unfortunately, the falseness of all point null hypotheses affects the operating characteristics of the directional two-tailed test, seeming to weaken certain of Leventhal's arguments in favor of this procedure.     

Apparent size as a function of vertical gaze direction: new tests of an old hypothesis

The hypothesis was tested that the decline of apparent size with elevated gaze results from a latent tendency of the eyes to diverge and thus increased vergence effort. Through the use of a method of category estimation, the decline of apparent size on elevation or depression of gaze was found not to be different between subjects with larger or smaller changes of dark vergence and thus vergence effort. In a 2nd experiment, vergence effort was varied by varying gaze elevation and the angle of convergence. With vergence effort constant, apparent size was dependent on the angle of convergence rather than being constant. It is concluded that apparent size does not depend on vergence effort and that the effect of gaze elevation on apparent size cannot be attributed to its concomitant effect on dark vergence.     

For a psycholinguistic model of handwriting production: testing the syllable-bigram controversy

This study examined the theoretical controversy on the impact of syllables and bigrams in handwriting production. French children and adults wrote words on a digitizer so that we could collect data on the local, online processing of handwriting production. The words differed in the position of the lowest frequency bigram. In one condition, it coincided with the word's syllable boundary. In the other condition, it was located before the syllable boundary. The results yielded higher movement durations at the position where the low-frequency bigram coincided with the syllable boundary compared to where the low-frequency bigram appeared before the syllable boundary. Syllable-oriented strategies failed with the presence of a very low-frequency bigram within the initial syllable. Further analysis showed that children in grades 3 and 4 privileged syllable-oriented programming strategies. The production times of children in grade 4 were also affected by syllable frequency and, to a lesser extent, bigram frequency. The adults writing durations were modulated by bigram frequency. Therefore, both bigrams and syllables regulate handwriting production although the influence of bigrams was stronger in adults than children. In the light of these results, we propose a psycholinguistic model of handwriting production.     

Fisher与Neyman–Pearson的分歧与心理统计中的假设检验争议

假设检验思想的提出者Fisher与Neyman–Pearson在统计模型的方法论基础、两类错误的性质、显著性水平的理解、以及假设检验的功能等方面存在诸多分歧, 使得心理统计中最常用的原假设显著性检验模式呈现出隐含的各种矛盾, 从而引发了应用上的争议。心理统计不仅需要检讨现有检验模型的模糊之处和提出其他补充性的统计推论方式,更应注重反思心理统计的教育传统, 以建立更加开放和多元的统计应用视野, 使心理统计为更好地心理学研究服务。     

Oculomotor control and occipital alpha activity: a review and a hypothesis

Findings relating occipital alpha rhythm with the visual system appear to be difficult to interprete. This is due to the fact that the discrimination between oculomotor control monitored by visual input and monitored by other factors — a distinction which is independent of the presence or absence of visual stimulation — has been widely neglected. A review of the available evidence leads to the conclusion that the inflow of visual information itself has no relation with occipital alpha activity. Blocking occurs only when such information serves also as a monitoring principle in oculomotor position control. Some implications of this hypothesis especially in relation to the concept of alertness and arousal in tasks demanding a high level of visual attention, such as car driving, are discussed.     

Sensory discrimination and intelligence: testing Spearman's other hypothesis

At the centenary of Spearman's seminal 1904 article, his general intelligence hypothesis remains one of the most influential in psychology. Less well known is the article's other hypothesis that there is "a correspondence between what may provisionally be called 'General Discrimination' and 'General Intelligence' which works out with great approximation to one or absoluteness" (Spearman, 1904, p. 284). Studies that do not find high correlations between psychometric intelligence and single sensory discrimination tests do not falsify this hypothesis. This study is the first directly to address Spearman's general intelligence-general sensory discrimination hypothesis. It attempts to replicate his findings with a similar sample of schoolchildren. In a well-fitting structural equation model of the data, general intelligence and general discrimination correlated .92. In a reanalysis of data published byActon and Schroeder (2001), general intelligence and general sensory ability correlated .68 in men and women. One hundred years after its conception, Spearman's other hypothesis achieves some confirmation. The association between general intelligence and general sensory ability remains to be replicated and explained.     

Strategies of clinical hypothesis testing

This research examines whether Psychology students, when they test clinical hypotheses, follow either confirmatory or disconfirmatory reasoning strategies. Two hundred and six psychology students, divided in four groups, participated. One group received information about the probability that the hypothesis was correct by means of verbal labels, and another group, by means of numerical expressions. An additional group received the information that getting a precise diagnosis was clinically important. In a last group, diagnostic tests allowed them to increase certainty about the hypothesis. Results show a partial use of confirmatory strategies because, although participants did not seek confirming information, they indeed avoided collecting disconfirming information. When the information increased certainty about the hypothesis, confirmatory strategies became more likely. Neither the increase in the task importance nor the numerical expression of the likelihood that the hypothesis was correct seemed to affect the testing strategy used.     

关于假设检验的争议:问题的澄清与解决

对传统的假设检验作为心理学实验的数据分析工具的评价涉及两个标准:首先是它是否合法,其次是它是否有用。置于频率学派统计学框架中的传统假设检验在逻辑上实际上是合法的;但它在效用性方面则有着备择假设不可证伪以及只能提供定性结论这两方面的缺陷。置信区间能够集中地改进和弥补这些缺陷。对传统假设检验使用中错误的澄清也使得研究者们开始重视PSI问题,这使得心理学实验的设计和数据分析从关注总体转向关注个体。