Use of the extreme groups approach: a critical reexamination and new recommendations

Analysis of continuous variables sometimes proceeds by selecting individuals on the basis of extreme scores of a sample distribution and submitting only those extreme scores to further analysis. This sampling method is known as the extreme groups approach (EGA). EGA is often used to achieve greater statistical power in subsequent hypothesis tests. However, there are several largely unrecognized costs associated with EGA that must be considered. The authors illustrate the effects EGA can have on power, standardized effect size, reliability, model specification, and the interpretability of results. Finally, the authors discuss alternative procedures, as well as possible legitimate uses of EGA. The authors urge researchers, editors, reviewers, and consumers to carefully assess the extent to which EGA is an appropriate tool in their own research and in that of others.     

Relative costs and statistical power in the extreme groups approach

The relationship between variables in applied and experimental research is often investigated by the use of extreme (i.e., upper and lower) groups. Recent analytical work has provided an extreme groups procedure that is more powerful than the standard correlational approach for all values of the correlation and extreme group size. The present article provides procedures to optimize power by determining the relative number of subjects to use in each of two stages of data collection given a fixed testing budget.The opinions or assertions contained herein are those of the writers and are not to be construed as official or reflecting the views of the Navy Department.Edward F. Alf, Jr. is also affiliated with the Department of Psychology, San Diego State University.     

Predicting group-level outcome variables: An empirical comparison of analysis strategies

This study provides a review of two methods for analyzing multilevel data with group-level outcome variables and compares them in a simulation study. The analytical methods included an unadjusted ordinary least squares (OLS) analysis of group means and a two-step adjustment of the group means suggested by Croon and van Veldhoven (2007). The Type I error control, power, bias, standard errors, and RMSE in parameter estimates were compared across design conditions that included manipulations of number of predictor variables, level of correlation between predictors, level of intraclass correlation, predictor reliability, effect size, and sample size. The results suggested that an OLS analysis of the group means, with White’s heteroscedasticity adjustment, provided more power for tests of group-level predictors, but less power for tests of individual-level predictors. Furthermore, this simple analysis avoided the extreme bias in parameter estimates and inadmissible solutions that were encountered with other strategies. These results were interpreted in terms of recommended analytical methods for applied researchers.     

Simultaneous factor analysis in several populations

This paper is concerned with the study of similarities and differences in factor structures between different groups. A common situation occurs when a battery of tests has been administered to samples of examinees from several populations.A very general model is presented, in which any parameter in the factor analysis models (factor loadings, factor variances, factor covariances, and unique variances) for the different groups may be assigned an arbitrary value or constrained to be equal to some other parameter. Given such a specification, the model is estimated by the maximum likelihood method yielding a large samplex ² of goodness of fit. By computing several solutions under different specifications one can test various hypotheses.The method is capable of dealing with any degree of invariance, from the one extreme, where nothing is invariant, to the other extreme, where everything is invariant. Neither the number of tests nor the number of common factors need to be the same for all groups, but to be at all interesting, it is assumed that there is a common core of tests in each battery that is the same or at least content-wise comparable.This research was supported by grant NSF-GB-12959 from National Science Foundation. My thanks are due to Michael Browne for his comments on an earlier draft of this paper and to Marielle van Thillo who checked the mathematical derivations and wrote and debugged the computer program SIFASP.Now at Statistics Department, University of Uppsala, Sweden.     

Exploring MMPI-Rorschach relationships

Limitations exist in the current literature examining Minnesota Multiphasic Personality Inventory (MMPI)-Rorschach interrelationships. In this article I attempt to identify several issues to be addressed by researchers in this area. These include restricting conclusions about MMPI-Rorschach relationships to research using the Comprehensive System; considering the relative reliability and validity of these tests; comparing the predictive power of the Rorschach and MMPI; examining the effects extreme outliers have on statistical analyses; and determining the appropriate level of analysis for research, the level of individual test scores, or psychological constructs. The implications of these issues for research design, statistical analysis, and applied clinical work are discussed.     

Extent of psychological differentiation and creativity.

Among the extensive array of studies relating Witkin's construct of psychological differentiation to many psychological functions and variables, none seem to have adequately tested its relationship to creativity, although these two variables have personality traits and development characteristics in common and require similar skills for effective performance. Eighty-five male high school students were separated according to Embedded Figures Test, and the extreme groups tested for creativity by eight factorially pure tests developed by Guilford, and by Schaefer and Anastasi's Biographical Inventory. Statistical procedures included analysis of covariance, analysis of variance, and a matrix of intercorrelations. With intelligence controlled, field-independent persons performed significantly better than field-dependent persons on measures of visualization, spontaneous flexibility, adaptive flexibility, and creativity in mathematics and science but not significantly better on the other six creativity tests.     

Exploring MMPI-Rorschach Relationships

Limitations exist in the current literature examining Minnesota Multiphasic Personality Inventory (MMPI)—Rorschach interrelationships. In this article I attempt to identify several issues to be addressed by researchers in this area. These include restricting conclusions about MMPI—Rorschach relationships to research using the Comprehensive System; considering the relative reliability and validity of these tests; comparing the predictive power of the Rorschach and MMPI; examining the effects extreme outliers have on statistical analyses; and determining the appropriate level of analysis for research, the level of individual test scores, or psychological constructs. The implications of these issues for research design, statistical analysis, and applied clinical work are discussed.     

The use of extreme groups in assessing relationships

The relationship between variables in applied and experimental research is often investigated by the use of extreme (i.e., upper and lower) groups. Earlier analytical work has demonstrated that the extreme groups procedure is more powerful than the standard correlational approach for some values of the correlation and extreme group size. The present article provides methods for using the covariance information that is usually discarded in the classical extreme groups approach. Essentially, then, the new procedure combines the extreme groups approach and the correlational approach. Consequently, it includes the advantages of each and is shown to be more powerful than either approach used alone.     

Individual differences in anterior brain asymmetry and fundamental dimensions of emotion.

This research assessed whether individual differences in anterior brain asymmetry are linked to differences in basic dimensions of emotion. In each of 2 experimental sessions, separated by 3 weeks, resting electroencephalogram (EEG) activity was recorded from female adults during 8 60-s baselines. Mean alpha power asymmetry across both sessions was extracted in mid-frontal and anterior temporal sites. Across both regions, groups demonstrating stable and extreme relative left anterior activation reported increased generalized positive affect (PA) and decreased generalized negative affect (NA) compared with groups demonstrating stable and extreme relative right anterior activation. Additional correlational analyses revealed robust relations between anterior asymmetry and PA and NA, particularly among subjects who demonstrated stable patterns of EEG activation over time. Anterior asymmetry was unrelated to individual differences in generalized reactivity.     

Neuropsychological assessment in illiterates : II. Language and praxic abilities

A basic neuropsychological examination of language and praxic abilities was administered to extreme educational groups (100 illiterates and 100 professionals). Subjects were matched according to sex and age (16-25, 26-35, 36-45, 46-55, and 56-65). The following tasks were included: language comprehension, phonological discrimination, naming (objects, figures, and body parts), repetition of words, verbal fluency, calculation, buccofacial and ideomotor praxis, finger alternating movements, meaningless movements, cancellation task, coordinated movements with both hands, and motor impersistence tasks. All the differences between the two educational groups were statistically significant. Two of the eight language tests (phonological discrimination and naming figures) and three of the seven praxic tests (buccofacial praxis, coordinated movements, and cancellation task) showed differences between age groups with a better performance in the younger groups. Calculation tasks and ideomotor praxis showed differences between sexes with a better performance in males. Influence of educational factors in performing routine neuropsychological tests is analyzed.     

Attitude Strength and Perceived Prevalence of Attitude Positions

Two studies are reported that evaluate the hypothesis that people with involved or extreme (as opposed to moderate) attitudes on an issue will tend to overestimate the extent to which others' attitudes are polarized toward both extremes (cf. Judd & Johnson, 1981). Unlike previous tests of this hypothesis, the present studies included a complete range of subject attitude groups and target categories, obtaining prevalence estimates on a variety of nuclear-related issues. However, support for the hypothesis was limited, being clearest for pronuclear subjects' estimates of pronuclear positions but almost nonexistent for other subjects and target categories. It is suggested that the cognitive mechanism proposed by Judd and Johnson needs to be supplemented by broader social, functional, and knowledge-based considerations.     

NOTE ON THE PREDICTION OF FACTORIALLY COMPLEX CLASSES

Indices of the discriminatory power of items, based on the differences in performances of groups, can be influenced by the degree of factorial mixture with respect to such groups. An illustrative example is presented, using hypothetical data. Items, normally rejected on the basis of such discriminatory tests, may be of value for predictive purposes. An empirical example of such a use is offered.     

An adjusted measure of achievement motivation for males and females and effects of future orientation on level of performance

An adjusted measure of achievement motivation was devised with the combined use of the Mehrabian Achievement Scale (Educational and Psychological Measurement, 1968, 28, 493–502) and the Mandler and Sarason (Journal of Abnormal and Social Psychology, 1952, 47, 166–173) Test Anxiety Questionnaire. Contingent instructions, in which Ss were told success on a test was necessary to qualify for opportunities to succeed on subsequent tests, were contrasted to performance in equivalent noncontingent situations and relative to adjusted achievement motivation for both male and female subjects. In noncontingent situations Ss were told performance on any one test did not affect their opportunity to take subsequent tests. Support was obtained for the use of the above mentioned scales as a predictor of performance for both sexes but not for the Mehrabian by itself. Both male and female Ss high in achievement motivation consistently scored significantly higher than Ss low in achievement motivation in all conditions. The expected interactions between treatments and motive groups were not found but t tests between extreme motive groups within conditions revealed significant differences in the expected directions. These results do not provide statistically reliable support to the elaborated theory of achievement motivation.     

Evaluating socioeconomic status bias in the WISC-R

The use of intelligence tests in making special education placement decisions for children has come under repeated scrutiny in recent years. Arguments for and against the use of IQs have centered around the issue of test bias. In California a permanent moratorium on the use of intelligence tests in placing minority group children into classes for the educable retarded has been handed down by the court. One defense proposed by advocates of mental testing has been that the tests are biased against socially and economically disadvantaged groups irrespective of race. According to Clarizio (1978), if a test predicts equally well for two groups it can not legitimately be described as biased. The purpose of this investigation was to evaluate the test bias of the Verbal IQ as measured by the Wechsler Intelligence Scale for Children-Revised in relation to predictions of academic achievement in two different socioeconomic status groups. The results indicated that Verbal IQs are not biased with respect to socioeconomic status. Derived regression equations are presented along with an analysis to inspect the statistical power of the original tests.     

The effect of variations in answer-sheet format on aptitude test performance

Three different answer sheets were used in the administration of the General Aptitude Test Battery to groups of young adults (n = 302). The answer sheets, designed for use with different brands of Optical Mark Reader, varied in the shape of the response space. A two-factor, repeated measures analysis of variance revealed a significant interaction between subtests and type of answer sheet. Further analyses showed that performance on ‘speed’ tests was affected by the format of answer sheet used, whereas performance on mainly ‘power’ tests was not. The implications of the results for testing practice are discussed.     

The Road to Hell: Neutralization of Killing in War

We explore how societies convince citizens to “go to hell,” where they kill strangers and risk their own lives. Our premise is that killing is difficult for most, regardless of context and widely considered a form of extreme deviance. Using Sykes and Matza’s techniques of neutralization, we show how this aversion to killing is neutralized by placing individuals in organized—both macro-level bureaucratic and micro-level small primary groups—to justify killing. Consequently, the neutralization of mass killing illustrates the role of power in defining deviance, and the failure of neutralizations may sometimes explain Post-Traumatic Stress Disorder.     

Sensitivity of MRQAP Tests to Collinearity and Autocorrelation Conditions

Multiple regression quadratic assignment procedures (MRQAP) tests are permutation tests for multiple linear regression model coefficients for data organized in square matrices of relatedness among n objects. Such a data structure is typical in social network studies, where variables indicate some type of relation between a given set of actors. We present a new permutation method (called “double semi-partialing”, or DSP) that complements the family of extant approaches to MRQAP tests. We assess the statistical bias (type I error rate) and statistical power of the set of five methods, including DSP, across a variety of conditions of network autocorrelation, of spuriousness (size of confounder effect), and of skewness in the data. These conditions are explored across three assumed data distributions: normal, gamma, and negative binomial. We find that the Freedman–Lane method and the DSP method are the most robust against a wide array of these conditions. We also find that all five methods perform better if the test statistic is pivotal. Finally, we find limitations of usefulness for MRQAP tests: All tests degrade under simultaneous conditions of extreme skewness and high spuriousness for gamma and negative binomial distributions. Special thanks go to Cajo Ter Braak, Philip Hans Franses, Patrick Houweling, Pierre Legendre, three anonymous reviewers, the associate editor, and the editor for comments.     

Testing for Measurement Invariance with Respect to an Ordinal Variable

Researchers are often interested in testing for measurement invariance with respect to an ordinal auxiliary variable such as age group, income class, or school grade. In a factor-analytic context, these tests are traditionally carried out via a likelihood ratio test statistic comparing a model where parameters differ across groups to a model where parameters are equal across groups. This test neglects the fact that the auxiliary variable is ordinal, and it is also known to be overly sensitive at large sample sizes. In this paper, we propose test statistics that explicitly account for the ordinality of the auxiliary variable, resulting in higher power against “monotonic” violations of measurement invariance and lower power against “non-monotonic” ones. The statistics are derived from a family of tests based on stochastic processes that have recently received attention in the psychometric literature. The statistics are illustrated via an application involving real data, and their performance is studied via simulation.     

Person-Centered Assessment

The use of tests and assessment in person-centered counseling is discussed. In person-centered counseling, the most fundamental value is honoring the client as the authority on his or her own life. The extreme focus of person-centered theory on the “self-authority” tends to militate against the use of tests and assessment. Person-centered counseling includes unsystematic counselor responses, such as reactions, questions, and suggestions. This increasing involvement of the counselor as a person has implications for the use of tests in person-centered counseling.     

Neuropsychological abilities of preschool-aged children who display hyperactivity and/or oppositional-defiant behavior problems.

This study focused on gaining a better understanding of the neuropsychological abilities of preschool-aged children who show elevated levels of hyperactivity and oppositional-defiance. It examined the performance of children aged 48 to 67 months on tests of attention/executive function, language, memory, and sensorimotor abilities, as measured by the NEPSY and Conners' K-CPT. Two hundred thirty-seven children were divided into four subgroups based on mothers' report of behavior using rating scales and a diagnostic interview: hyperactive only (HYP), oppositional-defiant only (OD), hyperactive and oppositional-defiant (HYP/OD), and nonproblem. Children in the HYP/OD group scored significantly worse than nonproblem children on four of nine subtests on the NEPSY, including one test of executive function, one test of language comprehension, and both tests of short-term verbal memory. However, only the test of executive function (Statue) showed significant predictive power, and, while specificity of this subtest was good, sensitivity was poor. On the K-CPT, a continuous performance test, children in both the HYP and HYP/OD groups performed worse than children in the OD and nonproblem groups. When the NEPSY Statue subtest and the K-CPT were used together, overall predictive power was .74. Results suggest that neuropsychological deficits can be observed among preschool children with hyperactivity, particularly when comorbid oppositional-defiance is present; however, moderate predictive power suggests that these tests should be used in conjunction with other methods of assessment.