Measurement Invariance,Predictive Invariance,and the Duality Paradox

The statistical literature on bias in psychological testing distinguishes at least two forms of bias: measurement bias and predictive bias. Measurement bias concerns group differences in the relationship between a test and the latent variable to be measured. Predictive bias concerns group differences in the relationship between a test and an external criterion. How are these two forms of bias related? For example. if a test is unbiased in the predictive sense, does this fact support the hypothesis that the test is unbiased in the measurement sense? A theorem is given that describes the conditions under which measurement invariance (lack of bias) is consistent with predictive invariance for the linear case. Paradoxically, these two forms of invariance are shown to be inconsistent under realistic conditions. This duality or inconsistency is illustrated in simulated data. The implications of the duality for group differences research are illustrated in real data involving gender and ethnic differences on the SAT. The phenomenon of duality may force a reinterpretation of common empirical findings of test criterion regression slope invariance. and of invariance in test validities. Other implications are discussed.     

On the relationship between sources of within- and between-group differences and measurement invariance in the common factor model

Investigating sources of within- and between-group differences and measurement invariance (MI) across groups is fundamental to any meaningful group comparison based on observed test scores. It is shown that by placing certain restrictions on the multigroup confirmatory factor model, it is possible to investigate the hypothesis that within- and between-group differences are due to the same factors. Moreover, the modeling approach clarifies that absence of measurement bias implies common sources of within- and between-group variation. It is shown how the influence of background variables can be incorporated in the model. The advantages of the modeling approach as compared with other commonly used methods for group comparisons is discussed and illustrated by means of an analysis of empirical data.     

Achievement motivation and problem solving

Abstract.— The relationships between achievement motivation and problem solving were discussed. It was suggested that problem solving may be differentially related to achievement motivation in high and low ability groups. In a group of 46 male students, scoring high on a test of intellectual flexibility, a positive relationship was found between a projective measure of fear of failure and solution of the Hatrack Problem. In a group of 49 male subjects with low flexibility scores, fear of failure was negatively related to problem solving. The results were taken to indicate that the ability measure represented general self confidence and expectancy. Thus subjects with high self confidence might attempt to avoid failure by trying hard to succeed, while a more probable strategy for subjects with low expectancy would be to exert little effort. A test of this hypothesis was made by analyzing a measure of overt activity in the Hatrack situation across high/low motivational and ability groups. Whereas in the high ability group subjects with high fear of failure showed a high level of activity, high fear of failure subjects in the low ability group showed a low level of activity, thus supporting the hypothesis.     

Age-related invariance of abilities measured with the Wechsler Adult Intelligence Scale--III

Examination of measurement invariance tests the assumption that the model underlying a set of test scores is directly comparable across groups. The observation of measurement invariance provides fundamental evidence for the inference that scores on a test afford equivalent measurement of the same psychological traits among diverse groups. Groups may be derived from different psychosocial backgrounds or different clinical presentations. In the Wechsler Adult Intelligence Scale-III (WAIS-III)/Wechsler Memory Scale-III (WMS-III) Technical Manual (Psychological Corporation, 2002), there appears to be a breakdown in factor structure among the standardization cases in older adults. In this study, the authors evaluated the invariance of the measurement model of the WAIS-III across 5 age bands. All components of the measurement model were examined. Overall, the evidence pointed to invariance across age of a modified 4-factor model that included cross-loadings for the Similarities and Arithmetic subtests. These results support the utility of the WAIS-III as a measure of stable intelligence traits across a wide age range.     

Factorial invariance in multilevel confirmatory factor analysis

This paper presents a procedure to test factorial invariance in multilevel confirmatory factor analysis. When the group membership is at level 2, multilevel factorial invariance can be tested by a simple extension of the standard procedure. However level‐1 group membership raises problems which cannot be appropriately handled by the standard procedure, because the dependency between members of different level‐1 groups is not appropriately taken into account. The procedure presented in this article provides a solution to this problem. This paper also shows Muthén's maximum likelihood (MUML) estimation for testing multilevel factorial invariance across level‐1 groups as a viable alternative to maximum likelihood estimation. Testing multilevel factorial invariance across level‐2 groups and testing multilevel factorial invariance across level‐1 groups are illustrated using empirical examples. SAS macro and Mplus syntax are provided.     

An estimate of the Flynn Effect: changes in IQ and subtest gains of 10-yr-old French children between 1965 and 1988

The significant increase in IQ in the industrialised countries during recent decades, known as the eponymous Flynn Effect, tends to be given numerous explanations (progress in schooling, television, nutrition, etc.). One way of studying this effect consists of assessing in which subtests the change in IQ takes place. Using the comparison among 120,000 French children on an intelligence test in 1965 and the results of 8640 children on the same test in 1988, Greenfield's hypothesis is discussed: that television and other media using graphical representations have induced an improvement in spatial and visual capacities to the detriment of capacities relating to vocabulary. The results are compatible with Greenfield's hypothesis, although other explanations are still possible.     

Factor and cluster analysis of the Intermediate Halstead Category Test

The factorial invariance of the six subtests of the Intermediate version of the Halstead Category Test (IHCT), as well as clusters of IHCT subtest patterns, were evaluated in a sample of 87 children with traumatic brain injuries. A two-factor solution was found which partially replicated that found in previous research with children with learning disabilities, supporting the assumption that the IHCT measures multiple underlying constructs in clinical populations. Four clusters were found, with differences between clusters determined to a substantial extent by relative accuracy on subtest 111, and to a less substantial extent by relative accuracy on subtests IV and V. Clinical and research implications are discussed.     

Non-verbal and verbal judgements of length invariance by young children.

One hundred and fifty non-conservers in length were assessed for their ability to make a non-verbal judgement of length invariance on a task employing a train-transfer design in which the stimuli were pairs of pencils. Seventy per cent of the children from the group who had to respond to length equality as opposed to inequality made the correct invariance judgement and could, moreover, characterize their choices in invariance language (e.g. 'same size') whilst still failing the standard verbal task in which such phrases were used by the experimenter. This is regarded as supporting the hypothesis that the non-conserver's linguistic difficulty is not with framing the verbal judgement but in interpreting a question which he believes has to be interpreted unidimensionally in such a context of perceptual change.     

Empirically Derived Ability-Achievement Subtypes in a Clinical Sample

The purpose of the present study was to identify reliable and clinically meaningful patterns of ability and achievement using the WISC-III and WIAT. Cluster analysis was used to group the 182 WISC-III and WIAT profiles (10 WISC-III subtests and 4 WIAT subtests) of children between the ages of 9 and 14 years. Theoretical and empirical considerations were used to identify a cluster solution, which involved comparison of several five-, six- and eight-cluster solutions. A five-cluster solution was selected as being representative of the data, which was well replicated across three hierarchical clustering methods (i.e., complete linkage, average linkage-within groups, and average linkage-between groups (UPGMA)). The clusters were labeled based on their most salient characteristics, which included a group of predominantly low ability and achievement, a group demonstrating a pattern of verbal processing deficits, a group demonstrating a pattern of visual spatial/processing speed deficits, a group with low ability and achievement with average processing speed, and a group with deficits consistent with an ACID pattern. The external validity of the five subtypes was assessed through an evaluation of the relationship between cluster membership and neuropsychological test data. Most predictions regarding neuropsychological performance were supported by the data, providing further evidence of the validity of the five-cluster solution. Clinical implications of the ability-achievement typology and suggestions for future research are discussed.     

Empirically derived ability-achievement subtypes in a clinical sample.

The purpose of the present study was to identify reliable and clinically meaningful patterns of ability and achievement using the WISC-III and WIAT. Cluster analysis was used to group the 182 WISC-III and WIAT profiles (10 WISC-III subtests and 4 WIAT subtests) of children between the ages of 9 and 14 years. Theoretical and empirical considerations were used to identify a cluster solution, which involved comparison of several five-, six- and eight-cluster solutions. A five-cluster solution was selected as being representative of the data, which was well replicated across three hierarchical clustering methods (i.e., complete linkage, average linkage-within groups, and average linkage-between groups (UPGMA)). The clusters were labeled based on their most salient characteristics, which included a group of predominantly low ability and achievement, a group demonstrating a pattern of verbal processing deficits, a group demonstrating a pattern of visual spatial/processing speed deficits, a group with low ability and achievement with average processing speed, and a group with deficits consistent with an ACID pattern. The external validity of the five subtypes was assessed through an evaluation of the relationship between cluster membership and neuropsychological test data. Most predictions regarding neuropsychological performance were supported by the data, providing further evidence of the validity of the five-cluster solution. Clinical implications of the ability-achievement typology and suggestions for future research are discussed.     

WAIS-III task perception and performance as a function of stopwatch display

Examiner's instructions for the Wechsler Adult Intelligence Scale-Third Edition indicate that stopwatch display is optional. Since verbal instructions do not consistently inform examinees whether a subtest is timed, examinees may rely upon the visual cue of the stopwatch. Furthermore, examinees' understanding of the actual subtest demands may affect their performances. In the present study, 58 volunteers were randomly assigned to two groups, one in which the examiner displayed the stopwatch and one in which the stopwatch was concealed. Participants were administered the first five subtests. Predictions were that (1) participants to whom the stopwatch was displayed would more accurately perceive whether the subtests were timed, and (2) participants who accurately understood the demands of subtests would score higher than those who did not. Results supported the first hypothesis for three subtests out of five, one remaining inconclusive due to a ceiling effect. The second hypothesis was not supported. Both the implications and limitations of the present study are discussed.     

Children's subjective ratings of their performance on a standardized achievement test

Relationships between children's perceptions of performance on a standardized achievement test and the actual test performance were examined. A total of 311 children in grades 4, 5, and 6 were administered the Iowa Tests of Basic Skills on four separate testing days. At the end of the testing session on Day 4, the children completed a questionnaire in which they rated their performance on each of the subtests as well as their overall score on the test. Students also were asked to make judgments about the difficulty of the test, the effort expended, the extent that ability versus luck was a factor in their test performance, and the importance of the test to them, their parents, and their teachers. The results showed that students were quite accurate in predicting their scores on the subtests and that a personal belief that ability decisively influences achievement associated with their test performance. These findings are discussed as they relate to the effect of self-statements on test performance.     

IRT框架下追踪数据的测量不变性分析——————以4至5岁儿童认知能力测验为例

本研究以4岁~5岁儿童认知能力测验为例,在IRT框架下探讨了如何进行追踪数据的测量不变性分析。分析模型采用项目间多维项目反应理论模型(between-item MIRT model)和项目内（within-item MIRT model）多维two-tier model,被试为来自全国的882名48个月的儿童,工具为自编4岁~5岁儿童认知能力测验。经测验水平 分析和项目水平分析,结果表明：(1)本文对追踪数据的测量不变性分析方法合理有效; (2)该测验在两个时间点上满足部分测量不变性要求,测验的潜在结构稳定; (3)“方位题”的区分度和难度参数都发生变化,另有4题难度参数出现浮动; (4)儿童在4岁~5岁期间认知能力总体呈快速发展趋势,能力增长显著。     

Decreasing scoring errors on Wechsler Scale Vocabulary, Comprehension, and Similarities subtests: a preliminary study

Studies of graduate students learning to administer the Wechsler scales have generally shown that training is not associated with the development of scoring proficiency. Many studies report on the reduction of aggregated administration and scoring errors, a strategy that does not highlight the reduction of errors on subtests identified as most prone to error. This study evaluated the development of scoring proficiency specifically on the Wechsler (WISC-IV and WAIS-III) Vocabulary, Comprehension, and Similarities subtests during training by comparing a set of 'early test administrations' to 'later test administrations.' Twelve graduate students enrolled in an intelligence-testing course participated in the study. Scoring errors (e.g., incorrect point assignment) were evaluated on the students' actual practice administration test protocols. Errors on all three subtests declined significantly when scoring errors on 'early' sets of Wechsler scales were compared to those made on 'later' sets. However, correcting these subtest scoring errors did not cause significant changes in subtest scaled scores. Implications for clinical instruction and future research are discussed.     

Preschoolers' construction order of event sequences related to mental ability and degree of familiarity.

Preschoolers' understanding of sequential relations was investigated in two experiments. Experiment 1 showed that appreciation of sequential relations depends on the level of mental ability and the degree of familiarity displayed with the social or behavioral phenomena under investigation. Experiment 2 reproduced the findings of Experiment 1 and, further, demonstrated that an increase in experience and familiarity through an interaction phase between children with higher and those with lower mental abilities led to a significant exchange of information and performance that was retained over the posttest phase. The data of both experiments are discussed within the framework of the "invariance hypothesis" (Bullock, 1985) in contrast to Piagetian theory (Piaget, 1946/1969).     

Sex differences in impulsivity and intellectual ability

Literature on sex differences in impulsivity has tended to focus on differences between groups, while ignoring relationships beyond zero order correlations. The purpose of this study was to evaluate Maccoby's hypothesis (1966) of opposite-direction correlations where the relationship between a set of variables (e.g., impulsivity and intellectual ability) is curvilinear, with males and females being systematically distributed on opposite sides of the curve. The Primary Mental Abilities (PMA) and Matching Familiar Figures tests were administered to 44 males and 49 females. The relationship between all of the PMA subtests and impulsivity was curvilinear for males, but linear for females on all but one of the subtests. The importance and implications of examining distribution differences rather than group differences are discussed.Support for this study has come from the Office of Research Administration, University of Kansas, Grant Number 3581-5038. This paper was presented in part at the meeting of the American Psychological Association, Chicago, August 1975, Division 7.     

Word memory test performance in children.

One-hundred and thirty-five children between the ages of 7 and 18 years were evaluated clinically. Their diagnoses included Fetal Alcohol Syndrome (FAS) or Effects, Schizophrenia, Bipolar Mood Disorder, various neurological diseases, Attention Deficit Hyperactivity Disorder (ADHD), Conduct Disorder, Oppositional-Defiant Disorder and learning disabilities. As part of a comprehensive neuropsychological assessment, the children were given the Word Memory Test (WMT; Green, Allen, & Astner, 1996; Green & Astner, 1995), containing various subtests which measure, respectively, effort and verbal memory. Although age and verbal intelligence are known to affect scores on most ability tests, they were not found to be significant determinants of WMT effort scores. Younger children did not score any lower on the effort subtests than older children. The children scored as well as a group of parents seeking custody of their children and they scored higher than adult patients with mild head injuries. The computerized WMT requires some basic reading skills and some children with lower than a grade 3 reading level scored at a relatively low level on the effort subtests. The current data suggest that most children with at least a grade 3 reading level can pass the WMT using the adult criteria. It is concluded that the WMT is potentially useful in the evaluation of effort during pediatric neuropsychological evaluations. Further research is needed to replicate these findings and to develop child norms for the memory subtests.     

Word Memory Test Performance in Children

One-hundred and thirty-five children between the ages of 7 and 18 years were evaluated clinically. Their diagnoses included Fetal Alcohol Syndrome (FAS) or Effects, Schizophrenia, Bipolar Mood Disorder, various neurological diseases, Attention Deficit Hyperactivity Disorder (ADHD), Conduct Disorder, Oppositional-Defiant Disorder and learning disabilities. As part of a comprehensive neuropsychological assessment, the children were given the Word Memory Test (WMT; Green, Allen, & Astner, 1996; Green & Astner, 1995), containing various subtests which measure, respectively, effort and verbal memory. Although age and verbal intelligence are known to affect scores on most ability tests, they were not found to be significant determinants of WMT effort scores. Younger children did not score any lower on the effort subtests than older children. The children scored as well as a group of parents seeking custody of their children and they scored higher than adult patients with mild head injuries. The computerized WMT requires some basic reading skills and some children with lower than a grade 3 reading level scored at a relatively low level on the effort subtests. The current data suggest that most children with at least a grade 3 reading level can pass the WMT using the adult criteria. It is concluded that the WMT is potentially useful in the evaluation of effort during pediatric neuropsychological evaluations. Further research is needed to replicate these findings and to develop child norms for the memory subtests.     

Measurement invariance of WISC-IV across normative and clinical samples

Measurement invariance of the WISC-IV second-order factorial structure between normative and clinical samples was investigated using WISC-IV core subtests and a total of 1100 children aged 6-16. Multi-group higher order analysis of mean and covariance structure (MG-MACS) models were used to analyze these data. Results supported measurement invariance. Only Coding and Comprehension subtest intercepts varied slightly between groups. The hypothesized WISC-IV factor model described the data well. Factor patterns, first- and second-order factor loadings, intercepts, residual variances of measured subtests, and disturbances of first-order factors of the WISC-IV were generally invariant. Results suggested that WISC-IV index scores and subtests have the same meaning for children in both normative and clinical groups.     

Ability and effort weights in pay level and pay increase decisions

It was hypothesized that ability, effort, and the interaction between ability and effort would influence pay level and pay increase decisions. An experiment was conducted with 66 human resource professionals to test this hypothesis. The results indicate that pay level decisions are influenced primarily by ability, and pay increase decisions are influenced primarily by the interaction between ability and effort. The results are discussed in the context of the traditional compensation model which suggests that ability should influence pay level, but not pay increase, and that effort should influence pay increase, but not pay level. It was concluded that the distinction currently being made between person and job based pay plans may not be meaningful.Portions of this paper were presented to the PHR division of the Southern Management Association, Atlanta, November, 1991.The authors wish to thank Marcia Miceli, Eileen Hogan, and Don Eskew, for their helpful comments on earlier drafts.