A Comparative Investigation of Rotation Criteria Within Exploratory Factor Analysis

Exploratory factor analysis (EFA) is a commonly used statistical technique for examining the relationships between variables (e.g., items) and the factors (e.g., latent traits) they depict. There are several decisions that must be made when using EFA, with one of the more important being choice of the rotation criterion. This selection can be arduous given the numerous rotation criteria available and the lack of research/literature that compares their function and utility. Historically, researchers have chosen rotation criteria based on whether or not factors are correlated and have failed to consider other important aspects of their data. This study reviews several rotation criteria, demonstrates how they may perform with different factor pattern structures, and highlights for researchers subtle but important differences between each rotation criterion. The choice of rotation criterion is critical to ensure researchers make informed decisions as to when different rotation criteria may or may not be appropriate. The results suggest that depending on the rotation criterion selected and the complexity of the factor pattern matrix, the interpretation of the interfactor correlations and factor pattern loadings can vary substantially. Implications and future directions are discussed.     

MBTI人格类型量表的效度分析

目的探讨中文版MBTI人格类型量表的内容效度、效标关联效度和结构效度,为其在中国应用提供操作性技术,方法大学本科和专科学生2123名,陆军初级军官276名;MBTI-G量表中文修订版;效标测验包括EPQ、16PF、MMPI-2、A-Type和PM测验。结果(1)经专家评判、中英文版相关分析、自评他评和信度分析,表明中文版MBTI有较好的内容效度。(2)效标关联效度研究发现EI维度具有明显的内外向人格特征;感觉型个体温和、现实和谨慎,直觉型个体则恃强、敢为、果断和中强度A型行为特征;对事型个体稳重、安详、恃强、自律;判断型个体善于交往和社会化程度高,做事有强的责任感、计划性和有恒性,适应新环境能力较强,成就感强。以上发现与MBTI原设计和国外研究吻合。(3)97项题目因子分析最大负荷落在主因素上平均占82．81％,次级负荷占11.02％,仅6题因子分析不理想。(4)修订版MBTI人格类型测验与PM领导行为类型测验间有一定相关;中国军队初级指挥员以ESFJ、ISTJ人格类型为主。结论本研究修订的中文版MBTI具有较好的内容效度、效标关联蚊度和结构效度。     

Construct interpretation of Minnesota Multiphasic Personality Inventory (MMPI) clinical scales: An orthogonal transformation

The implications for personality test construction of the revolution in testing caused by construct validity considerations are outlined, with particular relevance to the assessment of psychopathology. These include (a) substantive definition of constructs; (b) concern for internal consistency reliability as well as generalizability; (c) evaluation of structural relationships among items and scales; (d) suppression of response biases; (e) emphasis on minimum redundancy among scales; (f) evaluation of convergent and discriminant validity of scales and profiles; and (g) evaluation of criterion validity for configurations of scales and profiles, as well as single scales. Benefits are seen as accruing to an increased understanding of psychopathology and higher levels of validity. Prior, and subsequent, to the forthcoming revision of the Minnesota Multiphasic Personality Inventory (MMPI), one approach to realizing some of the aims of construct measurement with an empirically based test is through an orthogonal transformation of the scales. Preliminary results for the extant MMPI clinical scales are reported, yielding evidence of (a) scale independence while retaining high correlations with uncorrected scales, (b) an appropriate pattern of correlations with a separate set of new scales of psychopathology, (c) a possible basis for new item analyses, and (d) freedom from correlations with a putative measure of response bias. Implications of the orthogonal transformation for profile interpretation are discussed.Portions of this paper were presnted at an invited address, 18th Annual Symposium on Recent Developments in the Use of the MMPI, Minneapolis, April 9, 1983. This paper was written while Douglas N. Jackson was distinguished visiting professor at the College of Education, The University of Iowa. This research has been supported by Research Grant 895-84/86 from the Ontario Mental Health Foundation, Research Grant 411-83-0014 from the Social Sciences and Humanities Research Council of Canada, and the Alberta Hospital Edmonton.     

Simple Structure Detection Through Bayesian Exploratory Multidimensional IRT Models

In modern validity theory, a major concern is the construct validity of a test, which is commonly assessed through confirmatory or exploratory factor analysis. In the framework of Bayesian exploratory Multidimensional Item Response Theory (MIRT) models, we discuss two methods aimed at investigating the underlying structure of a test, in order to verify if the latent model adheres to a chosen simple factorial structure. This purpose is achieved without imposing hard constraints on the discrimination parameter matrix to address the rotational indeterminacy. The first approach prescribes a 2-step procedure. The parameter estimates are obtained through an unconstrained MCMC sampler. The simple structure is, then, inspected with a post-processing step based on the Consensus Simple Target Rotation technique. In the second approach, both rotational invariance and simple structure retrieval are addressed within the MCMC sampling scheme, by introducing a sparsity-inducing prior on the discrimination parameters. Through simulation as well as real-world studies, we demonstrate that the proposed methods are able to correctly infer the underlying sparse structure and to retrieve interpretable solutions.     

Overclaiming as a measure of faking

Researchers have recently asserted that popular measures of response distortion (i.e., socially desirable responding scales) lack construct validity (i.e., measure traits rather than test faking) and that applicant faking on personality tests remains a serious concern ( [Griffith and Peterson, 2008] and [Holden, 2008]). Thus, although researchers and human resource (HR) selection specialists have been attempting to find measures which readily capture individual differences in faking that increase personality test validity, to date such attempts have rarely, if ever succeeded. The current study, however, finds that the overclaiming technique captures individual differences in faking and subsequently increases personality test score validity via suppressing unwanted error variance in personality test scores. Implications of this research on the overclaiming technique for improving HR selection decisions are illustrated and discussed.     

On the myth and the reality of the temporal validity degradation of general mental ability test scores

Claims of changes in the validity coefficients associated with general mental ability (GMA) tests due to the passage of time (i.e., temporal validity degradation) have been the focus of an on-going debate in applied psychology. To evaluate whether and, if so, under what conditions this degradation may occur, we integrate evidence from multiple sub-disciplines of psychology. The temporal stability of construct validity is considered in light of the evidence regarding the differential stability of g and the invariance of measurement properties of GMA tests over the adult life-span. The temporal stability of criterion-related validity is considered in light of evidence from long-term predictive validity studies in educational and occupational realms. The evidence gained from this broad-ranging review suggests that temporal degradation of the construct- and criterion-related validity of ability test scores may not be as ubiquitous as some have previously concluded. Rather, it appears that both construct and criterion-related validity coefficients are reasonably robust over time and that any apparent degradation of criterion-related validity coefficients has more to do with changes in the determinants of task performance and changes in the nature of the criterion domain rather temporal degradation per se (i.e., the age of the test scores). A key exception to the conclusion that temporal validity degradation is more myth than reality concerns decision validity. Although the evidence is sparse, it is likely that the utility of a given GMA test score for making diagnostic decisions about an individual deteriorates over time. Importantly, we also note several areas in need of additional and more rigorous research before strong conclusions can be supported.     

An examination of the construct validity of posttraumatic stress disorder with veterans using a revised criterion set

Ongoing concerns exist in the literature regarding the construct of posttraumatic stress disorder (PTSD) and how to best conceptualize and measure this disorder. We compared the traditional DSM-IV PTSD symptom criteria (i.e., symptoms from clusters B, C, and D) to a revised criterion set that omits overlapping mood and other anxiety symptoms on PTSD prevalence, PTSD diagnostic caseness, associated psychiatric comorbidity, functional status, and structural validity using a cross-sectional, multi-site primary care sample of 747 veterans. After removing items theorized to overlap with mood and other anxiety disorders, PTSD prevalence was identical using both criterion sets (i.e., 12%). Overall, there were few statistically significant differences in PTSD caseness, associated psychiatric comorbidity, functional status, and structural validity across the two diagnostic criterion sets. These data provide further support that removing items that overlap with other psychiatric disorders does not significantly impact the prevalence of PTSD, its associated comorbidity and functional impairment, or its structural validity. Although the revised criterion set represents a more parsimonious model, the current study findings generally support the strong construct validity of PTSD. The implications of these study findings for research and clinical practice are discussed.     

The Ability to Identify Criteria: Its Relationship with Social Understanding,Preparation, and Impression Management in Affecting Predictor Performance in a High-Stakes Selection Context

The ability to correctly identify the criteria (ATIC) that are being evaluated in a selection procedure is a relatively new construct attracting attention because of its significant relationship with performance in selection assessments and its potential for explaining their criterion validity. Multiple source data from 319 applicants undergoing a high-stakes medical student selection procedure were used to test a moderated mediation model that extends current understanding of the nomological net of this construct. Results show that ATIC fully mediated the relationship between interview preparation and performance, partially mediated the effect of social understanding on performance, and moderated the effect of impression management on performance. Findings are explained in terms of cognitive theories of cue recognition, and practical implications are discussed.     

Optimizing the validity of personality assessments: The importance of aggregation and item content

The predictive validity of a psychological measure can be improved by minimizing measurement errors through increases in the length of the assessment (aggregation) and, for an assessment of finite length, by making use of objective strategies for choosing from all available component measures. Two prominent considerations in selecting individual measures to be aggregated involve standards of (a) item content (construct approach) and (b) item/criterion association (empirical approach). Personality trait scales of different lengths were assembled for this study in order to represent features of the construct and empirical methods of selection. It was observed that (a) although reliability and validity generally increased with test length, aggregation beyond a certain point can fail to be expedient; and (b) although the prediction performance of empirically derived measures initially surpassed that of construct based assessments, the superiority of the empirical scales did not generalize to trait criteria that were not used as a basis for item selection. The data are interpreted as providing support for a theory-based program of test development where substantive considerations involving item content play a major role. The findings are also viewed as encouragement for conventional conceptualizations about organized dimensions of behavior.     

The family of origin scale: A construct validation study

Data from 69 women and 63 men along with ratings from a panel of expert judges were used to assess the construct validity of the Family Of Origin Scale as a measure of family health. Loevinger's conception of construct validity, focusing on the substantive, structural, and external components of validity, was employed to organize the research procedures followed and the data analytic techniques used. Results showed that the instrument appears to be a useful measure of a warmth-coldness affect dimension in the family of origin, but the multi-dimensional structure of the test, as a measure of family-of-origin health, could not be validated. Suggestions for use, score interpretation, and further development are discussed.     

Measuring the automatic negative self-schema: New evidence for the construct and criterion validity of the Depression Implicit Association

The construct and criterion validity of the Depression Implicit Association Test (Depression IAT) as a marker of an automatic negative self-schema was investigated. The Depression IAT and other measures were administered to a sample of 116 participants (72 females) aged 37.28 (SD = 15.69) that was composed by 56 patients with an history of suicide ideation (SI) and by 60 university students. Combining students and patients’ sub-samples, results revealed that the Depression IAT was significantly and positively correlated with self-report scales of depression and hopelessness as well as with implicit and explicit measures of death-life identification. On the contrary, it was negatively correlated with life satisfaction, optimism, and self-esteem scales. These results give new evidence for the construct validity of the Depression IAT. Moreover, considering only the patients’ sub-sample, significant correlations between Depression IAT scores and SI in the last year, month, and week, as well as in a follow-up observation two months later, were found. These correlations remain unchanged even when the Death IAT was controlled for, supporting the incremental validity of the Depression IAT. Overall, these results provide new evidence for the construct and criterion validity of the Depression IAT as a marker of an automatic negative self-schema.     

Flexible Computerized Adaptive Tests to Detect Misconceptions and Estimate Ability Simultaneously

The Scaling Individuals and Classifying Misconceptions (SICM) model is an advanced psychometric model that can provide feedback to examinees’ misconceptions and a general ability simultaneously. These two types of feedback are represented by a discrete and a continuous latent variable, respectively, in the SICM model. The complex structure of the SICM model brings difficulties in estimating both misconception profile and ability efficiently in a linear test. To overcome this challenge, this study proposes a flexible computerized adaptive test (FCAT) design as a new test delivery method to increase test efficiency by administering an individualized test to examinees. We propose three item selection methods and two transition criteria to determine adaptive steps based on the needs of estimating one or two latent variables. Through two simulation studies, we demonstrate how to select an appropriate item selection method for an adaptive step and what transition criterion should be used between two adaptive steps. Results reveal the combination of the item selection method and the transition criterion could improve the estimation accuracy of a specific latent variable to a different extent and thus provide further guidance in designing an FCAT.     

A survey of neuropsychologists’ use of validity tests with children and adolescents

A web-based survey of validity test use by North American neuropsychologists was conducted, with 282 participants meeting inclusion criteria. Respondents indicated that they use a median of one stand-alone performance validity test (PVT), one embedded PVT, and one symptom validity test (SVT) per pediatric assessment. The vast majority of respondents indicated they give at least one PVT (92%) and at least one SVT (88%) during each pediatric assessment. A meaningful difference in validity use (i.e., at least a medium effect size) was only found for those who engage in forensic work, with those clinicians giving more stand-alone PVTs than those who do not conduct forensic work. The most frequently used validity measures in pediatric assessments are presented, as are reasons participants reported for both using and not using validity tests. Limitations and qualitative comparisons to other surveys on validity test use with adults are discussed.     

JOB-RELATEDNESS OF A POSTTRAINING JOB KNOWLEDGE CRITERION USE TO ASSESS VALIDITY AND TEST FAIRNESS

Significant job-relatedness was found for a posttraining job knowledge test criterion using an application of Lawshe's content validity method. The aide test was used as a criterion to assess the predictive validity of a vocabulary test and a civil service test with samples of black ( N = 43) and white ( N = 62) psychiatric aides. Significant validities were found on both tests, but a vocabulary test proved to be the better predictor of the criterion in both samples. The obtained validities were discussed in terms of differential validity, test fairness, and sample size. This study demonstrated that a content validity method could be applied to criteria as well as selection tests. It was concluded that content validity methods may be able to help solve the problem of criterion relevance in validation research by providing quantitative evidence of the job-relatedness of criteria.     

The concurrent validity of three self-report measures of borderline personality

The recent accumulation of self-report measures of borderline personality disorder (BPD) affords the opportunity to evaluate both the construct validity of the concept and the quality of these measures. This study examines the relationship among three recently developed self-report instruments for assessing BPD from the Personality Assessment Inventory (PAI; Morey, 1991), the MMPI Personality Disorders Scales (MPD; Morey, Waugh, & Blashfield, 1985), and the Bell Object Relations Inventory (BORI: Bell, Billington, & Becker, 1986). Data on the three measures were provided by 119 undergraduate subjects from a southeastern university. A correlational analysis addresses the convergence of these measures of BPD, their divergence from measures of different but related traits, and their independence from variance due to method. Application of the Campbell-Fiske (1959) criteria indicates adequate convergence for all the BPD measures but a lack of discriminant validity for the BORI scales. The fit of the data to a structural model of construct validity is tested using confirmatory factor analysis, and these results are consistent with the hypothesis of a latent borderline trait factor independent of measurement method factors. In sum, the construct validity of the borderline personality concept using self-report methodologies receives support, and a strong association between borderline personality and paranoid phenomena is also suggested.     

Is the Political Skill Inventory Fit for Personnel Selection? An Experimental Field Study

The political skill inventory (PSI) assesses social effectiveness in organizations by self‐reports and has demonstrated strong evidence of validity. It was the purpose of this experimental field study to investigate construct and criterion‐related validity of the PSI when used under conditions of personnel selection. In the experimental group (n=102), the instructions asked job incumbents to work on the PSI, a social desirability scale, and a Big‐Five personality inventory as if they took part in a personnel selection procedure for a personally very attractive position. Additionally, they were asked to report yearly income. In the control group (n=110), job incumbents were asked to answer the items honestly. As expected, in both conditions, the PSI did not correlate with social desirability, but it correlated positively with extraversion, conscientiousness, and income, and negatively with neuroticism, thus demonstrating construct and incremental criterion‐related validity under both conditions. Implications and limitations are discussed.     

Latent variable selection in multidimensional item response theory models using the expectation model selection algorithm

The aim of latent variable selection in multidimensional item response theory (MIRT) models is to identify latent traits probed by test items of a multidimensional test. In this paper the expectation model selection (EMS) algorithm proposed by Jiang et al. (2015) is applied to minimize the Bayesian information criterion (BIC) for latent variable selection in MIRT models with a known number of latent traits. Under mild assumptions, we prove the numerical convergence of the EMS algorithm for model selection by minimizing the BIC of observed data in the presence of missing data. For the identification of MIRT models, we assume that the variances of all latent traits are unity and each latent trait has an item that is only related to it. Under this identifiability assumption, the convergence of the EMS algorithm for latent variable selection in the multidimensional two-parameter logistic (M2PL) models can be verified. We give an efficient implementation of the EMS for the M2PL models. Simulation studies show that the EMS outperforms the EM-based L₁ regularization in terms of correctly selected latent variables and computation time. The EMS algorithm is applied to a real data set related to the Eysenck Personality Questionnaire.     

面试预测效度和构想效度研究述评

面试是人才甄选中最常用的测量工具。大量研究证实,面试的预测效度比较理想,但不同类型面试的预测效度存在差异。虽然能够证实面试的预测效度较好,但对面试的测量构想却知之甚少。研究面试的构想效度,对于提高面试的递增效度有着重要的实践价值。相对人格成分而言,以往研究对面试能够测量到认知成分形成了更加一致地认识。     

中文版过剩适应量表在大学生群体中的效度和信度

为修订中文版过剩适应量表(OAS-C),并检验其在中国大学生群体中的信效度,抽取589名大学生(样本1),278名大学生(样本2)和174名大学生(样本3)进行施测.效度分析结果表明,OAS-C为二因子结构,包括外部适应过剩和内部适应匮乏.该二因子模型拟合良好,且与各效标间呈显著正相关.信度分析结果显示,内部一致性系数...     

Text anxiety on cognitive ability test can result in differential predictive validity of academic performance

This paper investigates whether test anxiety leads to differential predictive validity in academic performance. Our results show that the predictive validity of a cognitive ability test, using final exam performance as a criterion, decreased a small amount as Worry (the cognitive aspect of anxiety) increased but was unaffected by Emotionality (the physiological aspect of anxiety). These results suggest that cognitive ability tests may be more useful as predictors of performance for low anxiety test-takers. These findings are discussed in the context of the interference and deficit perspectives of test anxiety.