The Content Validity of Cognitively Oriented Tests: Commentary on Schmidt ()

Schmidt (International Journal of Selection and Assessment, 20, 1–13 (2012)) argues that it is possible for scores based on measures of general cognitive ability (GCA) to have content validity evidence. This commentary examines this argument further. I first decompose the various lines of validity evidence that may exist for GCA scores. Next, I consider whether GCA scores can have content validity evidence and whether they cannot. I conclude with several observations about the meaning of content validity within GCA research and practice. The bottom line is that although I agree with Schmidt that GCA scores can have content validity evidence, I am not sure such evidence tells us much about the overall validity of GCA.     

结构化面试预测效度的贡献成分分析

以情境面试和行为面试为主要形式的结构化面试,被广泛证明能够预测员工绩效,但其构念效度一直没被清晰地证明.面试的结构性、应聘者印象管理行为、动机、认知能力和人格特征等成分对结构化面试的预测效度都有积极贡献,这种对结构化面试预测效度贡献成分的分析有助于认清结构化面试的构念效度.未来的研究需关注结构化面试构念是否以人际技能为主,以及人际技能、动机、认知能力各自所占比重.     

Content Validity and Cognitive Tests: Response to Kehoe (), Ployhart (), and Sackett ()

The most important fact emerging from the combination of my article and the three commentaries is the consensus judgment that content validity is appropriate scientifically and professionally for use with tests of specific cognitive skills used in job performance. This is important because the 1978 Uniform Guidelines on Employee Selection Procedures have typically been interpreted as not permitting such usage, and this is particularly the case in the interpretation given to the Guidelines by federal government enforcement agencies. Although the Society for Industrial and Organizational Psychology Principles and the Standards do not prohibit such usage, many industrial–organizational psychologists believe that it is not professionally or scientifically appropriate to employ content validity methods with cognitive measures. The hope is that this series will convince them otherwise. On this point, all four authors in the series are in agreement. The major disagreement among us concerns whether specific cognitive skills used in content valid tests must be considered constructs or not. My position, and apparently that of Kehoe, is that they need not be so considered. I argue that constructs must be invoked only in the context of a substantive theory. Sackett and Ployhart, on the other hand, argue that all measures taken on people must be viewed as constructs, regardless of whether any theoretical propositions and assumptions are involved. In this response, I present reasons why this need not be the case.     

Criterion‐Related Validity of Dutch Police‐Selection Measures and Differences Between Ethnic Groups

This study investigated the criterion‐related validity of cognitive ability as well as non‐cognitive ability measures and differences between ethnic majority (N=2365) and minority applicants (N=682) in Dutch police officer selection. Findings confirmed the relatively low predictive validity of cognitive ability generally found for police jobs. Previous research reported no differential prediction. The present study, however, found small but systematic evidence for differences in validity for the ethnic majority and minority group of both cognitive and non‐cognitive measures. For the minority group, training performance appeared to be mainly predicted by the cognitive ability test. For the majority group, cognitive ability showed very little predictive power. Non‐cognitive ability variables appeared to be somewhat more predictive in this group.     

Validation of personality and cognitive measures for insurance claims examiners

Claims Examiners are critical for the profitability of large insurance companies. Mistakes in hiring these people can result in improperly processed claims and/or turnover, and this yields negative financial consequences for the company. This paper describes research evaluating the validity of selection procedures for the position of claims examiner at Blue Cross and Blue Shield of Oklahoma. A job analysis indicated that both cognitive skills and personal abilities are necessary for successful performance as a claims examiner. The job analysis also indicated that both subjective and objective criterion measures reflect important aspects of job performance. The criterion data included nine supervisory ratings, employee nominations, average percent of performance, absences, disciplinary actions, and sick leave. In a concurrent validation strategy, we tested a sample of experienced incumbents (N=50) using four cognitive ability tests (Basic Skills Tests: BST) and two measures of personality (Hogan Personality Inventory and PROFILE). Scales from two personality inventories were combined to form the Claims Examiner Inventory (CEI). Scores from the CEI and three of the four cognitive tests correlated significantly with the average percent of performance and six of the supervisory ratings, including the average supervisory rating. In addition to the cognitive measures, the personality measure contributed significantly to the prediction of percent of performance achieved with R=.64. These results are consistent with other research findings that support the use of both personality and cognitive measures as predictors of job performance.     

INTERNATIONAL VALIDITY GENERALIZATION OF GMA AND COGNITIVE ABILITIES: A EUROPEAN COMMUNITY META-ANALYSIS

This article reports on a series of meta-analyses into the criterion validity of general menatl ability (GMA) and specific cognitive ability tests for predicting job performance ratings and training success in the European Community (EC). Meta-analyses were computed on a large EC database examining the operational validity of GMA and other specific cognitive abilities, including verbal, numerical, spatial-mechanical, perceptual and memory ( N ranged from 946 to 16,065) across 10 EC member countries. The results showed that tests of GMA and specific cognitive ability are very good predictors of job performance and training success across the EC. Evidence for the international validity generalization of GMA and specific cognitive abilities was presented. The results for the EC meta-analyses showed a larger operational validity than previous meta-analyses in the U.S. for predicting job performance. For training success, the European and American results are very similar. Implications for the international generalizability of GMA test validities, practical use of cognitive ability tests for personnel selection, and directions for future research are discussed.     

Assessing the validity of inferences from scores on the cognitive reflection test

Decision‐making researchers purport that a novel cognitive ability construct, cognitive reflection, explains variance in intuitive thinking processes that traditional mental ability constructs do not. However, researchers have questioned the validity of the primary measure because of poor construct conceptualization and lack of validity studies. Prior studies have not adequately aligned the analytical techniques with the theoretical basis of the construct, dual‐processing theory of reasoning. The present study assessed the validity of inferences drawn from the cognitive reflection test (CRT) scores. We analyzed response processes with an item response tree model, a method that aligns with the dual‐processing theory in order to interpret CRT scores. Findings indicate that the intuitive and reflective factors that the test purportedly measures were indistinguishable. Exploratory, post hoc analyses demonstrate that CRT scores are most likely capturing mental abilities. We suggest that future researchers recognize and distinguish between individual differences in cognitive abilities and cognitive processes.     

Subject Matter Expert Judgments Regarding the Relative Importance of Competencies are not Useful for Choosing the Test Batteries that Best Predict Performance

Several recent articles have suggested that assessments of the relative importance of different abilities or competencies to a job have little bearing on the criterion‐related validity of these selection tests that measure those abilities. We hypothesize that selection test batteries chosen to maximize the judged importance of knowledge, skills, and abilities will not predict performance better than batteries of tests chosen at random. The results in two independent samples consistently show that the validity of test batteries chosen based on subject matter expert judgments of importance is not different from the validity of batteries of a comparable number of tests chosen at random from a set of intercorrelated tests, or even those chosen to provide the worst possible match between test content and job content.     

More than g: Evidence for the Incremental Validity of Performance-Based Assessments for Predicting Training Performance

Past research has consistently shown that tests measuring specific cognitive abilities provide little if any incremental validity over tests of general mental ability when predicting performance on the job. In this study, we suggest that the seeming lack of incremental validity may have been due to the type of content that has traditionally been assessed. Therefore, we hypothesised that incremental validity can be obtained using specific cognitive abilities that are less highly correlated with g and are matched to the tasks performed on the job. To test this, we examined a recently developed performance-based measure that assesses a number of cognitive abilities related to training performance. In a sample of 310 US Navy student pilots, results indicated that performance-based scores added sizeable incremental validity to a measure of g. The significant increases in R² ranged from .08 to .10 across criteria. Similar results were obtained after correcting correlations for range restriction, though the magnitude of incremental validity was slightly smaller (ΔR² ranged from .05 to .07).     

The Concept of Personal Initiative: An Overview of Validity Studies

Construct validity of an interview measure of personal initiative (PI) is examined in two parts. The first part assembles the results from 11 samples, showing that PI is meaningfully related to a nomological network of variables, based on environmental supports; knowledge, skills, and cognitive abilities; personality variables and orientations; and behavior and performance, confirming our hypotheses. In the second part, the article presents a new analysis that looks at the influence of motivational parameters (control aspiration, self-efficacy, and change orientation) and cognitive ability on PI within a longitudinal study in East Germany.     

履历数据测评的效度分析

履历数据是人员测评的重要手段, 但其组织情景性限制使得国外同类工具不能适用于国内人事选拔, 国内也缺乏实证研究验证其信度、效度。本研究结合某通信企业的招募选拔工作, 开发履历数据分析工具, 选取250名应聘者的履历数据、一般认知能力、大五个性和面试结果数据, 通过效标关联效度、增量效度思想, 分析履历数据有效性。结果表明, 以面试结果为效标, 履历数据具有良好的效标关联效度, 与其他测评工具组合使用时有良好的增量效度。     

Construct-oriented Biodata: Capturing Change-related and Contextually Relevant Future Performance

This study highlights an approach to the development of performance and predictor constructs that has the potential to permit context and culture relevant selection. Task, contextual and adaptive measures of performance were used as criterion data in a selection study involving 325 staff in the hotel industry. Construct-oriented biodata were developed to predict adaptive and contextual performance and the validities were contrasted with measures of cognitive ability and personality. The results provided construct support for the separation of adaptive performance from task and contextual performance and for the predictive validity of change-related biodata after controlling for cognitive ability. The pattern of correlations among the personality measures, biodata scales and cognitive tests supported the construct validity of the biodata scales. Results are discussed in relation to the context in which the data were collected, and the general relevance of the procedure across countries or cultures.     

Relative Validity of Distinct Spatial Abilities: An example with implications for diversity

Despite the increasingly recognized importance of spatial ability in STEM (science, technology, engineering, and mathematics) fields, research to date has largely ignored distinctions in the criterion‐related validity and subgroup differences associated with theoretically distinct spatial abilities. This study aims to enhance practical understanding by spotlighting a prominent military context as an example wherein specific spatial abilities may be both differentially relevant and differentially apt to result in adverse impact. Analyses compared the relative importance of (i) spatial orientation, (ii) visualization, (iii) closure flexibility, and (iv) perceptual speed for predicting U.S. Air Force pilot performance (N = 1440), and compared subgroup differences in a large applicant sample (N = 10,643). Given that specific spatial abilities vary substantially in the extent to which gender differences exist, this study highlights the practical implications associated with the choice of particular spatial ability measures in personnel selection.     

Examining the Construct Validity for the Multiple-Content Testing Programs

The primary objective of this study was to examine the construct validity for the 2 multiple-content testing programs-the multiple-choice Comprehensive Tests of Basic Skills (CTBS/5) together with the performance-based Maryland School Performance Assessment Program (MSPAP)-by evaluating the true-score longitudinal associations among multiple-content scores in 1 school district. Each content area measure in the 2 testing programs was closely examined by the following criterion: The true-score correlation between 2 time-period measures of the same content area is higher than its longitudinal true-score correlations with other content areas. This criterion was achieved in 2 (Reading and Mathematics) of 5 CTBS/5 content subtests, as well as 1 (Language) of 6 MSPAP content subtests. The structural equation modeling has been conducted on a multitrait-multimethod correlation dataset, where the traits of Reading and Mathematics were assessed by MSPAP and the old version of CTBS/4. Although convergent validity existed in these 2 measures, there was little evidence to support discriminant validity in both measures.     

Cognitive ability in virtual reality: Validity evidence for VR game‐based assessments

Virtual reality (VR) technology may provide unique benefits for assessing individual characteristics, in contrast with traditional assessments. This study examined validity evidence for the use of VR game (VRG) scores to indicate specific cognitive abilities. Participants completed three VRGs, three computer‐based assessments (CBAs), and additional scales. To evaluate convergent validity, VRG scores were compared with CBA scores. For divergent validity, VRG scores were compared with five factors of personality based on the Five Factor Model. For criterion‐related validity, VRG scores were compared with academic achievement (i.e., grade point average). Findings generally provided support that the VRG scores are related to certain spatial reasoning subdimensions of cognitive ability. However, further research is needed to examine the qualities of VRGs as assessment tools.     

Assessing mental flexibility with a new word recognition test

This paper presents the development and preliminary evaluation of a new word recognition test (WRT) designed to measure individual differences in mental flexibility, defined as the ability to solve novel problems in unfamiliar settings. Conceptually designed to simulate problem solving in real world performance situations, the test was developed to recruit fluid and reproductive abilities and the interplay between convergent and divergent thinking. It is based on a framework that integrates and extends previous theoretical and methodological approaches to the study of cognitive ability and creative cognition. The WRT was administered with various cognitive ability and criterion measures to an undergraduate student sample (n = 266). Results provide preliminary evidence of construct validity. WRT scores correlated as expected with reference measures of cognitive ability, creative performance, and college performance (GPA). Regression analyses showed the WRT explained an additional 4.5% of variance in college performance over and above traditional cognitive ability measures that take up to five times as long to administer. Results suggest further study is warranted given the potential for its contribution to basic research and applied use.     

Validity evidence for the situational judgment test paradigm in emotional intelligence measurement

To date, various measurement approaches have been proposed to assess emotional intelligence (EI). Recently, two new EI tests have been developed based on the situational judgment test (SJT) paradigm: the Situational Test of Emotional Understanding (STEU) and the Situational Test of Emotion Management (STEM). Initial attempts have been made to examine the construct‐related validity of these new tests; we extend these findings by placing the tests in a broad nomological network. To this end, 850 undergraduate students completed a personality inventory, a cognitive ability test, a self‐report EI test, a performance‐based EI measure, the STEU, and the STEM. The SJT‐based EI tests were not strongly correlated with personality and fluid cognitive ability. Regarding their relation with existing EI measures, the tests did not capture the same construct as self‐report EI measures, but corresponded rather to performance‐based EI measures. Overall, these results lend support for the SJT paradigm for measuring EI as an ability.     

It Was Nice to Be There: Construct Validity Then and Now

I appear to be one of the few persons who attended both the colloquium at Princeton titled Construct Validity in Psychological Measurement held in 1979 and the conference at Newport Beach titled Construct Validity: Issues and Opportunities held in 1990. As a discussant for both meetings, I have compared construct validity as seen by participants at the 1979 meetings with construct validity as seen by participants at the 1990 meeting. Comparisons are drawn according to conceptualizations of construct validity, construct va- lidity as scientific inquiry, and construct validity in assessment technology. A most noteworthy conclusion is that these three aspects no longer apply very well to the articles herein that were originally delivered at the 1990 meeting. Instead these articles show strongly that classification according to scientific inquiry versus applications is not possible. Such a conceptual gap no longer seems to exist for construct validity. Instead, these articles as well as the one by Hogan (1991) all emphasize large scale programmatic efforts characterized by approaches driven primarily by conceptualizations derived from construct validity. A secondary theme in these articles emphasizes analysis according to taxonomies at both the predictor and criterion levels. A primary conclusion from reviewing these articles is that construct validity no longer revolves around issues, but it clearly does connote opportunities for future research and applications.     

Predicting Training and Job Performance for Transit Operators

Although considerable meta‐analytic research has validated the use of cognitive ability tests, structured interviews, and personality tests with training and job performance criteria, few studies have investigated the validity of these measures with transit operators. There are the only two single studies of concurrent validation research specifically with transit operators. This article presents the results of a predictive validation study conducted with transit operator applicants for a large urban transit authority in Canada. Key knowledge, skills, and abilities were determined for the role and used as a basis for the design and choice of predictors and criteria. Four predictors were used in the study: education, cognitive ability, personality assessment, and structured interview. Criteria included training performance (formative and summative), probationary performance, preventable accidents, and lost time injuries. Validation results supported cognitive ability, structured interview, and several personality factors as predictors of training performance, but less so for job performance. The use of formative training ratings greatly augmented the evidence supporting the predictors beyond typical organizational criteria.     

面试预测效度和构想效度研究述评

面试是人才甄选中最常用的测量工具。大量研究证实,面试的预测效度比较理想,但不同类型面试的预测效度存在差异。虽然能够证实面试的预测效度较好,但对面试的测量构想却知之甚少。研究面试的构想效度,对于提高面试的递增效度有着重要的实践价值。相对人格成分而言,以往研究对面试能够测量到认知成分形成了更加一致地认识。