The Validity of Job Knowledge Measures

The validity of written job knowledge tests as predictors of job performance and training success was examined. Based on a sample of 363,528 persons and 502 coefficients, meta-analyses were performed to investigate the extent of validity generalization and the effects of two hypothesized moderator variables: content similarity and job complexity. Corrected mean validities were 0.45 for studies predicting job performance and 0.47 for studies of training success. Support was found for both moderators since validities were higher for high complexity jobs and when job-test content similarity was high. The implications of this study for personnel selection and research in the area of work performance are discussed.     

The predictive validity of cognitive ability tests: A UK meta‐analysis

A meta‐analysis on the validity of tests of general mental ability (GMA) and specific cognitive abilities for predicting job performance and training success in the UK was conducted. An extensive literature search resulted in a database of 283 independent samples with job performance as the criterion (N=13,262), and 223 with training success as the criterion (N=75,311). Primary studies were also coded by occupational group, resulting in seven main groups (clerical, engineer, professional, driver, operator, manager, and sales), and by type of specific ability test (verbal, numerical, perceptual, and spatial). Results indicate that GMA and specific ability tests are valid predictors of both job performance and training success, with operational validities in the magnitude of .5–.6. Minor differences between these UK findings and previous US meta‐analyses are reported. As expected, operational validities were moderated by occupational group, with occupational families possessing greater job complexity demonstrating higher operational validities between cognitive tests and job performance and training success. Implications for the practical use of tests of GMA and specific cognitive abilities in the context of UK selection practices are discussed in conclusion.     

INTERNATIONAL VALIDITY GENERALIZATION OF GMA AND COGNITIVE ABILITIES: A EUROPEAN COMMUNITY META-ANALYSIS

This article reports on a series of meta-analyses into the criterion validity of general menatl ability (GMA) and specific cognitive ability tests for predicting job performance ratings and training success in the European Community (EC). Meta-analyses were computed on a large EC database examining the operational validity of GMA and other specific cognitive abilities, including verbal, numerical, spatial-mechanical, perceptual and memory ( N ranged from 946 to 16,065) across 10 EC member countries. The results showed that tests of GMA and specific cognitive ability are very good predictors of job performance and training success across the EC. Evidence for the international validity generalization of GMA and specific cognitive abilities was presented. The results for the EC meta-analyses showed a larger operational validity than previous meta-analyses in the U.S. for predicting job performance. For training success, the European and American results are very similar. Implications for the international generalizability of GMA test validities, practical use of cognitive ability tests for personnel selection, and directions for future research are discussed.     

The effect of time constraints and personality facets on general cognitive ability (GCA) assessment

General cognitive ability (GCA) is a recognized construct for predicting job performance and capacity to learn. However, it has recently been argued that the time constraints under which GCA is assessed might provoke test anxiety, which negatively biases GCA scores. This can then lead to erroneous rejection of qualified candidates in personnel hiring contexts. This paper aimed to investigate: (1) to what extent candidates’ GCA scores increase when tested without time constraints and the ability of this GCA score to predict job performance; and (2) the personality characteristics that hinder GCA test performance under time constraints. Results from two field studies conducted in an actual personnel selection context partially confirmed the hypotheses. They revealed that, aside from the improvement of all candidates’ GCA scores when time constraint was removed, only GCA assessed without time constraints predicts job performance. Furthermore, while all candidates’ scores were influenced by the time constraint condition, individuals who are anxious, low-impulse, low value-questioning and deliberating are more penalized by the time constraint condition of such testing and, thus, are more likely to be erroneously eliminated in a selection process.     

Hello From the New Editor

The research goal was to identify potential “contextual” predictors of job performance for military officers. Exploratory factor analysis of job component ratings by military officers (n = 105) in the Irish Defence Forces revealed two “contextual” and two task-related factors. There were no differences in the extent to which contextual job components were a part of the jobs across four military units, but for one contextual factor, contextual components were a significant part of some jobs. Other research shows contextual components increment validity beyond both g and task-related job components. In this first reported study using a sample of military officers to examine contextual job behaviors, the findings suggest contextual components may also increment validity in officer selection models.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.     

VALIDATING SYNTHETIC VALIDATION: COMPARING TRADITIONAL AND SYNTHETIC VALIDITY COEFFICIENTS

We describe a unique application of a synthetic validation technique to a selection system development project in a large organization. Job analysis data were collected from 4,725 job incumbents and 619 supervisors, and were used to identify 11 job families and 27 job components. We developed 12 tests to predict performance on these job components and conducted a concurrent validation study to collect test and job component data for 1,926 incumbents. We created a test composite for each job component and then chose a test battery for each job family based on its relevant job components. Synthetic validity coefficients were computed for each test battery and compared to traditional validity coefficients that were computed within job families with large sample sizes. The synthetic validity coefficient was very close to the within-family validity coefficient for most job families and was within the bounds of sampling error for all job families. Validities tended to be highest when predictors were weighted according to the number of job components to which they were relevant and job component criterion measures were unit weighted.     

IMPLICATIONS OF THE MULTIDIMENSIONAL NATURE OF JOB PERFORMANCE FOR THE VALIDITY OF SELECTION TESTS: MULTIVARIATE FRAMEWORKS FOR STUDYING TEST VALIDITY

Although most studies of criterion-related validity focus on univariate relationships, the complex and multidimensional nature of the performance construct and the widespread use of multiple selection devices argue in favor of multivariate frameworks for evaluating validity. Using a Monte Carlo simulation we estimated the validity of general cognitive ability tests and personality tests in predicting "job performance," where performance is conceptualized as a composite of multiple performance measures (i.e., individual job task performance and organizational citizenship behaviors). The validity of a selection battery varies substantially as a function of the relative weight given to both predictors and criteria; the 95% confidence interval for validities ranged from .20 to .78. The effective weights given to performance dimensions accounted for 34% of the variance in selection battery validities; depending on precisely how "performance" is defined, the same test battery can have relatively high or relatively low levels of validity. Our model suggests that the way an organization defines job performance is a source of true and important variability in validities, and that the validity of selection tests for predicting complex performance criteria may show considerably less generalizability that current metaanalysis of univariate validities would suggest.     

Revisiting predictive bias of cognitive ability tests against Hispanic American job applicants

Previous studies have concluded that cognitive ability tests are not predictively biased against Hispanic American job applicants because test scores generally overpredict, rather than underpredict, their job performance. However, we highlight two important shortcomings of these past studies and use meta-analytic and computation modeling techniques to address these two shortcomings. In Study 1, an updated meta-analysis of the Hispanic–White mean difference (d-value) on job performance was carried out. In Study 2, computation modeling was used to correct the Study 1 d-values for indirect range restriction and combine them with other meta-analytic parameters relevant to predictive bias to determine how often cognitive ability test scores underpredict Hispanic applicants’ job performance. Hispanic applicants’ job performance was underpredicted by a small to moderate amount in most conditions of the computation model. In contrast to previous studies, this suggests cognitive ability tests can be expected to exhibit predictive bias against Hispanic applicants much of the time. However, some conditions did not exhibit underprediction, highlighting that predictive bias depends on various selection system parameters, such as the criterion-related validity of cognitive ability tests and other predictors used in selection. Regardless, our results challenge “lack of predictive bias” as a rationale for supporting test use.     

Predictor content matters for knowledge testing: Evidence supporting content validation

Organizations often rely on the match between job requirements and test content to justify test use. This practice has been questioned on the grounds that content validation has little relevance to criterion-related validation due to positive manifold among predictors. We analyze two large databases to assess the implications of test content for (a) test interchangeability and (b) criterion-related validity. Analyses of 15 knowledge tests administered (N = 80,394) as part of Project Talent demonstrate that test content is related to predictor interchangeability. Analyses of SAT and Advanced Placement test data compare correlations among predictors and criteria drawn from matched and unmatched content domains. We conclude that test-criterion content match is likely to result in stronger criterion-related validity.     

Expressed vs inventoried interests as predictors of paramedical effectiveness

Two interest measures, a composite of six items which provide a direct assessment of an individual's preferences for Navy health care work and a Navy Vocational Interest Inventory (NVII) scale which assessed the degree of similarity between an individual's activity preferences and those of hospital corpsmen (HMs), were considered along with aptitude test scores as potential predictors of effectiveness among 2429 Navy enlisted personnel assigned to training for HM jobs. Effectiveness was defined as completion of HM training and remaining on the job for at least 2 years. The six-item composite and NVII HM scale scores were correlated .51 with each other and were correlated .27 and .28, respectively, with the effectiveness criterion. When these measures were added to the aptitude measure by using multiple regression procedures, the validity for predicting effectiveness was increased significantly from .36 to .42 with a cross-validity of .42 for the three-variable equation. These findings provide support for the contention that interest measures can be useful predictors of occupational success as well as occupational entry. Furthermore, since most of the interest-related attrition that was found in this study occurred during HM training, these findings suggest that job withdrawal attributable to mismatches in vocational interests may tend to occur very soon after the individual enters the occupational field.     

THE VALIDATION OF A MINICOURSE FOR TELEPHONE COMPANY SWITCHING TECHNICIANS1

The introduction of electronic switching equipment has changed the nature of the telephone company switching job. A lengthy and complex training program must be completed before an employee can perform the electronic switching job. Because of the high cost of this training a more elaborate, second-stage selection procedure was developed. The ESS Minicourse was designed to be a self-paced content valid sample of ESS training which would be suitable for use with job candidates without any previous telephone company experience. A criterion-related validity study was undertaken to provide further evidence of validity as well as data helpful in setting a cutting score. Results showed that a combination of time to complete the Minicourse and performance on the objective tests was predictive of time to complete self-paced training in electronic switching. Cross- validated estimates of validity were used to develop estimates of u'tility given different selection ratios.     

Selection Practices in Canadian Firms: An empirical investigation

Using 7 years of data from Statistics Canada's Workplace and Employee Survey, this study examined the types of selection tools used with 23,639 employees in 6,693 Canadian firms. While 79% of these employees were given an interview during the selection process, only 10% were given a test on job‐related knowledge and 9% were given a personality test. Using logit analysis, job‐ and organization‐level variables were examined as predictors of the type of selection tools used. The size of the organization, an in‐house human resource department, the presence of a union and occupation were significant predictors of the use of a test on job‐related knowledge in the selection process. The implications and plausible explanations of this theory to practice gap are discussed.     

Applications of Personality Assessment to the Workplace: A Review

Our review of the relevant research literature in the recent past strongly supports the view that personality assessment has high utility in the workplace. We review the evidence that personality assessment measures, especially those based upon the Big Five factors of personality, can effectively predict job performance and thus can be used for personnel selection. The validity of integrity testing in predicting counterproductivity on the job has been demonstrated both for overt measures of integrity and, to a lesser degree, for more general (subtle) measures of personality. We also found good evidence of validity for measures of (supervisory) management and (transformational) leadership, a complex field which includes a number of multidimensional instruments each built around its own theory. Finally, we review the empirical literature that supports the use of assessment centers and the use of personality assessment in training and development.     

An Examination of the Validity and Incremental Value of Needed‐at‐Entry Ratings for a Customer Service Job

We examined needed‐at‐entry ratings of knowledge, skills, abilities, and other characteristics (KSAOs) provided by job experts (N= 285) during a job analysis for a customer service manager position. To serve as an external referent, a group of industrial and organisational psychologists (N= 31) rated the perceived trainability of each KSAO. Analyses revealed only limited support for the validity of inferences drawn from job experts' needed‐at‐entry ratings. Consistent with our hypotheses, less validity evidence was found for ratings of the more person‐oriented and abstract “AO” attributes than for the more job‐oriented and concrete “KS” attributes. In general, job experts tended to rate attributes as needed‐at‐entry that psychologists thought could be developed on the job. We also found that failing to collect needed‐at‐entry ratings, and relying on only the more common importance‐to‐the‐job ratings, would have resulted in a rather different set of critical KSAOs identified for assessment during the selection process.     

PERSONALITY TESTS AT THE CROSSROADS: A RESPONSE TO MORGESON, CAMPION, DIPBOYE, HOLLENBECK, MURPHY, AND SCHMITT (2007)

Despite widespread and growing acceptance that published personality tests are valid predictors of job performance, Morgeson et al. (2007) propose they be abandoned in personnel selection because average validity estimates are low. Our review of the literature shows that Morgeson et al.'s skepticism is unfounded. Meta-analyses have demonstrated that published personality tests, in fact, yield useful validity estimates when validation is based on confirmatory research using job analysis and taking into account the bidirectionality of trait–performance linkages. Further gains are likely by use of narrow over broad measures, multivariate prediction, and theory attuned to the complexities of trait expression and evaluation at work. Morgeson et al. also suggest that faking has little, if any, impact on personality test validity and that it may even contribute positively to job performance. Job applicant research suggests that faking under true hiring conditions attenuates personality test validity but that validity is still sufficiently strong to warrant personality test use in hiring. Contrary to Morgeson et al., we argue that the full value of published personality tests in organizations has yet to be realized, calling for programmatic theory-driven research.     

A Concurrent Validity Study Applied to a Secretarial Position in a State Universities Civil Service System

In a concurrent validity study, a comprehensive job analysis of a mid-level secretarial position resulted in the development of highly valid employment selection instruments. Six hundred fifty-nine supervisors and 883 incumbents in 20 locations participated in the job analysis. Scores from the selection test correlated with composite ratings from a research performance appraisal (RPA) .41 (p<.001). Two forms of the test (A and B) were developed. Form A and Form B test scores and RPA composite ratings correlated .55 and .48 (p<.001) respectively. The unbiased estimate of equivalence reliability of Form A and Form B was .94. The two tests correlated .89 (p<.001).     

The Content Validity of Cognitively Oriented Tests: Commentary on Schmidt ()

Schmidt (International Journal of Selection and Assessment, 20, 1–13 (2012)) argues that it is possible for scores based on measures of general cognitive ability (GCA) to have content validity evidence. This commentary examines this argument further. I first decompose the various lines of validity evidence that may exist for GCA scores. Next, I consider whether GCA scores can have content validity evidence and whether they cannot. I conclude with several observations about the meaning of content validity within GCA research and practice. The bottom line is that although I agree with Schmidt that GCA scores can have content validity evidence, I am not sure such evidence tells us much about the overall validity of GCA.     

Alternative Predictors for Dealing with the Diversity–Validity Dilemma in Personnel Selection: The constructed response multimedia test

In the context of the diversity–validity dilemma in personnel selection, the present field study compared ethnic subgroup differences on an innovative constructed response multimedia test to other commonly used selection instruments. Applicants (N = 245, 27% ethnic minorities) for entry‐level police jobs completed a constructed response multimedia test, cognitive ability test, language proficiency test, personality inventory, structured interview, and role play. Results demonstrated minor ethnic subgroup differences on constructed response multimedia test scores as compared to other instruments. Constructed response multimedia test scores were related to the selection decision, and no evidence for predictive bias was found. Subgroup differences were also examined on the dimensional level, with cognitively loaded dimension scores displaying larger differences.     

PERSONALITY MEASURES AS PREDICTORS OF JOB PERFORMANCE: A META-ANALYTIC REVIEW

The purpose of this study was to investigate conflicting findings in previous research on personality and job performance. Meta-analysis was used to (a) assess the overall validity of personality measures as predictors of job performance, (b) investigate the moderating effects of several study characteristics on personality scale validity, and (c) appraise the predictability of job performance as a function of eight distinct categories of personality content, including the "Big Five" personality factors. Based on review of 494 studies, usable results were identified for 97 independent samples (total N = 13,521). Consistent with predictions, studies using confirmatory research strategies produced a corrected mean personality scale validity (.29) that was more than twice as high as that based on studies adopting exploratory strategies (.12). An even higher mean validity (.38) was obtained based on studies using job analysis explicitly in the selection of personality measures. Validities were also found to be higher in longer tenured samples and in published articles versus dissertations. Corrected mean validities for the "Big Five" factors ranged from .16 for Extroversion to .33 for Agreeableness. Weaknesses in the reporting of validation study characteristics are noted, and recommendations for future research in this area are provided. Contrary to conclusions of certain past reviews, the present findings provide some grounds for optimism concerning the use of personality measures in employee selection.