IDENTIFYING OPTIMAL PREDICTOR COMPOSITES AND TESTING FOR GENERALIZABILITY ACROSS JOBS AND PERFORMANCE FACTORS

The initial examination of validity generalization in the Army Selection and Classification Project used data from a concurrent validation sample of 4,039 job incumbents drawn from a representative sample of nine jobs. The available data consisted of 24 predictor scores and five job performance factor scores on each individual. The major objectives were to determine (a) the degree of validity generalization across the major components of performance, with the job held constant, and (b) the degree of validity generalization across jobs within each major performance factor. After reducing the predictor set by eliminating variables that added no information, a modified confirmatory analysis was used to test the hypotheses that one equation would fit the data from all performance components and that one equation would fit the data from all jobs, given a particular performance component. The major findings were that different predictor equations were needed for each of the five criterion factors. For generalization across jobs, within each criterion factor, one equation fit the data for four of the five performance components. Different prediction equations were required for the component that reflects proficiency on the technical tasks specific to each job.     

The Role of Cognitive Ability Tests in Employment Selection

Cognitive ability tests correlate with measures of job performance across many jobs. However, cognitive ability tests produce racial differences that are 3 to 5 times larger than other predictors-such as biodata, personality inventories, and the structured interview-that are valid predictors of job performance. Given that (a) cognitive ability tests can be combined with other predictors such that adverse impact is reduced while overall validity is increased, and (b) alternative predictors with less adverse impact can produce validity coefficients comparable to those obtained with cognitive ability tests alone, sole reliance on cognitive ability tests when alternatives are available is unwarranted.     

Characteristics of Successful Direct Support Professionals: An examination of personality and cognitive ability requirements

Service jobs in the health care industry continue to face difficulties in recruiting, selecting, and retaining quality employees. We conducted two studies to examine potential cognitive and personality predictors of job performance, satisfaction, and turnover among direct support professionals (i.e., individuals providing long‐term residential care and support to persons with disabilities). Data from a concurrent validation study was used to create a unit‐weighted combination of agreeableness, conscientiousness, optimism, and numerical ability. This composite predictor was significantly correlated with job performance and job satisfaction. The composite was further supported in a follow‐up predictive validation study of turnover.     

Situational Judgment and Job Performance

Data from 160 civil service employees demonstrate the validity of a situational judgment test in predicting overall job performance as well as three performance dimensions: task performance (core technical proficiency), motivational contextual performance (job dedication), and interpersonal contextual performance (interpersonal facilitation). Situational judgment also provided incremental validity over the prediction provided jointly by cognitive ability, the Big Five personality traits, and job experience. These findings extended the work of Clevenger, Pereira, Wiechmann, Schmitt, and Harvey (2001) on the incremental validity of situational judgment tests as well as the meta-analytic results reported by McDaniel, Morgeson, Finnegan, Campion, and Braverman (2001). Implications are discussed in terms of research on the prediction and understanding of job performance.     

Predicting Adaptive Performance: Further Tests of a Model of Adaptability

This research further investigates an 8-dimension taxonomy of adaptive performance developed by Pulakos, Arad, Donovan, and Plamondon (2000) and the usefulness of past experience, interest, and self-efficacy predictors developed to measure these same 8 dimensions for predicting adaptive job performance. Participants in the concurrent, criterion-related validation study included 739 military personnel. They completed the 3 adaptability measures as well as more traditional cognitive and noncognitive predictors. Supervisors of the study participants rated their adaptive job performance. Results showed support for the 8-dimension model of adaptability. In addition, cognitive ability, personality, and the new adaptability predictors were shown to predict adaptive performance, with some past experience items adding incremental validity beyond the more traditional cognitive ability and personality measures. Results are discussed.     

The Validity of Conscientiousness for Predicting Job Performance: A meta‐analytic test of two hypotheses

This study examined two hypotheses regarding the moderating effects of job characteristics on the validity of personality. Using meta‐analytic techniques, the authors explored the extent to which the structural characteristics and cognitive ability requirements of jobs influence the role of conscientiousness in predicting performance. The results suggest that conscientiousness is a stronger predictor of performance in jobs that are highly routinized, and a weaker predictor of performance in jobs with high levels of cognitive ability requirements. Implications for theory and future research are discussed.     

Prediction of situational awareness in F-15 pilots

Situational awareness (SA) is a skill often deemed essential to pilot performance in both combat and noncombat flying. A study was conducted to determine if SA in U.S. Air Force F-15 pilots could be predicted. The participants were 171 active duty F-15 A/C pilots who completed a test battery representative of various psychological constructs proposed or demonstrated to be valid for the prediction of performance in a wide variety of military and civilian jobs. These predictors encompassed measures of cognitive ability, psychomotor ability, and personality. Supervisor and peer ratings of SA were collected. Supervisors and peers showed substantial agreement on the SA ratings of the pilots. The first unrotated principle component extracted from the supervisor and peer ratings accounted for 92.5% of the variability of ratings. The unrotated first principal component served as the SA criterion. Flying experience measured in number of F-15 hours was the best predictor of SA. After controlling for the effects of F-15 flying hours, the measures of general cognitive ability based on working memory, spatial reasoning, and divided attention were found to be predictive of SA. Psychomotor and personality measures were not predictive. With additional F-15 flying hours it is expected that pilots would improve their ratings of SA.     

PROJECT A: SPECIFICATION OF THE PREDICTOR DOMAIN AND DEVELOPMENT OF NEW SELECTION/ CLASSIFICATION TESTS

A major goal of the Army Selection and Classification Project was to develop an experimental predictor battery that would best supplement the Armed Forces Vocational Aptitude Battery for making selection and classification decisions for entry-level enlisted personnel. That is, what predictor measures would best serve the needs of all the jobs in an entire selection/classification system? This paper describes the characteristics of the new test battery and the procedures that were used to develop it. The major steps in the procedure were a structured literature search using a standard protocol, an extensive expert judgment study of expected true validities for a population of predictor variables against a population of performance components, fabrication of modularized software and a special response pedestal for computerized measurement of perceptual and psychomotor abilities, evaluations of experimental measures in three iterative pilot tests and one major field test, and a series of reviews by a panel of scientific advisers. The test battery that resulted from this 2 1/2-year development effort is described. The basic psychometric properties of each measure, as determined in a large concurrent validation sample, are also described.     

Progress Toward Understanding the Structure and Determinants of Job Performance: A focus on task and citizenship performance

Job performance criterion models have become more popular and more often investigated. The current research adds to this literature by testing performance models focusing on ability and personality predictors of task and citizenship job performance. Air traffic controllers (N = 928) completed cognitive ability and personality predictor measures, were administered a performance test assessing two possible mediators, and were evaluated using supervisor and peer ratings of two task performance, one citizenship performance, and an overall performance dimension. Path analyses largely confirmed that ability predicts primarily task performance and personality predicts primarily citizenship performance. The mediation related to task knowledge and skill on the task performance side was confirmed, and the overall fit of the performance models was quite good. Discussion focuses on how the present research furthers progress toward mapping relations among individual differences and individual performance constructs, beyond using overall performance as the endogenous performance variable.     

Do Birds of a Feather Flock Together? An Examination of Calling,Congruence, Job Design and Personality as Predictors of Job Satisfaction and Tenure

This study explored whether congruence, calling, job characteristics or personality were better predictors of job satisfaction and tenure. The sample consisted of 1968 employees across four different job roles: sales engineers (N=309), graphic designers (N=383), teachers (N=481) and clergy (N=795). Data was collected as part of a selection and development centre battery. Results found evidence of calling, with clergy reporting significantly higher levels of work satisfaction and tenure, despite the absence of some personality predictors (i.e. conscientiousness) and job characteristics (task identity, feedback). In general personality [particularly conscientiousness (+) and neuroticism (-)] along with job characteristics (variety and autonomy) were the most likely predictors of satisfaction across the different roles. No evidence could be found that congruence predicted work satisfaction or tenure. Results have implications for renewed interest in the role of calling according to its original definition and question the role of congruence in determining best fit especially in the context of vocational assessments.     

Selection Tests for Firefighters: A Comprehensive Review and Meta-Analysis

This study reviewed and summarized the literature on the use of written tests in the selection of firefighters using a sample of 13,418 individuals drawn from 101 samples. For the prediction of job performance, cognitive tests showed substantial validity (.42), although mechanical comprehension tests showed even higher validity (.54). However, the best prediction was obtained by tests which were composites of cognitive and mechanical predictors (.56). Training criteria was best predicted by cognitive measures (.77), although mechanical comprehension predictors also showed substantial prediction value (.62). Tests which were composites of cognitive and mechanical measures showed validities equal to that of cognitive measures (.77).     

IMPLICATIONS OF THE MULTIDIMENSIONAL NATURE OF JOB PERFORMANCE FOR THE VALIDITY OF SELECTION TESTS: MULTIVARIATE FRAMEWORKS FOR STUDYING TEST VALIDITY

Although most studies of criterion-related validity focus on univariate relationships, the complex and multidimensional nature of the performance construct and the widespread use of multiple selection devices argue in favor of multivariate frameworks for evaluating validity. Using a Monte Carlo simulation we estimated the validity of general cognitive ability tests and personality tests in predicting "job performance," where performance is conceptualized as a composite of multiple performance measures (i.e., individual job task performance and organizational citizenship behaviors). The validity of a selection battery varies substantially as a function of the relative weight given to both predictors and criteria; the 95% confidence interval for validities ranged from .20 to .78. The effective weights given to performance dimensions accounted for 34% of the variance in selection battery validities; depending on precisely how "performance" is defined, the same test battery can have relatively high or relatively low levels of validity. Our model suggests that the way an organization defines job performance is a source of true and important variability in validities, and that the validity of selection tests for predicting complex performance criteria may show considerably less generalizability that current metaanalysis of univariate validities would suggest.     

The stability of validity coefficients over time: Ackerman's (1988) model and the general aptitude test battery

This study examined P. L. Ackerman's (1988) model of skill acquisition within an applied setting. Differences were examined between jobs on the basis of task consistency: changes in performance variability across experience, learning curves, and stability of ability-performance correlations across experience. Results showed the degree of task consistency influenced the shape of learning curves, with jobs composed of primarily consistent tasks improving more rapidly and reaching asymptote sooner. In addition, trends in ability-performance correlations were moderated by the degree of task consistency within ajob. Specifically, forjobs with primarily consistent tasks, general cognitive ability best predicted early performance whereas psychomotor ability best predicted later performance. In contrast, general cognitive ability was the strongest predictor across experience for jobs with primarily inconsistent tasks.     

The relationship of specific mental ability measures compared to a general mental ability measure to quality and quantity performance on a clerical job sample

Barrett (1993) presented evidence that mental ability measures designed specifically for the prediction of performance in tasks and jobs have validity at least equal to that of tests of general mental ability, while having advantages such as lower adverse impact. The current study demonstrates that the match in specificity of predictor and criterion can be attained through simple changes in the scoring of standard mental and clerical ability tests. A sample of 51 university students completed a battery of pre-employment tests and, a week or two later, worked on a clerical job sample. The results indicated that quality of performance (number of errors on the task) was best predicted using the number of errors made on the predictor tests while quantity of work was only predicted by the number correct. A test of general mental ability did not correlate with either criterion. These results contradicted the conclusions of Schmidt (1993) that specific ability tests have no incremental validity above a general ability test.Ralph Alexander died before this report was complete. Consequently, the conclusions are those of the first two authors.     

The Incremental Validity of Honesty–Humility Over Cognitive Ability and the Big Five Personality Traits

The present study examines the incremental validity of Honesty–Humility (H-H), a measure of the tendency to be fair and genuine in dealing with others, for supervisory ratings of job performance (including both task and contextual performance) over cognitive ability and the Big Five personality traits. Specifically, we explore the incremental validity of H-H in predicting contextual performance. Results based on 217 South Korean military officer candidates are consistent with previous conclusions that Conscientiousness is the strongest predictor of contextual performance and that cognitive ability is the strongest predictor of task performance. More important, our results reveal that H-H offers moderate incremental validity for contextual performance but no incremental validity for task performance over the effects of cognitive ability and the Big Five personality traits.     

Predicting Training and Job Performance for Transit Operators

Although considerable meta‐analytic research has validated the use of cognitive ability tests, structured interviews, and personality tests with training and job performance criteria, few studies have investigated the validity of these measures with transit operators. There are the only two single studies of concurrent validation research specifically with transit operators. This article presents the results of a predictive validation study conducted with transit operator applicants for a large urban transit authority in Canada. Key knowledge, skills, and abilities were determined for the role and used as a basis for the design and choice of predictors and criteria. Four predictors were used in the study: education, cognitive ability, personality assessment, and structured interview. Criteria included training performance (formative and summative), probationary performance, preventable accidents, and lost time injuries. Validation results supported cognitive ability, structured interview, and several personality factors as predictors of training performance, but less so for job performance. The use of formative training ratings greatly augmented the evidence supporting the predictors beyond typical organizational criteria.     

Incremental validation of situational judgment tests.

Using 3 different samples, the authors assessed the incremental validity of situational judgment inventories (SJIs), relative to job knowledge, cognitive ability, job experience, and conscientiousness, in the prediction of job performance. The SJI was a valid predictor in all 3 samples and incrementally so in 2 samples. Relative to the other predictors, SJI's partial correlation with performance, controlling for the other 4 predictors, was superior in most comparisons. Subgroup differences on the SJI also appear to be less than those for cognitive ability and job knowledge, but greater than differences in conscientiousness. The SJI should prove to be a valuable additional measure in the prediction of job performance, but several additional areas of research are suggested.     

Validities of Unit-Weighted Composites of the ASVAB and the ECAT Battery

Currently, assignment to training in each entry-level enlisted job in the U.S. military is influenced by examinees’ scores on the classification composite for that job from the Armed Services Vocational Aptitude Battery (ASVAB). For a cross-service sample of the 18 occupations in the Enhanced Computer-Administered Test (ECAT) project, enhancements of their respective operational ASVAB composites (9 unique ones) are evaluated here. Analyses are reported on the gains in validity from adding 1 test with unit weight to each composite, the addition coming either from the other ASVAB tests or from the 9 tests of the ECAT battery. In more than half the schools, the best ASVAB additions were Coding Speed or Mechanical Comprehension. In the ECAT battery, 1 test of psychomotor ability (Two-Hand Tracking) and 2 tests of working memory (Mental Counters and Sequential Memory) were the most promising additions. The utility of the new composites for person–job matching is discussed.     

Measuring adaptability demands of jobs and the adaptability of military and civilians

Jobs within the military and in civil society differ in character and requirements. It is assumed that differences in job types both within civil and military domains require different dimensions of adaptability competency. The present study investigates this assumption. The Dutch Adaptability Dimensions And Performance Test (D-ADAPT) was used to measure both the adaptability demands of jobs, and of self-assessed adaptability competency of employees. In addition, the predictive value of certain personality traits (openness, neuroticism, and conscientiousness) and other relevant personal characteristics (cognitive flexibility, learning goal orientation) on various adaptability dimensions was explored. Military personnel were found to estimate their adaptability competency as higher than civilians did. With respect to the civilian jobs, the adaptability demands varied across job types. This was not found in the set of military jobs, possibly due to a smaller variety of jobs included. This study also shows that a variety of personal characteristics predict different dimensions of adaptability. Cognitive flexibility was found to be the best predictor for adaptability in general. Openness turned out to be a good predictor within the civilian group, whereas neuroticism was a better predictor within the military group. Results of this study contribute to a better understanding of work adaptability and influencing personality characteristics. Practical implications for selection and training of personnel can be derived, based on the acquisition of adaptability competency that matches the type of job.     

Validation of personality and cognitive measures for insurance claims examiners

Claims Examiners are critical for the profitability of large insurance companies. Mistakes in hiring these people can result in improperly processed claims and/or turnover, and this yields negative financial consequences for the company. This paper describes research evaluating the validity of selection procedures for the position of claims examiner at Blue Cross and Blue Shield of Oklahoma. A job analysis indicated that both cognitive skills and personal abilities are necessary for successful performance as a claims examiner. The job analysis also indicated that both subjective and objective criterion measures reflect important aspects of job performance. The criterion data included nine supervisory ratings, employee nominations, average percent of performance, absences, disciplinary actions, and sick leave. In a concurrent validation strategy, we tested a sample of experienced incumbents (N=50) using four cognitive ability tests (Basic Skills Tests: BST) and two measures of personality (Hogan Personality Inventory and PROFILE). Scales from two personality inventories were combined to form the Claims Examiner Inventory (CEI). Scores from the CEI and three of the four cognitive tests correlated significantly with the average percent of performance and six of the supervisory ratings, including the average supervisory rating. In addition to the cognitive measures, the personality measure contributed significantly to the prediction of percent of performance achieved with R=.64. These results are consistent with other research findings that support the use of both personality and cognitive measures as predictors of job performance.