Selection Tests for Firefighters: A Comprehensive Review and Meta-Analysis

This study reviewed and summarized the literature on the use of written tests in the selection of firefighters using a sample of 13,418 individuals drawn from 101 samples. For the prediction of job performance, cognitive tests showed substantial validity (.42), although mechanical comprehension tests showed even higher validity (.54). However, the best prediction was obtained by tests which were composites of cognitive and mechanical predictors (.56). Training criteria was best predicted by cognitive measures (.77), although mechanical comprehension predictors also showed substantial prediction value (.62). Tests which were composites of cognitive and mechanical measures showed validities equal to that of cognitive measures (.77).     

Review of psychomotor skills in pilot selection research of the U.S. military services

This review provides an historical perspective of the use of psychomotor, perceptual--cognitive paper-and-pencil, and automated tests for the selection of pilot trainees by the U.S. military services. Automated versions of vintage psychomotor tests (developed in the 1930s and 1940s) seem to be as predictive of military pilot/aviator performance today as in the past. The psychomotor tests receiving the most attention today are the Complex Coordination and Two-Hand Coordination tests originally developed by Mashburn and colleagues [correction of colleges] before World War II (Mashburn, 1934). These tests were significant predictors of Air Force and Navy pass--fail criteria in the past, and automated versions are similarly predictive today. The U.S. Army and Air Force are now using a combination of paper-and-pencil and automated psychomotor--cognitive tests for initial selection (Air Force) or helicopter assignment (Army). It appears that the Navy is considering the use of automated cognitive and psychomotor tests in a selection battery of the future.     

PROJECT A VALIDITY RESULTS: THE RELATIONSHIP BETWEEN PREDICTOR AND CRITERION DOMAINS

A predictor battery of cognitive ability, perceptual-psychomotor ability, temperament/personality, interest, and job outcome preference measures was administered to enlisted soldiers in nine Army jobs. These measures were summarized in terms of 24 composite scores. The relationships between the predictor composite scores and five components of job performance were analyzed. Scores from the cognitive and perceptual-psychomotor ability tests provided the best prediction of job-specific and general task proficiency, while the temperament/personality composites were the best predictors of giving extra effort, supporting peers, and exhibiting personal discipline. Composite scores derived from the interest inventory were correlated more highly with task proficiency than with demonstrating effort and peer support. In particular, vocational interests were among the best predictors of task proficiency in combat jobs. The results suggest that the Army can improve the prediction of job performance by adding non-cognitive predictors to its present battery of predictor tests.     

The Role of General and Specific Psychomotor Tracking Ability in Validity

A study of 1,099 males and females in pilot training was conducted to investigate the relative importance of general and specific factors of psychomotor tracking ability in the prediction of several pilot training criteria. Measures of psychomotor tracking ability were derived from the Basic Attributes Test (Carretta and Ree 1994). The criteria were the dichotomous passing or failing pilot training and the average of six flying work samples. Comparisons of linear models indicated that general psychomotor tracking ability was the source of most of the validity and that only the specific measure of reaction time was incremental. Reaction time, although part of the psychomotor taxonomy, has frequently been identified as a measure of cognitive ability.     

Relationship of laboratory and on-road tests for driving-school students and experienced drivers

There is a pressing need to evaluate driving ability without an actual driving test but no reliable, standardised procedure is available. This study investigated the resulting distortion performance evaluation for different driver groups in on-road tests due to driving skills and driving style. Two groups, 15 female driving school students and 25 experienced male drivers were examined using cognitive and psychomotor tests and standard personality questionnaires as well as an on-road driving test. The students made significantly fewer errors on the driving test than the experienced drivers. The former group's driving performance could be predicted by errors on the laboratory tests, along with personality factors, while experienced drivers' driving errors were best predicted by slowness on laboratory tests. This study suggests that, when a series of standard cognitive and psychomotor tests are considered, different measures may apply in predicting young people's ability to drive safely based on assessing driving ability of experienced drivers, assumed lowered due to distorted driving style.     

Expanding the Nexus of Cognitive and Psychomotor Abilities

A study was conducted to expand the nexus of cognitive and psychomotor abilities. A cognitive aptitude battery and a psychomotor battery were administered to 429 military recruits. A confirmatory factor analysis yielded higher-order factors of general cognitive ability (g) and psychomotor/technical knowledge (PM/TK). PM/TK was interpreted as Vernon's (1969) practical factor (k:m). In the joint analysis of these batteries, g and PM/TK each accounted for about 31% of the common variance. No residualized lower-order factor accounted for more than 7% PM/TK influenced a broad range of lower-order psychomotor factors. The first practical implication of these findings is that psychomotor tests are expected to be at least generally interchangeable. A second implication is that the incremental validity of psychomotor tests beyond cognitive tests is expected to be small. These findings should help guide test developers and inform personnel selecting agencies regarding the expected utility of psychomotor tests.     

Introducing a forced-choice recognition task to the California Verbal Learning Test – Children’s Version

The importance of performance validity tests (PVTs) is increasingly recognized in pediatric neuropsychology. To date, research has focused on investigating whether PVTs designed for adults function similarly in children. The downward extension of adult cutoffs is counter-intuitive considering the robust effect of age-related changes in basic cognitive skills in children and adolescents. The purpose of this study was to examine the signal detection properties of a forced-choice recognition trial (FCR-C) for the California Verbal Learning Test – Children’s Version. A total of 72 children aged 6–15 years (M = 11.1 , SD = 2.6) completed the FCR-C as part of a larger neuropsychological assessment battery. Cross-validation analyses revealed that the FCR-C had good signal detection performance against reference PVTs. The first level of failure (≤14/15) produced the best combination of overall sensitivity (.31) and specificity (.87). A more conservative FCR-C cutoff (≤13) resulted in a predictable trade-off between sensitivity (.15) and specificity (.94), but also a net loss in discriminant power. Lowering the cutoff to ≤12 resulted in a slight improvement in specificity (.97) but further deterioration in sensitivity (.14). These preliminary findings suggest that the FCR-C has the potential to become the newest addition to a growing arsenal of pediatric PVTs.     

Organization and components of psychomotor ability

We investigated the organization and components of psychomotor abilities by administering a diverse set of cognitive and psychomotor tasks to a group of recent high school graduates (N = 161). Confirmatory factor analyses identified two psychomotor factors: a general factor associated with all psychomotor tests and an orthogonal psychomotor learning factor associated exclusively with practiced psychomotor tests. Path analyses suggested that the general psychomotor factor could be largely accounted for by two cognitive factors, general working-memory capacity (r = .67) and an orthogonal time estimation factor (r = .32). Most of the psychomotor learning factor variance was unique, but psychomotor learning was somewhat related to processing speed (r = .49). We conclude that initial psychomotor performance is constrained by working-memory limits and the ability to keep track of time. Practiced psychomotor skill is additionally limited by processing speed, consistent with the literature on the development of process automaticity. The Discussion addresses (a) the small dimensionality of the psychomotor abilities space, (b) our discovery of the importance of time estimation and other cognitive factors in psychomotor learning, and (c) the changing nature of psychomotor skill with practice.     

Prediction of situational awareness in F-15 pilots

Situational awareness (SA) is a skill often deemed essential to pilot performance in both combat and noncombat flying. A study was conducted to determine if SA in U.S. Air Force F-15 pilots could be predicted. The participants were 171 active duty F-15 A/C pilots who completed a test battery representative of various psychological constructs proposed or demonstrated to be valid for the prediction of performance in a wide variety of military and civilian jobs. These predictors encompassed measures of cognitive ability, psychomotor ability, and personality. Supervisor and peer ratings of SA were collected. Supervisors and peers showed substantial agreement on the SA ratings of the pilots. The first unrotated principle component extracted from the supervisor and peer ratings accounted for 92.5% of the variability of ratings. The unrotated first principal component served as the SA criterion. Flying experience measured in number of F-15 hours was the best predictor of SA. After controlling for the effects of F-15 flying hours, the measures of general cognitive ability based on working memory, spatial reasoning, and divided attention were found to be predictive of SA. Psychomotor and personality measures were not predictive. With additional F-15 flying hours it is expected that pilots would improve their ratings of SA.     

The Correlation of General Cognitive Ability and Psychomotor Tracking Tests

A study was conducted to investigate the nexus of cognitive and psychomotor tests as might be used for personnel selection and assessment. These two domains are frequently seen as independent. A multiple aptitude cognitive test battery and a psychomotor test battery were administered to 354 United States Air Force recruits. The average multiple correlation of the cognitive tests and each psychomotor score as a criterion was 0.34, corrected for range restriction. Confirmatory factor analyses disclosed general cognitive and general psychomotor factors, three lower-order psychomotor factors, and two lower-order cognitive factors. The general cognitive factor accounted for 39% of the variance and the general psychomotor factor accounted for 29% of the variance. Residualized, the lower-order factors accounted for between 10% and 3% of the variance. The average g saturations (loadings) of the cognitive and psychomotor tests were 0.82 and 0.34 respectively. An implication for personnel selection is that the incremental validity of psychomotor tracking tests beyond the validity of cognitive tests will be small due to the commonality of measurement. A further implication of findings is the need to study the validity of the general and specific psychomotor factors.     

Psychomotor abilities via touch‐panel testing: Measurement innovations,construct, and criterion validity

Assessment of psychomotor abilities for prediction of human performance is briefly reviewed. Reasons for the abandonment of psychomotor testing for selection applications are described. We review innovations in touch‐sensitive computer monitors as a methodology for relatively low‐cost, highly flexible test development, validation, and application of standard psychomotor tests. The development and evaluation of 5 psychomotor test types are described including discrete response tests (choice‐simple reaction time [RT], serial RT, and tapping) and continuous‐response tests (maze tracing and mirror tracing). Two empirical studies of the new psychomotor tests are presented, with a broad array of perceptual speed and cognitive abilities providing evidence for construct validity. In addition, some of the psychomotor tests are validated against a real‐time simulation criterion (the Kanfer‐Ackerman Air Traffic Controller Task©). We argue that these new innovations provide a means toward revisiting psychomotor testing to augment employee selection batteries.     

DYNAMIC CRITERIA REVISITED: A LONGITUDINAL STUDY OF PERFORMANCE STABILITY AND PREDICTIVE VALIDITY

The concept of dynamic criteria has been the subject of a recent debate regarding both the definition and prevalence of the phenomenon (Austin, Humphreys, & Hulin, 1989; Barrett & Alexander, 1989; Barrett, Caldwell, & Alexander, 1985). The present paper questions the adequacy of the conceptual framework underlying the debate and provides data supporting a refined concept of dynamic criteria. The incidence and possible causes of change in relative performance were investigated using weekly performance data from 509 sewing machine operators. Analyses were conducted to determine the degree of performance consistency, potential moderators of consistency, and the stability of predictor-criteria relationships using multiple predictors and criteria. Results revealed a steady decline in performance stability coefficients as the interval between measures increased. This decay was evident regardless of employees' prior job experience, cognitive ability, or psychomotor ability. Analyses of predictive validity coefficients revealed temporal changes in validity for both objective and subjective criteria, but not in the expected direction. The validity of cognitive ability increased, the validity of psychomotor ability was stable, and that of prior job experience decreased over time. Implications for theory and research are discussed.     

Genetic and environmental influences on antisocial behavior: a meta-analysis of twin and adoption studies

A meta-analysis of 51 twin and adoption studies was conducted to estimate the magnitude of genetic and environmental influences on antisocial behavior. The best fitting model included moderate proportions of variance due to additive genetic influences (.32), nonadditive genetic influences (.09), shared environmental influences (.16), and nonshared environmental influences (.43). The magnitude of familial influences (i.e., both genetic and shared environmental influences) was lower in parent-offspring adoption studies than in both twin studies and sibling adoption studies. Operationalization, assessment method, zygosity determination method, and age were significant moderators of the magnitude of genetic and environmental influences on antisocial behavior, but there were no significant differences in the magnitude of genetic and environmental influences for males and females.     

Predicting Training and Job Performance for Transit Operators

Although considerable meta‐analytic research has validated the use of cognitive ability tests, structured interviews, and personality tests with training and job performance criteria, few studies have investigated the validity of these measures with transit operators. There are the only two single studies of concurrent validation research specifically with transit operators. This article presents the results of a predictive validation study conducted with transit operator applicants for a large urban transit authority in Canada. Key knowledge, skills, and abilities were determined for the role and used as a basis for the design and choice of predictors and criteria. Four predictors were used in the study: education, cognitive ability, personality assessment, and structured interview. Criteria included training performance (formative and summative), probationary performance, preventable accidents, and lost time injuries. Validation results supported cognitive ability, structured interview, and several personality factors as predictors of training performance, but less so for job performance. The use of formative training ratings greatly augmented the evidence supporting the predictors beyond typical organizational criteria.     

A reexamination of black-white mean differences in work performance: more data, more moderators

This study is the largest meta-analysis to date of Black-White mean differences in work performance. The authors examined several moderators not addressed in previous research. Findings indicate that mean racial differences in performance favor Whites (d = 0.27). Effect sizes were most strongly moderated by criterion type and the cognitive loading of criteria, whereas data source and measurement level were influential moderators to a lesser extent. Greater mean differences were found for highly cognitively loaded criteria, data reported in unpublished sources, and for performance measures consisting of multiple item scales. On the basis of these findings, the authors hypothesize several potential determinants of mean racial differences in job performance.     

Relationships Among Selected Tests of Spatial Orientation Ability

The purpose of this study was to investigate the relationships between selected tests of spatial orientation ability. 2 tests which have been used as predictors of spatial ability (Guilford-Zimmerman Aptitude Survey, Parts V and VI) plus 2 newly developed tests designed to measure that ability were given to 202 Junior high school boys aged 11-15 yr. 3 of these tests were of the paper and pencil type while one was a physical performance (tumbling) test. The paper-and-pencil tests correlated significantly with each other (.62, .61, .44) but the physical performance test did not correlate significantly with any of the other tests.     

Subvocalization rate versus other predictors of the memory span

Two experiments were performed to determine how accurately the immediate memory span may be predicted from the subject's subvocalization rate, as compared with other subject and stimulus variables. In the first, span and free-recall measures were obtained for 24 subjects, each tested with four types of spoken material (nonsense syllables, random words, fourth-order approximations to English, and normal prose). Silent subvocalization rate, whispered subvocalization rate, speaking rate, and silent reading rate were also measured for each subject. The span (3.9-14 items) showed the highest zero-order correlation with silent subvocalization rate: r = .80. Multiple regression analysis confirmed this as the span's best predictor, partial r = .31. Similar results were found for free-recall scores. In Exp. 2 the digit span was correlated against measures of age, intelligence, subvocalization rate, perceptual speed, and memory search rate, for 40 subjects aged 7 to 17 yr. The best zero-order predictor was age (.62), followed by subvocalization rate (.57) and intelligence (.39). Multiple regression analysis indicated that the span was best predicted by age, partial r = .37, and subvocalization rate, partial r = .29. The span appears more closely related to subjects' internal speech rate than to other cognitive functions.     

Individual prediction of dyslexia by single versus multiple deficit models

The overall goals of this study were to test single versus multiple cognitive deficit models of dyslexia (reading disability) at the level of individual cases and to determine the clinical utility of these models for prediction and diagnosis of dyslexia. To accomplish these goals, we tested five cognitive models of dyslexia--two single-deficit models, two multiple-deficit models, and one hybrid model--in two large population-based samples, one cross-sectional (Colorado Learning Disability Research Center) and one longitudinal (International longitudinal Twin Study). The cognitive deficits included in these cognitive models were in phonological awareness, language skill, and processing speed and/or naming speed. To determine whether an individual case fit one of these models, we used two methods: 1) the presence or absence of the predicted cognitive deficits, and 2) whether the individual's level of reading skill best fit the regression equation with the relevant cognitive predictors (i.e., whether their reading skill was proportional to those cognitive predictors.) We found that roughly equal proportions of cases met both tests of model fit for the multiple deficit models (30-36%) and single deficit models (24-28%); hence, the hybrid model provided the best overall fit to the data. The remaining roughly 40% of cases in each sample lacked the deficit or deficits that corresponded with their best-fitting regression model. We discuss the clinical implications of these results for both diagnosis of school-age children and preschool prediction of children at risk for dyslexia.     

Role of age in relation between two kinds of abilities and performance in acquisition of new motor skill

Better understanding of age-related differences in skilled performance was the focus of analyses of cognitive-performance scores-relationships in acquisition of a new motor skill. 31 younger adults and 33 older adults were tested on both a cognitive and a psychomotor test. Then, they were asked to learn a juggling task over 12 sessions of 20 min. Analysis indicated age-related differences in the rate of learning. Acquisition by the younger adult group was significantly faster than that by the older adult group. This difference was also reflected in the relationship of cognition and performance for the two age groups. Motor execution for the older adults seemed to require more psychomotor ability, especially at the end of the learning sessions, and was dependent on cognitive control. This trend is consistent with the perspective that cognitive predictors of performance are related to age.     

Self-Efficacy and the Prediction of Domain-Specific Cognitive Abilities

ABSTRACT We evaluated predictors of performance in 4 specific cognitive ability domains: verbal, numerical, spatial, and mechanical. The predictors were individual differences in self-efficacy beliefs, self-enhancement tendencies, and cross-domain abilities. Our university students' beliefs about their verbal, numerical, and spatial capabilities correlated well with their actual performance on standardized tests (verbal r =.33, numerical r =.27, spatial r =.36). In contrast, the students' self-efficacy for mechanical tasks did relatively poorly in predicting mechanical test performance ( r =.10). Most interesting were two other findings: (a) The best predictor of domain performance was level of cross-domain performance by far, even for mechanical tasks, and (b) self-enhancement tendencies added to cross-domain abilities and self-efficacy beliefs in the prediction of performance. The results are discussed in terms of possible mechanisms explaining how one's score on a maximal performance task can be affected by self-efficacy beliefs and self-enhancement tendencies.