Incremental Validity of Leaderless Group Discussion Ratings Over and Above General Mental Ability and Personality in Predicting Promotion

Leaderless group discussions (LGDs) constitute one of the oldest assessment center exercises. In recent times, their added value has sometimes been questioned in light of trends to streamline assessment centers. The purpose of the present study is to examine the incremental validity of LGD ratings over cognitive ability scores and personality ratings for the prediction of extrinsic career success (i.e., promotion speed and number of promotions). We investigated this issue in the context of the promotion of French naval officers (N = 93) in an academy for high‐level executive positions over a 10‐year period. Results indicated that LGD ratings accounted for incremental variance in the prediction of promotion criterion measures, beyond cognitive ability and personality test scores. These results confirm that LGD ratings provide a unique contribution to the prediction of extrinsic career success in high‐level executive positions.     

Predicting Training and Job Performance for Transit Operators

Although considerable meta‐analytic research has validated the use of cognitive ability tests, structured interviews, and personality tests with training and job performance criteria, few studies have investigated the validity of these measures with transit operators. There are the only two single studies of concurrent validation research specifically with transit operators. This article presents the results of a predictive validation study conducted with transit operator applicants for a large urban transit authority in Canada. Key knowledge, skills, and abilities were determined for the role and used as a basis for the design and choice of predictors and criteria. Four predictors were used in the study: education, cognitive ability, personality assessment, and structured interview. Criteria included training performance (formative and summative), probationary performance, preventable accidents, and lost time injuries. Validation results supported cognitive ability, structured interview, and several personality factors as predictors of training performance, but less so for job performance. The use of formative training ratings greatly augmented the evidence supporting the predictors beyond typical organizational criteria.     

INTERNATIONAL VALIDITY GENERALIZATION OF GMA AND COGNITIVE ABILITIES: A EUROPEAN COMMUNITY META-ANALYSIS

This article reports on a series of meta-analyses into the criterion validity of general menatl ability (GMA) and specific cognitive ability tests for predicting job performance ratings and training success in the European Community (EC). Meta-analyses were computed on a large EC database examining the operational validity of GMA and other specific cognitive abilities, including verbal, numerical, spatial-mechanical, perceptual and memory ( N ranged from 946 to 16,065) across 10 EC member countries. The results showed that tests of GMA and specific cognitive ability are very good predictors of job performance and training success across the EC. Evidence for the international validity generalization of GMA and specific cognitive abilities was presented. The results for the EC meta-analyses showed a larger operational validity than previous meta-analyses in the U.S. for predicting job performance. For training success, the European and American results are very similar. Implications for the international generalizability of GMA test validities, practical use of cognitive ability tests for personnel selection, and directions for future research are discussed.     

ENTRY‐LEVEL POLICE CANDIDATE ASSESSMENT CENTER: AN EFFICIENT TOOL OR A HAMMER TO KILL A FLY?

The study examined the validity of the assessment center (AC) as a selection process for entry-level candidates to die police and its unique value beyond cognitive ability tests. The sample included 712 participants who responded to personality and cognitive ability testing (CAT), and underwent an AC procedure. AC results included the overall assessment rating (OAR) and peer evaluation (PE). Seven criterion measures were collected for 585 participants from a training stage and on-the-job performance. Results showed that the selection system was valid. Findings yielded significant unique validities of OAR and PE beyond CAT and of PE beyond OAR even after corrections for restriction of range. Results support the use of ACs for entry-level candidates.     

PREDICTING LONG-TERM FIREFIGHTER PERFORMANCE FROM COGNITIVE AND PHYSICAL ABILITY MEASURES

Firefighters from 1 academy training class were observed for 23 years, beginning with their selection test consisting of a g-saturated written exam (GCA) and firefighting simulations loaded on a strength/endurance (SE) factor. Operational validity coefficients for both GCA and SE were high for training success and remained consistently high for job performance ratings throughout the study. The operational validity for combined GCA and SE predictors was .86 for a composite job rating measure covering 21 years of service. A structural model produced similar results for more broadly defined GCA and SE latent variables. Both analyses suggested approximately equal weighting for GCA and SE for a fire service selection test. Results indicate considerable latitude in choosing cognitive and physical predictors for firefighter screening if the predictors are highly loaded on GCA and SE.     

Intelligence,Executive Functions,and Decision Making as Predictors of Antisocial Behavior in an Adolescent Sample of Justice‐Involved Youth and a Community Comparison Group

A clinical sample of justice‐involved male adolescents and a community comparison group were compared on a battery of cognitive ability tasks (intelligence and executive functions), decision making measures, and other individual difference measures, including ratings of self‐control, recognition of morally debatable behaviors, and antisocial beliefs. The clinical sample displayed lower performance on cognitive abilities and decision making than the community comparison group. In particular, the clinical group displayed less otherside thinking and more hostile attribution biases in unintentional situations compared with the community comparison group. Cognitive abilities and the decision making performance predicted group membership. Then, group membership, ratings of self‐control, attitudes about morally debatable behaviors, and antisocial beliefs predicted ratings of antisocial behavior in the full sample. These findings suggest that measures of cognitive ability and decision making make separate contributions to explaining antisocial behaviors. In addition, the predictors of group membership and antisocial behavior did not overlap, suggesting that antisocial behavior engagement in clinical samples may be separable from the continuum of antisocial behavior across the full sample. Cognitive science models of decision making can provide a framework for understanding antisocial behavior in clinical and community samples of adolescents. Copyright © 2015 John Wiley & Sons, Ltd.     

Assessment Center Dimensions: Individual differences correlates and meta‐analytic incremental validity

This study provides an investigation of the nomological net for the seven primary assessment center (AC) dimensions identified by Arthur, Day, McNelly, and Eden (Personnel Psychology, 56, 125–154, 2003). In doing so, the authors provide the first robust estimates of the relationships between all primary AC dimensions with cognitive ability and the Big 5 factors of personality. Additionally, intercorrelations between AC dimensions based on sample sizes much larger than those previously available in the meta‐analytic literature are presented. Data were obtained from two large managerial samples (total N=4985). Primary data on AC dimensions, personality, and cognitive ability interrelationships were subsequently integrated with meta‐analytic data to estimate incremental validity for optimally and unit‐weighted AC dimension composites as well as overall AC ratings over psychometric tests of personality and cognitive ability. Results show that unit‐ and optimally weighted composites of construct‐based AC dimensions add incremental validity over tests of personality and cognitive ability, while overall AC ratings (including those obtained using subjective methods of data combination) do not.     

Transitioning from recruit to officer: An investigation of how stress appraisal and coping influence work engagement

This study investigated stress, coping, and work engagement among Portuguese police officers while undergoing academy training and then 1 year later, when on duty. It was hypothesized that stress appraisal and coping preferences predicted engagement. Additionally, in order to test a full cross‐lagged prediction model, it was hypothesized that stress, coping, and engagement in recruits predicted these variables later when working as police officers. Structural equation modeling was used to test the research hypotheses. Results suggest that coping and stress appraisals do not seem to be strong predictors of work engagement among recruits and police officers on duty. With the exception of self‐blame, that seems to be a strong predictor of work engagement among police officers on duty. These results highlight the need to investigate other potential variables such as working conditions that may better explain work engagement. Considering the positive influence of engagement on health, wellbeing, and performance of police recruits and officers future applied and theoretical implications are discussed.     

The predictive validity of cognitive ability tests: A UK meta‐analysis

A meta‐analysis on the validity of tests of general mental ability (GMA) and specific cognitive abilities for predicting job performance and training success in the UK was conducted. An extensive literature search resulted in a database of 283 independent samples with job performance as the criterion (N=13,262), and 223 with training success as the criterion (N=75,311). Primary studies were also coded by occupational group, resulting in seven main groups (clerical, engineer, professional, driver, operator, manager, and sales), and by type of specific ability test (verbal, numerical, perceptual, and spatial). Results indicate that GMA and specific ability tests are valid predictors of both job performance and training success, with operational validities in the magnitude of .5–.6. Minor differences between these UK findings and previous US meta‐analyses are reported. As expected, operational validities were moderated by occupational group, with occupational families possessing greater job complexity demonstrating higher operational validities between cognitive tests and job performance and training success. Implications for the practical use of tests of GMA and specific cognitive abilities in the context of UK selection practices are discussed in conclusion.     

Transparency of Assessment Centers: Lower Criterion‐related Validity but Greater Opportunity to Perform?

Assessment centers (ACs) are popular selection devices in which assessees are assessed on several dimensions during different exercises. Surveys indicate that ACs vary with regard to the transparency of the targeted dimensions and that the number of transparent ACs has increased during recent years. Furthermore, research on this design feature has put conceptual arguments forward regarding the effects of transparency on criterion‐related validity, impression management, and fairness perceptions. This study is the first to examine these effects using supervisor‐rated job performance data as the criterion. We conducted simulated ACs with transparency as a between‐subjects factor. The sample consisted of part‐time employed participants who would soon be applying for a new job. In line with our hypothesis, results showed that ratings from an AC with nontransparent dimensions were more criterion valid than ratings from an AC with transparent dimensions. Concerning impression management, our results supported the hypothesis that transparency moderates the relationship between self‐promotion and job performance, such that self‐promotion in the nontransparent AC was more positively related to job performance than self‐promotion in the transparent AC. The data lent no support for the hypothesis that participants’ perceptions of their opportunity to perform are higher in the transparent AC.     

Criterion‐Related Validity of Dutch Police‐Selection Measures and Differences Between Ethnic Groups

This study investigated the criterion‐related validity of cognitive ability as well as non‐cognitive ability measures and differences between ethnic majority (N=2365) and minority applicants (N=682) in Dutch police officer selection. Findings confirmed the relatively low predictive validity of cognitive ability generally found for police jobs. Previous research reported no differential prediction. The present study, however, found small but systematic evidence for differences in validity for the ethnic majority and minority group of both cognitive and non‐cognitive measures. For the minority group, training performance appeared to be mainly predicted by the cognitive ability test. For the majority group, cognitive ability showed very little predictive power. Non‐cognitive ability variables appeared to be somewhat more predictive in this group.     

Designing Selection Systems for Medicine: The importance of balancing predictive and political validity in high‐stakes selection contexts

Selection into medical education and training is a high‐stakes process. A key unanswered issue is the effectiveness of measuring noncognitive predictors via both low‐fidelity and high‐fidelity selection approaches in this high‐stakes context. We review studies investigating the effectiveness of multiple selection instruments in terms of predictive validity, incremental validity, and applicant reactions in both entry‐level and advanced‐level medical selection. Our results show that the situational judgment test (SJT) is the best single predictor of performance, operationalized in multiple ways. In addition, the low‐fidelity SJT has incremental predictive power over cognitively oriented tests, and high‐fidelity assessment center (AC) exercises add incremental validity over the low‐fidelity (and less costly) selection methods. Concerning applicant reactions, results show that overall, the selection system is positively received. However, the method with the highest predictive validity – the SJT – received comparatively lower face validity ratings which may present a ‘justice dilemma’ for employers. Furthermore, various other stakeholders have a political interest in the selection methods used (e.g., government, the regulators and trade unions).     

An investigation into assessment centre validity,fairness, and selection drivers

This study examined the construct‐related validity of an assessment centre (AC) developed by a national distribution company for the selection and development of lower‐grade managers. In five locations throughout Britain, 487 individuals were observed on nine dimensions, each of which was measured through six distinct exercises. Multitrait‐multimethod analyses conducted to investigate the convergent and discriminant validity of the AC revealed strong exercise (“method”) effects. This finding was corroborated by an exploratory factor analysis showing that AC ratings clustered into factors according to exercises, rather than according to performance dimensions. A series of MANOVAs and chi‐squared tests demonstrated that neither the exercise ratings nor the selection decision were biased by sex, ethnicity, or training location, and a logistic regression determined which exercises had most impact on the final decision.     

The Swedish Hayling task,and its relation to working memory,verbal ability,and speech‐recognition‐in‐noise

Cognitive functions and speech‐recognition‐in‐noise were evaluated with a cognitive test battery, assessing response inhibition using the Hayling task, working memory capacity (WMC) and verbal information processing, and an auditory test of speech recognition. The cognitive tests were performed in silence whereas the speech recognition task was presented in noise. Thirty young normally‐hearing individuals participated in the study. The aim of the study was to investigate one executive function, response inhibition, and whether it is related to individual working memory capacity (WMC), and how speech‐recognition‐in‐noise relates to WMC and inhibitory control. The results showed a significant difference between initiation and response inhibition, suggesting that the Hayling task taps cognitive activity responsible for executive control. Our findings also suggest that high verbal ability was associated with better performance in the Hayling task. We also present findings suggesting that individuals who perform well on tasks involving response inhibition, and WMC, also perform well on a speech‐in‐noise task. Our findings indicate that capacity to resist semantic interference can be used to predict performance on speech‐in‐noise tasks.     

Book Review

Performance-based measures and ratings of executive functions were examined in a sample of adolescents with attention deficit/hyperactivity disorder (ADHD) and comparison controls. Performance-based measures of executive function included inhibition, working memory, set shifting, and planning, and ratings of these same executive functions were completed by parents and teachers. Adolescents with ADHD demonstrated lower executive function performance than controls and displayed elevated ratings on the executive function ratings by parents and teachers. Significant associations were obtained between the performance-based measures and the parent and teacher ratings, but each measure was not uniquely associated with its respective scale on the rating scales. When performance-based measures and ratings were examined as predictors of ADHD status, the parent and teacher ratings entered as significant predictors of ADHD status. Further commonality analyses indicated that performance-based measures accounted for little unique variance in predicting ADHD status and also displayed little overlap with the behavioral ratings. These findings highlight the diagnostic utility of behavioral ratings of executive function in predicting ADHD status; however, behavioral ratings should not be assumed to be a proxy for performance on measures of executive function in clinical practice.     

The Role of Cognitive Ability Tests in Employment Selection

Cognitive ability tests correlate with measures of job performance across many jobs. However, cognitive ability tests produce racial differences that are 3 to 5 times larger than other predictors-such as biodata, personality inventories, and the structured interview-that are valid predictors of job performance. Given that (a) cognitive ability tests can be combined with other predictors such that adverse impact is reduced while overall validity is increased, and (b) alternative predictors with less adverse impact can produce validity coefficients comparable to those obtained with cognitive ability tests alone, sole reliance on cognitive ability tests when alternatives are available is unwarranted.     

Multisource Assessments of Behavioral Competencies and Selection Interview Performance

This study sought to determine whether and how ratings of leadership in a developmental multisource feedback program compare against those obtained by leadership selection decision makers. Seventy‐seven senior managers received feedback on eight competencies that also formed the basis of a subsequent promotional interview for senior executive positions. Each rating source displayed a unique perspective on the participants, but only the supervisor and peer ratings predicted performance on the interview. As well, peer, subordinate, and self‐ratings failed to improve upon the supervisors' predictions. The results also suggested a general performance factor underlying both the multisource assessments and the interview ratings.     

Executive Attention: An Alternative Perspective on General Mental Ability,Performance, and Subgroup Differences

The validity‐adverse impact tradeoff associated with the relationships among general mental ability (GMA), ethnicity, and employee performance represents one of the most pressing concerns in organizational staffing. We conducted 4 studies with 273 bank employees and 197 university students designed to assess the extent to which executive attention (EA) and GMA predict simulation performance and supervisory ratings of performance. We also assess the extent to which measures of EA and GMA are associated with subgroup differences. Results indicate that, like GMA, EA positively predicts managerial simulation and supervisory ratings of performance. In addition, although reaching statistical significance in only 1 of our 4 studies, EA was generally associated with smaller subgroup differences than GMA, and meta‐analysis across our samples supports this reduced subgroup difference. Moreover, advantages of EA tend to increase as studies move from the laboratory with undergraduate students to a concurrent validation organizational setting with employees. We discuss implications for a theory‐based view of cognitive ability in employee selection and implications for managerial practice.     

Police Officer Selection validation project: The multijurisdictional Police Officer Examination

This study presents the results of a predictive validity study of the Multijurisdictional Police Officer Examination (MPOE). The MPOE is a paper-and-pencil test of cognitive abilities developed to select entry level police officers. Criterion measures for the study included training performance, supervisory performance ratings and a file composite measure of negative work related incidents (e.g., use of force). There were 159 (120 majority and 39 minority group) members in the final validation sample. Results indicated that the MPOE was predictive of the three performance indices. Cumulation of results across seven studies (including the present effort) revealed a weighted average correlation between the MPOE and performance ratings of .20 (N=913). The potential generalizability of the results to other police departments is discussed.     

A META-ANALYSIS OF WORK SAMPLE TEST VALIDITY: UPDATING AND INTEGRATING SOME CLASSIC LITERATURE

Work sample tests have been used in applied psychology for decades as important predictors of job performance, and they have been suggested to be among the most valid predictors of job performance. As we examined classic work sample literature, we found the narrative review by Asher and Sciarrino (1974) to be plagued by many methodological problems. Further, it is possible that data used in this study may have influenced the results (e.g., r = .54) reported by Hunter and Hunter in their seminal work in 1984. After integrating all of the relevant data, we found an observed mean correlation between work sample tests and measures of job performance of .26. This value increased to .33 when measures of job performance (e.g., supervisory ratings) were corrected for attenuation. Our results suggest that the level of the validity for work sample tests may not be as large as previously thought (i.e., approximately one third less than previously thought). Further, our work also summarizes the relationship of work sample exams to measures of general cognitive ability. We found that work sample tests were associated with an observed correlation of .32 with tests of general cognitive ability.