Initial Validation of a Computer-Based Secondary Selection System for Student Naval Aviators

This research describes the evaluation of a recently developed experimental computer-based aircrew secondary selection test battery. Student naval avia- tors selected for flight training on the basis of current paper-and-pencil screening tools were administered a series of performance tasks. The tasks assessed abilities in problem solving, spatial reasoning, perceptual motor skills, and memory. Performance data included measures of both accuracy and reaction time. The results indicate that performance-based test measures can be used after initial selection to predict flight training performance. Several test measures were reliably related to both pass/fail and primary flight grade (FG) criteria. Age and previous flight training also were found to contribute to prediction of the criteria. The results provide support for the prediction of performance in addition to passing or failing in flight training.     

Incremental Validity of Assessment Center Ratings Over Cognitive Ability Tests: A Study at the Executive Management Level

Both tests of cognitive ability and assessment center (AC) ratings of various performance attributes have proven useful in personnel selection and promotion contexts. To be of theoretical or practical value, however, the AC method must show incremental predictive accuracy over cognitive ability tests given the cost disparities between the two predictors. In the present study, we investigated this issue in the context of promotion of managers in German police departments into a training academy for high‐level executive positions. Candidates completed a set of cognitive ability tests and a 2‐day AC. The criterion measure was the final grade at the police academy. Results indicated that AC ratings of managerial abilities were important predictors of training success, even after accounting for cognitive ability test scores. These results confirm that AC ratings provide unique contribution to the understanding and prediction of training performance of high‐level executive positions beyond cognitive ability tests.     

IMPLICATIONS OF THE MULTIDIMENSIONAL NATURE OF JOB PERFORMANCE FOR THE VALIDITY OF SELECTION TESTS: MULTIVARIATE FRAMEWORKS FOR STUDYING TEST VALIDITY

Although most studies of criterion-related validity focus on univariate relationships, the complex and multidimensional nature of the performance construct and the widespread use of multiple selection devices argue in favor of multivariate frameworks for evaluating validity. Using a Monte Carlo simulation we estimated the validity of general cognitive ability tests and personality tests in predicting "job performance," where performance is conceptualized as a composite of multiple performance measures (i.e., individual job task performance and organizational citizenship behaviors). The validity of a selection battery varies substantially as a function of the relative weight given to both predictors and criteria; the 95% confidence interval for validities ranged from .20 to .78. The effective weights given to performance dimensions accounted for 34% of the variance in selection battery validities; depending on precisely how "performance" is defined, the same test battery can have relatively high or relatively low levels of validity. Our model suggests that the way an organization defines job performance is a source of true and important variability in validities, and that the validity of selection tests for predicting complex performance criteria may show considerably less generalizability that current metaanalysis of univariate validities would suggest.     

Pilot selection in the Norwegian Air Force: a validation and meta-analysis of the test battery

The purpose of this study was to evaluate the test battery currently used for pilot selection to the Norwegian Air Force. Selection is currently based on a standard battery of 20 different psychological tests as well as on medical tests and on an interview by a licensed psychologist. First, two-factor analyses were conducted to examine the relation between the tests in the battery. Then, a correlation study was conducted to evaluate the predictive validity of the tests against two criteria of pilot performance collected during the basic training period. Finally, a small-scale meta-analysis of previous validation studies in Norway was conducted. me best predictors of success in training, based on the meta-analysis, were Instrument Comprehension (mean r = .29), Mechanical Principles (mean r = .23), and Aviation Information (mean r = .22)     

PROJECT A VALIDITY RESULTS: THE RELATIONSHIP BETWEEN PREDICTOR AND CRITERION DOMAINS

A predictor battery of cognitive ability, perceptual-psychomotor ability, temperament/personality, interest, and job outcome preference measures was administered to enlisted soldiers in nine Army jobs. These measures were summarized in terms of 24 composite scores. The relationships between the predictor composite scores and five components of job performance were analyzed. Scores from the cognitive and perceptual-psychomotor ability tests provided the best prediction of job-specific and general task proficiency, while the temperament/personality composites were the best predictors of giving extra effort, supporting peers, and exhibiting personal discipline. Composite scores derived from the interest inventory were correlated more highly with task proficiency than with demonstrating effort and peer support. In particular, vocational interests were among the best predictors of task proficiency in combat jobs. The results suggest that the Army can improve the prediction of job performance by adding non-cognitive predictors to its present battery of predictor tests.     

Understanding how and why adding valid predictors can decrease the validity of selection composites: A generalization of Sackett,Dahlke, Shewach,and Kuncel (2017)

It is usually assumed that adding more valid predictors will increase the predictive power of a selection test battery. Sackett, Dahlke, Shewach, and Kuncel showed that when selection tests are combined using unit weights, adding a valid predictor can lead to a decrease in validity. Situating the Sackett et al. approach in a more general multivariate framework I show how: (a) it is the tradeoff between predictor validity and predictor intercorrelations, and not the differences in predictor validities that determines whether adding a valid predictor to a composite will cause the validity of that composite to increase or decrease; and (b) this same dynamic applies across a wide range of non‐optimal schemes for weighting predictors and/or criteria.     

Selecting Predictor Subsets: Considering validity and adverse impact

The paper proposes a procedure for designing Pareto‐optimal selection systems considering validity, adverse impact and constraints on the number of predictors from a larger subset that can be included in an operational selection system. The procedure determines Pareto‐optimal composites of a given maximum size thereby solving the dual task of identifying the predictors that will be included in the reduced set and determining the weights with which the retained predictors will be combined to the composite predictor. Compared with earlier proposals, the simultaneous consideration of both tasks makes it possible to combine several strategies for reducing adverse impact in a single procedure. In particular, the present approach allows integrating (a) investigating a large number of possible predictors (such as multitest battery of ability tests, or a collection of ability and nonability measures); (b) explicit predictor weighting within feasible test procedures of a given limited size.     

VALIDATING SYNTHETIC VALIDATION: COMPARING TRADITIONAL AND SYNTHETIC VALIDITY COEFFICIENTS

We describe a unique application of a synthetic validation technique to a selection system development project in a large organization. Job analysis data were collected from 4,725 job incumbents and 619 supervisors, and were used to identify 11 job families and 27 job components. We developed 12 tests to predict performance on these job components and conducted a concurrent validation study to collect test and job component data for 1,926 incumbents. We created a test composite for each job component and then chose a test battery for each job family based on its relevant job components. Synthetic validity coefficients were computed for each test battery and compared to traditional validity coefficients that were computed within job families with large sample sizes. The synthetic validity coefficient was very close to the within-family validity coefficient for most job families and was within the bounds of sampling error for all job families. Validities tended to be highest when predictors were weighted according to the number of job components to which they were relevant and job component criterion measures were unit weighted.     

Developing a generic approach to online automated analysis of writing and drawing tests in clinical patient profiling

Writing and drawing tests are widely used in the clinical environment for the diagnosis of a variety of neuropsychological conditions. Conventional assessment of these tests involves the inspection by trained assessors of the completed patient response. This article describes the development of a computer-based framework for data capture, automated feature analysis, and result reporting for a range of drawing- and writing-based test batteries. In developing this framework, we have exploited the commonality between tasks while allowing for both flexibility in configuration across condition-specific testing requirements and extensibility for future test development. Using the two example clinical conditions of visuospatial neglect and dyspraxia, we illustrate the advantages of utilizing a computer-based analysis system, describe a structured approach to system implementation, and demonstrate the generality of this implementation for different conditions of interest, which extends to feature selection and design.     

Administration and Environment Considerations in Computer-Based Sports-Concussion Assessment

Computer-based testing has become a vital tool for the assessment of sport-related concussion (SRC). An increasing number of papers have been published on this topic, focusing on subjects such as the purpose and validity of baseline testing, the performance of special populations on computer-based tests, the psychometric properties of different computerized neurocognitive tools, and considerations for valid and reliable administration of these tools. The current paper describes several considerations regarding computerized test design, input and output devices, and testing environment that should be described explicitly when administering computer-based cognitive testing, regardless of whether the assessment is used for clinical or research purposes. The paper also reviews the conclusions of recent literature (2007–2013) using computer-based testing for the assessment of SRC, with special attention to the methods used in these studies. We also present an appendix checklist for clinicians and researchers that may be helpful in ensuring proper attention to factors that could influence the reliability and validity of computer-based cognitive testing. We believe that explicit attention to these technological factors may lead to the development of standards for the development and implementation of computer-based tests. Such standards have the potential to enhance the accuracy and utility of computer-based tests in SRC.     

Development of a Cyber/Information Technology Knowledge Test for Military Enlisted Technical Training Qualification

An Armed Services Vocational Aptitude Battery (ASVAB) Review Panel, with expertise in personnel selection, job classification, psychometrics, and cognitive psychology developed recommendations for changes to the military enlistment test battery. One recommendation was to develop and evaluate a test of cyber/information and communications technology literacy to supplement current ASVAB content. This article summarizes a multiphased Cyber Test development process: (a) a review of information/computer technology literacy definitions and measures, (b) development and pilot testing of a cyber knowledge measure, (c) validation of test scores against final school grades (FSGs) for selected technical training courses, (d) development of an operational reporting metric and subgroup norms, and (e) examination of construct validity. Results indicate the Cyber Test has predictive validity versus technical training school grades and incremental validity comparable to the ASVAB technical knowledge tests when used with the ASVAB Armed Forces Qualification Test (AFQT) verbal/math composite as a baseline.     

Test selection and suppressor variables

A theoretical discussion of the factor pattern of predictor tests and criterion shows that ordinary test selection methods break down under certain circumstances. It is shown that maximal resultsmay not occur if suppressor variables are present among the predictors. Suggested solutions to the problem include: (1) prior item analysis of tests against the criterion, (2) selection of several trial batteries including some with suppressor variables on the basis of a factor analysis of tests and criterion, (3) modification of the usual test selection procedures to include separate solutions based upon each of several starting variables, or (4) the cumbersome and tedious solution of all possible combinations of predictors. The solutions are recommended in the order named above. Although all of the suggested solutions involve added labor and may not be necessary, the test or battery constructor should at least be aware of the problem.     

Validities of Unit-Weighted Composites of the ASVAB and the ECAT Battery

Currently, assignment to training in each entry-level enlisted job in the U.S. military is influenced by examinees’ scores on the classification composite for that job from the Armed Services Vocational Aptitude Battery (ASVAB). For a cross-service sample of the 18 occupations in the Enhanced Computer-Administered Test (ECAT) project, enhancements of their respective operational ASVAB composites (9 unique ones) are evaluated here. Analyses are reported on the gains in validity from adding 1 test with unit weight to each composite, the addition coming either from the other ASVAB tests or from the 9 tests of the ECAT battery. In more than half the schools, the best ASVAB additions were Coding Speed or Mechanical Comprehension. In the ECAT battery, 1 test of psychomotor ability (Two-Hand Tracking) and 2 tests of working memory (Mental Counters and Sequential Memory) were the most promising additions. The utility of the new composites for person–job matching is discussed.     

Internet Recruitment and Selection: Kissing Frogs to find Princes

The Internet has already had a dramatic impact on the way in which recruitment and selection are carried out in North America, and the impact is increasingly being felt in terms of changes in practice in Europe and Asia-Pacific. The paper presents a picture of the current development of the Internet as a medium in general and as a recruitment and selection medium in particular. The new medium has enabled the widespread adoption of computer-based assessment and it is predicted that it will replace paper as the default medium before very long. A range of issues are raised and discussed. These include security, confidentiality, authentication, control of assessment conditions, control over practice and equality of access. It is argued that as the second generation of users takes over from the first generation, so inequality of skill and access are becoming less and less of an issue. Finally, some potential areas of abuse of the system are noted and a call is made for the development of international standards to protect the rights and interests of test providers, test users and test takers.     

Large sample validations of three general predictors of pilot training success

This article addresses one of the perennial problems of pilot selection research: obtaining an adequate sample size for reliable statistical analysis of predictive validity. Results from three studies involving the same computerized tests of instrument comprehension and psychomotor ability were combined in a meta-analysis to determine whether the validities of these tests generalized across three contexts. These were Royal Air Force and Turkish Air Force fixed-wing pilot training and British Army Air Corps rotary-wing pilot training. In this article, we discuss the adequacy of samples for estimating the validity of the tests, and the persistence of predictive validity to later stages of training as shown by British Army Air Corps data. Reference is also made to data from a fourth independent study of Qantas pilot training.     

Salesperson Selection,Training, and Development: Trends,Implications, and Research Opportunities

Building on the forces of change in selling identified in the introductory paper of this Special Issue of JPSSM, we identify and explicate trends, managerial implications, and research opportunities in two important domains of the selling field: training and development (TD) and selection. Part One focuses on TD of salespeople, and argues why a fresh perspective on salesperson TD is essential. Then, taking a knowledge, skills, and ability (KSA) approach, a framework for future TD content is proposed around three levels of KSAs: task-related KSAs, growth-related KSAs, and meta-KSAs. Part Two, on selection, focuses on several important predictors of salesperson success: cognitive ability tests, personality inventories, and biodata. Then, assessment methods are reviewed. Finally, we summarize with a discussion of emerging trends and research opportunities. The focus of the discussion is on both enhancing salesperson performance and affording scholars new research opportunities.     

Air force officer qualifying test validity for predicting pilot training performance

The AFOQT was validated for the prediction of pilot training criteria. Subjects were 7,563 men and women selected for pilot training on the basis of educational attainment and AFOQT scores. Criterion variables included daily flight training grades, check flight grades in subsonic and transonic aircraft, and overall academic performance in the 53 week pilot training course. Test validities were presented as observed, corrected for multivariate range restriction, and corrected for multivariate range restriction and unreliability. The Aviation Information and Instrument Comprehension tests, measures of job knowledge, were most predictive of daily and check flights in the initial subsonic jet aircraft. This reflects the relative greater importance of prior job knowledge early in training. The Scale Reading test, a measure of perceptual speed, was most predictive for daily and check flights in the advanced transonic training aircraft. The Arithmetic Reasoning test, a good measure of general cognitive ability, was most predictive of aeronautics in ground school. The development of an improved pilot selection composite is suggested by the results of the validity analyses.The views expressed are those of the authors and not necessarily those of the Department of the Air Force, Department of Defense, or the Government of the United States.     

PREDICTING LONG-TERM FIREFIGHTER PERFORMANCE FROM COGNITIVE AND PHYSICAL ABILITY MEASURES

Firefighters from 1 academy training class were observed for 23 years, beginning with their selection test consisting of a g-saturated written exam (GCA) and firefighting simulations loaded on a strength/endurance (SE) factor. Operational validity coefficients for both GCA and SE were high for training success and remained consistently high for job performance ratings throughout the study. The operational validity for combined GCA and SE predictors was .86 for a composite job rating measure covering 21 years of service. A structural model produced similar results for more broadly defined GCA and SE latent variables. Both analyses suggested approximately equal weighting for GCA and SE for a fire service selection test. Results indicate considerable latitude in choosing cognitive and physical predictors for firefighter screening if the predictors are highly loaded on GCA and SE.     

Prediction of situational awareness in F-15 pilots

Situational awareness (SA) is a skill often deemed essential to pilot performance in both combat and noncombat flying. A study was conducted to determine if SA in U.S. Air Force F-15 pilots could be predicted. The participants were 171 active duty F-15 A/C pilots who completed a test battery representative of various psychological constructs proposed or demonstrated to be valid for the prediction of performance in a wide variety of military and civilian jobs. These predictors encompassed measures of cognitive ability, psychomotor ability, and personality. Supervisor and peer ratings of SA were collected. Supervisors and peers showed substantial agreement on the SA ratings of the pilots. The first unrotated principle component extracted from the supervisor and peer ratings accounted for 92.5% of the variability of ratings. The unrotated first principal component served as the SA criterion. Flying experience measured in number of F-15 hours was the best predictor of SA. After controlling for the effects of F-15 flying hours, the measures of general cognitive ability based on working memory, spatial reasoning, and divided attention were found to be predictive of SA. Psychomotor and personality measures were not predictive. With additional F-15 flying hours it is expected that pilots would improve their ratings of SA.     

Selling points: What cognitive abilities are tapped by casual video games?

The idea that video games or computer-based applications can improve cognitive function has led to a proliferation of programs claiming to “train the brain.” However, there is often little scientific basis in the development of commercial training programs, and many research-based programs yield inconsistent or weak results. In this study, we sought to better understand the nature of cognitive abilities tapped by casual video games and thus reflect on their potential as a training tool. A moderately large sample of participants (n = 209) played 20 web-based casual games and performed a battery of cognitive tasks. We used cognitive task analysis and multivariate statistical techniques to characterize the relationships between performance metrics. We validated the cognitive abilities measured in the task battery, examined a task analysis-based categorization of the casual games, and then characterized the relationship between game and task performance. We found that games categorized to tap working memory and reasoning were robustly related to performance on working memory and fluid intelligence tasks, with fluid intelligence best predicting scores on working memory and reasoning games. We discuss these results in the context of overlap in cognitive processes engaged by the cognitive tasks and casual games, and within the context of assessing near and far transfer. While this is not a training study, these findings provide a methodology to assess the validity of using certain games as training and assessment devices for specific cognitive abilities, and shed light on the mixed transfer results in the computer-based training literature. Moreover, the results can inform design of a more theoretically-driven and methodologically-sound cognitive training program.