The Predictive Role of Simulations in Assessing Military Performance

The current study assessed the predictive validity of simulations to improve the military selection system. Four navy simulations were developed and their predictive validity was measured. The performance of 1007 Israeli navy soldiers was measured in a longitudinal study, which was carried on for almost two years. Participants’ performance in four simulations (naval-navigation test, raft sailing, rubber boat [zodiac] mounting, and military tent assembly) was measured and used as a behavioral predictor for their performance at the end of their first year of active military service on combat ships. All but the raft sailing simulation predicted participants’ performance.     

Predictive validity of the WISC-R with Mexican-American children

The reliability and predictive validity of the WISC-R was investigated with 46 Mexican-American children. Approximately a year and a half after the administration of the WISC-R, scores on the Iowa Tests of Basic Skills (ITBS) were collected. Internal consistency estimates for WISC-R subtests and composites were found to be comparable to those reported in the WISC-R manual. Correlations between WISC-R subtests and those of the ITBS ranged from modest for verbal subtests to mild when performance subtests were used as predictors. The best single predictor proved to be that of Verbal IQ, which accounted for some 40% of the variance in future achievement when measured by the ITBS. It was concluded that the WISC-R meets minimum requirements of reliability and predictive validity with Mexican-American children.     

Construct-oriented Biodata: Capturing Change-related and Contextually Relevant Future Performance

This study highlights an approach to the development of performance and predictor constructs that has the potential to permit context and culture relevant selection. Task, contextual and adaptive measures of performance were used as criterion data in a selection study involving 325 staff in the hotel industry. Construct-oriented biodata were developed to predict adaptive and contextual performance and the validities were contrasted with measures of cognitive ability and personality. The results provided construct support for the separation of adaptive performance from task and contextual performance and for the predictive validity of change-related biodata after controlling for cognitive ability. The pattern of correlations among the personality measures, biodata scales and cognitive tests supported the construct validity of the biodata scales. Results are discussed in relation to the context in which the data were collected, and the general relevance of the procedure across countries or cultures.     

Text anxiety on cognitive ability test can result in differential predictive validity of academic performance

This paper investigates whether test anxiety leads to differential predictive validity in academic performance. Our results show that the predictive validity of a cognitive ability test, using final exam performance as a criterion, decreased a small amount as Worry (the cognitive aspect of anxiety) increased but was unaffected by Emotionality (the physiological aspect of anxiety). These results suggest that cognitive ability tests may be more useful as predictors of performance for low anxiety test-takers. These findings are discussed in the context of the interference and deficit perspectives of test anxiety.     

The problem of classification of personnel

The personnel classification problem arises in its pure form when all job applicants must be used, being divided among a number of job categories. The use of tests for classification involves problems of two types: (1) problems concerning the design, choice, and weighting of tests into a battery, and (2) problems of establishing the optimum administrative procedure of using test results for assignment. A consideration of the first problem emphasizes the desirability of using simple, factorially pure tests which may be expected to have a wide range of validities for different job categories. In the use of test results for assignment, an initial problem is that of expressing predictions of success in different jobs in comparable score units. These units should take account of predictor validity and of job importance. Procedures are described for handling assignment either in terms of daily quotas or in terms of a stable predicted yield.Address of the President of the Division on Evaluation and Measurement of the American Psychological Association, delivered at Denver, Colorado, September 9, 1949.     

TWO-STAGE SEQUENTIAL SELECTION PROCEDURES USING ABILITY AND TRAINING PERFORMANCE: INCREMENTAL VALIDITY OF BEHAVIORAL CONSISTENCY MEASURES

A behavioral consistency model was used to evaluate the predictive validity of ability and training performance measures as components in a two-stage pre-reject sequential selection procedure. Participants were 91 college students who completed ability tests and training relevant to an air intercept and traffic control operator task. A simulation study was conducted so that two groups could be examined: a high ability group, analogous to a screened hired employee group in an organization; and a quasi-random ability group, analogous to an applicant pool as a validation sample. The incremental validity of training performance was practically and statistically significant in the prediction of component and overall task performance after including ability as a predictor in both groups. Adding ability to the prediction of task performance after training performance had been used as a predictor had no practical effects in either sample. The validity and likely utility of using sequential procedures to select employees as well as implications of behavioral consistency measures for privacy and discrimination in employee testing are discussed. Future research directions using different types of training and employee samples are also described.     

Validity of USES Aptitude Test Batteries for Predicting MDTA Training Success

As a result of the continuing Federal-State cooperative test research program in the Employment Service, aptitude test batteries for a large number of specific occupations have been developed. These test batteries are in daily use by Employment Service counselors of candidates for training programs established under the Manpower Development and Training Act (MDTA) of 1962. To what extent are these test batteries predictive of success of enrollees in MDTA courses? This article summarizes the available evidence on this question, based on data for 875 MDTA trainees in 12 courses. In 10 of the 12 samples the operational test norms showed significant cross validity.     

JOB-RELATEDNESS OF A POSTTRAINING JOB KNOWLEDGE CRITERION USE TO ASSESS VALIDITY AND TEST FAIRNESS

Significant job-relatedness was found for a posttraining job knowledge test criterion using an application of Lawshe's content validity method. The aide test was used as a criterion to assess the predictive validity of a vocabulary test and a civil service test with samples of black ( N = 43) and white ( N = 62) psychiatric aides. Significant validities were found on both tests, but a vocabulary test proved to be the better predictor of the criterion in both samples. The obtained validities were discussed in terms of differential validity, test fairness, and sample size. This study demonstrated that a content validity method could be applied to criteria as well as selection tests. It was concluded that content validity methods may be able to help solve the problem of criterion relevance in validation research by providing quantitative evidence of the job-relatedness of criteria.     

EFFECTS OF ALTERNATIVE WEIGHTING METHODS IN A POLICY- CAPTURING APPROACH TO JOB EVALUATION: A REVIEW AND EMPIRICAL INVESTIGATION

This research was designed to examine differences in the predictive power of alternative scale weighting methods in the context of job evaluation. Two different point-factor job evaluation instruments were used to evaluate 71 managerial and service jobs in a metropolitan university, and five different weighting models were compared in terms of predictive validity and salary classification. For the job evaluation system having high multicollinearity and validity concentration, no significant differences in accuracy were found among the weighting methods. However, in the more heterogeneous system, prediction models based upon unit weights, correlational weights, and multiple regression had significantly higher predictive validity than models based upon equal raw score weights or rational weights developed by a job evaluation committee. In addition, the weighting models differed substantially in terms of the predicted policy wages and classification structures.     

Study of the adjustment of Western expatriates in Taiwan ROC with the Multicultural Personality Questionnaire

The present paper examined the validity of the Multicultural Personality Questionnaire (MPQ). As criteria of validity three levels of adjustment were used. The study took place among a sample of expatriates ( N = 102) during their assignment in Taiwan. The MPQ has scales for cultural empathy, open-mindedness, social initiative, emotional stability and flexibility. The MPQ scales appeared to be predictive of expatriates' personal, professional and social adjustment. In all three domains, emotional stability appeared most consistently as a predictor of adjustment. Social initiative was an additional strong predictor of psychological well-being, as was cultural empathy of satisfaction with life and of the amount of social support in the host country. Flexibility was a predictor of job satisfaction and social support. The study also examined the effects of marital status on adjustment. Married expatriates showed higher levels of adjustment than expatriates who were single or separated.     

Examining the reliability and predictive validity of performance assessments by soccer coaches and scouts: The influence of structured collection and mechanical combination of information

Soccer coaches and scouts typically assess in-game soccer performance to predict players’ future performance. However, there is hardly any research on the reliability and predictive validity of coaches’ and scouts’ performance assessments, or on strategies they can use to optimize their predictions. In the current study, we examined whether robust principles from psychological research on selection – namely structured information collection and mechanical combination of predictor information through a decision-rule – improve soccer coaches’ and scouts’ performance assessments. A total of n = 96 soccer coaches and scouts participated in an elaborate within-subjects experiment. Participants watched soccer players’ performance on video, rated their performance in both a structured and unstructured manner, and combined their ratings in a holistic and mechanical way. We examined the inter-rater reliability of the ratings and assessed the predictive validity by relating the ratings to players’ future market values. Contrary to our expectations, we did not find that ratings based on structured assessment paired with mechanical combination of the ratings showed higher inter-rater reliability and predictive validity. In contrast, unstructured-holistic ratings yielded the highest reliability and predictive validity, although differences were marginal. Overall, reliability was poor and predictive validities small-to-moderate, regardless of the approach used to rate players’ performance. The findings provide insights into the difficulty of predicting future performance in soccer.     

Validity and Adverse Impact Potential of Predictor Composite Formation

Previous research on the validity and adverse impact (AI) of predictor composite formation focused on the merits of regression‐based or ad hoc composites. We argue for a broader focus. Ad hoc chosen composites are usually not Pareto‐optimal, whereas the regression‐based composite represents only one element from the total set of Pareto‐optimal composites and can, therefore, provide only limited information on the potential for validity and AI reduction of forming predictor composites when both validity and AI are of concern. In that case, other Pareto‐optimal composites may provide a better benchmark to decide on the merits of the predictor composite formation. We summarize a method to determine the set of Pareto‐optimal composites and apply the method to a representative collection of selection predictors. The application shows that the assessment of the AI and validity of predictor composite formation can differ substantially from the one arrived at when considering only regression‐based composites.     

AN OBJECTIVE GROUP QUESTIONNAIRE AS A SUBSTITUTE FOR A PERSONAL INTERVIEW IN THE PREDICTION OF SUCCESS IN MILITARY TRAINING IN ISRAEL

The need for reliable and valid measures of personality and motivational factors in the prediction of success in military training was discussed. The personnel classification system currently used by the Israeli Army was briefly described. The personality factors used in that system are measured by an interview, which is individually administered to each enlisted man. The goal of the present study was to replace this interview by an objective group questionnaire, with the hope of saving time, manpower and effort without any loss to predictive validity. The criterion for validation of the system was the performance of the soldiers in elementary training. This performance was assessed by commanding officers and by peers. The results showed that the questionnaire is eventually equivalent to the interview as a predictor of performance in military training. It was concluded that the questionnaire should be preferred for economical reasons.     

Pilot selection batteries: shortcomings and perspectives

This article presents a critical examination of pilot selection batteries. The first part of the article focuses on two problems. First, the vast majority of pilot selection batteries predict training performance rather than operational performance; second, the batteries have low correlations between the predictors and the criterion. The second part of the article examines why these two problems occur. Last, a number of suggestions for improving the predictive validity of the selection batteries are offered.     

Designing Selection Systems for Medicine: The importance of balancing predictive and political validity in high‐stakes selection contexts

Selection into medical education and training is a high‐stakes process. A key unanswered issue is the effectiveness of measuring noncognitive predictors via both low‐fidelity and high‐fidelity selection approaches in this high‐stakes context. We review studies investigating the effectiveness of multiple selection instruments in terms of predictive validity, incremental validity, and applicant reactions in both entry‐level and advanced‐level medical selection. Our results show that the situational judgment test (SJT) is the best single predictor of performance, operationalized in multiple ways. In addition, the low‐fidelity SJT has incremental predictive power over cognitively oriented tests, and high‐fidelity assessment center (AC) exercises add incremental validity over the low‐fidelity (and less costly) selection methods. Concerning applicant reactions, results show that overall, the selection system is positively received. However, the method with the highest predictive validity – the SJT – received comparatively lower face validity ratings which may present a ‘justice dilemma’ for employers. Furthermore, various other stakeholders have a political interest in the selection methods used (e.g., government, the regulators and trade unions).     

Forming Composites of Cognitive Ability and Alternative Measures to Predict Job Performance and Reduce Adverse Impact: Corrected Estimates and Realistic Expectations

Although there has been empirical attention paid to the criterion‐related validity of predictor composites, there has been much less attention paid to the standardized ethnic group differences associated with these composites. One important area of inquiry in predictor composite research is the influence of adding predictors to a test of general mental ability. The limited empirical literature on this practice is mixed, but the prevailing expectation is that there is likely to be higher validity and less adverse impact. Unfortunately, much of the previous work is limited by the presence of inaccurate validity and standardized ethnic group difference values. In this analysis we formed meta‐analytic matrices to more accurately estimate the validity and standardized ethnic group differences of several composites that combine a measure of cognitive ability with measures of conscientiousness, a structured interview, or biodata. While results were somewhat complex, we found that adding alternative predictors does not result in a situation in which validity automatically goes up and adverse impact potential automatically goes down. In fact, the reductions in adverse impact (if any) from adding “non‐cognitive” predictors were more modest than much of the literature suggests.     

THE WARTEGG DRAWING COMPLETION TEST AS A PREDICTOR OFADJUSTMENT AND SUCCESS IN INDUSTRIAL SCHOOL

Mellberg K. The Wartegg Drawing Completion Test as a predictor of adjustment and success in industrial school. Scand. J. Psychol ., 1972, 13 , 34–38.—The WDCT, a projective personality test, was administered to 284 elementary-school boys when they were seeking entry to the industrial school. Two years later, their school performance and teacher-rated adjustment were recorded. The validity of the prediction from the WDCT relative to these criteria was tested on the basis of analytical and intuitive interpretations of the WDCT. The analytical interpretations showed only slight predictive validity. Even intuitive interpretations had a low validity. In the intuitive interpretations trained psychologists used only 4 of 53 analytical WDCT-variables.