Challenges in Transforming Military Training: Research and Application of Advanced Simulation and Training Technologies and Methods

This study addresses the need to validate estimates of cross-job retraining times against measures of actual retraining success. Specialty-level cross-job retraining time estimates were computed for 43 U.S. Air Force (USAF) enlisted specialties. The estimates were based on profiles of mean subject- matter-expert ratings of months to proficiency across 26 task categories. Individual-level criteria indicating previous job skill utilization, current skills and abilities, and current job performance were obtained from a separate study conducted 10 years earlier. Results were supportive: Correlations between retraining time estimates and performance criteria were negative and statistically significant. Future research needs and potential applications of retraining time estimates are discussed.     

Development and convergent validation of a methodology for estimating cross-job retraining times

Cross-jobtransferability of skills is defined in terms of the ease with which individuals can apply knowledge and skills acquired in a previous job in learning to perform a new job. This study reports the development of a prototype methodology for estimating retraining times based on analyses of transferability of skills. Using this methodology which assesses interjob similarity in task content and task learning time, times to retrain across 41 different U.S. Air Force enlisted jobs were estimated. Convergent validities of retraining time estimates were tested in terms of correlations with differences in jobs': (a) general job learning difficulty, and (b) aptitude requirements. Results supported predictions that cross-job retraining time would be longer (a)into jobs that generally aremore difficult to learn, (b)from jobs that generally areless difficult to learn, and (c) across jobs having different, rather than similar, aptitude requirements. Implications for training, human resources planning, and coping with increasingly rapid technological changes in the workplace are discussed.Research sponsored by the Air Force Office of Scientific Research/AFSC, United States Air Force, under contract F49620-87-R-0004. The United States Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notation hereon.     

The criterion-related validity of integrity tests: an updated meta-analysis

Integrity tests have become a prominent predictor within the selection literature over the past few decades. However, some researchers have expressed concerns about the criterion-related validity evidence for such tests because of a perceived lack of methodological rigor within this literature, as well as a heavy reliance on unpublished data from test publishers. In response to these concerns, we meta-analyzed 104 studies (representing 134 independent samples), which were authored by a similar proportion of test publishers and non-publishers, whose conduct was consistent with professional standards for test validation, and whose results were relevant to the validity of integrity-specific scales for predicting individual work behavior. Overall mean observed validity estimates and validity estimates corrected for unreliability in the criterion (respectively) were .12 and .15 for job performance, .13 and .16 for training performance, .26 and .32 for counterproductive work behavior, and .07 and .09 for turnover. Although data on restriction of range were sparse, illustrative corrections for indirect range restriction did increase validities slightly (e.g., from .15 to .18 for job performance). Several variables appeared to moderate relations between integrity tests and the criteria. For example, corrected validities for job performance criteria were larger when based on studies authored by integrity test publishers (.27) than when based on studies from non-publishers (.12). In addition, corrected validities for counterproductive work behavior criteria were larger when based on self-reports (.42) than when based on other-reports (.11) or employee records (.15).     

A WORKER MOTIVATION SCALE

A 70 item questionnaire was constructed using the rationale of several current value/expectancy theories of motivation and the results of two prior factor analyses of motivation items. It was completed anonymously by 240 employed adults in 35 organizations and the resulting factor analysis yielded a hierarchical structure of motivation and eight reliable subscales. Direct and indirect criterion measures also obtained via the questionnaire and covering job characteristics/components; satisfaction (need) levels for n ach, n affil, n power ; the four elements of Heider's attribution theory of motivation; and salary, education and job classification were used to validate the subscales. Support for the criteria was found and promising validities were indicated. Lastly, personal, job and organizational variables were included with the motivation subscales and criteria in a subsequent factor analysis to examine moderator effects of the obtained validities. Results are discussed in terms of motivation theories, validities, moderator effects in study designs, and future use of the questionnaire by other researchers with business-industrial samples.     

INCREASED ACCURACY FOR RANGE RESTRICTION CORRECTIONS: IMPLICATIONS FOR THE ROLE OF PERSONALITY AND GENERAL MENTAL ABILITY IN JOB AND TRAINING PERFORMANCE

A recently developed procedure produces substantial improvements in the accuracy of corrections for range restriction and reveals that predictive validities of employment selection methods are higher than previously thought. This study applied this procedure to meta‐analytic validities of 2 personality measures (Conscientiousness and Emotional Stability) and general cognitive ability measures. Results show that the increases in validity estimates are greater for cognitive ability than for personality, reducing the relative validity of personality. In addition, the incremental validity of these personality measures over that provided by cognitive ability is smaller than previously estimated. Path analyses based on the more accurate data reveal relatively smaller path coefficients from these personality traits to job and training performance. These findings have implications for theories of job performance and for the practical value in selection of personality measures relative to cognitive ability measures.     

EXPERT AND NAIVE RATERS USING THE PAQ: DOES IT MATTER?

A finding by Smith and Hakel (1979) is that job expert ratings on the PAQ correlate highly with ratings obtained from college students who are given no information about jobs other than their titles. One possible explanation for this finding is that the PAQ measures only trivial or common knowledge about work that both experts and naive observers possess. This particular view, of course, has serious implications for the use of the PAQ. In this paper we point out a problem in the way Smith and Hakel calculated the convergent validity between expert raters and naive raters. We also present the results from a replication study that indicate convergent validities are much lower than those reported by Smith and Hakel. Additional points are presented in order to caution against the interpretation that the PAQ measures only common knowledge about jobs.     

The Effects of Social Desirability and Faking on Personality and Integrity Assessment for Personnel Selection

A review of the extant literature and new empirical research suggests that social desirability is not much of a concern in personality and integrity testing for personnel selection. In particular, based on meta-analytically derived evidence, it appears that social desirability influences do not destroy the convergent and discriminant validity of the Big Five dimensions of personality (Emotional Stability, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness). We also present new empirical evidence regarding gender and age differences in socially desirable re- sponding. Although social desirability predicts a number of important work variables such as job satisfaction, organizational commitment, and supervisor ratings of training success, social desirability does not seem to be a predictor of overall job performance and is only very weakly related to specific dimensions of job performance such as technical proficiency (r = -.07) and personal discipline ( r = .05). Large sample investigations of the moderating influences of social desirability in actual work settings indicate that social desirability does not moderate the criterion-related validities of personality variables or integrity tests. The criterion-related validity of integrity tests for overall job performance with applicant samples in predictive studies is .41. Controlling for social desirability in integrity or personality test scores leaves the operational validities intact, thereby suggesting that social desirability functions neither as a mediator nor as a suppressor variable in personality-performance.     

USING A META-ANALYTIC PERSPECTIVE TO ENHANCE JOB COMPONENT VALIDATION

This paper develops synthetic validity estimates based on a meta-analytic-weighted least squares (WLS) approach to job component validity (JCV), using position analysis questionnaire (PAQ) estimates of job characteristics, and the Data, People, & Things ratings from the  Dictionary of Occupational Titles  as indices of job complexity. For the general aptitude test battery database of 40,487 employees, nine validity coefficients were estimated for 192 positions. The predicted validities from the WLS approach had lower estimated variability than would be obtained from either the classic JCV approach or local criterion-related validity studies. Data, People, & Things summary ratings did not consistently moderate validity coefficients, whereas the PAQ data did moderate validity coefficients. In sum, these results suggest that synthetic validity procedures should incorporate a WLS regression approach. Moreover, researchers should consider a comprehensive set of job characteristics when considering job complexity rather than a single aggregated index.     

META-ANALYSIS OF PERSONALITY-JOB PERFORMANCE RELATIONS: A REPLY TO ONES, MOUNT, BARRICK, AND HUNTER (1994)

Tett, Jackson, and Rothstein's (1991) meta-analysis identified higher average correlations for personality in predicting job proficiency than did Barrick and Mount's (1991). Ones et al. suggest that discrepancies between the two studies involving the Big Five personality dimensions are due to certain procedural differences. In this reply, we show that their arguments do not adequately explain the noted discrepancies. We also show that, because personality traits correlate significantly with job performance both positively and negatively beyond chance levels, use of absolute values, contrary to Ones et al., is important in meta-analyses involving personality. Addressing all of Ones et al.3 statistical concerns, re-analysis of Tett et al.k main data set results in slightly lower mean validities (e.g., .24 vs.29 for fully corrected values based on confirmatory estimates), and renders non-significant the job analysis/no job analysis distinction found to be significant in the original study. Tett et al.'s main conclusions, however, remain unchanged. We suggest that Barrick and Mount's lower mean validities may be due to their averaging signed correlations, pooling exploratory and confirmatory findings, and to the use of different inclusion criteria for selecting source studies.     

"TO SEE OURSELVES AS OTHERS SEE US!": RELATIONS AMONG SELF-PERCEPTIONS,PEER-PERCEPTIONS,AND EXPECTED PEER-PERCEPTIONS OF PERSONALITY

The need for convergent validity in the measurement of personality variables and past failures to demonstrate such properties are briefly reviewed. Two apppoaches to the problem using several types of data obtained from various subject populations and employing alternative factor analytic methods and factor score estimation procedulres are described. The results indicate that convergent factor patterns and substantial convergence and discriminant validities among factor score estimates are obtainable when sufficient care is taken in the design and development of measure variables .of each type and when analysis methods appropriate for the problem are used.     

Estimating Coefficients of Equivalence and Stability for Job Performance Ratings: The importance of controlling for transient error on criterion measurement

Previous research on measurement error in job performance ratings estimated reliability using coefficients: alpha, test–retest, and interrater correlation. None of these three coefficients control for the four main sources of error in performance ratings. For this reason, coefficient of equivalence and stability (CES) has been suggested as the ideal estimate of reliability. This article presents the estimates of CES for a time interval of 1, 2, and 3 years. The values obtained for a single rater were .51, .48, and .44, respectively. For two raters, the values were .59, .55, and .51. The findings suggest that previous reliability estimates based on alpha, test–retest, and interrater coefficients overestimated the reliability of job performance ratings. In the present study, the interrater coefficient overestimates reliability by 13.6–25.4% for an interval time of 1–3 years, as it does not control for transient error. Results also showed that the importance of transient error increases as the length of the interval between the measures increases. Based on the results, it is suggested that corrected validities based on interrater reliability underestimate the magnitude of the validity. The implications of these findings for future efforts to estimate criterion reliability and predictor validity are discussed.     

Psychometric properties of a Chinese translation of the political skill inventory

Ferris and colleagues defined political skill in organizations as "the ability to effectively understand others at work and to use such knowledge to influence others to act in ways that enhance one's personal and/or organizational objectives." In this study, the psychometric properties of a Chinese translation of the Political Skill Inventory were investigated, supporting construct, convergent, discriminant, and criterion validities. The results suggested that the Chinese translation retained a four-factor structure. Political skill was positively correlated with self-monitoring, conscientiousness, political savvy, emotional intelligence, extraversion, agreeableness, and proactive personality, and was negatively correlated with trait anxiety and external locus of control. After controlling for age, sex, and job tenure, political skill was predictive of task performance, work contribution, and interpersonal help.     

An openness scale for the California Psychological Inventory

We developed a 36-item scale to measure Openness, using items on the California Psychological Inventory (CPI; Gough, 1957, 1987, 1996), Form 434. Items were initially chosen on the basis of content validity. Five samples (N = 2,375) were used to establish reliability, validity, and norms; 4 samples consisted of university undergraduate students, and 1 comprised applicants for nonmanagement call centerjobs. Internal consistency estimates obtained in each sample averaged approximately .75, and test-retest stability, assessed in 1 sample, was estimated at .84. Cross-correlations with related scales, for example, the NEO Personality Inventory-Revised Openness scale (Costa & McCrae, 1992) and other CPI-based scales, provided evidence of construct validity. Statistically significant predictive validities were obtained in 2 call centerjob-incumbent samples, with range-corrected true validities of .20 to .36 for a number of job performance criteria. Construct and predictive validity were found to be higher than for other scales consisting of CPI items designed to measure Openness or a related construct. Finally, norms were prepared for university undergraduate students (n = 1,847) and nonmanagement service-sector job applicants (n = 528).     

PREDICTION OF TRADES APPRENTICES'PERFORMANCE ON JOB SAMPLE CRITERIA

A criterion-related validation was conducted to assess the validity of four aptitude tests and five tests of content taken directly from job tasks in predicting job sample performance of apprentices in eight skilled trades. Observed validities were above .40 (corrected for range restriction, validities averaged .52). Though there were large subgroup mean differences on both predictor and criterion measures, there was no evidence of significant differential prediction.     

The predictive validity of cognitive ability tests: A UK meta‐analysis

A meta‐analysis on the validity of tests of general mental ability (GMA) and specific cognitive abilities for predicting job performance and training success in the UK was conducted. An extensive literature search resulted in a database of 283 independent samples with job performance as the criterion (N=13,262), and 223 with training success as the criterion (N=75,311). Primary studies were also coded by occupational group, resulting in seven main groups (clerical, engineer, professional, driver, operator, manager, and sales), and by type of specific ability test (verbal, numerical, perceptual, and spatial). Results indicate that GMA and specific ability tests are valid predictors of both job performance and training success, with operational validities in the magnitude of .5–.6. Minor differences between these UK findings and previous US meta‐analyses are reported. As expected, operational validities were moderated by occupational group, with occupational families possessing greater job complexity demonstrating higher operational validities between cognitive tests and job performance and training success. Implications for the practical use of tests of GMA and specific cognitive abilities in the context of UK selection practices are discussed in conclusion.     

A trilevel approach to motivators for retraining

A three-level taxonomy of the reasons for vocational retraining is proposed. The compliance level is characterized by the necessity of immediate acquisition of a profession. At the identification level retraining is motivated by the need for a satisfying self-definition through the profession. With the internalization level the profession is chosen to suit one's capabilities and self-fulfillment. Participants in retraining programs filled out the following questionnaires: locus of control, vocational maturity attitudes, level of motivation for retraining, evaluation of the retraining program, and new job satisfaction. Internalization level motivators positively correlated with internal locus of control and vocational maturity attitudes. No such correlations were found for identification or compliance level. Subjects who retrained because of high compliance motivation were less satisfied by the programs, more inclined to drop out, and less apt to seek employment in jobs for which they had retrained than those high on the other levels of motivation. No difference in job satisfaction was determined for those who did work in the new profession.     

Are you interested? A meta-analysis of relations between vocational interests and employee performance and turnover

A common belief among researchers is that vocational interests have limited value for personnel selection. However, no comprehensive quantitative summaries of interests validity research have been conducted to substantiate claims for or against the use of interests. To help address this gap, we conducted a meta-analysis of relations between interests and employee performance and turnover using data from 74 studies and 141 independent samples. Overall validity estimates (corrected for measurement error in the criterion but not for range restriction) for single interest scales were .14 for job performance, .26 for training performance, -.19 for turnover intentions, and -.15 for actual turnover. Several factors appeared to moderate interest-criterion relations. For example, validity estimates were larger when interests were theoretically relevant to the work performed in the target job. The type of interest scale also moderated validity, such that corrected validities were larger for scales designed to assess interests relevant to a particular job or vocation (e.g., .23 for job performance) than for scales designed to assess a single, job-relevant realistic, investigative, artistic, social, enterprising, or conventional (i.e., RIASEC) interest (.10) or a basic interest (.11). Finally, validity estimates were largest when studies used multiple interests for prediction, either by using a single job or vocation focused scale (which tend to tap multiple interests) or by using a regression-weighted composite of several RIASEC or basic interest scales. Overall, the results suggest that vocational interests may hold more promise for predicting employee performance and turnover than researchers may have thought.     

Do Impression Management Scales in Personality Inventories Predict Managerial Job Performance Ratings?

Theorists have argued that impression management scales should predict job performance in jobs where interpersonal interactions are important. This research examined the correlation between impression management scale scores and overall job performance as well as its component dimensions for managerial jobs (one type of job requiring considerable interpersonal interaction). First, a meta‐analysis was conducted to determine the relationship between impression management scale scores and overall managerial performance. Results indicated that the correlation was .04 (k = 17, N = 20,069) for managers. Two additional studies were conducted to investigate the relationship between impression management and specific facets or dimensions of managerial performance. In one of these two studies, the relationships between impression management scores and supervisory ratings on 24 performance facets were investigated for 826 managers. Criterion‐related validities ranged from ‐.06 to .07. In the second study, correlations of an impression management scale with ratings on 22 dimensions of performance were examined using a sample of 257 executives. The validities ranged from .01 to .11. There appears to be little evidence that impression management scales predict job performance in at least one job category where interpersonal interactions are important (management). Potential relevance of these results for understanding social desirability in personality measurement for personnel selection is discussed.     

The problem of classification of personnel

The personnel classification problem arises in its pure form when all job applicants must be used, being divided among a number of job categories. The use of tests for classification involves problems of two types: (1) problems concerning the design, choice, and weighting of tests into a battery, and (2) problems of establishing the optimum administrative procedure of using test results for assignment. A consideration of the first problem emphasizes the desirability of using simple, factorially pure tests which may be expected to have a wide range of validities for different job categories. In the use of test results for assignment, an initial problem is that of expressing predictions of success in different jobs in comparable score units. These units should take account of predictor validity and of job importance. Procedures are described for handling assignment either in terms of daily quotas or in terms of a stable predicted yield.Address of the President of the Division on Evaluation and Measurement of the American Psychological Association, delivered at Denver, Colorado, September 9, 1949.