IMPLICATIONS OF THE MULTIDIMENSIONAL NATURE OF JOB PERFORMANCE FOR THE VALIDITY OF SELECTION TESTS: MULTIVARIATE FRAMEWORKS FOR STUDYING TEST VALIDITY

Although most studies of criterion-related validity focus on univariate relationships, the complex and multidimensional nature of the performance construct and the widespread use of multiple selection devices argue in favor of multivariate frameworks for evaluating validity. Using a Monte Carlo simulation we estimated the validity of general cognitive ability tests and personality tests in predicting "job performance," where performance is conceptualized as a composite of multiple performance measures (i.e., individual job task performance and organizational citizenship behaviors). The validity of a selection battery varies substantially as a function of the relative weight given to both predictors and criteria; the 95% confidence interval for validities ranged from .20 to .78. The effective weights given to performance dimensions accounted for 34% of the variance in selection battery validities; depending on precisely how "performance" is defined, the same test battery can have relatively high or relatively low levels of validity. Our model suggests that the way an organization defines job performance is a source of true and important variability in validities, and that the validity of selection tests for predicting complex performance criteria may show considerably less generalizability that current metaanalysis of univariate validities would suggest.     

THE VALIDITY OF A MEASURE OF JOB COMPATIBILITY IN THE PREDICTION OF JOB PERFORMANCE AND TURNOVER OF MOTION PICTURE THEATER PERSONNEL

The predictive validity of a measure of job compatibility was studied for theater personnel. Scores on a forced-choice instrument, developed from the Job Compatibility Questionnaire (JCQ), predicted employee performance (r = .22, p< .05), turnover (r = -.35, p< .01), and scores on a "value composite" (reflecting a combination of job performance and employee retention criteria) as defined by the research sponsor (P = .41, p< .01). Furthermore, job compatibility scores explained a statistically significant increment in turnover and value composite score variance when analyzed concomitantly with verbal and numerical ability test scores. Finally, job compatibility scores were shown to be nonredundant with hiring decisions based on an application review, reference check, and interview, whereas the cognitive ability test scores shared considerable redundancy with hiring decisions based on the current selection system.     

PREDICTING JOB PERFORMANCE OF YOUTH TRAINEES UNDER A JOB TRAINING PARTNERSHIP ACT PROGRAM (JTPA): CRITERION VALIDATION OF A BEHAVIOR-BASED MEASURE OF WORK MATURITY

Current concepts of the construct of work maturity have focused on a cognitive developmental perspective. This study demonstrates the criterion-related validity of a behavior-based measure of work maturity. The work-maturity levels of 142 youth enrollees in a job training program sponsored by the Job Training Partnership Act (JTPA) were measured using a locally relevant, behavior-based instrument. Through factor and linear regression analyses, the construct of work maturity was defined and related to later job performance. Two work-maturity factors (Time Consciousness and Sociability) were found to predict a weighted composite criterion composed of ten job performance dimensions. In addition, the ability of individual dimensions to predict job performance was also examined. Future studies of work maturity that combine cognitive and behavioral measures are suggested.     

DYNAMIC CRITERIA REVISITED: A LONGITUDINAL STUDY OF PERFORMANCE STABILITY AND PREDICTIVE VALIDITY

The concept of dynamic criteria has been the subject of a recent debate regarding both the definition and prevalence of the phenomenon (Austin, Humphreys, & Hulin, 1989; Barrett & Alexander, 1989; Barrett, Caldwell, & Alexander, 1985). The present paper questions the adequacy of the conceptual framework underlying the debate and provides data supporting a refined concept of dynamic criteria. The incidence and possible causes of change in relative performance were investigated using weekly performance data from 509 sewing machine operators. Analyses were conducted to determine the degree of performance consistency, potential moderators of consistency, and the stability of predictor-criteria relationships using multiple predictors and criteria. Results revealed a steady decline in performance stability coefficients as the interval between measures increased. This decay was evident regardless of employees' prior job experience, cognitive ability, or psychomotor ability. Analyses of predictive validity coefficients revealed temporal changes in validity for both objective and subjective criteria, but not in the expected direction. The validity of cognitive ability increased, the validity of psychomotor ability was stable, and that of prior job experience decreased over time. Implications for theory and research are discussed.     

SELF-REPORTED ASSERTIVE JOB-SEEKING BEHAVIORS OF MINIMALLY EDUCATED JOB HUNTERS

Little research attention has focused on the job-search behavior of minimally educated workers. The primary objective of the studies reported was to examine the relationship of self-reported assertive job-hunting behavior to acquisition of employment among minimally educated workers. The results of three studies provide: (a) construct validity evidence for the Assertive Job-Hunting Survey (AJHS; Becker, 1980), (b) evidence for construct similarity across minimally educated job seeker and college student populations, (c) evidence of a significant relationship between the AJHS and both subjective and objective job acquisition criteria in postdictive and predictive research designs, and (d) evidence that the AJHS, a measure of a non-cognitive variable, can make a unique contribution to the prediction of job acquisition beyond cognitive ability measures.     

Identification and meta-analytic assessment of psychological constructs measured in employment interviews

There has been a growing interest in understanding what constructs are assessed in the employment interview and the properties of those assessments. To address these issues, the authors developed a comprehensive taxonomy of 7 types of constructs that the interview could assess. Analysis of 338 ratings from 47 actual interview studies indicated that basic personality and applied social skills were the most frequently rated constructs in this taxonomy, followed by mental capability and job knowledge and skills. Further analysis suggested that high- and low-structure interviews tend to focus on different constructs. Taking both frequency and validity results into consideration, the findings suggest that at least part of the reason why structured interviews tend to have higher validity is because they focus more on constructs that have a stronger relationship with job performance. Limitations and directions for future research are discussed.     

A Framework for the Study of Relationships Between Organizational Characteristics and Organizational Innovation

This paper presents a taxonomy of organizational characteristics that was developed as part of a large scale job analysis project conducted for the Department of Labor (DOL). Based on research and theory related to organizations, a hierarchical taxonomy of organizational characteristics was developed with six construct domains at the highest level: organizational structure, leadership, human resources (HR) systems and practices, goals, and organizational values. The taxonomy has been empirically tested using data from over 300 organizations, and a summary of these results is presented. This taxonomy is then used to organize and review research on the relationships between organizational variables and innovation. Implications of the taxonomy for understanding relationships between organizational size, industry type, “high-performance” practices, and innovation are discussed.     

EXPLORING BLACK-WHITE SUBGROUP DIFFERENCES OF MANAGERIAL COMPETENCIES

This study investigates whether different job-relevant competencies vary in terms of Black-White subgroup differences exhibited. There were 633 participants (545 Whites, 88 Blacks) who completed a managerial assessment center that evaluated 13 competency dimensions across 8 assessment exercises. Participants also completed a cognitive ability test. The results suggest that subgroup differences vary by the content domain of the competency. As predicted, significant subgroup differences emerged for a majority of the more cognitively loaded competencies (e.g., judgment) while nonsignificant differences were associated with a majority of the less cognitively loaded competencies (e.g., human relations). Furthermore, when cognitive ability was controlled, 12 of 13 competency scores demonstrated incremental validity in predicting supervisory job performance ratings. In addition, competencies with greater cognitive load tended to more strongly predict cognitive aspects of job performance as compared to noncognitive aspects. However, competencies with less cognitive load did not differentially predict cognitive and noncognitive aspects of job performance.     

PERSON-SITUATION SELECTION: A REVIEW OF SOME ABILITY-SITUATION INTERACTION RESEARCH

A review of literatures thought to be relevant for understanding situational contributions to ability-performance relationships was conducted. The review suggested that only three sets of variables, incentive systems, job characteristics, and leadership style/management philosophy ("climates") have recieved research attention from an ability/situation vantage point. It was also shown that in both experimental and survey research these three factors can moderate ability-performance relationships. Because these same three sets of variables are also known to have main effects on average performance levels, an hypothesis is presented that when organizations establish work conditions which facilitate the display of task-relevant individual ability, then validity for ability measures, average performance levels, and general levels of satisfaction will all be high.     

Predicting Adaptive Performance: Further Tests of a Model of Adaptability

This research further investigates an 8-dimension taxonomy of adaptive performance developed by Pulakos, Arad, Donovan, and Plamondon (2000) and the usefulness of past experience, interest, and self-efficacy predictors developed to measure these same 8 dimensions for predicting adaptive job performance. Participants in the concurrent, criterion-related validation study included 739 military personnel. They completed the 3 adaptability measures as well as more traditional cognitive and noncognitive predictors. Supervisors of the study participants rated their adaptive job performance. Results showed support for the 8-dimension model of adaptability. In addition, cognitive ability, personality, and the new adaptability predictors were shown to predict adaptive performance, with some past experience items adding incremental validity beyond the more traditional cognitive ability and personality measures. Results are discussed.     

A JOB SAMPLING APPROACH TO MERIT SYSTEM EXAMINING1

The application of content validity to a merit examining process or rating schedule requires an extension of the concept beyond that of work samples and tests of knowledges or skills to measures of ability and personal characteristics. A method for accomplishing this without violating the principles of content validity is presented. This technique, called the job sampling approach, is a task-based, structured system of eliciting the information necessary to construct the rating schedule from sources most able to provide that information and for using the information to construct the rating schedule and linking it to job performance. The steps include: definition of the performance domain of the job in terms of process statements; identification of the selection and measurement objectives of the organization; development of the measurement domain in relation to the performance domain and to the selection and measurement objectives; and demonstration that a close match between the performance domain and the measurement domain was in fact achieved.     

THE INCREMENTAL VALIDITY OF INTERVIEW SCORES OVER AND ABOVE COGNITIVE ABILITY AND CONSCIENTIOUSNESS SCORES

Recent research has suggested that scores on measures of cognitive ability, measures of Conscientiousness, and interview scores are positively correlated with job performance. There remains, however, a question of incremental validity: To what extent do interviews predict above and beyond cognitive ability and Conscientiousness? This question was addressed in this paper by (a) conducting meta‐analyses of the relationships among cognitive ability, Conscientiousness, and interviews, (b) combining these results with predictive validity results from previous meta‐analyses to form a “meta‐correlation matrix” representing the relationships among cognitive ability, Conscientiousness, interviews, and job performance, and (c) performing 9 hierarchical regressions to examine the incremental validity of 3 levels of structured interviews in best, actual, and worst case scenarios for prediction. Results suggested that interview scores contribute to the prediction of job performance over and above cognitive ability and Conscientiousness to the extent that they are structured, with scores from highly structured interviews contributing substantially to prediction. Directions for future research are discussed.     

责任心子维度与绩效的关系研究中存在的争议

在责任心及其子维度与工作绩效的关系研究中,研究者在下面四个方面还存在争议：1．责任心的结构定义,也就是责任心总维度包含哪些子维度？2．为什么子维度的信度系数比责任心总维度的低？3．对于具体的,较细的工作绩效效标,大维度还是小维度有更好的预测效度？4．用于检验哪一种预测源更有效的方法,计算简单相关系数还是增量效度？文章主要就这些问题进行讨论。     

ARE WE GETTING FOOLED AGAIN? COMING TO TERMS WITH LIMITATIONS IN THE USE OF PERSONALITY TESTS FOR PERSONNEL SELECTION

We recently published an article in which we highlighted a number of issues associated with the use of self-report personality tests in personnel selection contexts ( Morgeson et al., 2007 ). Both Ones, Dilchert, Viswesvaran, and Judge (2007) and Tett and Christiansen (2007) have written responses to this article. In our response to these articles we address many of the issues raised by Ones et al. and Tett and Christiansen. In addition to a detailed response, we make the following 4 key points: (1) Our criticisms of personality testing apply only to the selection context, not to all research on personality; (2) the observed validities of personality tests predicting job performance criteria are low and have not changed much over time; (3) when evaluating the usefulness of using personality tests to select applicants, one must not ignore the observed, uncorrected validity; and (4) when discussing the value of personality tests for selection contexts, the most important criteria are those that reflect job performance. Implications for personality testing research and practice are discussed.     

Time Is Money: Polychronicity as a Predictor of Performance Across Job Levels

This series of studies examined the construct and criterion-related validity of polychronicity (i.e., preferences and attitudes associated with multitasking) in managerial and nonmanagerial jobs. A person–situation approach was taken to understand the individual differences and job demand characteristics that help explain the relationship between polychronicity and job performance. Hypotheses were tested with two samples of employees in different organizations (N?=?222 and 168, respectively). Results provide support for polychronicity as a predictor of job performance and a correlate of other individual differences measures (e.g., personality, job-relevant skills). In addition, polychronicity accounted for variance in job performance beyond personality, biodata, and ability. Results are discussed in the context of future research and applications for identifying contexts in which polychronicity is a salient predictor.     

Social Networking Websites,Personality Ratings,and the Organizational Context: More Than Meets the Eye?1

We examined the psychometric properties of the Big Five personality traits assessed through social networking profiles in 2 studies consisting of 274 and 244 social networking website (SNW) users. First, SNW ratings demonstrated sufficient interrater reliability and internal consistency. Second, ratings via SNWs demonstrated convergent validity with self‐ratings of the Big Five traits. Third, SNW ratings correlated with job performance, hirability, and academic performance criteria; and the magnitude of these correlations was generally larger than for self‐ratings. Finally, SNW ratings accounted for significant variance in the criterion measures beyond self‐ratings of personality and cognitive ability. We suggest that SNWs may provide useful information for potential use in organizational research and practice, taking into consideration various legal and ethical issues.     

EXERCISES AND DIMENSIONS ARE THE CURRENCY OF ASSESSMENT CENTERS

A novel assessment center (AC) structure that models broad dimension factors, exercise factors, and a general performance factor is proposed and supported in 4 independent samples of AC ratings. Consistent with prior research, the variance attributable to dimension and exercise factors varied widely across ACs. To investigate the construct validity of these empirically supported components of AC ratings, the nomological network of broad dimensions, exercises, and general performance was examined. Results supported the criterion‐related validity of broad dimensions and exercises as predictors of effectiveness and success criteria as well as the incremental validity of broad dimensions beyond exercises and general performance. Finally, the relationships between individual differences and AC factors supported the construct validity of broad dimension factors and provide initial insight as to the meaning of exercise specific variance and general AC performance.     

INCREASED ACCURACY FOR RANGE RESTRICTION CORRECTIONS: IMPLICATIONS FOR THE ROLE OF PERSONALITY AND GENERAL MENTAL ABILITY IN JOB AND TRAINING PERFORMANCE

A recently developed procedure produces substantial improvements in the accuracy of corrections for range restriction and reveals that predictive validities of employment selection methods are higher than previously thought. This study applied this procedure to meta‐analytic validities of 2 personality measures (Conscientiousness and Emotional Stability) and general cognitive ability measures. Results show that the increases in validity estimates are greater for cognitive ability than for personality, reducing the relative validity of personality. In addition, the incremental validity of these personality measures over that provided by cognitive ability is smaller than previously estimated. Path analyses based on the more accurate data reveal relatively smaller path coefficients from these personality traits to job and training performance. These findings have implications for theories of job performance and for the practical value in selection of personality measures relative to cognitive ability measures.     

INTERNATIONAL VALIDITY GENERALIZATION OF GMA AND COGNITIVE ABILITIES: A EUROPEAN COMMUNITY META-ANALYSIS

This article reports on a series of meta-analyses into the criterion validity of general menatl ability (GMA) and specific cognitive ability tests for predicting job performance ratings and training success in the European Community (EC). Meta-analyses were computed on a large EC database examining the operational validity of GMA and other specific cognitive abilities, including verbal, numerical, spatial-mechanical, perceptual and memory ( N ranged from 946 to 16,065) across 10 EC member countries. The results showed that tests of GMA and specific cognitive ability are very good predictors of job performance and training success across the EC. Evidence for the international validity generalization of GMA and specific cognitive abilities was presented. The results for the EC meta-analyses showed a larger operational validity than previous meta-analyses in the U.S. for predicting job performance. For training success, the European and American results are very similar. Implications for the international generalizability of GMA test validities, practical use of cognitive ability tests for personnel selection, and directions for future research are discussed.     

Developing Measures of Basic Job-Relevant English Proficiency for the Prediction of Job Performance and Promotability

The present study is an attempt to devise reliable and valid measures of job-relevant basic English proficiency to be applied in the prediction of job performance and promotability in an entry-level manual labor job dominated by workers whose native language is not English. An audio test and a reading test were developed to assess the ability to understand spoken English and written English respectively using job relevant material. Results showed that both tests demonstrated high predictive validity for the criteria of supervisory assessments of incumbents' English proficiency on the job, overall job performance, and promotability. In predicting ratings of overall job performance and promotability, each test provided incremental validity over the predictability provided by the other test. In addition, the tests also provided incremental validity in predicting promotability over the predictability provided by overall job performance.