IMPLICATIONS OF THE MULTIDIMENSIONAL NATURE OF JOB PERFORMANCE FOR THE VALIDITY OF SELECTION TESTS: MULTIVARIATE FRAMEWORKS FOR STUDYING TEST VALIDITY

Although most studies of criterion-related validity focus on univariate relationships, the complex and multidimensional nature of the performance construct and the widespread use of multiple selection devices argue in favor of multivariate frameworks for evaluating validity. Using a Monte Carlo simulation we estimated the validity of general cognitive ability tests and personality tests in predicting "job performance," where performance is conceptualized as a composite of multiple performance measures (i.e., individual job task performance and organizational citizenship behaviors). The validity of a selection battery varies substantially as a function of the relative weight given to both predictors and criteria; the 95% confidence interval for validities ranged from .20 to .78. The effective weights given to performance dimensions accounted for 34% of the variance in selection battery validities; depending on precisely how "performance" is defined, the same test battery can have relatively high or relatively low levels of validity. Our model suggests that the way an organization defines job performance is a source of true and important variability in validities, and that the validity of selection tests for predicting complex performance criteria may show considerably less generalizability that current metaanalysis of univariate validities would suggest.     

A Matter of Context: A Meta‐Analytic Investigation of the Relative Validity of Contextualized and Noncontextualized Personality Measures

The empirical evidence that has accumulated in support of the notion that personality is a valid predictor of employee performance is vast, yet debate on the matter continues. This study investigates frame‐of‐reference effects as they relate to the validity of self‐report measures of personality. Specifically, we compare the validities of general, noncontextualized personality measures and work‐specific, contextualized measures. The findings suggest that personality measures are a more valid predictor of performance when the scale items or instructions are framed specifically so as to reference work‐specific behaviors. We found that the validities for noncontextualized measures of personality ranged from .02 to .22, with a mean validity of .11. The validities for contextualized measures ranged from .14 to .30, with a mean of .24 . Additional moderator analyses were conducted in an effort to examine several alternate explanations for these validity differences. Specifically, we examined differences between the developmental purpose (general use vs. workplace use) and reliabilities of each type of personality measure. We also compared the validities from published studies to those from unpublished studies. Results suggest that these moderators did not have an impact on the validity differences between noncontextualized and contextualized measures.     

INCREASED ACCURACY FOR RANGE RESTRICTION CORRECTIONS: IMPLICATIONS FOR THE ROLE OF PERSONALITY AND GENERAL MENTAL ABILITY IN JOB AND TRAINING PERFORMANCE

A recently developed procedure produces substantial improvements in the accuracy of corrections for range restriction and reveals that predictive validities of employment selection methods are higher than previously thought. This study applied this procedure to meta‐analytic validities of 2 personality measures (Conscientiousness and Emotional Stability) and general cognitive ability measures. Results show that the increases in validity estimates are greater for cognitive ability than for personality, reducing the relative validity of personality. In addition, the incremental validity of these personality measures over that provided by cognitive ability is smaller than previously estimated. Path analyses based on the more accurate data reveal relatively smaller path coefficients from these personality traits to job and training performance. These findings have implications for theories of job performance and for the practical value in selection of personality measures relative to cognitive ability measures.     

Validation through self and other ratings on dimensions of six nonstressful multi-scaled personality instruments

Six nonstressful personality instruments were concurrently validated using respondent and close friend ratings on specific scale dimensions. Tests were administered to 203 predominantly Caucasian college students. Prior to testing, the individual and a close friend through self-ratings and ratings of the respondents had estimated the strengths of the tested personality Variables on a seven-point scale. The definitions of the various personality dimensions were taken from publisher's manuals. An estimated whole test validity was obtained by an average of the individual scale validities using a conversion of Pearson's r to z'. Significant differences among and between subtest validities were found by ANOVA. All "self-rating" validities were significant (.001), with two of these self-rating validities significantly different (.01) than the other four tests. Only two "other-rating" validities were significant. All six tests appeared to be valid for college population use From preliminary analysis, counselees could rate themselves accurately on most test measures. In some cases self-ratings might be used in lieu of giving the test.     

PERSONALITY MEASURES AS PREDICTORS OF JOB PERFORMANCE: A META-ANALYTIC REVIEW

The purpose of this study was to investigate conflicting findings in previous research on personality and job performance. Meta-analysis was used to (a) assess the overall validity of personality measures as predictors of job performance, (b) investigate the moderating effects of several study characteristics on personality scale validity, and (c) appraise the predictability of job performance as a function of eight distinct categories of personality content, including the "Big Five" personality factors. Based on review of 494 studies, usable results were identified for 97 independent samples (total N = 13,521). Consistent with predictions, studies using confirmatory research strategies produced a corrected mean personality scale validity (.29) that was more than twice as high as that based on studies adopting exploratory strategies (.12). An even higher mean validity (.38) was obtained based on studies using job analysis explicitly in the selection of personality measures. Validities were also found to be higher in longer tenured samples and in published articles versus dissertations. Corrected mean validities for the "Big Five" factors ranged from .16 for Extroversion to .33 for Agreeableness. Weaknesses in the reporting of validation study characteristics are noted, and recommendations for future research in this area are provided. Contrary to conclusions of certain past reviews, the present findings provide some grounds for optimism concerning the use of personality measures in employee selection.     

APPLICATION OF CONTENT VALIDITY METHODS TO THE DEVELOPMENT OF A JOB-RELATED PERFORMANCE RATING CRITERION

This study demonstrated the use of quantitative content validity procedures in the development of a job-related behavioral rating scale criterion for entry-level psychiatric aides. Work behavior items were developed by staff from 6 state psychiatric hospitals, placed in a content validity questionnaire using the Lawshe format, and given to a representative sample of 38 aides and supervisors. Seventy-eight of 83 items were found to be significantly job-relevant using the computation procedures of both Lawshe and Aiken. After the significant items were grouped into 4 categories with high interjudge agreement and placed in a rating scale format, ratings were obtained on 72 psychiatric aides from 4 hospitals. Items in the 4 categories were found to be internally consistent using coefficient alpha. Significant but low concurrent validities were established for 2 verbal ability selection tests using the rating criterion. The validities found were interpreted to be especially significant when the factors of low selection ratio, restriction in range, and limited rater training were considered.     

CONSTRUCT VALIDATION OF THE COMREY PERSONALITY SCALES

To investigate the construct validity of the Comrey Personality Scales, this personality test and a biographical data inventory inquiring about past and present life circumstances were administered to 209 volunteer U.C.L.A. students. Biographical data variables were correlated with scores on the eight personality scales. Personality scale score estimates by the authors, based on examination of the respondents' biographical data responses, were also correlated with actual scale scores. The personality scales which received the best confirmation were Social Conformity vs Rebelliousness and Orderliness vs. Lack of Compulsion. Support for the construct validity of the other personality scales varied from moderate to negligible.     

Be Careful What You Ask For: Effects of Response Instructions on the Construct Validity and Reliability of Situational Judgment Tests

The aim of this study was to examine how six different types of situational judgment test (SJT) instructions, used frequently in practice, influence the psychometric characteristics of SJTs. The six SJT versions used the exact same items and differed only in their instructions; these versions were administered in two phases. Phase I was a between–subjects design (n= 486) that had participants complete one version of the SJTs. Phase II was a within–subjects design (n= 231) held several weeks later that had participants complete all six versions of the SJTs. Further, 146 of these individuals completed both phases, allowing for an assessment of test–retest reliability. A variety of objective and subjective criteria were collected, including self and peer ratings. Results indicated that instructions had a large effect on SJT responses, reliability, and validity. In general, instructions asking what one ‘would do’ showed more favorable characteristics than those that asked what one ‘should do’. Correlations between these two types were relatively low despite the fact that the same items were used, and criterion–related validities differed substantially in favor of the ‘would do’ instructions. Overall, this study finds that researchers and practitioners must give careful consideration to the types of SJT instructions used; failing to do so could influence criterion–related validity and cloud inferences of construct validity.     

DEVELOPMENT OF A GLOBAL MEASURE OF PERSONALITY

A common practice among practitioners and researchers doing cross-cultural work is to transport personality inventories developed in one country to another country of interest. This transport process is done with varying degrees of rigor and success. This paper describes the development of the Global Personality Inventory (GPI). The development process involved cross-cultural input from psychologists around the world following best practices that have been identified to date for each step of the development of a cross-cultural measure of personality. Construct validity studies including data from 11 countries and 10 languages are reported. Initial criterion-related validity evidence for inferences made from the GPI about job performance is presented. Examples of challenges and best practices are illustrated in the areas of construct development, item writing, item selection, translation, and data analysis.     

The Big Five personality factors as a model for the structure of children's peer nominations

The validity and usefulness of the five‐factor theory of personality as a model for children's peer nominations is assessed. Eighty‐nine groups of ten school children, attending the same class, aged 9 to 12 years, nominated the pupil that was the most or the least typical example for 25 bipolar trait pairs, covering the ‘Big Five’ personality factors. Factor analysis of summed and transformed peer nominations for the total sample shows that children's nominations can be represented by three factors: Agreeableness, a combined Extraversion–Emotional Instability component, and a combined Intellect–Conscientiousness factor. A second study with a smaller but independent sample confirms the factorial structure of the peer nomination scales. The reasons for the less differentiated peer nomination structure of children and the relevance of multidimensional assessment of peer perception are discussed. Copyright © 2000 John Wiley & Sons, Ltd.     

INDIVIDUAL DIFFERENCES IN INTERVIEWER RATINGS: THE IMPACT OF STANDARDIZATION, CONSENSUS DISCUSSION, AND SAMPLING ERROR ON THE VALIDITY OF A STRUCTURED INTERVIEW

This research investigated empirical issues regarding the validity of individual interviewers'( N = 62) ratings collected after a structured interview. Each interviewer rated an average of 25 interviewees. One hypothesis examined but not supported was that systematic interviewer errors will attenuate interview validity when data are aggregated across interviewers. Also investigated was the validity of ratings averaged across interviewers compared to consensus ratings; consensus ratings were shown to have significantly but probably not practically higher validities. Third, a meta-analysis of individual interviewer validities revealed that all of the variance in validities could be attributed to sampling error. Results and implications are discussed.     

VALIDITY OF SELF-ASSESSMENT: A FIELD EVALUATION

This study evaluated the validity of self-assessment in a natural setting. It also addressed the possibility of improving the utility of self-assessment by cautioning subjects that their responses could be compared with other existing data on them. Subjects were 357 males, aged 18–19 years, who were being screened for a prestigious military course. During the screening, they participated in small groups for nine days of tests, class exercises, and field simulations supervised by veteran commanders. On the ninth day, ratings were collected from commanders, peer group members, and the subjects themselves on dimensions related to eventual success in the training course. The experimental group was told their reports would be compared with those from other sources; the control group was not. Predictive and convergent validities were examined on three criteria: course success, commander ratings, and peer evaluations. Self-assessments from both experimental and control group were valid; however, those of the experimental group did not yield consistently higher validities. Findings are discussed in regard to their practical ramifications.     

Reference Group Effects in the Measurement of Personality and Attitudes

Reference-group effects (discovered in cross-cultural settings) occur when responses to self-report items are based not on respondents’ absolute level of a construct but rather on their level relative to a salient comparison group. In this article, we examine the impact of reference-group effects on the assessment of self-reported personality and attitudes. Two studies illustrate that a reference-group effect can be induced by small changes to instruction sets, changes that mirror the instruction sets of commonly used measures of personality. Scales that specified different reference groups showed substantial reductions in criterion-related validities for academic performance, self-reported counterproductive behaviors, and self-reported health outcomes relative to reference-group-free versions of those scales.     

METAANALYSES OF VALIDITY STUDIES PUBLISHED BETWEEN 1964 AND 1982 AND THE INVESTIGATION OF STUDY CHARACTERISTICS

Review and metaanalyses of published validation studies for the years 1964-1982 of Journal of Applied Psychology and Personnel Psychology were undertaken to examine the effect of (1) research design; (2) criterion used; (3) type of selection instrument used; (4) occupational group studies; and (5) predictor-criterion combination on the level of observed validity coefficients. Results indicate that concurrent validation designs produce validity coefficients roughly equivalent to those obtained in predictive validation designs and that both of these designs produce higher validity coefficients than does a predictive design which includes use of the selection instrument. Of the criteria examined, performance rating criteria generally produced lower validity coefficients than did the use of other more "objective" criteria. In comparing the validities of various types of predictors, it was found cognitive ability tests were not superior to other predictors such as assessment centers, work samples, and supervisory/peer evaluations as has been found in previous metaanalytic work. Personality measures were clearly less valid. Compared to previous validity generalization work, much unexplained variance in validity coefficients remained after corrections for differences in sample size. Finally, the studies reviewed were deficient for our purposes with respect to the data reported. Selection ratios, standard deviations, reliabilities, predictor and criterion intercorrelations were rarely and inconsistently reported. There are also many predictor-criterion relationships for which very few validation efforts have been undertaken.     

The Moderating Effect of Raters’ Opportunities to Observe Ratees' Job Performance on the Validity of an Assessment Centre

Previous research has suggested that the quality of judgmental validation criteria may improve depending on the increase of opportunity for the rater to observe ratees’ job performances. This study investigated whether the validity of an assessment centre increases with opportunity to observe. Subjects were engineers and applied scientists from the Research and Development (R&D) units of seven German high-technology companies. A total of 6 personality scales, 10 cognitive ability tests and 9 work samples was administered to 155 subjects. Performance appraisals were collected from their supervisors. Assessors were trained organizational psychologists who were not members of the participating organizations. A total score (‘assessment centre score’) was computed by averaging the 25 predictors after standardization of each predictor. The (uncorrected) validity of this assessment centre was r = 0.37. In addition, opportunity to observe (i.e. amount of time the performance rater was the ratee’s supervisor) emerged as a moderator: for example validities were r = 0.09 (opportunity to observe ≤ 2 years) and r = 0.50 (opportunity to observe> 2 years).     

ENTRY‐LEVEL POLICE CANDIDATE ASSESSMENT CENTER: AN EFFICIENT TOOL OR A HAMMER TO KILL A FLY?

The study examined the validity of the assessment center (AC) as a selection process for entry-level candidates to die police and its unique value beyond cognitive ability tests. The sample included 712 participants who responded to personality and cognitive ability testing (CAT), and underwent an AC procedure. AC results included the overall assessment rating (OAR) and peer evaluation (PE). Seven criterion measures were collected for 585 participants from a training stage and on-the-job performance. Results showed that the selection system was valid. Findings yielded significant unique validities of OAR and PE beyond CAT and of PE beyond OAR even after corrections for restriction of range. Results support the use of ACs for entry-level candidates.     

Generalization of Cognitive and Noncognitive Validities across Personality‐based Job Families

The positive relationship between complexity of work and the validity of general mental ability (GMA) measures across a variety of occupations is well supported by research and provides important practical and theoretical support for cognitive ability measures. However, there is currently no research demonstrating a systematic relationship between the size of the validities of any personality measure and the personality requirements of jobs, thus leaving open to question the predictive and construct validity of personality measures for applicant selection. We compared the validities of two biodata measures – one scored to measure social competence and one to measure GMA – across six job families that varied in social requirements. The validity of the GMA measure generalized at approximately the same level across the job families while the validity of the social competence measure decreased as social requirements diminished.     

A META-ANALYSIS OF THE CRITERION-RELATED VALIDITY OF ASSESSMENT CENTER DIMENSIONS

We used meta-analytic procedures to investigate the criterion-related validity of assessment center dimension ratings. By focusing on dimension-level information, we were able to assess the extent to which specific constructs account for the criterion-related validity of assessment centers. From a total of 34 articles that reported dimension-level validities, we collapsed 168 assessment center dimension labels into an overriding set of 6 dimensions: (a) consideration/awareness of others, (b) communication, (c) drive, (d) influencing others, (e) organizing and planning, and (f) problem solving. Based on this set of 6 dimensions, we extracted 258 independent data points. Results showed a range of estimated true criterion-related validities from .25 to .39. A regression-based composite consisting of 4 out of the 6 dimensions accounted for the criterion-related validity of assessment center ratings and explained more variance in performance (20%) than Gaugler, Rosenthal, Thornton, and Bentson (1987) were able to explain using the overall assessment center rating (14%).     

Comparing the Eysenck and HEXACO Models of Personality in the Prediction of Adult Delinquency

Drawing from self and peer reports of personality, the present study compares the structures of the HEXACO and Eysenck models of personality and the models' capacity to predict self reported acts of delinquency. Correlations amongst scales revealed that Psychoticism captures elements of both HEXACO Emotionality and Conscientiousness. The Eysenck Lie scale correlated positively with both self and peer reported HEXACO Honesty–Humility and Conscientiousness, suggesting that this validity scale includes substantive variance relating to the latter factors. Regression analyses of personality data from both rater sources revealed that Honesty–Humility and Psychoticism were strong predictors of delinquency that independently offered substantial incremental validity. For self reports, the Extraversion and Lie scales were also strong unique predictors of delinquency. Copyright © 2011 John Wiley & Sons, Ltd.