Measurement Equivalence Between Applicant and Incumbent Groups: An IRT Analysis of Personality Scales

Incumbents are often used in the development and validation of a wide variety of personnel selection instruments, including noncognitive instruments such as personality tests. However, the degree to which assumed motivational factors impact the measurement equivalence and validity of tests developed using incumbents has not been adequately addressed. This study addressed this issue by examining the measurement equivalence of 6 personality scales between a group applying for jobs as sales managers in a large retail organization (N = 999) and a group of sales managers currently employed in that organization (N = 796). A graded item response theory model (Samejima, 1969) was fit to the personality scales in each group. Results indicated that moderately large differences existed in personality scale scores (approximately 1/2 standard deviation units) but only one of the six scales contained any items that evidenced differential item functioning and no scales evidenced differential test functioning. In addition, person-level analyses showed no apparent differences across groups in aberrant responding. The results suggest that personality measures used for selection retain similar psychometric properties to those used in incumbent validation studies.     

PERSONALITY TESTS AT THE CROSSROADS: A RESPONSE TO MORGESON, CAMPION, DIPBOYE, HOLLENBECK, MURPHY, AND SCHMITT (2007)

Despite widespread and growing acceptance that published personality tests are valid predictors of job performance, Morgeson et al. (2007) propose they be abandoned in personnel selection because average validity estimates are low. Our review of the literature shows that Morgeson et al.'s skepticism is unfounded. Meta-analyses have demonstrated that published personality tests, in fact, yield useful validity estimates when validation is based on confirmatory research using job analysis and taking into account the bidirectionality of trait–performance linkages. Further gains are likely by use of narrow over broad measures, multivariate prediction, and theory attuned to the complexities of trait expression and evaluation at work. Morgeson et al. also suggest that faking has little, if any, impact on personality test validity and that it may even contribute positively to job performance. Job applicant research suggests that faking under true hiring conditions attenuates personality test validity but that validity is still sufficiently strong to warrant personality test use in hiring. Contrary to Morgeson et al., we argue that the full value of published personality tests in organizations has yet to be realized, calling for programmatic theory-driven research.     

Correcting Personality Tests for Faking: A Review of Popular Personality Tests and an Initial Survey of Researchers

We reviewed a number of personality tests that are commonly used for personnel selection and found that some of the more popular tests provide a “correction” for faking. Additionally, a survey of researchers' preferences regarding correcting personality test scores for faking was conducted. The respondents, who were experienced in using personality tests for industrial‐organizational purposes, generally favored the use of faking corrections (69% were in favor). The apparently common practice of correcting personality scores for faking was contrasted with relevant conceptual, empirical, and statistical concerns as to the advisability of applying the correction for faking.     

Big Five traits: Predictors of retesting propensity and score improvement

Although most high‐stakes admissions, credentialing, and pre‐employment tests allow candidates to retest, relatively little is known about the personal traits of candidates who persist in retesting upon initial failure. In this study we investigated whether Big Five traits may predispose initially unsuccessful applicants to retest and subsequently improve on high‐stakes cognitive ability and knowledge tests required for personnel selection. In this study personality measures (unlike the cognitive tests) did not affect selection outcomes and hence did not provide applicants incentive to distort their personality responses to gain entry into the organization. Applicants higher in conscientiousness were more likely to retest, and emotional stability positively predicted cognitive test score improvement. We discuss implications of these results for organizations considering retesting policies.     

Hope in Personnel Selection

This study examined the potential role of hope as a personality trait in performance in personnel selection batteries. We hypothesized that hope would be associated with overall success in the selection process, mediated by coping. One hundred twenty job applicants who went through a full day of individual psychological assessment also filled out measures of hope and coping style. Structural equation modeling analysis provided support for a partial mediation model, with problem‐solving‐oriented coping mediating the association between hope and the psychologists’ general recommendation based on various psychological measures (though hope and coping were not included in the psychologists’ considerations). The model remained significant after controlling for general mental ability, personality, interview, and group simulation measures used in the assessment. The potential implications and limitations of the study are discussed in light of existing research on hope and performance.     

Reconsidering vocational interests for personnel selection: the validity of an interest-based selection test in relation to job knowledge, job performance, and continuance intentions

Although vocational interests have a long history in vocational psychology, they have received extremely limited attention within the recent personnel selection literature. We reconsider some widely held beliefs concerning the (low) validity of interests for predicting criteria important to selection researchers, and we review theory and empirical evidence that challenge such beliefs. We then describe the development and validation of an interests-based selection measure. Results of a large validation study (N = 418) reveal that interests predicted a diverse set of criteria—including measures of job knowledge, job performance, and continuance intentions—with corrected, cross-validated Rs that ranged from .25 to .46 across the criteria (mean R = .31). Interests also provided incremental validity beyond measures of general cognitive aptitude and facets of the Big Five personality dimensions in relation to each criterion. Furthermore, with a couple exceptions, the interest scales were associated with small to medium subgroup differences, which in most cases favored women and racial minorities. Taken as a whole, these results appear to call into question the prevailing thought that vocational interests have limited usefulness for selection.     

The Inwald Personality Inventory (IPI) and Hilson Research inventories: Development and rationale

This article provides a “behind-the-scenes” account of how and why the Inwald/ Hilson tests were developed. Since the 1970s, personality testing has been adapted and customized for use in selecting applicants for different occupations, including police, public safety, fire and emergency services personnel. The author developed the Inwald Personality Inventory (IPI) in 1979 as the first comprehensive behaviorally-based personality measure designed and validated specifically for use in high risk occupations, such as law enforcement. Over time, studies consistently demonstrated that the IPI was a better predictor of poor job performance than were traditional tests of psychopathology. While antisocial behavior patterns and characteristics measured by the IPI predicted poor job performance, there also developed a need for measures that would predict above-average/excellent performance in the workplace. In 1988, the author designed and validated the first comprehensive test of “positive” work-related characteristics or “emotional IQ,” the Hilson Personnel Profile/Success Quotient (HPP/SQ). Other instruments followed, including the Inwald Survey 5 — Revised (IS5-R) with questions added on domestic violence. This article describes the theoretical rationale, development and validation of several instruments developed by the author and now widely used for high risk occupations and for screening corporate managers and key personnel in occupational settings.     

Using Blatant Extreme Responding for Detecting Faking in High‐stakes Selection: Construct validity,relationship with general mental ability,and subgroup differences

Although there has been a steady growth in research and use of self‐report measures of personality in the last 20 years, faking in personality testing remains as a major concern. Blatant extreme responding (BER), which includes endorsing desirable extreme responses (i.e., 1 and 5 s), has recently been identified as a potential faking detection technique. In a large‐scale (N = 358,033), high‐stakes selection context, we investigate the construct validity of BER, the extent to which BER relates to general mental ability, and the extent to which BER differs across jobs, gender, and ethnic groups. We find that BER reflects applicant faking by showing that BER relates to a more established measure of faking, an unlikely virtue (UV) scale, and that applicants score higher than incumbents on BER. BER is (slightly) positively related to general mental ability whereas UV is negatively related to it. Applicants for managerial positions score slightly higher on BER than applicants for nonmanagerial positions. In addition, there was no gender or racial differences on BER. The implications of these findings for detecting faking in personnel selection are delineated.     

Emotional Intelligence in Selection Contexts: Measurement method,criterion‐related validity,and vulnerability to response distortion

Two studies examined aspects of the validity of self‐report and performance‐based measures of emotional intelligence (EI) relevant to their use in personnel selection. In Study 1, structural equation modeling indicated that a two‐factor model with separate factors for the two types of EI measures fit better than a one‐factor solution. The performance‐based EI factor was more related to cognitive ability (R=.38) than personality (R=.26), whereas the self‐report EI factor was more related to personality (R=.85) than cognitive ability (R=.09). Although the performance‐based EI factor correlated more strongly with job performance (ρ=.24) than did that of the self‐report (ρ=.05), it provided little incremental validity beyond cognitive ability and conscientiousness. In Study 2, participants were asked to complete the measures as if applying for a job, and mean scores were then compared with those of Study 1. Results indicated that self‐report EI measures were more vulnerable to distortion than were the performance‐based measures. Implications for the assessment of EI in personnel selection contexts are discussed.     

Personality and absenteeism: a meta‐analysis of integrity tests

Until recently, research focus has been on a variety of demographic, attitudinal, and organizational variables in predicting and explaining absenteeism. If personality traits predict absenteeism, then it may be possible to use measures of these traits to identify and select job applicants and thereby reduce absenteeism rates. In this research, our goal was to examine whether integrity tests could be used to predict absenteeism. Meta‐analysis was applied to studies of the validity of pre‐employment integrity tests for predicting voluntary absenteeism. Twenty‐eight studies based on a total sample of 13 972 were meta‐analysed. The estimated mean predictive validity of personality‐based integrity tests was 0.33. This operational validity generalized across various predictor scales, organizations, settings, and jobs (SD_ρ = 0.00). Overt integrity tests, however, showed much lower predictive validity for absenteeism and greater variability than personality‐based tests (ρ = 0.09; SD_ρ = 0.16). The results indicate that a personnel selection approach to reducing absenteeism in organizations may be a useful strategy, particularly if personality‐based integrity tests are utilized. Potential explanations for differences between these results and those found for Big Five measures of personality are offered. Future research investigating models of absenteeism should incorporate the personality constructs assessed by integrity tests. Copyright © 2003 John Wiley & Sons, Ltd.     

Validation of personality and cognitive measures for insurance claims examiners

Claims Examiners are critical for the profitability of large insurance companies. Mistakes in hiring these people can result in improperly processed claims and/or turnover, and this yields negative financial consequences for the company. This paper describes research evaluating the validity of selection procedures for the position of claims examiner at Blue Cross and Blue Shield of Oklahoma. A job analysis indicated that both cognitive skills and personal abilities are necessary for successful performance as a claims examiner. The job analysis also indicated that both subjective and objective criterion measures reflect important aspects of job performance. The criterion data included nine supervisory ratings, employee nominations, average percent of performance, absences, disciplinary actions, and sick leave. In a concurrent validation strategy, we tested a sample of experienced incumbents (N=50) using four cognitive ability tests (Basic Skills Tests: BST) and two measures of personality (Hogan Personality Inventory and PROFILE). Scales from two personality inventories were combined to form the Claims Examiner Inventory (CEI). Scores from the CEI and three of the four cognitive tests correlated significantly with the average percent of performance and six of the supervisory ratings, including the average supervisory rating. In addition to the cognitive measures, the personality measure contributed significantly to the prediction of percent of performance achieved with R=.64. These results are consistent with other research findings that support the use of both personality and cognitive measures as predictors of job performance.     

GROUP DIFFERENCES IN PERSONALITY: META‐ANALYSES COMPARING FIVE U.S. RACIAL GROUPS

This research poses 2 applied questions: How large are racial group differences on personality scales and are these differences likely to cause adverse impact in personnel selection? We examined the extent to which racial groups differ across Big 5 personality factors and facets. Large‐scale, quantitative estimates based on over 700 effect sizes were meta‐analytically summarized. Multiple personality instruments and understudied racial groups, particularly Asian Americans and American Indians, were included in the meta‐analyses. Most group comparisons and personality scales yielded negligible differences and are not likely to cause adverse impact in selection. However, facet‐level analyses produced different d‐values, with some group comparisons showing moderate differences, suggesting that the use of personality measures in selection does not uniformly circumvent adverse impact concerns. The veracity of this conclusion depends on (a) the particular trait in question, (b) the composition of the applicant pool (i.e., which groups are to be compared with one another), (c) the effect size (i.e., d‐value), and (d) the selection ratio. For practitioners, we present a quantitatively informed summary of when and where to expect adverse impact to result from personality measures used in personnel selection.     

Option weights should be determined empirically and not by experts when assessing knowledge with multiple‐choice items

Multiple‐choice tests are frequently used in personnel selection contexts to measure knowledge and abilities. Option weighting is an alternative multiple‐choice scoring procedure that awards partial credit for incomplete knowledge reflected in applicants’ distractor choices. We investigated whether option weights should be based on expert judgment or on empirical data when trying to outperform conventional number‐right scoring in terms of reliability and validity. To obtain generalizable results, we used repeated random sub‐sampling validation and found that empirical option weighting, but not expert option weighting, increased the reliability of a knowledge test. Neither option weighting procedure improved test validity. We recommend to improve the reliability of existing ability and knowledge tests used for personnel selection by computing and publishing empirical option weights.     

IN SUPPORT OF PERSONALITY ASSESSMENT IN ORGANIZATIONAL SETTINGS

Personality constructs have been demonstrated to be useful for explaining and predicting attitudes, behaviors, performance, and outcomes in organizational settings. Many professionally developed measures of personality constructs display useful levels of criterion-related validity for job performance and its facets. In this response to Morgeson et al. (2007) , we comprehensively summarize previously published meta-analyses on (a) the optimal and unit-weighted multiple correlations between the Big Five personality dimensions and behaviors in organizations, including job performance; (b) generalizable bivariate relationships of Conscientiousness and its facets (e.g., achievement orientation, dependability, cautiousness) with job performance constructs; (c) the validity of compound personality measures; and (d) the incremental validity of personality measures over cognitive ability. Hundreds of primary studies and dozens of meta-analyses conducted and published since the mid 1980s indicate strong support for using personality measures in staffing decisions. Moreover, there is little evidence that response distortion among job applicants ruins the psychometric properties, including criterion-related validity, of personality measures. We also provide a brief evaluation of the merits of alternatives that have been offered in place of traditional self-report personality measures for organizational decision making. Given the cumulative data, writing off the whole domain of individual differences in personality or all self-report measures of personality from personnel selection and organizational decision making is counterproductive for the science and practice of I-O psychology.     

NEW DEVELOPMENTS IN THE USE OF MEASURES OF HONESTY INTEGRITY, CONSCIENTIOUSNESS, DEPENDABILITY TRUSTWORTHINESS, AND RELIABILITY FOR PERSONNEL SELECTION

This paper is the fourth in a series of reviews of the use of measures of honesty, integrity, conscientiousness, dependability, trustworthiness, and reliability for personnel selection (see Sackett & Decker, 1979; Sackett & Harris, 1984; Sackett, Burris, & Callahan, 1989). New developments reviewed include an examination of professional and congressional inquiry into this area of testing, rapid growth of the validity data base, new insight into similarities and differences between different tests, and links to the Big Five personality dimensions. Inquiries into relationships with other constructs, including cognitive ability, moral reasoning, and social desirability are reviewed, as are applicant reactions to these tests. The effects of the Civil Rights Act of 1991 and the Americans with Disabilities Act are considered.     

Using the BIDR to Distinguish the Effects of Impression Management and Self‐Deception on the Criterion Validity of Personality Measures: A Meta‐Analysis

Although the use of personality tests for personnel selection has gained increasing acceptance, researchers have raised concerns that job applicants may distort their responses to inflate their scores. In the present meta‐analysis, we examined the effects of the two dimensions of social desirability, impression management and self‐deception, on the criterion validity of personality constructs using the balanced inventory of desirable responding (BIDR). The results indicate that impression management and self‐deception did not create spurious effects on the relationship between personality measures and performance, nor did they function as performance predictors. Moreover, removing the influence of impression management or self‐deception from personality measures did not substantially attenuate the criterion validity of personality variables. Implications of the results and directions for future research are also discussed.     

They're Right,Well ... Mostly Right: Research Evidence and an Agenda to Rescue Personality Testing From 1960s Insights

We comment on the major points made by Murphy and Dzieweczynski (this issue) and Hogan (this issue), and although we agree with much of what they say, we go on to provide solid reasons for hope in the development and use of personality tests in personnel selection settings. We review current theorizing and research in this area, which suggests that the pessimistic view-that personality tests necessarily yield low-to-no criterion-related validity-is unwarranted and untrue. The points of caution offered by Murphy, Dzieweczynski, and Hogan should therefore also be viewed as points of scientific challenge, and to that end we conclude by outlining an agenda of major topics for future research.     

Predicting Adaptive Performance: Further Tests of a Model of Adaptability

This research further investigates an 8-dimension taxonomy of adaptive performance developed by Pulakos, Arad, Donovan, and Plamondon (2000) and the usefulness of past experience, interest, and self-efficacy predictors developed to measure these same 8 dimensions for predicting adaptive job performance. Participants in the concurrent, criterion-related validation study included 739 military personnel. They completed the 3 adaptability measures as well as more traditional cognitive and noncognitive predictors. Supervisors of the study participants rated their adaptive job performance. Results showed support for the 8-dimension model of adaptability. In addition, cognitive ability, personality, and the new adaptability predictors were shown to predict adaptive performance, with some past experience items adding incremental validity beyond the more traditional cognitive ability and personality measures. Results are discussed.     

ARE WE GETTING FOOLED AGAIN? COMING TO TERMS WITH LIMITATIONS IN THE USE OF PERSONALITY TESTS FOR PERSONNEL SELECTION

We recently published an article in which we highlighted a number of issues associated with the use of self-report personality tests in personnel selection contexts ( Morgeson et al., 2007 ). Both Ones, Dilchert, Viswesvaran, and Judge (2007) and Tett and Christiansen (2007) have written responses to this article. In our response to these articles we address many of the issues raised by Ones et al. and Tett and Christiansen. In addition to a detailed response, we make the following 4 key points: (1) Our criticisms of personality testing apply only to the selection context, not to all research on personality; (2) the observed validities of personality tests predicting job performance criteria are low and have not changed much over time; (3) when evaluating the usefulness of using personality tests to select applicants, one must not ignore the observed, uncorrected validity; and (4) when discussing the value of personality tests for selection contexts, the most important criteria are those that reflect job performance. Implications for personality testing research and practice are discussed.