FURTHER WITHIN-SETTING EMPIRICAL TESTS OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION

The situational specificity hypothesis in personnel selection holds that variation in observed validity coefficients across studies for the same test and job is due to subtle variations from setting to setting in what constitutes job performance. This hypothesis therefore predicts that, if the setting does not vary, validity will not vary. Using data from a single large-sample validity study ( N = 1,455), this research generated numerous small-sample studies for which the setting (organization, job, test, criterion measure, applicant pool, time period, and sample size) was held constant. It was found that even under these circumstances there was substantial variability across studies in (a) observed validity coefficients, (b) significance levels, and (c) (using traditional data analytic methods) conclusions about the presence or absence of validity. These findings disconfirm the situational specificity hypothesis and argue strongly against traditional data-analytic procedures and the practice of reliance on single small-sample studies. In contrast to the erroneous conclusions produced by traditional data-analytic procedures, meta-analytic methods correctly estimated the population observed validity at .22 and correctly indicated that all between-study variance in observed validities was due to sampling error alone.     

PERSONALITY MEASURES AS PREDICTORS OF JOB PERFORMANCE: A META-ANALYTIC REVIEW

The purpose of this study was to investigate conflicting findings in previous research on personality and job performance. Meta-analysis was used to (a) assess the overall validity of personality measures as predictors of job performance, (b) investigate the moderating effects of several study characteristics on personality scale validity, and (c) appraise the predictability of job performance as a function of eight distinct categories of personality content, including the "Big Five" personality factors. Based on review of 494 studies, usable results were identified for 97 independent samples (total N = 13,521). Consistent with predictions, studies using confirmatory research strategies produced a corrected mean personality scale validity (.29) that was more than twice as high as that based on studies adopting exploratory strategies (.12). An even higher mean validity (.38) was obtained based on studies using job analysis explicitly in the selection of personality measures. Validities were also found to be higher in longer tenured samples and in published articles versus dissertations. Corrected mean validities for the "Big Five" factors ranged from .16 for Extroversion to .33 for Agreeableness. Weaknesses in the reporting of validation study characteristics are noted, and recommendations for future research in this area are provided. Contrary to conclusions of certain past reviews, the present findings provide some grounds for optimism concerning the use of personality measures in employee selection.     

The Role of Person Characteristics in Perceptions of the Validity of Cognitive Ability Testing

Individuals vary in how they perceive cognitive ability tests; thus, it is useful for organizations to consider how individual differences influence applicant perceptions of selection tools. The present study examined the influence of implicit theories of ability and locus of control on perceptions of face validity and predictive validity for two cognitive ability tests. Relationships between perceptions and test experience, job‐relevant experience, and job familiarity were also examined. Interactions between implicit theories and self‐assessed performance in predicting perceptions were found, although not of the form hypothesized. Furthermore, job familiarity and prior success in selection contexts were related to perceptions. Finally, sample type interacted with test type to influence perceptions. Implications for selection system design and research on applicant perceptions are discussed.     

PERSONALITY TESTS AT THE CROSSROADS: A RESPONSE TO MORGESON, CAMPION, DIPBOYE, HOLLENBECK, MURPHY, AND SCHMITT (2007)

Despite widespread and growing acceptance that published personality tests are valid predictors of job performance, Morgeson et al. (2007) propose they be abandoned in personnel selection because average validity estimates are low. Our review of the literature shows that Morgeson et al.'s skepticism is unfounded. Meta-analyses have demonstrated that published personality tests, in fact, yield useful validity estimates when validation is based on confirmatory research using job analysis and taking into account the bidirectionality of trait–performance linkages. Further gains are likely by use of narrow over broad measures, multivariate prediction, and theory attuned to the complexities of trait expression and evaluation at work. Morgeson et al. also suggest that faking has little, if any, impact on personality test validity and that it may even contribute positively to job performance. Job applicant research suggests that faking under true hiring conditions attenuates personality test validity but that validity is still sufficiently strong to warrant personality test use in hiring. Contrary to Morgeson et al., we argue that the full value of published personality tests in organizations has yet to be realized, calling for programmatic theory-driven research.     

PREDICTING TEST PERFORMANCE: A CONTENT VALID APPROACH TO SCREENING APPLICANTS

This study examined the validity of a rationally developed biographical questionnaire for predicting content valid test performance for electrician applicants ( N = 221). Job analysis task statements were utilized as biographical items, and applicants were required to rate their previous work experience regarding each task. The utility of the questionnaire in screening applicant populations was shown to be both statistically and practically significant. EEOC implications are discussed.     

The Validity of the Employment Interview: A Meta-Analysis

The interview is the most widely used personnel selection method, but has revealed low reliability and validity compared with other selection methods (Mayfield 1964; Ulrich and Trumbo 1965; Schmitt 1976; Arvey 1979). Thirty-one studies on the validity of the interview were meta-analyzed. The result was an average validity coefficient of .27. The estimated true validity of the interview was calculated to be .38. These values indicated that the interview has moderate validity as a personnel selection device. Six characteristics of the interview were also examined in relation to the validity of the interview: structure of the interview; number of interviewers; length of the interview; gender of the applicant pool; blue-collar/white-collar jobs; and use of college students versus job applicants. The six study characteristics accounted for 30.9% of the variance in the validity of the interview. Structure of the interview appeared to be the only characteristic that moderated the validity of the interview. The relationship of this study to other meta-analyses of the employment interview is discussed.     

Effects of Individual Differences on the Perceived Job Relatedness of a Cognitive Ability Test and a Multimedia Situational Judgment Test

Although there is a growing number of publications concerning applicant reactions to different selection instruments, the relationships between individual differences and applicant reactions have largely remained unexplored. The aim of the present study was to examine the effects of several testing‐related and general individual differences (anxiety, self‐evaluations, and personality) on the most commonly studied dimension of applicant reactions, namely the perceived job relatedness of selection instruments. Participants were 153 psychology students, who completed a cognitive ability test and a multimedia situational judgment test as part of their educational program. Our results indicated that computer anxiety negatively affected perceived job relatedness and core self‐evaluations, subjective well‐being, agreeableness, emotional stability, and openness to experience positively affected perceived job relatedness. Openness to experience was the most consistent predictor of perceived job relatedness. The results of our study suggest that certain individuals may be more predisposed to react positively to selection instruments. Therefore, we concluded that the nature of the applicant pool should be carefully considered when designing interventions to improve applicant reactions.     

APPLICANT AGE AND SELECTION INTERVIEW DECISIONS: EFFECT OF INFORMATION EXPOSURE ON AGE DISCRIMINATION IN PERSONNEL SELECTION

Two research questions pertaining to the applicant age bias effect were addressed in this study: the effect of age-related information exposure on selection decisions and the issue of between-sample generalizability with reference to the age bias effect. Manager and student selectors made selection decisions after viewing one videotape of a simulated selection interview of either an old or a young job applicant. Before viewing the interview tape, half of each subject group was given an item of age-related information and the other half was given neutral information. The results showed that, under the neutral information condition, managers preferred hiring the young applicant for the low-status job, and students favored the old candidate for the high-status position. Under the age-related information condition, managers shifted to favoring the old candidate for the low-status job, and students preferred the young applicant for both the low- and high-status positions. Age discrimination in selection decisions and the lack of external validity of student results in selection research are then discussed.     

Offsetting Performance Losses Due to Cheating in Unproctored Internet‐based Testing by Increasing the Applicant Pool

A concern about unproctored Internet‐based testing (UIT) without follow‐up verification is that because applicants are able to cheat when completing tests, test validity decreases to an unknown extent. However, this view ignores a potential advantage to UIT: Should UIT increase applicant pool size, an organization with a fixed goal in terms of the number of candidates to advance to the next stage of the hiring process can increase the cutoff score and thus be more selective. In this study, a simulation was conducted to determine if this advantage outweighs the cheating disadvantage in the prediction of job performance. When the applicant pool is increased substantially through UIT, job performance outcomes are often higher even when many applicants increase their test scores through cheating.     

TWO-STAGE SEQUENTIAL SELECTION PROCEDURES USING ABILITY AND TRAINING PERFORMANCE: INCREMENTAL VALIDITY OF BEHAVIORAL CONSISTENCY MEASURES

A behavioral consistency model was used to evaluate the predictive validity of ability and training performance measures as components in a two-stage pre-reject sequential selection procedure. Participants were 91 college students who completed ability tests and training relevant to an air intercept and traffic control operator task. A simulation study was conducted so that two groups could be examined: a high ability group, analogous to a screened hired employee group in an organization; and a quasi-random ability group, analogous to an applicant pool as a validation sample. The incremental validity of training performance was practically and statistically significant in the prediction of component and overall task performance after including ability as a predictor in both groups. Adding ability to the prediction of task performance after training performance had been used as a predictor had no practical effects in either sample. The validity and likely utility of using sequential procedures to select employees as well as implications of behavioral consistency measures for privacy and discrimination in employee testing are discussed. Future research directions using different types of training and employee samples are also described.     

EFFECTS OF JOB ATTRIBUTES AND RECRUITING PRACTICES ON APPLICANT DECISIONS: A COMPARISON1

Previous research on the factors which affect applicant decisions concerning jobs has focused on the effects of either job attributes or recruiting practices. The present study examined the simultaneous impact of job attributes and recruiting practices on the likelihood of job acceptance by actual job applicants. Path analysis revealed that job attributes but not recruiting practices influenced the likelihood of job acceptance by graduating college students (N= 200). Given that college students might be expected to be the job applicants most swayed by recruiting practices because of their lack of work experience, the results suggest that the emphasis placed on recruiting practices as determinants of applicant decisions may be overstated in the literature.     

The Validity of Verifiable and Non‐verifiable Biodata Items: An Examination Across Applicants and Incumbents

This study examines the influence of item verifiability (non‐verifiable vs. verifiable), context (applicant vs. incumbent), and keying procedure on biodata mean test scores and validity. Concurrent and predictive validation studies were conducted using a sample of 425 call center incumbents and a sample of 410 call center applicants. Although applicants did not obtain significantly higher mean biodata scores, results provide support for the hypothesis that the non‐verifiable biodata composite would be less valid in the applicant context, while the verifiable biodata composite would be equally valid across both the applicant and incumbent contexts. The same pattern of results was obtained using both item‐ and option‐keying procedures. Implications for research and practice are discussed.     

CORRESPONDENCE BETWEEN ESTIMATES OF CONTENT AND CRITERION-RELATED VALIDITY VALUES

The correspondence between inferences made using two validation strategies–content and criterion-related–were examined in a specific personnel selection application. Empirical validity values and Law-she's (1975) content validity ratios (CVR) were obtained for items from three structured interview guides used in the selection of insurance agents. Ratings of each item by over 300 field managers were used to calculate the CVR values. Statistically significant, yet modest correlations were found between empirical item validities and content validities for an interview guide used to select applicants with prior insurance sales experience. No significant differences were found among these correlations by comparing job experts of different levels of managerial experience and experience in selection. Data for the interview guide used to select experienced applicants also indicated that a content validity approach can be useful in developing a selection instrument with an empirically valid composite rating. The hypotheses were not confirmed for interview guides used to select applicants with no prior insurance sales experience. The practical importance of these results are discussed, as are plans for future research.     

Implementation intentions as a predictor of applicant withdrawal

Despite withdrawal from the job application being a serious concern for organizations, limited research exists that examines this phenomenon. Utilizing the broad intention-behavior framework as suggested in the Theory of Planned Behavior, we examined whether the type of intentions formed when applying for a job (i.e., goal vs. implementation intentions) predicts later withdrawal from the job application. Using a field experimental strategy, half of the applicants in a large applicant pool (N = 5,346) were randomly made to form implementation intentions. It was found that forming implementation intentions was indeed effective in reducing applicant withdrawal, albeit with a small effect size. Theoretical and practical implications of this finding are discussed.     

Readministration of the MMPI-2 following defensive invalidation in a military job applicant sample

Job applicants whose response style on the Minnesota Multiphasic Personality Inventory-2 (MMPI-2; Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer) is excessively defensive create a dilemma for psychologists working in personnel selection settings. Clinical guidelines recommend the profile be considered uninterpretable, but the absence of usable testing data could lead to the elimination of otherwise qualified candidates. Readministering the MMPI-2 with altered instructions to reduce defensive responding has been suggested as an alternative. This option has lacked empirical support until recently. One study (Butcher, Morfitt, Rouse, & Holden, 1997) evaluated the effects of altered instructions on retest validity in a civilian job applicant sample, with the majority of participants obtaining valid and normal profiles on retest. The purpose of this study was to determine if these results would be replicated in a military job applicant sample. Participants were 97 military personnel who completed the MMPI-2 as part of application for selection to instructor duty. Forty-seven participants had obtained invalid profiles and retook the MMPI-2 after receiving instructions intended to reduce defensiveness. The results showed that 83% of retested participants obtained valid profiles on the second MMPI-2. The second test results were very similar in profile to those obtained from a comparison group of 50 participants whose initial MMPI-2 results were valid. The findings are discussed in terms of study limitations and future research directions.     

Impression management and overclaiming on job applications: Related to future sales performance?

Despite decades of research on the role of socially desirable responding (SDR) in hiring contexts, it remains unclear whether job applicant SDR is related to future employee performance. Drawing on a large sample of 1,084 retail sales employees and their original job application data, this study examined the predictive validity of two operationalizations of SDR: (a) impression management captured using a personality scale and (b) overclaiming familiarity with a list of products that were directly relevant to the job. Neither measure was used to inform the hiring decisions. Results showed that job applicant impression management and overclaiming were unrelated to four indicators of future sales performance as employees. Implications for organizational selection policies involving applicant SDR are discussed.     

The criterion-related validity of integrity tests: an updated meta-analysis

Integrity tests have become a prominent predictor within the selection literature over the past few decades. However, some researchers have expressed concerns about the criterion-related validity evidence for such tests because of a perceived lack of methodological rigor within this literature, as well as a heavy reliance on unpublished data from test publishers. In response to these concerns, we meta-analyzed 104 studies (representing 134 independent samples), which were authored by a similar proportion of test publishers and non-publishers, whose conduct was consistent with professional standards for test validation, and whose results were relevant to the validity of integrity-specific scales for predicting individual work behavior. Overall mean observed validity estimates and validity estimates corrected for unreliability in the criterion (respectively) were .12 and .15 for job performance, .13 and .16 for training performance, .26 and .32 for counterproductive work behavior, and .07 and .09 for turnover. Although data on restriction of range were sparse, illustrative corrections for indirect range restriction did increase validities slightly (e.g., from .15 to .18 for job performance). Several variables appeared to moderate relations between integrity tests and the criteria. For example, corrected validities for job performance criteria were larger when based on studies authored by integrity test publishers (.27) than when based on studies from non-publishers (.12). In addition, corrected validities for counterproductive work behavior criteria were larger when based on self-reports (.42) than when based on other-reports (.11) or employee records (.15).     

Living and Learning: Reproducing Beliefs in Selective Experience

People's decisions shape their experience. For example, a recruitment officer decides between job applicants and cannot evaluate the suitability of rejected applicants. The selection decisions thus affect the content of the officer's experience of suitable and unsuitable applicants, and experiential learning is achieved from a selective sample of experiences. It is suggested that people's beliefs are sensitive to the content of the experienced sample, but the mind cannot adjust for the selectivity of the sample even when it results from the individual's own decisions. Two experiments with a recruitment task showed that incorrect prior beliefs survive experiential learning when the beliefs are reproduced and thus appear to be confirmed, in actual experience. When the task was to achieve high performance, incorrect prior beliefs persisted because they were reproduced in a smaller sample of selected job applicants. In contrast, when the task was focused on learning, a greater number of applicants were selected, and a more representative experience therefore revised incorrect beliefs. The actual content of the experienced sample is thus crucial for the persistence, as well as for the revision, of incorrect beliefs. Further, as predicted by the hypothesis of constructivist coding, when feedback was absent for rejected applicants, participants constructed “internal feedback” in line with the expectation that the rejected applicant was unsuitable. Thus, when fewer applicants were hired, participants came to believe that the actual proportion of suitable applicants was low. Finally, the implications for efforts to reduce bias and improve experiential learning are discussed. Copyright © 2012 John Wiley & Sons, Ltd.     

Recruiters’ Inferences of Applicant Personality Based on Resume Screening: Do Paper People have a Personality?

Research shows recruiters infer dispositional characteristics from job applicants’ resumes and use these inferences in evaluating applicants’ employability. However, the reliability and validity of these inferences have not been empirically tested. Using data collected from 244 recruiters, we found low levels of estimated interrater reliability when they reviewed entry-level applicants’ resumes and made inferences regarding applicants’ personality traits. Moreover, when recruiters’ inferences of applicant personality were correlated with applicants’ actual Big Five personality scores, results indicated that recruiters’ inferences lacked validity, with the possible exceptions of extraversion and openness to experience. Finally, despite being largely unreliable and invalid, recruiters’ inferences of applicants’ extraversion, openness to experience, and conscientiousness predicted the recruiters’ subsequent employability assessments of the applicants.