A WITHIN SETTING EMPIRICAL TEST OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION

The situational specificity hypothesis of selection procedure validity makes two predictions. The first is that variation in observed validities across settings is caused by real differences in what constitutes job performance. Validity generalization studies to date have provided disconfirming evidence for this prediction by showing that the observed variation is due to statistical and measurement artifacts. The second prediction is that if the situation (i.e., the organization, the setting, the job, the test and the criterion) is held constant, then validity findings will not vary from study to study. This article tests this prediction empirically using data from a unique series of studies reported by Bender and Loveless (1958). The results are contrary to the situational specificity hypothesis and consistent with the hypothesis that variation in observed validities is due to statistical and measurement artifacts that are unrelated to situations and settings. It is concluded that both major predictions of the situational specificity hypothesis have now been empirically disconfirmed.     

The Within‐Setting Variability of Validity in Cognitive Ability Tests

This study was directed to check the second prediction of the situational specificity hypothesis (SSH). According to this prediction, if the setting is held constant, then validity coefficients will not vary from study to study. This prediction was tested using data from a series of studies in which cognitive ability tests were used. These tests are subjected to a cultural bias that could affect the validity variability. The results are contrary to the second prediction of the SSH because a great variability was found in the validity coefficients within‐setting. However, the findings are consistent with the hypothesis of validity generalization (VGH) that predicts similar variability in both across settings and within‐setting. Also, the cultural context does not seem to affect the variability of validity and this variability is similar to that observed in previous studies of validity generalization.     

Validity generalization: Some critical remarks on the Schmidt-Hunter procedure*

This article presents some critical comments on the validity generalization procedure which has been presented by Schmidt, Hunter and others. They have put forward a method for testing the hypothesis that the variance in validity coefficients across situations for job-test combinations is due to what they consider to be statistical artifacts. The Schmidt-Hunter approach is criticized on the following points: the compilation of validity data, the use of criterion measures, and the test of the hypothesis of no situational specificity. Further, the relation between the concepts ‘situational specificity’ and ‘validity generalization’ is considered. In addition, it is noted that Schmidt, Hunter and others have defined the concept ‘situation’ in a different way than classical writers. It is concluded that the Schmidt-Hunter approach to validity generalization shows fundamental shortcomings. As a consequence their far-reaching conclusions for the practice of personnel selection should be considered premature.     

VALIDITY GENERALIZATION RESULTS FOR LAW ENFORCEMENT OCCUPATIONS

The Schmidt-Hunter interactive validity generalization procedure was applied to validity data for cognitive abilities tests for law enforcement occupations. Both assumed artifact distributions, and distributions of artifacts constructed from information contained in the current sample of studies were used to test the hypothesis of situational specificity and to estimate validity generalizability. Results for studies using a criterion of performance in training programs showed that validities ranged from .41 to .71, and for four test types the hypothesis of situational specificity could be rejected using the 75% decision rule. For the remaining test types, validity was generalizable, based on 90% credibility values ranging from .37 to .71. Results for studies using a criterion of performance on the job indicated that the hypothesis of situational specificity was not tenable for three test types, which had validities between .17 and .31. For the remaining test types, estimated mean true validities ranged from .10 to .26 and were generalizable to a majority of situations. Results for both groups of studies were essentially identical for the two types of artifact distribution. Possible reasons for the apparently lower validities and lesser generalizability for job performance criteria are discussed, including possible low validity of the criterion (due to lack of opportunity by supervisors to observe behavior) and the potential role of noncognitive factors in the determination of law enforcement job success. Suggestions for specifically targeted additional research are made.     

FURTHER TESTS OF THE SCHMIDT-HUNTER BAYESIAN VALIDITY GENERALIZATION PROCEDURE1

Using an improved version of the Bayesian validity generalization model presented in Schmidt and Hunter (1977), this study showed that most of the between-study variation in observed validity coefficients of various aptitude and ability tests for two types of clerical work and for the job of first-line supervisor is artifactual in nature. These results cast additional doubt on the traditional belief that employment test validities are situationally specific. In addition, the results showed that generalization of validities to new settings was justified even where the hypothesis of situational specificity could not be rejected under the standards used in this study. Implications for the use of aptitude and ability tests in selection and for theory development in industrial-organizational psychology are discussed.     

USING RANDOM RATHER THAN FIXED EFFECTS MODELS IN META-ANALYSIS: IMPLICATIONS FOR SITUATIONAL SPECIFICITY AND VALIDITY GENERALIZATION

Combining statistical information across studies (i.e., meta-analysis) is a standard research tool in applied psychology. The most common meta-analytic approach in applied psychology, the fixed effects approach, assumes that individual studies are homogeneous and are sampled from the same population. This model assumes that sampling error alone explains the majority of observed differences in study effect sizes and its use has lead some to challenge the notion of situational specificity in favor of validity generalization. We critique the fixed effects methodology and propose an advancement–the random effects model (RE) which provides estimates of how between-study differences influence the relationships under study. RE models assume that studies are heterogeneous since they are often conducted by different investigators under different settings. Parameter estimates of both models are compared and evidence in favor of the random effects approach is presented. We argue against use of the fixed effects model because it may lead to misleading conclusions about situational specificity.     

GENERALIZABLE BIOGRAPHICAL DATA VALIDITY CAN BE ACHIEVED WITHOUT MULTI-ORGANIZATIONAL DEVELOPMENT AND KEYING

This research showed that the validity of a biodata instrument developed and keyed within a single organization can generalize to other organizations. It also examines a criterion measure-rate of promotional progress-that has not been used extensively in biodata research, but has several characteristics that make its use attractive. The validity of the biodata component of the Manager Profile Record (MPR), developed and keyed within a single organization, as a predictor of rate of managerial progress was cross-validated on a sample of 7,334 managers and staff professionals in 24 organizations. Results indicate the MPR was a valid predictor of rate of promotional progress across all organizations and that validity did not vary greatly across organizations ( p = .53, SD_p= .05). The MPR was also a valid predictor for both sexes, as well as for managers of all age groups, lengths of company service, and education levels. These findings demonstrate that multiple-firm development and keying of a biodata instrument was not required for generalizable validity, and argue against the hypothesis of situational specificity. Suggestions for developing biodata instruments in single organizations that will generalize to other organizations include careful attention to the validity and reliability of criterion measures and developing validity at the item level.     

The Situational Specificity of Validities: Correcting for Statistical Artifacts Does Not Always Reduce the Trans-Situational Variability of Correlation Coefficients

Validity generalization research suggests that much of the variability in validity coefficients is due to statistical artifacts, such as sampling error. Correcting for these artifacts usually reduces the trans-situational variability in validity coefficients, but the reverse can happen. Conditions under which corrections for statistical artifacts can increase the variability in validity coefficients are described, examples of meta-analyses reporting larger corrected than observed variance in validity are cited, and implications for the situational specificity of validity coefficients are discussed. In general, the true variability in validity coefficients based on large samples is often larger than the observed variability in test validities.     

USING A META-ANALYTIC PERSPECTIVE TO ENHANCE JOB COMPONENT VALIDATION

This paper develops synthetic validity estimates based on a meta-analytic-weighted least squares (WLS) approach to job component validity (JCV), using position analysis questionnaire (PAQ) estimates of job characteristics, and the Data, People, & Things ratings from the  Dictionary of Occupational Titles  as indices of job complexity. For the general aptitude test battery database of 40,487 employees, nine validity coefficients were estimated for 192 positions. The predicted validities from the WLS approach had lower estimated variability than would be obtained from either the classic JCV approach or local criterion-related validity studies. Data, People, & Things summary ratings did not consistently moderate validity coefficients, whereas the PAQ data did moderate validity coefficients. In sum, these results suggest that synthetic validity procedures should incorporate a WLS regression approach. Moreover, researchers should consider a comprehensive set of job characteristics when considering job complexity rather than a single aggregated index.     

Development and validation of a HEXACO situational judgment test

The purpose of this study was to develop and validate a construct-based situational judgment test of the HEXACO personality dimensions. In four studies, among applicants, employees, and Amazon Mechanical Turk participants (Ns = 72–305), we showed that it is possible to assess the six personality dimensions with a situational judgment test and that the criterion-related validity of the situational judgment test is comparable to the criterion-related validity of traditional self-reports but lower than the criterion-related validity of other-reports of personality. Test–retest coefficients (with a time interval of 2 weeks) varied between .55 and .74. Considering personality is the most commonly assessed construct in employee selection contexts (Ryan et al., 2015), this situational judgment test may provide human resources professionals with an alternative assessment tool.     

Evidence of Predictive and Incremental Validity of Personality Factors,Vocal Attractiveness and the Situational Interview

This study evaluates the impact of the situational interview, personality measures, vocal attractiveness, and interactions between personality and vocal attractiveness on job performance. In a concurrent validity study with 154 customer service employees, a structured, situational interview was conducted and three of the Big Five personality factors (agreeableness, conscientiousness, and extraversion) are measured with two different frames‐of‐reference: at‐work and typical. Among the results are the validity of the situational interview and at‐work personality measures is established; vocal attractiveness correlates with both interview ratings and job performance; and finally, agreeableness and conscientiousness are found to vary in a pattern related to levels of vocal attractiveness such that both personality factors predict performance more strongly for people with more attractive voices.     

评价中心的结构效度研究

评价中心虽然具备很高的预测效度,但其结构效度指标却不太理想,如研究普遍发现其汇聚效度和区分效度较低。影响评价中心结构效度的因素众多,如评分维度因素（数量和类型）、评价者因素（培训方式和人员类型）、测评方法因素（情景导向特征、特质激活潜力、测评活动形式）以及系统的观察与评价程序。该文从上述因素出发,综述了评价中心结构效度的相关研究,总结了提高评价中心结构效度的措施,并指出了未来的研究方向     

情景判断测验的开发程序、构思效度及研究趋势

章详细介绍情景判断测验开发的一般程序,对情景判断测验的多种形式和记分方法进行了总结和比较。同时,从情景判断测验的结果与认知能力、个性和工作经验等关系的角度出发,分析了情景判断测验的构思效度,认为情景判断测验测量的是多维构思。文章最后认为,需要从与其他构思的关系、测评指定构思、影响效度的因素以及跨文化比较等四个方面对情景判断测验开展进一步研究。     

A counterintuitive hypothesis about employment interview validity and some supporting evidence

This study found mixed support for the hypothesis that the difference in criterion-related validity between unstructured and structured employment interviews is due solely to the greater reliability of structured interviews. Using data from prior meta-analyses, this hypothesis was tested in 4 data sets by using standard psychometric procedures to remove the effects of measurement error in interview scores from correlations with rated job performance and training performance. In the 1st data set. support was found for this hypothesis. However, in a 2nd data set structured interviews had higher true score correlations with performance ratings, and in 2 other data sets unstructured interviews had higher true score correlations. We also found that averaging across 3 to 4 independent unstructured interviews provides the same level of validity for predicting job performance as a structured interview administered by a single interviewer. Practical and theoretical implications are discussed.     

SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIES

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct-level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007 ; McDaniel & Ngyuen, 2001 ; Schmitt & Chan, 2006 ) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content-based typology. We then conducted a meta-analysis to determine the criterion-related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion-related validity; and (b) video-based situational judgment tests tended to have stronger criterion-related validity than pencil-and-paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.     

EXPLORING OPTIONS FOR SUPPORTING TEST USE IN SITUATIONS PRECLUDING LOCAL VALIDATION

This paper examines some of the options available to practitioners interested in supporting the use of selection measures in an organization, including test transportability, validity generalization (VG), and synthetic validation, reviewing some of the advantages, disadvantages and requirements of each approach. Results of four unpublished, proprietary validation studies are reported which compare validity estimates provided by the job component validation (JCV; a type of synthetic validation) routine inherent in the Position Analysis Questionnaire (PAQ) with observed validity coefficients for a variety of criterion measures. We then examine the accuracy of the JCV procedure in predicting validity coefficients for 51 clerical classifications extracted from an existing PAQ job evaluation database of a large utility company. Predicted JCVs are compared to mean observed validity coefficients for five DOT clerical categories provided by Pearlman, Schmidt, and Hunter (1980). The VG and JCV methods provided highly similar and converging estimates of the validity of cognitive ability tests for predicting performance in clerical occupations. Implications for practice are discussed, particularly the need to use multiple, converging lines of evidence to support test use.     

Situational Judgment and Job Performance

Data from 160 civil service employees demonstrate the validity of a situational judgment test in predicting overall job performance as well as three performance dimensions: task performance (core technical proficiency), motivational contextual performance (job dedication), and interpersonal contextual performance (interpersonal facilitation). Situational judgment also provided incremental validity over the prediction provided jointly by cognitive ability, the Big Five personality traits, and job experience. These findings extended the work of Clevenger, Pereira, Wiechmann, Schmitt, and Harvey (2001) on the incremental validity of situational judgment tests as well as the meta-analytic results reported by McDaniel, Morgeson, Finnegan, Campion, and Braverman (2001). Implications are discussed in terms of research on the prediction and understanding of job performance.     

Gender bias in employment contexts: A closer examination of the role incongruity principle

This research extends the role incongruity analysis of employment-related gender bias by investigating the role of dispositional and situational antecedents, specifically political ideology and the salience of cues to the traditional female gender role. The prediction that conservatives would show an anti-female candidate bias and liberals would show a pro-female bias when the traditional female gender role is salient was tested across three experimental studies. In Study 1, 126 participants evaluated a male or a female job applicant with thoughts of the traditional female gender role activated or not. Results showed that when the gender role is salient, political ideology moderates evaluations of the female candidates such that conservatives evaluate her negatively and liberals evaluate her positively. Study 2 (89 participants) replicated this effect and showed that this political ideology-based bias does not occur when the non-traditional female gender role is made salient. Study 2 also demonstrated that the observed effects are not driven by liberals' and conservatives' differing perceptions regarding the female applicant's qualifications for the job. Finally, Study 3 (159 participants) both replicated the political ideology-based evaluation bias for female candidates and demonstrated that this bias is mediated by conservatives' and liberals' attitudes toward the roles of women in society.     

Impact of Assessments of Validity Generalization and Situational Specificity on the Science and Practice of Personnel Selection

The application of meta-analysis, in particular validity generalization (VG) analysis, to the cumulative literature on the validity of selection tests has fundamentally changed the science and practice of personnel selection. VG analyses suggest that the validities of standardized tests and other structured assessments are both higher and more consistent across jobs and organizations than was previously believed. As a result, selection researchers and practitioners can draw on the research literature to make reasonably accurate forecasts about the validity and usefulness of different tests in particular applications. Distinctions between tests of validity generalization and tests of situational specificity are described, and difficulties in demonstrating that validity is constant across the different settings where tests are used are outlined.