SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIES

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct-level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007 ; McDaniel & Ngyuen, 2001 ; Schmitt & Chan, 2006 ) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content-based typology. We then conducted a meta-analysis to determine the criterion-related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion-related validity; and (b) video-based situational judgment tests tended to have stronger criterion-related validity than pencil-and-paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.     

The team role test: development and validation of a team role knowledge situational judgment test

The main objectives in this research were to introduce the concept of team role knowledge and to investigate its potential usefulness for team member selection. In Study 1, the authors developed a situational judgment test, called the Team Role Test, to measure knowledge of 10 roles relevant to the team context. The criterion-related validity of this measure was examined in 2 additional studies. In a sample of academic project teams (N = 93), team role knowledge predicted team member role performance (r = .34). Role knowledge also provided incremental validity beyond mental ability and the Big Five personality factors in the prediction of role performance. The results of Study 2 revealed that the predictive validity of role knowledge generalizes to team members in a work setting (N = 82, r = .30). The implications of the results for selection in team environments are discussed.     

Situational Judgment and Job Performance

Data from 160 civil service employees demonstrate the validity of a situational judgment test in predicting overall job performance as well as three performance dimensions: task performance (core technical proficiency), motivational contextual performance (job dedication), and interpersonal contextual performance (interpersonal facilitation). Situational judgment also provided incremental validity over the prediction provided jointly by cognitive ability, the Big Five personality traits, and job experience. These findings extended the work of Clevenger, Pereira, Wiechmann, Schmitt, and Harvey (2001) on the incremental validity of situational judgment tests as well as the meta-analytic results reported by McDaniel, Morgeson, Finnegan, Campion, and Braverman (2001). Implications are discussed in terms of research on the prediction and understanding of job performance.     

Use of situational judgment tests to predict job performance: a clarification of the literature

Although situational judgment tests have a long history in the psychological assessment literature and continue to be frequently used in employment contexts, there has been virtually no summarization of this literature. The purpose of this article is to review the history of such tests and present the results of a meta-analysis on criterion-related and construct validity. On the basis of 102 coefficients and 10,640 people, situational judgment tests showed useful levels of validity (rho = .34) that were generalizable. A review of 79 correlations between situational judgment tests and general cognitive ability involving 16,984 people indicated that situational judgment tests typically evidence relationships with cognitive ability (rho = .46). On the basis of the literature review and meta-analytic findings, implications for the continued use of situational judgment tests are discussed, particularly in terms of recent investigations into tacit knowledge.     

Measuring Procedural Knowledge More Simply with a Single-Response Situational Judgment Test

Purpose  This paper describes the development of a situational judgment test (SJT) based on single-response options developed directly from critical incidents and reports a study that tested the SJT’s concurrent validity against ratings of job performance. Design/Methodology/Approach  Situational judgment test items were developed from critical incidents provided by administrators of volunteer agencies. Volunteers who worked at another agency completed the SJT and a self-report personality test. Supervisors rated their job performance on three dimensions. Findings  Situational judgment test scores representing procedural knowledge about work effort were significantly correlated with ratings of work effort performance (r = .28). Conscientiousness was correlated with work effort knowledge (r = .26), but not with work effort performance (r = −.02). Implications  These results provide some preliminary evidence that a single-response SJT format based upon critical incidents can produce valid measures of procedural knowledge and might be a useful alternative to the traditional multiple-response format. Originality/Value  This paper presents a novel way to construct SJTs using single-response options that are less laborious to develop than the multiple-response options in traditional SJT formats. Results of the validity study suggest that this new single-response format can predict job performance and encourage further research on the viability of this approach.     

Personality and Situational Judgment Tests Across Applicant and Incumbent Settings: An Examination of Validity,Measurement, and Subgroup Differences

The effects of concurrent versus predictive designs were examined in 2 large samples of incumbents (N = 2,989) and applicants (N = 7,259) for 3 measures of personality and a situational judgment test (SJT). Results showed that applicants scored significantly higher than incumbents on the personality measures and significantly lower on the SJT. Regarding measurement equivalence, the results indicate that the same form of factor structure is found in both settings, with the factor structures themselves being generally invariant. Importantly, the criterion-related validities were equivalent across samples. Subgroup differences were small to moderate in both settings, though varying across race and gender. Limitations and directions for future research are offered.     

PERSONALITY DIMENSIONS EXPLAINING RELATIONSHIPS BETWEEN INTEGRITY TESTS AND COUNTERPRODUCTIVE BEHAVIOR: BIG FIVE,OR ONE IN ADDITION?

Although the criterion-related validity of integrity tests is well established, there has not been enough research examining which personality constructs contribute to their criterion-related validity. Moreover, evidence of how well findings on integrity tests in North America generalize to non-English speaking countries is virtually absent. This research addressed these issues with data obtained from employees and students in Canada and Germany (total N = 853). Specifically, we tested the hypotheses that (a) Honesty–Humility, as specified in the HEXACO model of personality, is relatively more important than the Big 5 dimensions of personality in accounting for the criterion-related validity of overt integrity tests, whereas (b) the Big 5 are relatively more important in explaining the validity of personality-based integrity tests. These predictions were tested using 2 criteria (counterproductive work behavior and counterproductive academic behavior) as well as 2 overt and 2 personality-based integrity tests. We found evidence of the expected differences between types of integrity tests largely regardless of culture of the sample, specific test, criterion, or population under research, pointing to some degree of generalizability of findings in integrity testing research. Implications include theoretical refinements in research on integrity testing and encouragement of practical applications beyond North America.     

Situational judgment tests in high-stakes settings: issues and strategies with generating alternate forms

This study used principles underlying item generation theory to posit competing perspectives about which features of situational judgment tests might enhance or impede consistent measurement across repeat test administrations. This led to 3 alternate-form development approaches (random assignment, incident isomorphism, and item isomorphism). The effects of these approaches on alternate-form consistency, mean score changes, and criterion-related validity were examined in a high-stakes context (N = 3,361). Generally, results revealed that even small changes in the context of the situations presented resulted in significantly lower alternate-form consistency. Conversely, placing more constraints on the alternate-form development process proved beneficial. The contributions, implications, and limitations of these results for the development of situational judgment tests and high-stakes testing are discussed.     

情景判断测验的开发程序、构思效度及研究趋势

章详细介绍情景判断测验开发的一般程序,对情景判断测验的多种形式和记分方法进行了总结和比较。同时,从情景判断测验的结果与认知能力、个性和工作经验等关系的角度出发,分析了情景判断测验的构思效度,认为情景判断测验测量的是多维构思。文章最后认为,需要从与其他构思的关系、测评指定构思、影响效度的因素以及跨文化比较等四个方面对情景判断测验开展进一步研究。     

IN SUPPORT OF PERSONALITY ASSESSMENT IN ORGANIZATIONAL SETTINGS

Personality constructs have been demonstrated to be useful for explaining and predicting attitudes, behaviors, performance, and outcomes in organizational settings. Many professionally developed measures of personality constructs display useful levels of criterion-related validity for job performance and its facets. In this response to Morgeson et al. (2007) , we comprehensively summarize previously published meta-analyses on (a) the optimal and unit-weighted multiple correlations between the Big Five personality dimensions and behaviors in organizations, including job performance; (b) generalizable bivariate relationships of Conscientiousness and its facets (e.g., achievement orientation, dependability, cautiousness) with job performance constructs; (c) the validity of compound personality measures; and (d) the incremental validity of personality measures over cognitive ability. Hundreds of primary studies and dozens of meta-analyses conducted and published since the mid 1980s indicate strong support for using personality measures in staffing decisions. Moreover, there is little evidence that response distortion among job applicants ruins the psychometric properties, including criterion-related validity, of personality measures. We also provide a brief evaluation of the merits of alternatives that have been offered in place of traditional self-report personality measures for organizational decision making. Given the cumulative data, writing off the whole domain of individual differences in personality or all self-report measures of personality from personnel selection and organizational decision making is counterproductive for the science and practice of I-O psychology.     

An Ideal Student Factor and the validity of noncognitive measures of student potential

The presence of a general factor has been found in previous research on the utilization of personality tests in selection contexts. Termed the Ideal Employee Factor (IEF), this factor is thought to represent an effort to present oneself as an ideal employee to some degree on all items in the test. Examining the responses to biodata and situational judgment items, we provide evidence for the existence of a factor similar to the IEF within the college admissions context, termed the Ideal Student Factor (ISF), but the magnitude of this factor relative to specific factors was small. Mixed support was found for the hypothesized differences in the factor loadings on the ISF for different constructs included in the model. The presence of the ISF was not found to substantially impact the validity of other predictors within this study. The overall conclusion is that this general factor would not impact the use of these biodata and situational judgment items in college admissions.     

情境判断测验的效度研究述评

情境判断测验是一种为作答者呈现工作相关的典型情境以及该情境下可能的行为反应, 要求根据指导语提示进行选择或评价的测验形式。随着其理论和实践的发展, 研究者越来越关注情境判断测验的效度研究, 包括对其构想效度、效标关联效度和递增效度的探讨, 以及指导语类型、情境保真度以及计分方式等因素对其效度的影响。基于这些研究进展, 未来情境判断测验实践领域可能的方向是：(1)开发针对特定构想的情境判断测验; (2)结合具体构想选用相应的指导语; (3)应用作假和培训对效度影响的研究结果指导实践。     

The Effects of Social Desirability and Faking on Personality and Integrity Assessment for Personnel Selection

A review of the extant literature and new empirical research suggests that social desirability is not much of a concern in personality and integrity testing for personnel selection. In particular, based on meta-analytically derived evidence, it appears that social desirability influences do not destroy the convergent and discriminant validity of the Big Five dimensions of personality (Emotional Stability, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness). We also present new empirical evidence regarding gender and age differences in socially desirable re- sponding. Although social desirability predicts a number of important work variables such as job satisfaction, organizational commitment, and supervisor ratings of training success, social desirability does not seem to be a predictor of overall job performance and is only very weakly related to specific dimensions of job performance such as technical proficiency (r = -.07) and personal discipline ( r = .05). Large sample investigations of the moderating influences of social desirability in actual work settings indicate that social desirability does not moderate the criterion-related validities of personality variables or integrity tests. The criterion-related validity of integrity tests for overall job performance with applicant samples in predictive studies is .41. Controlling for social desirability in integrity or personality test scores leaves the operational validities intact, thereby suggesting that social desirability functions neither as a mediator nor as a suppressor variable in personality-performance.     

Incremental Validity of Situational Judgment Tests for Task and Contextual Job Performance

This paper has three goals. First, it responds to calls for additional research on subgroup differences in situational judgment tests. Second, it expands the cumulative knowledge on the incremental validity of situational judgment tests beyond cognitive ability and personality. Third, it examines the validity and incremental validity of various predictors for both task and contextual performance.     

Personality-Based Job Analysis and The Self-Serving Bias

This study tested the hypothesis that personality-based job analysis (PBJA) ratings are correlated with subject matter expert (SME) personality scale scores (through a self-serving bias). Some support was found for the hypothesis. However, PBJA ratings were somewhat predictive of the criterion-related validity of scale scores for the big five personality dimensions.     

Constructing personality scales under the assumptions of an ideal point response process: toward increasing the flexibility of personality measures

The main aim of this article is to explicate why a transition to ideal point methods of scale construction is needed to advance the field of personality assessment. The study empirically demonstrated the substantive benefits of ideal point methodology as compared with the dominance framework underlying traditional methods of scale construction. Specifically, using a large, heterogeneous pool of order items, the authors constructed scales using traditional classical test theory, dominance item response theory (IRT), and ideal point IRT methods. The merits of each method were examined in terms of item pool utilization, model-data fit, measurement precision, and construct and criterion-related validity. Results show that adoption of the ideal point approach provided a more flexible platform for creating future personality measures, and this transition did not adversely affect the validity of personality test scores.     

The Short Form of the Revised Almost Perfect Scale

We created a shorter and more refined item set from the Almost Perfect Scale–Revised (APS–R; Slaney, Mobley, Trippi, Ashby, & Johnson, 1996; Slaney, Rice, Mobley, Trippi, & Ashby, 2001) to measure 2 major dimensions of perfectionism: standards (high performance expectations) and discrepancy (self-critical performance evaluations). In Study 1, after testing the internal structure of the measure (N = 749), a subset of the current APS–R items was derived (Short Almost Perfect Scale [SAPS]) that possessed good psychometric features, such as strong item–factor loadings, score reliability, measurement invariance between women and men, and criterion-related validity through associations with neuroticism, conscientiousness, academic performance, and depression. Controlling for neuroticism and conscientiousness, factor mixture modeling supported a 2-factor, 3-class model of perfectionism, and results were consistent with labeling the classes as nonperfectionists and adaptive and maladaptive perfectionists. Measurement results were cross-validated in a separate sample (N = 335). Study 2 also provided substantial evidence for the convergent, discriminant, and criterion-related validity of SAPS scores. Both studies supported the SAPS as a brief and psychometrically strong measure of major perfectionism factors and classes of perfectionists.     

Predictor content matters for knowledge testing: Evidence supporting content validation

Organizations often rely on the match between job requirements and test content to justify test use. This practice has been questioned on the grounds that content validation has little relevance to criterion-related validation due to positive manifold among predictors. We analyze two large databases to assess the implications of test content for (a) test interchangeability and (b) criterion-related validity. Analyses of 15 knowledge tests administered (N = 80,394) as part of Project Talent demonstrate that test content is related to predictor interchangeability. Analyses of SAT and Advanced Placement test data compare correlations among predictors and criteria drawn from matched and unmatched content domains. We conclude that test-criterion content match is likely to result in stronger criterion-related validity.     

Be Careful What You Ask For: Effects of Response Instructions on the Construct Validity and Reliability of Situational Judgment Tests

The aim of this study was to examine how six different types of situational judgment test (SJT) instructions, used frequently in practice, influence the psychometric characteristics of SJTs. The six SJT versions used the exact same items and differed only in their instructions; these versions were administered in two phases. Phase I was a between–subjects design (n= 486) that had participants complete one version of the SJTs. Phase II was a within–subjects design (n= 231) held several weeks later that had participants complete all six versions of the SJTs. Further, 146 of these individuals completed both phases, allowing for an assessment of test–retest reliability. A variety of objective and subjective criteria were collected, including self and peer ratings. Results indicated that instructions had a large effect on SJT responses, reliability, and validity. In general, instructions asking what one ‘would do’ showed more favorable characteristics than those that asked what one ‘should do’. Correlations between these two types were relatively low despite the fact that the same items were used, and criterion–related validities differed substantially in favor of the ‘would do’ instructions. Overall, this study finds that researchers and practitioners must give careful consideration to the types of SJT instructions used; failing to do so could influence criterion–related validity and cloud inferences of construct validity.     

Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis

In this article, we describe a systematic review of mean race and sex differences in situational judgment test (SJT) performance. On average, White test takers perform better on SJTs than Black, Hispanic, and Asian test takers. Female examinees perform slightly better than male test takers on SJTs. We investigate two moderators of these differences: loading of g or personality on the SJT, and response instructions. Mean race differences between Black, Hispanic, Asian and White examinees in SJT performance are largely explained by the cognitive loading of the SJT such that the larger the cognitive load, the larger the mean race differences. Regarding the effect of personality loadings of SJTs on race differences, Black–White and Asian–White differences are smaller to the extent that the SJT is correlated with emotional stability and Hispanic–White differences are smaller to the extent that SJTs are correlated with conscientiousness and agreeableness. Cognitive loading has minimal effect on male–female SJT score differences; however, SJT score differences are larger, favoring women, when SJTs are correlated with conscientiousness and agreeableness. Concerning response instructions, knowledge response instructions showed greater race differences than behavioral tendency instructions. The mean correlations show that these differences are largely because of the greater g loading of knowledge instructions. A second study showed that when used in hiring, SJTs are likely to have adverse impact by race but not by sex.