SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIES

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct-level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007 ; McDaniel & Ngyuen, 2001 ; Schmitt & Chan, 2006 ) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content-based typology. We then conducted a meta-analysis to determine the criterion-related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion-related validity; and (b) video-based situational judgment tests tended to have stronger criterion-related validity than pencil-and-paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.     

VIDEO-BASED SITUATIONAL TESTING

In two separate studies, video-based tests of situational judgment were developed and validated against measures of performance for hourly service workers. In the first study, 684 employees were used to develop a test of retail associate judgment and 787 newly hired employees were used to cross-validate the instrument. In the second study, 412 current employees were used to develop a test of nursing home caregiver judgment and 148 newly hired caregivers were used to cross-validate this video-based test. In both studies, responses to video-based situational vignettes were empirically keyed against supervisory ratings of performance. The resulting keys produced uncorrected cross-validities in the low .20s. The video-based test scores were also found to be related to measures of cognitive ability and, to a lesser extent, experience. These results suggest that cognitive ability and possibly experience account for some but not all of the predictiveness of video-based situational tests. Video-based situational tests demonstrated score differences between Whites and non-Whites of roughly one-half a standard deviation, indicating that the use of such tests could produce adverse impact against non-Whites. In the second study, customer preferences regarding desired behavior were also used to develop a rational scoring key. This customer driven key was significantly related to performance ( r = .33) in the cross-validation sample, demonstrating that customers' preferences can provide the basis for developing valid predictors. Potential benefits of video-based situational tests in selection and directions for future research are discussed.     

Incremental Validity of Situational Judgment Tests for Task and Contextual Job Performance

This paper has three goals. First, it responds to calls for additional research on subgroup differences in situational judgment tests. Second, it expands the cumulative knowledge on the incremental validity of situational judgment tests beyond cognitive ability and personality. Third, it examines the validity and incremental validity of various predictors for both task and contextual performance.     

Establishing the Measurement Equivalence of Online Selection Assessments Delivered on Mobile versus Nonmobile Devices

Recent usage data suggest job applicants are completing online selection assessments using mobile devices (e.g., smartphones) in greater numbers. To determine the appropriateness of this new technology, this study examined the measurement equivalence of selection assessments delivered on mobile and nonmobile devices (e.g., personal computers). Measurement invariance tests conducted with multigroup confirmatory factor analysis suggest mobile versions of a cognitive ability‐type assessment, two biodata assessments, a multimedia work simulation, and a text‐based situational judgment test appear to be equivalent to nonmobile versions. However, mobile device user latent means were half a standard deviation lower than their nonmobile counterparts for the situational judgment test. Implications for mobile device usage within selection and assessment are discussed.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.     

An Ideal Student Factor and the validity of noncognitive measures of student potential

The presence of a general factor has been found in previous research on the utilization of personality tests in selection contexts. Termed the Ideal Employee Factor (IEF), this factor is thought to represent an effort to present oneself as an ideal employee to some degree on all items in the test. Examining the responses to biodata and situational judgment items, we provide evidence for the existence of a factor similar to the IEF within the college admissions context, termed the Ideal Student Factor (ISF), but the magnitude of this factor relative to specific factors was small. Mixed support was found for the hypothesized differences in the factor loadings on the ISF for different constructs included in the model. The presence of the ISF was not found to substantially impact the validity of other predictors within this study. The overall conclusion is that this general factor would not impact the use of these biodata and situational judgment items in college admissions.     

On the Construct Validity of Integrity Tests: Individual and Situational Factors as Predictors of Test Performance

Although integrity tests are widely applied in screening job applicants, there is a need for research for examining the construct validity of these tests. In the present study, a theoretical model examining the causes of destructive behavior in organizational settings was used to develop background data measures of individual and situational variables that might be related to integrity test scores. Subsequently, 692 undergraduates were asked to complete these background data scales along with (a) two overt integrity tests – the Reid Report and the Personnel Selection Inventory, and (b) two personality‐based measures – the delinquency and socialization scales of the California Psychological Inventory. When scores of these measures were correlated with and regressed on the background data scales, it was found that relevant individual variables, such as narcissism and power motives, and relevant situational variables, such as alienation and exposure to negative peer groups, were related to scores on both types of integrity tests. However, a stronger pattern of validity evidence was obtained for the personality‐based measures and, in all cases, situational variables were found to be better predictors than individual variables. The implications of these findings for the validity of inferences drawn from overt and personality‐based integrity tests are discussed.     

Developing a biodata measure and situational judgment inventory as predictors of college student performance

This article describes the development and validation of a biographical data (biodata) measure and situational judgment inventory (SJI) as useful predictors of broadly defined college student performance outcomes. These measures provided incremental validity when considered in combination with standardized college-entrance tests (i.e., SAT/ACT) and a measure of Big Five personality constructs. Racial subgroup mean differences were much smaller on the biodata and SJI measures than on the standardized tests and college grade point average. Female students tended to outperform male students on most predictors and outcomes with the exception of the SAT/ACT. The biodata and SJI measures show promise for student development contexts and for selecting students on a wide range of outcomes with reduced adverse impact.     

Situational Judgment and Job Performance

Data from 160 civil service employees demonstrate the validity of a situational judgment test in predicting overall job performance as well as three performance dimensions: task performance (core technical proficiency), motivational contextual performance (job dedication), and interpersonal contextual performance (interpersonal facilitation). Situational judgment also provided incremental validity over the prediction provided jointly by cognitive ability, the Big Five personality traits, and job experience. These findings extended the work of Clevenger, Pereira, Wiechmann, Schmitt, and Harvey (2001) on the incremental validity of situational judgment tests as well as the meta-analytic results reported by McDaniel, Morgeson, Finnegan, Campion, and Braverman (2001). Implications are discussed in terms of research on the prediction and understanding of job performance.     

Use of situational judgment tests to predict job performance: a clarification of the literature

Although situational judgment tests have a long history in the psychological assessment literature and continue to be frequently used in employment contexts, there has been virtually no summarization of this literature. The purpose of this article is to review the history of such tests and present the results of a meta-analysis on criterion-related and construct validity. On the basis of 102 coefficients and 10,640 people, situational judgment tests showed useful levels of validity (rho = .34) that were generalizable. A review of 79 correlations between situational judgment tests and general cognitive ability involving 16,984 people indicated that situational judgment tests typically evidence relationships with cognitive ability (rho = .46). On the basis of the literature review and meta-analytic findings, implications for the continued use of situational judgment tests are discussed, particularly in terms of recent investigations into tacit knowledge.     

Situational judgment tests in high-stakes settings: issues and strategies with generating alternate forms

This study used principles underlying item generation theory to posit competing perspectives about which features of situational judgment tests might enhance or impede consistent measurement across repeat test administrations. This led to 3 alternate-form development approaches (random assignment, incident isomorphism, and item isomorphism). The effects of these approaches on alternate-form consistency, mean score changes, and criterion-related validity were examined in a high-stakes context (N = 3,361). Generally, results revealed that even small changes in the context of the situations presented resulted in significantly lower alternate-form consistency. Conversely, placing more constraints on the alternate-form development process proved beneficial. The contributions, implications, and limitations of these results for the development of situational judgment tests and high-stakes testing are discussed.     

A situated process model of vocational achievement goal striving within members of the academic staff at university

Building on achievement goal approach and self-determination theory (SDT), we propose a concise process model of achievement goal striving within academic staff members working at universities. We investigate this model in a sample of 107 academic staff members, who we questioned twice a day over the course of 2 weeks resulting in 1653 measurement occasions. Using two-level structural equation modeling, we found that substantial amounts of variance in situated achievement goals could be attributed to personal goal orientations. Life aspirations indirectly predicted the general strength of situated achievement goal striving via personal goal orientations. Finally, the situational satisfaction of the basic psychological needs for autonomy, competence and relatedness was differentially predictive for fluctuations in situated achievement goal striving, which in turn predicted situational intrinsic work motivation and work engagement. Our research ties achievement goal approach more closely to SDT and delivers a first outlook on the importance of working conditions in academia for goal setting processes and vocational motivation.     

Situational Tests in Student Selection: An Examination of Predictive Validity,Adverse Impact,and Construct Validity

The Flemish Admission Exam ‘Medical and Dental Studies’ is comprised of four cognitive ability tests and four situational tests, namely two work samples (i.e., a lecture and a medical text) and two video‐based situational judgement tests (i.e., a physician–patient interaction and a medical expert discussion). On the basis of the Admission Exam scores of 941 candidates (359 men, 582 women) this study shows that situational tests significantly can predict better than cognitive ability tests, with lecture and text emerging as significant predictors. When situational tests are combined with cognitive ability tests, there are no mean gender differences. Situational tests also enable us to measure a broader range of constructs. For example, in this study, the personality factor Openness is related to better situational test performance. Overall, this study demonstrates that situational tests may be a useful complement to traditional student selection procedures.     

情景判断测验的开发程序、构思效度及研究趋势

章详细介绍情景判断测验开发的一般程序,对情景判断测验的多种形式和记分方法进行了总结和比较。同时,从情景判断测验的结果与认知能力、个性和工作经验等关系的角度出发,分析了情景判断测验的构思效度,认为情景判断测验测量的是多维构思。文章最后认为,需要从与其他构思的关系、测评指定构思、影响效度的因素以及跨文化比较等四个方面对情景判断测验开展进一步研究。     

The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance

This study provides conceptual and empirical arguments why an assessment of applicants' procedural knowledge about interpersonal behavior via a video-based situational judgment test might be valid for academic and postacademic success criteria. Four cohorts of medical students (N = 723) were followed from admission to employment. Procedural knowledge about interpersonal behavior at the time of admission was valid for both internship performance (7 years later) and job performance (9 years later) and showed incremental validity over cognitive factors. Mediation analyses supported the conceptual link between procedural knowledge about interpersonal behavior, translating that knowledge into actual interpersonal behavior in internships, and showing that behavior on the job. Implications for theory and practice are discussed.     

Validity evidence for the situational judgment test paradigm in emotional intelligence measurement

To date, various measurement approaches have been proposed to assess emotional intelligence (EI). Recently, two new EI tests have been developed based on the situational judgment test (SJT) paradigm: the Situational Test of Emotional Understanding (STEU) and the Situational Test of Emotion Management (STEM). Initial attempts have been made to examine the construct‐related validity of these new tests; we extend these findings by placing the tests in a broad nomological network. To this end, 850 undergraduate students completed a personality inventory, a cognitive ability test, a self‐report EI test, a performance‐based EI measure, the STEU, and the STEM. The SJT‐based EI tests were not strongly correlated with personality and fluid cognitive ability. Regarding their relation with existing EI measures, the tests did not capture the same construct as self‐report EI measures, but corresponded rather to performance‐based EI measures. Overall, these results lend support for the SJT paradigm for measuring EI as an ability.     

Development and validation of a HEXACO situational judgment test

The purpose of this study was to develop and validate a construct-based situational judgment test of the HEXACO personality dimensions. In four studies, among applicants, employees, and Amazon Mechanical Turk participants (Ns = 72–305), we showed that it is possible to assess the six personality dimensions with a situational judgment test and that the criterion-related validity of the situational judgment test is comparable to the criterion-related validity of traditional self-reports but lower than the criterion-related validity of other-reports of personality. Test–retest coefficients (with a time interval of 2 weeks) varied between .55 and .74. Considering personality is the most commonly assessed construct in employee selection contexts (Ryan et al., 2015), this situational judgment test may provide human resources professionals with an alternative assessment tool.     

Job Control and Job Stressors as Predictors of Proactive Work Behavior: Is Role Breadth Self-Efficacy the Link?

Previous research showed that job stressors are positively related to proactive work behavior. This study examines if these relations can be generalized to supervisor-rated proactive behavior and if relations of job stressors (time pressure, situational constraints) and job control with proactive behavior can be explained by role breadth self-efficacy as an underlying mechanism. We used supervisor ratings of proactive behavior (personal initiative, taking charge) from 140 employees working in small and medium-sized companies. Multilevel path analyses supported a partial mediation model. Job control and time pressure were positively related to role breadth self-efficacy, and role breadth self-efficacy was positively related to proactive behavior. Job control had also a direct relation with proactive behavior. The relation between situational constraints and proactive behavior was not significant.     

Incremental validation of situational judgment tests.

Using 3 different samples, the authors assessed the incremental validity of situational judgment inventories (SJIs), relative to job knowledge, cognitive ability, job experience, and conscientiousness, in the prediction of job performance. The SJI was a valid predictor in all 3 samples and incrementally so in 2 samples. Relative to the other predictors, SJI's partial correlation with performance, controlling for the other 4 predictors, was superior in most comparisons. Subgroup differences on the SJI also appear to be less than those for cognitive ability and job knowledge, but greater than differences in conscientiousness. The SJI should prove to be a valuable additional measure in the prediction of job performance, but several additional areas of research are suggested.     

情境判断测验的研究进展

目前,情境判断测验的相关研究主要呈现两大趋势———效度研究和跨文化比较研究。其中,效度研究主要分为效标关联效度和构想效度研究,跨文化比较研究则主要探讨测验的文化公平性,以及对不同种族员工工作绩效的预测作用。文章还将介绍研究呈现出的新趋势———对情境判断测验本质的研究,即探讨测验形式特点和被试信息加工过程对测验结果的影响。