Video-based versus written situational judgment tests: a comparison in terms of predictive validity

In this study, the authors examined whether video-based situational judgment tests (SJTs) have higher predictive validity than written SJTs (keeping verbal content constant). The samples consisted of 1,159 students who completed a video-based version of an SJT and 1,750 students who completed the same SJT in a written format. The study was conducted in a high stakes testing context. The video-based version of an interpersonally oriented SJT had a lower correlation with cognitive ability than did the written version. It also had higher predictive and incremental validity for predicting interpersonally oriented criteria than did the written version. In this high stakes context, applicants also reacted relatively favorably to the SJTs, although there was no significant difference in face validity between the formats. These findings suggest that SJT format changes be made with caution and that validation evidence is needed when changes are proposed.     

Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis

In this article, we describe a systematic review of mean race and sex differences in situational judgment test (SJT) performance. On average, White test takers perform better on SJTs than Black, Hispanic, and Asian test takers. Female examinees perform slightly better than male test takers on SJTs. We investigate two moderators of these differences: loading of g or personality on the SJT, and response instructions. Mean race differences between Black, Hispanic, Asian and White examinees in SJT performance are largely explained by the cognitive loading of the SJT such that the larger the cognitive load, the larger the mean race differences. Regarding the effect of personality loadings of SJTs on race differences, Black–White and Asian–White differences are smaller to the extent that the SJT is correlated with emotional stability and Hispanic–White differences are smaller to the extent that SJTs are correlated with conscientiousness and agreeableness. Cognitive loading has minimal effect on male–female SJT score differences; however, SJT score differences are larger, favoring women, when SJTs are correlated with conscientiousness and agreeableness. Concerning response instructions, knowledge response instructions showed greater race differences than behavioral tendency instructions. The mean correlations show that these differences are largely because of the greater g loading of knowledge instructions. A second study showed that when used in hiring, SJTs are likely to have adverse impact by race but not by sex.     

Estimating Trait and Situational Variance in a Situational Judgment Test

In organizational research, situational judgment tests (SJTs) consistently demonstrate incremental validity, yet our theoretical understanding of SJTs is limited. Our knowledge could be advanced by decomposing the variance of SJT items into trait variance and situation variance; we do that by applying statistical methods used to analyze multitrait–multimethod matrices. A college-student sample (N = 2,747) was administered an SJT of goal orientation traits (i.e., mastery, performance-approach, and performance-avoid). Structural equation modeling was used to estimate the proportions of item variance to attributable to situational differences (across students) and to trait-based differences in students (across situations). Situation factors accounted for over three times the amount of variance as did individual difference factors. We conclude with general implications for the design of SJTs in organizational research.     

Using Situational Judgment Tests to Assess Training Effectiveness: Lessons Learned Evaluating Military Equal Opportunity Advisor Trainees

Situational judgment tests (SJTs) are well suited for training evaluation, especially when measures of on-the-job behaviors/performances are unavailable. Yet, SJTs are underutilized in the training evaluation context. This research details the development and use of an SJT to evaluate equal opportunity/diversity training in the military. We focus on issues that differentiate the development of SJTs for training evaluation from the development of SJTs for selection. Finally, results are presented for two cycles of training evaluation using an SJT, and the strengths and limitations of using SJTs in the training context are discussed.     

Be Careful What You Ask For: Effects of Response Instructions on the Construct Validity and Reliability of Situational Judgment Tests

The aim of this study was to examine how six different types of situational judgment test (SJT) instructions, used frequently in practice, influence the psychometric characteristics of SJTs. The six SJT versions used the exact same items and differed only in their instructions; these versions were administered in two phases. Phase I was a between–subjects design (n= 486) that had participants complete one version of the SJTs. Phase II was a within–subjects design (n= 231) held several weeks later that had participants complete all six versions of the SJTs. Further, 146 of these individuals completed both phases, allowing for an assessment of test–retest reliability. A variety of objective and subjective criteria were collected, including self and peer ratings. Results indicated that instructions had a large effect on SJT responses, reliability, and validity. In general, instructions asking what one ‘would do’ showed more favorable characteristics than those that asked what one ‘should do’. Correlations between these two types were relatively low despite the fact that the same items were used, and criterion–related validities differed substantially in favor of the ‘would do’ instructions. Overall, this study finds that researchers and practitioners must give careful consideration to the types of SJT instructions used; failing to do so could influence criterion–related validity and cloud inferences of construct validity.     

On the Limitations of Using Situational Judgment Tests to Measure Interpersonal Skills: The Moderating Influence of Employee Anger

Many authors have suggested that situational judgment tests (SJTs) are useful tools for assessing applicants because SJT items can be written to assess a number of job‐related knowledges, skills, abilities and other characteristics (KSAOs). However, SJTs may not be appropriate for measuring certain KSAOs for some applicants. We posit that using SJTs to measure interpersonal skills may lead to invalid inferences about applicants with higher levels of angry hostility (AH), and thus, AH should moderate the relation between interpersonally oriented SJTs and job performance. Three studies, using samples of healthcare workers (n = 225), police officers (n = 54), and medical doctors (n = 92), provided support for hypotheses in that that relations between SJT scores and performance criteria were significantly weaker among employees higher in AH compared to those lower in AH. In addition, none of the other facets of neuroticism tested (self‐consciousness, anxiety, depression, immoderation, or vulnerability to stress) consistently moderated SJT validity, providing support for the uniqueness of AH. Implications for practice, and for future research studying the relations between interpersonal skills as measured by SJTs and job performance, are discussed.     

Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations

Assessing reliability of situational judgment tests (SJTs) in high‐stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of α = .46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT test–retest reliability was r = .82, compared with internal consistency, α = .46, and stratified alpha, α = .45 at Time 1 and α = .52 and stratified α = .51 at Time 2. We used a student sample (Time 1: n = 185; Time 2: n = 132) with items from a credentialing exam with ‘should do’ instructions. The SJT correlated significantly with cognitive ability, r = .30, and agreeableness, r = .24. In Study 2, we assessed test–retest reliability with Human Resource professionals (Time 1: n = 94; Time 2: n = 32) who had been recently credentialed and who participated in a pilot test of new SJT items with ‘most likely/least likely do’ response options. The SJT test–retest reliability was r = .66 compared with internal consistency, α = .43 and stratified α = .47 at Time 1 and α = .61 and stratified α = .67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.     

The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection

Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.     

Scoring Situational Judgment Tests: Once You Get the Data,Your Troubles Begin

Although situational judgment tests (SJTs) have been in use for decades, consensus has not been reached on the best way to score these assessments or others (e.g., biodata) whose items do not have a single demonstrably correct answer. The purpose of this paper is to review and to demonstrate the scoring strategies that have been described in the literature. Implementation and relative merits of these strategies are described. Then, several of these methods are applied to create 11 different keys for a video‐based SJT in order to demonstrate how to evaluate the quality of keys. Implications of scoring SJTs for theory and practice are discussed.     

Order Effects on Situational Judgment Test Items: A case of construct‐irrelevant difficulty

Situational judgment tests (SJTs) pose unique cognitive demands on test takers in that, when presented in written form, they require a great deal of reading and cognitive effort. Because of this cognitive demand, responses to test items toward the end of the test may be influenced by an order effect produced by responding to a large quantity of previous test items. This construct‐irrelevant order effect may increase measurement error and threaten the validity of SJT scores. To test this phenomenon, data were obtained from 1,089 applicants who had completed a lengthy SJT as part of a selection process for an hourly safety and surveillance job at a large international corporation. Results showed that local item dependence, item difficulty, and the rate of omitted responses all increased when items were placed toward the end. The order effect alone was not strong enough to influence subgroup mean score differences in the second half of the test. However, this effect did vary by race: African‐Americans were most strongly affected by the order effect, followed by Caucasians, in their number of omitted responses. Implications and future research of this effect for SJTs and similar types of assessments are discussed.     

Impact of Elaboration on Responding to Situational Judgment Test Items

Although faking has been identified as a potential problem in situational judgment tests (SJTs), no studies have investigated proactive approaches for controlling faking in SJTs. Therefore, this study examined the impact of elaboration on responding to SJT items. Elaboration was operationalized as reason‐giving. Two hundred and forty‐seven master students were assigned to either an honest or a fake condition, and to a non‐elaboration or an elaboration condition. Results showed that elaboration decreased the effect of faking for items with high familiarity. Elaboration on familiar items also decreased the percentage of fakers in the top of the distribution. Next, participants in the elaboration condition rated the SJT significantly higher in terms of allowing them to present themselves more realistically and to demonstrate their knowledge, skills, and abilities. Finally, there were no significant differences in participants' satisfaction with the SJT across the elaboration and non‐elaboration condition.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.     

The Cross‐cultural Transportability of Situational Judgment Tests: How does a US‐based integrity situational judgment test fare in Spain?

Despite the globalization of HRM, there is a dearth of research on the potential use of contextualized selection instruments such as situational judgment tests (SJTs) in other countries than those where the selection instruments were originally developed. Therefore, two studies are conducted to examine the transportability of an integrity SJT that was originally developed in the United States to a Spanish context. Study 1 showed that most SJT scenarios (16 out of 19) that were developed in the United States were also considered realistic in a Spanish context. In Study 2, the item option endorsement patterns converged to the original scoring scheme, with the exception of two items. In addition, there were high correlations between the original US empirical scoring scheme and two empirical scoring schemes that were tailored to the Spanish context (i.e., mode consensus scoring and proportional consensus scoring). Finally, correlations between the SJT integrity scores and ratings on a self‐report integrity measure did not differ significantly from each other according to the type of scoring key (original US scoring vs. Spanish scoring keys). Overall, these results shed light on potential issues and solutions related to the cross‐cultural use of contextualized selection instruments such as SJTs.     

Elaboration et validation de tests de jugement situationnel comme outil de sélection professionnelle : guide méthodologique

Current views in personnel selection recognize the necessity for situational judgements tests (SJT) which are closed to the real work context of companies. The SJTs have also become a popular selection tool across Europe. SJTs are attractive because they show good validity, positive applicants reactions and can be cost effective to test large group of applicants at once. The aim of this paper is to describe a structured methodology for the elaboration of such test. Each stage is illustrated by examples and psychometrics validation issues are discussed.     

Use of Situational Judgment Tests in Personnel Selection: Are the different methods for scoring the response options equivalent?

The different methods used to score the response options in situational judgment tests (SJTs) carried out as part of the personnel selection process were compared by creating different keys for a single SJT, and the potential benefits of an innovative method combining existing methods were examined. The results, based on a sample of 1,194 candidates, point to some interesting differences between scoring methods. First, the innovative method created the lowest mean, near 60%. Second, the single‐best‐answer method produced the largest variance. The curve of the rank‐ordering method was the closest to a normal distribution. Finally, evidence suggests that the best‐and‐worst‐answer method and the innovative method provide the best results regarding construct validity. In sum, although no clear conclusion could be drawn about which methods should be preferred to score SJTs, results indicate that the new method could prove to be very interesting.     

Measuring Relationships between Personality,Knowledge, and Performance using Single‐response Situational Judgment Tests

We report two studies that investigate single‐response situational judgment tests (SJTs) as measures of job knowledge. Study 1 examines relationships between job knowledge measured by a single‐response SJT, personality, and performance for museum tour guides. Study 2 extends Study 1's findings with a sample of volunteers using a single‐response SJT about volunteerism. In both studies, personality was related to knowledge, and knowledge predicted performance. In Study 2, knowledge accounted for incremental variance in performance beyond personality, but personality added no incremental variance beyond knowledge. Results suggest that knowledge of effective behavior and knowledge of ineffective behavior are separate constructs. These studies demonstrate the validity of single‐response SJTs and provide evidence that knowing what to do and what not to do are separate domains.     

Toward an understanding of situational judgment item validity and group differences

This paper evaluates 2 adjustments to common scoring approaches for situational judgment tests (SJTs). These adjustments can result in substantial improvements to item validity, reductions in mean racial differences, and resistance to coaching designed to improve scores. The first adjustment, applicable to SJTs that use Likert scales, controls for elevation and scatter (Cronbach & Gleser, 1953). This adjustment improves item validity. Also, because there is a White-Black mean difference in the preference for extreme responses on Likert scales (Bachman & O'Malley, 1984), these adjustments substantially reduce White-Black mean score differences. Furthermore, this adjustment often eliminates the score elevation associated with the coaching strategy of avoiding extreme responses (Cullen, Sackett, & Lievens, 2006). Item validity is shown to have a U-shaped relationship with item means. This holds both for SJTs with Likert score response formats and for SJTs where respondents identify the best and worst response option. Given the U-shaped relationship, the second adjustment is to drop items with midrange item means. This permits the SJT to be shortened, sometimes dramatically, without necessarily harming validity.