The Cross‐cultural Transportability of Situational Judgment Tests: How does a US‐based integrity situational judgment test fare in Spain?

Despite the globalization of HRM, there is a dearth of research on the potential use of contextualized selection instruments such as situational judgment tests (SJTs) in other countries than those where the selection instruments were originally developed. Therefore, two studies are conducted to examine the transportability of an integrity SJT that was originally developed in the United States to a Spanish context. Study 1 showed that most SJT scenarios (16 out of 19) that were developed in the United States were also considered realistic in a Spanish context. In Study 2, the item option endorsement patterns converged to the original scoring scheme, with the exception of two items. In addition, there were high correlations between the original US empirical scoring scheme and two empirical scoring schemes that were tailored to the Spanish context (i.e., mode consensus scoring and proportional consensus scoring). Finally, correlations between the SJT integrity scores and ratings on a self‐report integrity measure did not differ significantly from each other according to the type of scoring key (original US scoring vs. Spanish scoring keys). Overall, these results shed light on potential issues and solutions related to the cross‐cultural use of contextualized selection instruments such as SJTs.     

Knowing What NOT To Do Is a Critical Job Skill: Evidence from 10 different scoring methods

Situational judgment tests (SJTs) have become an increasingly important tool for predicting employee performance; however, at least two key areas warrant further investigation. First, prior studies of SJTs have generally relied on samples from the western world, leaving open the question of the validity of using SJTs in the developing world where the majority of the world's workforce resides. Second, there is currently no standardized, theoretically‐based method for the development and scoring of SJTs. Therefore, SJTs are highly domain‐specific and must be developed anew for each new context. We report the results of three studies, conducted in India, that aim to: (1) test the cross‐cultural validity of SJTs in a non‐western context, and (2) examine the differential validity of 10 different approaches to scoring SJTs, some of which have the potential to resolve the problem of developing a theoretically‐infused, standardized approach to scoring and future development.     

Use of Situational Judgment Tests in Personnel Selection: Are the different methods for scoring the response options equivalent?

The different methods used to score the response options in situational judgment tests (SJTs) carried out as part of the personnel selection process were compared by creating different keys for a single SJT, and the potential benefits of an innovative method combining existing methods were examined. The results, based on a sample of 1,194 candidates, point to some interesting differences between scoring methods. First, the innovative method created the lowest mean, near 60%. Second, the single‐best‐answer method produced the largest variance. The curve of the rank‐ordering method was the closest to a normal distribution. Finally, evidence suggests that the best‐and‐worst‐answer method and the innovative method provide the best results regarding construct validity. In sum, although no clear conclusion could be drawn about which methods should be preferred to score SJTs, results indicate that the new method could prove to be very interesting.     

The State of Research on Situational Judgment Tests: A Content Analysis and Directions for Future Research

Situational judgment tests (SJTs) have become popular selection methods, with 59 empirical studies having been published since 1990. In contrast to prior narrative reviews or meta-analyses, this study develops (a) a comprehensive structure of SJT features, or “attributes,” (b) uses this structure to quantitatively and qualitatively summarize existing research in a content analysis, and then (c) uses the content analysis to generate directions for future research. The review is organized around a single question: What are the current practices in SJT research? Using this question as a foundation, we focus the content analysis on three significant theoretical and practical themes: (a) SJT development, scoring methods, and uses; (2) types of reliability estimates reported for SJTs, and (3) attributes that enhance or reduce internal consistency reliability.     

Toward an understanding of situational judgment item validity and group differences

This paper evaluates 2 adjustments to common scoring approaches for situational judgment tests (SJTs). These adjustments can result in substantial improvements to item validity, reductions in mean racial differences, and resistance to coaching designed to improve scores. The first adjustment, applicable to SJTs that use Likert scales, controls for elevation and scatter (Cronbach & Gleser, 1953). This adjustment improves item validity. Also, because there is a White-Black mean difference in the preference for extreme responses on Likert scales (Bachman & O'Malley, 1984), these adjustments substantially reduce White-Black mean score differences. Furthermore, this adjustment often eliminates the score elevation associated with the coaching strategy of avoiding extreme responses (Cullen, Sackett, & Lievens, 2006). Item validity is shown to have a U-shaped relationship with item means. This holds both for SJTs with Likert score response formats and for SJTs where respondents identify the best and worst response option. Given the U-shaped relationship, the second adjustment is to drop items with midrange item means. This permits the SJT to be shortened, sometimes dramatically, without necessarily harming validity.     

Prosocial implicit trait policies underlie performance on different situational judgment tests with interpersonal content

ABSTRACT

This study tests the hypothesis that situational judgment tests (SJTs) with interpersonal content reflect implicit beliefs about the utility of prosocial action for job effectiveness and that agreeable people are more likely to believe that prosocial action is effective. Two hundred ninety-four undergraduates completed four different SJTs with interpersonal content and a measure of Agreeableness. Results show that the effectiveness of response options in these SJTs is positively correlated with the level of prosociality they express and that because of their prosocial elements, scores on different SJTs are correlated with one another and with Agreeableness. These results shed light on the construct-related validity of SJTs with interpersonal content and point to the possibility that they can assess prosociality in job settings different from those described in their items.     

Specification of Scales in Biodata Form Development: Rational vs. Empirical and Global vs. Specific

Although empirical keying has been the most popular scoring strategy with biodata, researchers have increasingly argued that rational approaches are better for advancing theory. Higher validities and less faking with empirical keys, however, have made many reluctant to abandon them. Research in the personality field provided support for the notion that many rational biodata scales may be multi-faceted. Development of more specific scales was suggested as a technique for creating rational scales with validities that more closely approximate that of empirical keys. Three different strategies (rational, internal, and external/empirical) were used in the scoring of a biodata inventory for use with mechanical equipment franchise owners. The rational and internal approaches were investigated within two samples and at two levels - very specific constructs and global constructs. In addition, two types of criteria were used, including an objective measure of sales and supervisory performance ratings. The specific rational scales were as predictive as the empirical item key. Strategies for developing and scoring a biodata form using a rational approach are discussed.     

Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis

In this article, we describe a systematic review of mean race and sex differences in situational judgment test (SJT) performance. On average, White test takers perform better on SJTs than Black, Hispanic, and Asian test takers. Female examinees perform slightly better than male test takers on SJTs. We investigate two moderators of these differences: loading of g or personality on the SJT, and response instructions. Mean race differences between Black, Hispanic, Asian and White examinees in SJT performance are largely explained by the cognitive loading of the SJT such that the larger the cognitive load, the larger the mean race differences. Regarding the effect of personality loadings of SJTs on race differences, Black–White and Asian–White differences are smaller to the extent that the SJT is correlated with emotional stability and Hispanic–White differences are smaller to the extent that SJTs are correlated with conscientiousness and agreeableness. Cognitive loading has minimal effect on male–female SJT score differences; however, SJT score differences are larger, favoring women, when SJTs are correlated with conscientiousness and agreeableness. Concerning response instructions, knowledge response instructions showed greater race differences than behavioral tendency instructions. The mean correlations show that these differences are largely because of the greater g loading of knowledge instructions. A second study showed that when used in hiring, SJTs are likely to have adverse impact by race but not by sex.     

The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection

Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.     

On the Limitations of Using Situational Judgment Tests to Measure Interpersonal Skills: The Moderating Influence of Employee Anger

Many authors have suggested that situational judgment tests (SJTs) are useful tools for assessing applicants because SJT items can be written to assess a number of job‐related knowledges, skills, abilities and other characteristics (KSAOs). However, SJTs may not be appropriate for measuring certain KSAOs for some applicants. We posit that using SJTs to measure interpersonal skills may lead to invalid inferences about applicants with higher levels of angry hostility (AH), and thus, AH should moderate the relation between interpersonally oriented SJTs and job performance. Three studies, using samples of healthcare workers (n = 225), police officers (n = 54), and medical doctors (n = 92), provided support for hypotheses in that that relations between SJT scores and performance criteria were significantly weaker among employees higher in AH compared to those lower in AH. In addition, none of the other facets of neuroticism tested (self‐consciousness, anxiety, depression, immoderation, or vulnerability to stress) consistently moderated SJT validity, providing support for the uniqueness of AH. Implications for practice, and for future research studying the relations between interpersonal skills as measured by SJTs and job performance, are discussed.     

Estimating Trait and Situational Variance in a Situational Judgment Test

In organizational research, situational judgment tests (SJTs) consistently demonstrate incremental validity, yet our theoretical understanding of SJTs is limited. Our knowledge could be advanced by decomposing the variance of SJT items into trait variance and situation variance; we do that by applying statistical methods used to analyze multitrait–multimethod matrices. A college-student sample (N = 2,747) was administered an SJT of goal orientation traits (i.e., mastery, performance-approach, and performance-avoid). Structural equation modeling was used to estimate the proportions of item variance to attributable to situational differences (across students) and to trait-based differences in students (across situations). Situation factors accounted for over three times the amount of variance as did individual difference factors. We conclude with general implications for the design of SJTs in organizational research.     

Using Situational Judgment Tests to Assess Training Effectiveness: Lessons Learned Evaluating Military Equal Opportunity Advisor Trainees

Situational judgment tests (SJTs) are well suited for training evaluation, especially when measures of on-the-job behaviors/performances are unavailable. Yet, SJTs are underutilized in the training evaluation context. This research details the development and use of an SJT to evaluate equal opportunity/diversity training in the military. We focus on issues that differentiate the development of SJTs for training evaluation from the development of SJTs for selection. Finally, results are presented for two cycles of training evaluation using an SJT, and the strengths and limitations of using SJTs in the training context are discussed.     

Determinants of Two-Choice Reaction-Time Patterns for Same-Hand and Different-Hand Finger Pairings

Several studies of two-choice reaction times have compared situations in which the alternative responses are fingers from one hand (the same-hand pairing) or one finger from each hand (the different-hand pairing). Two patterns of results have been obtained: (a) equivalent reaction times for the same-hand and different-hand pairings and (b) faster reaction times for the different-hand pairing. Previously, these outcomes have been attributed to the adoption of different response-preparation strategies when response pairs are constant (low response-pair uncertainty) versus when they are varied from trial to trial (high response-pair uncertainty). However, response-pair uncertainty has been confounded with whether only the two relevant fingers were placed on response keys or whether more than two fingers were. Experiment 1 of the present study demonstrated that finger placement, rather than response-pair uncertainty, determines which reaction-time pattern is obtained. Experiments 2 and 3 investigated the nature of the finger-placement effect by placing the fingers that were irrelevant for the task on response keys or on immovable blocks. The experiments indicated that the crucial factor is the number of fingers on active response keys, with the type of preparation being different when only two fingers are on keys rather than when more than two fingers are.     

Determinants of two-choice reaction-time patterns for same-hand and different-hand finger pairings

Several studies of two-choice reaction times have compared situations in which the alternative responses are fingers from one hand (the same-hand pairing) or one finger from each hand (the different-hand pairing). Two patterns of results have been obtained: (a) equivalent reaction times for the same-hand and different-hand pairings and (b) faster reaction times for the different-hand pairing. Previously, these outcomes have been attributed to the adoption of different response-preparation strategies when response pairs are constant (low response-pair uncertainty) versus when they are varied from trial to trial (high response-pair uncertainty). However, response-pair uncertainty has been confounded with whether only the two relevant fingers were placed on response keys or whether more than two fingers were. Experiment 1 of the present study demonstrated that finger placement, rather than response-pair uncertainty, determines which reaction-time pattern is obtained. Experiments 2 and 3 investigated the nature of the finger-placement effect by placing the fingers that were irrelevant for the task on response keys or on immovable blocks. The experiments indicated that the crucial factor is the number of fingers on active response keys, with the type of preparation being different when only two fingers are on keys rather than when more than two fingers are.     

Elaboration et validation de tests de jugement situationnel comme outil de sélection professionnelle : guide méthodologique

Current views in personnel selection recognize the necessity for situational judgements tests (SJT) which are closed to the real work context of companies. The SJTs have also become a popular selection tool across Europe. SJTs are attractive because they show good validity, positive applicants reactions and can be cost effective to test large group of applicants at once. The aim of this paper is to describe a structured methodology for the elaboration of such test. Each stage is illustrated by examples and psychometrics validation issues are discussed.     

Impact of Elaboration on Responding to Situational Judgment Test Items

Although faking has been identified as a potential problem in situational judgment tests (SJTs), no studies have investigated proactive approaches for controlling faking in SJTs. Therefore, this study examined the impact of elaboration on responding to SJT items. Elaboration was operationalized as reason‐giving. Two hundred and forty‐seven master students were assigned to either an honest or a fake condition, and to a non‐elaboration or an elaboration condition. Results showed that elaboration decreased the effect of faking for items with high familiarity. Elaboration on familiar items also decreased the percentage of fakers in the top of the distribution. Next, participants in the elaboration condition rated the SJT significantly higher in terms of allowing them to present themselves more realistically and to demonstrate their knowledge, skills, and abilities. Finally, there were no significant differences in participants' satisfaction with the SJT across the elaboration and non‐elaboration condition.     

情境判断测验的效度研究述评

情境判断测验是一种为作答者呈现工作相关的典型情境以及该情境下可能的行为反应, 要求根据指导语提示进行选择或评价的测验形式。随着其理论和实践的发展, 研究者越来越关注情境判断测验的效度研究, 包括对其构想效度、效标关联效度和递增效度的探讨, 以及指导语类型、情境保真度以及计分方式等因素对其效度的影响。基于这些研究进展, 未来情境判断测验实践领域可能的方向是：(1)开发针对特定构想的情境判断测验; (2)结合具体构想选用相应的指导语; (3)应用作假和培训对效度影响的研究结果指导实践。     

Video-based versus written situational judgment tests: a comparison in terms of predictive validity

In this study, the authors examined whether video-based situational judgment tests (SJTs) have higher predictive validity than written SJTs (keeping verbal content constant). The samples consisted of 1,159 students who completed a video-based version of an SJT and 1,750 students who completed the same SJT in a written format. The study was conducted in a high stakes testing context. The video-based version of an interpersonally oriented SJT had a lower correlation with cognitive ability than did the written version. It also had higher predictive and incremental validity for predicting interpersonally oriented criteria than did the written version. In this high stakes context, applicants also reacted relatively favorably to the SJTs, although there was no significant difference in face validity between the formats. These findings suggest that SJT format changes be made with caution and that validation evidence is needed when changes are proposed.     

Derivation and application of a unit scoring system for the strong vocational interest blank for women

Scoring keys, based upon unit weights, were made up for fourteen occupations of theStrong Vocational Interest Blank for Women. The study here presented of scores obtained in using these keys, in comparison with scores obtained from Dr. Strong's keys, indicates, for 551 women at the University of Rochester, that the new, unit-weighted keys are valid for all practical purposes and make possible a great saving in scoring time.     

Graduate student WAIS-III scoring accuracy is a function of full scale IQ and complexity of examiner tasks

Research on the Wechsler Adult Intelligence Scale-Revised and Wechsler Adult Intelligence Scale-Third Edition (WAIS-III) suggests that practicing clinical psychologists and graduate students make item-level scoring errors that affect IQ, index, and subtest scores. Studies have been limited in that Full-Scale IQ (FSIQ) and examiner administration, recording, and scoring tasks have not been systematically varied. In this study, graduate student participants score a high (FSIQ = 112) and low (FSIQ = 85) IQ record form in one of two stimulus conditions: digitized film clips (N = 13) or partially completed record forms (N = 11). Results demonstrate that examiners are less accurate in the high IQ condition, and that recording examinee responses from scoring video clips results in more scoring errors. Obtained FSIQs are significantly higher than criterion IQ scores in the high IQ condition (8.46 for video condition, 2.55 for record form condition). Self-reported proficiency in WAIS-III administration and scoring is positively related to number of scoring errors.