The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection

Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.     

Faking on a situational judgment test in a medical school selection setting: Effect of different scoring methods?

We examined the occurrence of faking on a rating situational judgment test (SJT) by comparing SJT scores and response styles of the same individuals across two naturally occurring situations. An SJT for medical school selection was administered twice to the same group of applicants (N = 317) under low‐stakes (T1) and high‐stakes (T2) circumstances. The SJT was scored using three different methods that were differentially affected by response tendencies. Applicants used significantly more extreme responding on T2 than T1. Faking (higher SJT score on T2) was only observed for scoring methods that controlled for response tendencies. Scoring methods that do not control for response tendencies introduce systematic error into the SJT score, which may lead to inaccurate conclusions about the existence of faking.     

On the Limitations of Using Situational Judgment Tests to Measure Interpersonal Skills: The Moderating Influence of Employee Anger

Many authors have suggested that situational judgment tests (SJTs) are useful tools for assessing applicants because SJT items can be written to assess a number of job‐related knowledges, skills, abilities and other characteristics (KSAOs). However, SJTs may not be appropriate for measuring certain KSAOs for some applicants. We posit that using SJTs to measure interpersonal skills may lead to invalid inferences about applicants with higher levels of angry hostility (AH), and thus, AH should moderate the relation between interpersonally oriented SJTs and job performance. Three studies, using samples of healthcare workers (n = 225), police officers (n = 54), and medical doctors (n = 92), provided support for hypotheses in that that relations between SJT scores and performance criteria were significantly weaker among employees higher in AH compared to those lower in AH. In addition, none of the other facets of neuroticism tested (self‐consciousness, anxiety, depression, immoderation, or vulnerability to stress) consistently moderated SJT validity, providing support for the uniqueness of AH. Implications for practice, and for future research studying the relations between interpersonal skills as measured by SJTs and job performance, are discussed.     

A context-independent situational judgment test to measure prosocial implicit trait policy

Two studies developed and validated a context-independent situational judgment test (SJT) of prosocial implicit trait policy (ITP). The first study developed a SJT based on critical incidents about the prosocial behavior of physicians, lawyers, community service volunteers, and human factors engineers. In a sample of 396 undergraduates, scores obtained from this SJT were internally consistent and correlated significantly with other trait constructs related to prosocial ITP. In the second study with 134 undergraduates, the SJT was significantly correlated with relevant trait constructs and prosocial performance in role-plays simulating scenarios in which others needed help. These results show that a generic SJT developed from items that describe situations and actions specific to several occupations can predict behavior in situations unlike any that appear in its items.     

Development and psychometric examination of a German video‐based situational judgment test for social competencies in medical school applicants

The present study describes the development and validation of a video‐based situational judgment test (SJT) assessing social competencies in applicants to medical school. Study 1 explored the psychometric properties of the SJT based on two applicant cohorts (N₁ = 769, N₂ = 787). Study 2 compared SJT data from applicants and 90 medical students. Tests for mean differences, correlation and factor analyses were used. In Study 1, the supposed two‐factor model for the SJT must be rejected. According to the minimum average partial test a single factor solution explaining 19–21% of the variance was suggested. The SJT showed satisfactory psychometric properties and stronger correlations to personality than to cognition as well as high acceptance. In study 2, applicants reached higher SJT scores than students.     

Measuring Relationships between Personality,Knowledge, and Performance using Single‐response Situational Judgment Tests

We report two studies that investigate single‐response situational judgment tests (SJTs) as measures of job knowledge. Study 1 examines relationships between job knowledge measured by a single‐response SJT, personality, and performance for museum tour guides. Study 2 extends Study 1's findings with a sample of volunteers using a single‐response SJT about volunteerism. In both studies, personality was related to knowledge, and knowledge predicted performance. In Study 2, knowledge accounted for incremental variance in performance beyond personality, but personality added no incremental variance beyond knowledge. Results suggest that knowledge of effective behavior and knowledge of ineffective behavior are separate constructs. These studies demonstrate the validity of single‐response SJTs and provide evidence that knowing what to do and what not to do are separate domains.     

Development and Validation of a Situational Judgment Test of Emotional Intelligence

The present study describes the development and validation of a situational judgment test (SJT) of emotional intelligence (EI). Initially, 80 situations and three response alternatives for each situation were created based on the available theoretical models. Principal component factor analysis with direct oblimin rotation of data (N = 213) yielded a three‐factor structure with 46 items. These factors were (1) utilizing own emotion, (2) sensing other's emotion, and (3) understanding emotional context. Additional studies showed that the measure had good internal consistency and test‐retest reliability. None of the three factors strongly correlated with the Big Five factors of personality (NEO Five‐Factor Inventory, thus establishing its identity as a construct distinct from personality. Findings of confirmatory factor analysis on secondary data reconfirm the three‐factor model for a 46‐item SJT of EI. The second study also found no correlation among these three factors, intelligence scores measured using Raven's Matrices, and trait EI score measured using the Trait Emotional Intelligence Questionnaire. The third study was conducted in order to determine the relationship of SJT of EI with academic achievement and life satisfaction. All three factors of SJT‐based EI measure were significantly associated with academic achievement and life satisfaction.     

THE SUSCEPTIBILITY OF OVERT AND COVERT INTEGRITY TESTS TO COACHING AND FAKING

Abstract— Although previous research has indicated that faking can affect integrity test scores, the effects of coaching on integrity test scores have never been examined We conducted a between-subjects experiment to assess the effects of coaching and faking instructions on an overt and a covert integrity test Coaching provided simple rules to follow when answering test items and instructions on how to avoid elevated validity scale scores There were five instruction conditions "just take,""fake good,""coach overt,""coach covert," and coach both All subjects completed both overt and covert tests and a measure of intelligence Results provided strong evidence for the coachability of the overt integrity test, over and above the much smaller elevation in the faking condition The covert test apparently could be neither coached nor faked successfully Scores on both integrity tests tended to be positively correlated with intelligence in the coaching and faking conditions We discuss the generalizability of these results to other samples and other integrity tests, and the relevance of the coachability of integrity tests to the ongoing debate concerning the prediction of counterproductive behavior     

The Cross‐cultural Transportability of Situational Judgment Tests: How does a US‐based integrity situational judgment test fare in Spain?

Despite the globalization of HRM, there is a dearth of research on the potential use of contextualized selection instruments such as situational judgment tests (SJTs) in other countries than those where the selection instruments were originally developed. Therefore, two studies are conducted to examine the transportability of an integrity SJT that was originally developed in the United States to a Spanish context. Study 1 showed that most SJT scenarios (16 out of 19) that were developed in the United States were also considered realistic in a Spanish context. In Study 2, the item option endorsement patterns converged to the original scoring scheme, with the exception of two items. In addition, there were high correlations between the original US empirical scoring scheme and two empirical scoring schemes that were tailored to the Spanish context (i.e., mode consensus scoring and proportional consensus scoring). Finally, correlations between the SJT integrity scores and ratings on a self‐report integrity measure did not differ significantly from each other according to the type of scoring key (original US scoring vs. Spanish scoring keys). Overall, these results shed light on potential issues and solutions related to the cross‐cultural use of contextualized selection instruments such as SJTs.     

Training teamwork in medicine: An active approach using role play and feedback

Recent reports in the field of medicine have recommended teamwork training as method of reducing human error. The current study evaluated a teamwork training protocol that used a lecture, several role plays, and guided discussion for feedback. Forty‐one medical residents participated in the training on 1 of 2 days (groups) and completed evaluation measures both prior to and immediately following the training program. Means for a situational judgment test (SJT) were higher for post‐training scores than pre‐training scores. However, mean comparisons with teamwork attitudes and learning goal orientation did not yield significant differences. An item analysis of the SJT responses indicated significant response shifts that corresponded to teamwork behaviors. Results suggested that teamwork training positively affected behavioral intentions for trauma room teamwork.     

Order Effects on Situational Judgment Test Items: A case of construct‐irrelevant difficulty

Situational judgment tests (SJTs) pose unique cognitive demands on test takers in that, when presented in written form, they require a great deal of reading and cognitive effort. Because of this cognitive demand, responses to test items toward the end of the test may be influenced by an order effect produced by responding to a large quantity of previous test items. This construct‐irrelevant order effect may increase measurement error and threaten the validity of SJT scores. To test this phenomenon, data were obtained from 1,089 applicants who had completed a lengthy SJT as part of a selection process for an hourly safety and surveillance job at a large international corporation. Results showed that local item dependence, item difficulty, and the rate of omitted responses all increased when items were placed toward the end. The order effect alone was not strong enough to influence subgroup mean score differences in the second half of the test. However, this effect did vary by race: African‐Americans were most strongly affected by the order effect, followed by Caucasians, in their number of omitted responses. Implications and future research of this effect for SJTs and similar types of assessments are discussed.     

Measuring Procedural Knowledge More Simply with a Single-Response Situational Judgment Test

Purpose  This paper describes the development of a situational judgment test (SJT) based on single-response options developed directly from critical incidents and reports a study that tested the SJT’s concurrent validity against ratings of job performance. Design/Methodology/Approach  Situational judgment test items were developed from critical incidents provided by administrators of volunteer agencies. Volunteers who worked at another agency completed the SJT and a self-report personality test. Supervisors rated their job performance on three dimensions. Findings  Situational judgment test scores representing procedural knowledge about work effort were significantly correlated with ratings of work effort performance (r = .28). Conscientiousness was correlated with work effort knowledge (r = .26), but not with work effort performance (r = −.02). Implications  These results provide some preliminary evidence that a single-response SJT format based upon critical incidents can produce valid measures of procedural knowledge and might be a useful alternative to the traditional multiple-response format. Originality/Value  This paper presents a novel way to construct SJTs using single-response options that are less laborious to develop than the multiple-response options in traditional SJT formats. Results of the validity study suggest that this new single-response format can predict job performance and encourage further research on the viability of this approach.     

FURTHER STUDIES OF SITUATIONAL TESTS

In the present paper, results are reported for two different situational judgment tests (SJT) used in validation studies with almost 4,000 employees in 7 different organizations. Across the 2 studies, it was shown that situational test scores were significantly related to performance (weighted average r = .19), cognitive ability (weighted average r = .45), and experience (weighted average r = .20). In one study, there was a slight tendency for experience and cognitive ability to interact in the prediction of situational judgment, such that cognitive ability became less predictive as experience increased. Situational judgment fully mediated the effects of cognitive ability in one study, but not in the other. Finally, SJT race effect sizes were consistent with past research and, while not trivial, smaller than those typically observed for cognitive ability tests. The studies yielded very similar results despite the use of different keying methods (empirical vs. subject matter expert opinion). The preponderance of the evidence indicates that situational judgment measures mediate avariety of job relevant skills. Limitations to these studies and directions for future research are reviewed.     

Optimizing Prediction of Attrition With the U.S. Army's Assessment of Individual Motivation (AIM)

The regression framework is often the method of choice used by psychologists for predicting organizationally relevant outcomes from test scores. However, alternatives to regression exist, and these techniques may provide better prediction of outcomes and a more effective means of classifying examinees for selection and placement. This research describes two of these alternatives—decision tree methodology and optimal appropriateness measurement (OAM)—and how they were used to optimize the prediction of attrition among a sample of first-term enlisted soldiers (N?=?22,537) using a temperament inventory called the Assessment of Individual Motivation (AIM). Results demonstrated that the OAM approach provided better differentiation between “stayers” and “leavers” after 12 months than either the traditional logistic regression or the decision tree methods.     

Validity evidence for the situational judgment test paradigm in emotional intelligence measurement

To date, various measurement approaches have been proposed to assess emotional intelligence (EI). Recently, two new EI tests have been developed based on the situational judgment test (SJT) paradigm: the Situational Test of Emotional Understanding (STEU) and the Situational Test of Emotion Management (STEM). Initial attempts have been made to examine the construct‐related validity of these new tests; we extend these findings by placing the tests in a broad nomological network. To this end, 850 undergraduate students completed a personality inventory, a cognitive ability test, a self‐report EI test, a performance‐based EI measure, the STEU, and the STEM. The SJT‐based EI tests were not strongly correlated with personality and fluid cognitive ability. Regarding their relation with existing EI measures, the tests did not capture the same construct as self‐report EI measures, but corresponded rather to performance‐based EI measures. Overall, these results lend support for the SJT paradigm for measuring EI as an ability.     

Effects of Response Instructions on Faking a Situational Judgment Test

A situational judgment test (SJT) and a Big 5 personality test were administered to 203 participants under instructions to respond honestly and to fake good using a within‐subjects design. Participants indicated both the best and worst response (i.e., Knowledge) and the most likely and least likely response (i.e., Behavioral Tendency) to each situation. Faking effect size for the SJT Behavioral Tendency response format was (d=.34) when participants responded first under honest instructions and (d=.15) when they responded first under faking instructions. Those for the Big 5 dimensions ranged from d=.26 to d=1.0. For the Knowledge response format results were inconsistent. Honest condition Knowledge SJT scores were more highly correlated with cognitive ability (r=.56) than were Behavioral Tendency SJT scores (r=.38). Implications for researchers and practitioners are discussed.     

The Effects of Coaching and Speeding on Big Five and Impression Management Scale Scores

We examined the effects of coaching and speeding on personality scale scores in a faking context (N = 192). A completely crossed 2 × 2 experimental design was used in which instructions (no coaching or coaching) and speeding (with or without a time limit) were manipulated. No statistically significant effects on scale scores were evidenced for speeding. Coaching participants significantly elevated scores (average d = .76) for each of the Big Five personality factors but did not significantly elevate the scores on the Impression Management scale (d = .06). Cognitive ability was significantly positively related to impression management for uncoached participants but not for coached participants. An exploratory simulation suggests that coaching would have an effect on who would be selected for a job.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.