Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations

Assessing reliability of situational judgment tests (SJTs) in high‐stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of α = .46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT test–retest reliability was r = .82, compared with internal consistency, α = .46, and stratified alpha, α = .45 at Time 1 and α = .52 and stratified α = .51 at Time 2. We used a student sample (Time 1: n = 185; Time 2: n = 132) with items from a credentialing exam with ‘should do’ instructions. The SJT correlated significantly with cognitive ability, r = .30, and agreeableness, r = .24. In Study 2, we assessed test–retest reliability with Human Resource professionals (Time 1: n = 94; Time 2: n = 32) who had been recently credentialed and who participated in a pilot test of new SJT items with ‘most likely/least likely do’ response options. The SJT test–retest reliability was r = .66 compared with internal consistency, α = .43 and stratified α = .47 at Time 1 and α = .61 and stratified α = .67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.     

Be Careful What You Ask For: Effects of Response Instructions on the Construct Validity and Reliability of Situational Judgment Tests

The aim of this study was to examine how six different types of situational judgment test (SJT) instructions, used frequently in practice, influence the psychometric characteristics of SJTs. The six SJT versions used the exact same items and differed only in their instructions; these versions were administered in two phases. Phase I was a between–subjects design (n= 486) that had participants complete one version of the SJTs. Phase II was a within–subjects design (n= 231) held several weeks later that had participants complete all six versions of the SJTs. Further, 146 of these individuals completed both phases, allowing for an assessment of test–retest reliability. A variety of objective and subjective criteria were collected, including self and peer ratings. Results indicated that instructions had a large effect on SJT responses, reliability, and validity. In general, instructions asking what one ‘would do’ showed more favorable characteristics than those that asked what one ‘should do’. Correlations between these two types were relatively low despite the fact that the same items were used, and criterion–related validities differed substantially in favor of the ‘would do’ instructions. Overall, this study finds that researchers and practitioners must give careful consideration to the types of SJT instructions used; failing to do so could influence criterion–related validity and cloud inferences of construct validity.     

Measuring Relationships between Personality,Knowledge, and Performance using Single‐response Situational Judgment Tests

We report two studies that investigate single‐response situational judgment tests (SJTs) as measures of job knowledge. Study 1 examines relationships between job knowledge measured by a single‐response SJT, personality, and performance for museum tour guides. Study 2 extends Study 1's findings with a sample of volunteers using a single‐response SJT about volunteerism. In both studies, personality was related to knowledge, and knowledge predicted performance. In Study 2, knowledge accounted for incremental variance in performance beyond personality, but personality added no incremental variance beyond knowledge. Results suggest that knowledge of effective behavior and knowledge of ineffective behavior are separate constructs. These studies demonstrate the validity of single‐response SJTs and provide evidence that knowing what to do and what not to do are separate domains.     

Development and psychometric examination of a German video‐based situational judgment test for social competencies in medical school applicants

The present study describes the development and validation of a video‐based situational judgment test (SJT) assessing social competencies in applicants to medical school. Study 1 explored the psychometric properties of the SJT based on two applicant cohorts (N₁ = 769, N₂ = 787). Study 2 compared SJT data from applicants and 90 medical students. Tests for mean differences, correlation and factor analyses were used. In Study 1, the supposed two‐factor model for the SJT must be rejected. According to the minimum average partial test a single factor solution explaining 19–21% of the variance was suggested. The SJT showed satisfactory psychometric properties and stronger correlations to personality than to cognition as well as high acceptance. In study 2, applicants reached higher SJT scores than students.     

Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis

In this article, we describe a systematic review of mean race and sex differences in situational judgment test (SJT) performance. On average, White test takers perform better on SJTs than Black, Hispanic, and Asian test takers. Female examinees perform slightly better than male test takers on SJTs. We investigate two moderators of these differences: loading of g or personality on the SJT, and response instructions. Mean race differences between Black, Hispanic, Asian and White examinees in SJT performance are largely explained by the cognitive loading of the SJT such that the larger the cognitive load, the larger the mean race differences. Regarding the effect of personality loadings of SJTs on race differences, Black–White and Asian–White differences are smaller to the extent that the SJT is correlated with emotional stability and Hispanic–White differences are smaller to the extent that SJTs are correlated with conscientiousness and agreeableness. Cognitive loading has minimal effect on male–female SJT score differences; however, SJT score differences are larger, favoring women, when SJTs are correlated with conscientiousness and agreeableness. Concerning response instructions, knowledge response instructions showed greater race differences than behavioral tendency instructions. The mean correlations show that these differences are largely because of the greater g loading of knowledge instructions. A second study showed that when used in hiring, SJTs are likely to have adverse impact by race but not by sex.     

Measuring Procedural Knowledge More Simply with a Single-Response Situational Judgment Test

Purpose  This paper describes the development of a situational judgment test (SJT) based on single-response options developed directly from critical incidents and reports a study that tested the SJT’s concurrent validity against ratings of job performance. Design/Methodology/Approach  Situational judgment test items were developed from critical incidents provided by administrators of volunteer agencies. Volunteers who worked at another agency completed the SJT and a self-report personality test. Supervisors rated their job performance on three dimensions. Findings  Situational judgment test scores representing procedural knowledge about work effort were significantly correlated with ratings of work effort performance (r = .28). Conscientiousness was correlated with work effort knowledge (r = .26), but not with work effort performance (r = −.02). Implications  These results provide some preliminary evidence that a single-response SJT format based upon critical incidents can produce valid measures of procedural knowledge and might be a useful alternative to the traditional multiple-response format. Originality/Value  This paper presents a novel way to construct SJTs using single-response options that are less laborious to develop than the multiple-response options in traditional SJT formats. Results of the validity study suggest that this new single-response format can predict job performance and encourage further research on the viability of this approach.     

An Examination of the Validity and Incremental Value of Needed‐at‐Entry Ratings for a Customer Service Job

We examined needed‐at‐entry ratings of knowledge, skills, abilities, and other characteristics (KSAOs) provided by job experts (N= 285) during a job analysis for a customer service manager position. To serve as an external referent, a group of industrial and organisational psychologists (N= 31) rated the perceived trainability of each KSAO. Analyses revealed only limited support for the validity of inferences drawn from job experts' needed‐at‐entry ratings. Consistent with our hypotheses, less validity evidence was found for ratings of the more person‐oriented and abstract “AO” attributes than for the more job‐oriented and concrete “KS” attributes. In general, job experts tended to rate attributes as needed‐at‐entry that psychologists thought could be developed on the job. We also found that failing to collect needed‐at‐entry ratings, and relying on only the more common importance‐to‐the‐job ratings, would have resulted in a rather different set of critical KSAOs identified for assessment during the selection process.     

Video-based versus written situational judgment tests: a comparison in terms of predictive validity

In this study, the authors examined whether video-based situational judgment tests (SJTs) have higher predictive validity than written SJTs (keeping verbal content constant). The samples consisted of 1,159 students who completed a video-based version of an SJT and 1,750 students who completed the same SJT in a written format. The study was conducted in a high stakes testing context. The video-based version of an interpersonally oriented SJT had a lower correlation with cognitive ability than did the written version. It also had higher predictive and incremental validity for predicting interpersonally oriented criteria than did the written version. In this high stakes context, applicants also reacted relatively favorably to the SJTs, although there was no significant difference in face validity between the formats. These findings suggest that SJT format changes be made with caution and that validation evidence is needed when changes are proposed.     

The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection

Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.     

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

To promote diversity in organizations it is important to have accurate knowledge about subgroup differences associated with selection procedures. However, current estimates of subgroup differences in situational judgment tests (SJTs) are overwhelmingly based on range‐restricted incumbent samples that are downwardly biased. This study provides much‐needed applicant level estimates of SJT subgroup differences (N = 37,530). As a key finding, Black‐White differences (d = 0.66) were higher than in incumbent samples (d = 0.38). Overall, sex differences were small. Females scored higher for management jobs (d = ?0.13) and males scored higher for administrative jobs (d = 0.15). By analyzing applicant samples that do not suffer from range restriction, this study adds knowledge about subgroup differences in SJTs.     

Order Effects on Situational Judgment Test Items: A case of construct‐irrelevant difficulty

Situational judgment tests (SJTs) pose unique cognitive demands on test takers in that, when presented in written form, they require a great deal of reading and cognitive effort. Because of this cognitive demand, responses to test items toward the end of the test may be influenced by an order effect produced by responding to a large quantity of previous test items. This construct‐irrelevant order effect may increase measurement error and threaten the validity of SJT scores. To test this phenomenon, data were obtained from 1,089 applicants who had completed a lengthy SJT as part of a selection process for an hourly safety and surveillance job at a large international corporation. Results showed that local item dependence, item difficulty, and the rate of omitted responses all increased when items were placed toward the end. The order effect alone was not strong enough to influence subgroup mean score differences in the second half of the test. However, this effect did vary by race: African‐Americans were most strongly affected by the order effect, followed by Caucasians, in their number of omitted responses. Implications and future research of this effect for SJTs and similar types of assessments are discussed.     

A meta-analysis of the criterion-related validity of prehire work experience

Organizations frequently screen or select job applicants based on their work experience. Despite this, surprisingly little is known about the criterion-related validity of prehire experience, which reflects the amount, duration, or type of experience workers have acquired before they enter a new organization. To address this critical gap in the literature, we used meta-analysis to synthesize data from 81 independent samples that reported relations between prehire experience and performance or turnover. Results revealed overall corrected correlations of .06 for job performance (k = 44, n = 11,785), .11 for training performance (k = 21, n = 8,176), and .00 for turnover (k = 32, n = 11,676). Measures that capture prehire experience with tasks, jobs, or occupations relevant to workers’ current position also are only weakly related to the outcomes (e.g.,  = .07 for job performance). Two exceptions to our main findings are that (a) prehire experience is somewhat more predictive of job performance when workers first start a new job and (b) measures of task-level experience predict training performance, although these results are based on small subsets of primary studies. Overall, the present findings suggest that the types of prehire experience measures organizations currently use to screen job applicants generally are poor predictors of future performance and turnover. We therefore caution organizations from selecting employees based on such measures unless more positive evidence emerges.     

The Role of Applicant Nonverbal Communication in the Employment Interview

This research examined the effects that the nonverbal behavior of job applicants had on subsequent hiring evaluations. Interviewees (n = 32) were instructed to use one of four styles of nonverbal behavior during the recording of short job interviews with a standard content. The videotaped interviews were watched by behavior raters (n = 22), who assessed nonverbal-cue usage, and by judges (n = 50), who evaluated the applicants as prospective employees. The applicants were also rated by judges (n = 30) for physical attractiveness. Results supported the hypothesis that nonverbal behavior in the job interview operates as subtle reinforcement. Implications for job-counseling programs are discussed.     

Estimating Trait and Situational Variance in a Situational Judgment Test

In organizational research, situational judgment tests (SJTs) consistently demonstrate incremental validity, yet our theoretical understanding of SJTs is limited. Our knowledge could be advanced by decomposing the variance of SJT items into trait variance and situation variance; we do that by applying statistical methods used to analyze multitrait–multimethod matrices. A college-student sample (N = 2,747) was administered an SJT of goal orientation traits (i.e., mastery, performance-approach, and performance-avoid). Structural equation modeling was used to estimate the proportions of item variance to attributable to situational differences (across students) and to trait-based differences in students (across situations). Situation factors accounted for over three times the amount of variance as did individual difference factors. We conclude with general implications for the design of SJTs in organizational research.     

Impact of Elaboration on Responding to Situational Judgment Test Items

Although faking has been identified as a potential problem in situational judgment tests (SJTs), no studies have investigated proactive approaches for controlling faking in SJTs. Therefore, this study examined the impact of elaboration on responding to SJT items. Elaboration was operationalized as reason‐giving. Two hundred and forty‐seven master students were assigned to either an honest or a fake condition, and to a non‐elaboration or an elaboration condition. Results showed that elaboration decreased the effect of faking for items with high familiarity. Elaboration on familiar items also decreased the percentage of fakers in the top of the distribution. Next, participants in the elaboration condition rated the SJT significantly higher in terms of allowing them to present themselves more realistically and to demonstrate their knowledge, skills, and abilities. Finally, there were no significant differences in participants' satisfaction with the SJT across the elaboration and non‐elaboration condition.     

A Meta‐Analytic Investigation of Job Applicant Faking on Personality Measures

This study investigates the extent to which job applicants fake their responses on personality tests. Thirty‐three studies that compared job applicant and non‐applicant personality scale scores were meta‐analyzed. Across all job types, applicants scored significantly higher than non‐applicants on extraversion (d=.11), emotional stability (d=.44), conscientiousness (d=.45), and openness (d=.13). For certain jobs (e.g., sales), however, the rank ordering of mean differences changed substantially suggesting that job applicants distort responses on personality dimensions that are viewed as particularly job relevant. Smaller mean differences were found in this study than those reported by Viswesvaran and Ones (Educational and Psychological Measurement, 59 (2), 197–210), who compared scores for induced “fake‐good” vs. honest response conditions. Also, direct Big Five measures produced substantially larger differences than did indirect Big Five measures.     

Extracurricular activities in young applicants’ résumés: What are the motives behind their involvement?

Applicants use résumés to demonstrate their knowledge, skills, abilities, and other personal characteristics (KSAOs) to recruiters, through education and job‐related or non‐job‐related experiences. But research suggests that the situation for young applicants is especially competitive, since they increasingly enter the labour market with similar educational credentials and limited job‐related experience. They may thus use non‐job‐related experiences, such as participation in extracurricular activities (ECAs) during their studies, to demonstrate KSAOs to recruiters, but also to add distinction and value to their credentials. ECAs may therefore become more important in the selection of young applicants. Yet few studies have undertaken a comprehensive and systematic analysis of the relationships students have with these activities. The purpose of this study was to investigate to what extent students’ involvement in ECAs is due to internal (e.g., passion) or external (e.g., résumé‐building) motives, and what factors influence these motives. Results from a study with 197 students suggest that students engage in ECAs mainly out of internal motives. But external motives are stronger for activities started closer to entering the labour market, for students active in associative or volunteering activities (as compared to sports or artistic activities), and for students holding leadership positions in their activities. Our results suggest that labour market pressure may be a key component of applicants’ involvement in ECAs. Also, organizations and recruiters may want to consider that students tend not to engage in ECAs purely out of internal motives, but also to add value to their credentials and match employers’ expectations.