RETEST EFFECTS IN OPERATIONAL SELECTION SETTINGS: DEVELOPMENT AND TEST OF A FRAMEWORK

This study proposes a framework for examining the effects of retaking tests in operational selection settings. A central feature of this framework is the distinction between within-person and between-person retest effects. This framework is used to develop hypotheses about retest effects for exemplars of 3 types of tests (knowledge tests, cognitive ability tests, and situational judgment tests) and to test these hypotheses in a high stakes selection setting (admission to medical studies in Belgium). Analyses of within-person retest effects showed that mean scores of repeat test takers were one-third of a standard deviation higher for the knowledge test and situational judgment test and one-half of a standard deviation higher for the cognitive ability test. The validity coefficients for the knowledge test differed significantly depending on whether examinees' test scores on the first versus second administration were used, with the latter being more valid. Analyses of between-person retest effects on the prediction of academic performance showed that the same test score led to higher levels of performance for those passing on the first attempt than for those passing on the second attempt. The implications of these results are discussed in light of extant retesting practice.     

FURTHER STUDIES OF SITUATIONAL TESTS

In the present paper, results are reported for two different situational judgment tests (SJT) used in validation studies with almost 4,000 employees in 7 different organizations. Across the 2 studies, it was shown that situational test scores were significantly related to performance (weighted average r = .19), cognitive ability (weighted average r = .45), and experience (weighted average r = .20). In one study, there was a slight tendency for experience and cognitive ability to interact in the prediction of situational judgment, such that cognitive ability became less predictive as experience increased. Situational judgment fully mediated the effects of cognitive ability in one study, but not in the other. Finally, SJT race effect sizes were consistent with past research and, while not trivial, smaller than those typically observed for cognitive ability tests. The studies yielded very similar results despite the use of different keying methods (empirical vs. subject matter expert opinion). The preponderance of the evidence indicates that situational judgment measures mediate avariety of job relevant skills. Limitations to these studies and directions for future research are reviewed.     

Selection Testing via the Internet: Practical Considerations and Exploratory Empirical Findings*

Despite a growing body of applied research on using the Internet for some human resource management practices, few studies have provided equivalence information or practical lessons concerning selection testing via the Internet. We identify several issues associated with measurement and validity, the role of several individual characteristics, respondents' reactions and behaviors, and other considerations concerning Internet test administration. We also report results from an exploratory study of the correlation between paper‐and‐pencil and Internet‐administered cognitively oriented selection tests (including timed and untimed, proctored tests). Our empirical results suggest modest degrees of cross‐mode equivalence for an untimed situational judgment test (r= .84) and for a timed cognitive ability test (r= .60). Further, some types of items (math, verbal, spatial) in the timed cognitive ability test seem to play a differential role in the reduced cross‐mode equivalence. New issues regarding the perception of, and reaction to, items presented via the Internet are presented, and a variety of practical issues are derived and discussed.     

SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIES

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct-level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007 ; McDaniel & Ngyuen, 2001 ; Schmitt & Chan, 2006 ) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content-based typology. We then conducted a meta-analysis to determine the criterion-related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion-related validity; and (b) video-based situational judgment tests tended to have stronger criterion-related validity than pencil-and-paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.     

情景判断测验的开发程序、构思效度及研究趋势

章详细介绍情景判断测验开发的一般程序,对情景判断测验的多种形式和记分方法进行了总结和比较。同时,从情景判断测验的结果与认知能力、个性和工作经验等关系的角度出发,分析了情景判断测验的构思效度,认为情景判断测验测量的是多维构思。文章最后认为,需要从与其他构思的关系、测评指定构思、影响效度的因素以及跨文化比较等四个方面对情景判断测验开展进一步研究。     

Use of situational judgment tests to predict job performance: a clarification of the literature

Although situational judgment tests have a long history in the psychological assessment literature and continue to be frequently used in employment contexts, there has been virtually no summarization of this literature. The purpose of this article is to review the history of such tests and present the results of a meta-analysis on criterion-related and construct validity. On the basis of 102 coefficients and 10,640 people, situational judgment tests showed useful levels of validity (rho = .34) that were generalizable. A review of 79 correlations between situational judgment tests and general cognitive ability involving 16,984 people indicated that situational judgment tests typically evidence relationships with cognitive ability (rho = .46). On the basis of the literature review and meta-analytic findings, implications for the continued use of situational judgment tests are discussed, particularly in terms of recent investigations into tacit knowledge.     

VIDEO-BASED SITUATIONAL TESTING

In two separate studies, video-based tests of situational judgment were developed and validated against measures of performance for hourly service workers. In the first study, 684 employees were used to develop a test of retail associate judgment and 787 newly hired employees were used to cross-validate the instrument. In the second study, 412 current employees were used to develop a test of nursing home caregiver judgment and 148 newly hired caregivers were used to cross-validate this video-based test. In both studies, responses to video-based situational vignettes were empirically keyed against supervisory ratings of performance. The resulting keys produced uncorrected cross-validities in the low .20s. The video-based test scores were also found to be related to measures of cognitive ability and, to a lesser extent, experience. These results suggest that cognitive ability and possibly experience account for some but not all of the predictiveness of video-based situational tests. Video-based situational tests demonstrated score differences between Whites and non-Whites of roughly one-half a standard deviation, indicating that the use of such tests could produce adverse impact against non-Whites. In the second study, customer preferences regarding desired behavior were also used to develop a rational scoring key. This customer driven key was significantly related to performance ( r = .33) in the cross-validation sample, demonstrating that customers' preferences can provide the basis for developing valid predictors. Potential benefits of video-based situational tests in selection and directions for future research are discussed.     

WEB‐BASED AND PAPER‐AND‐PENCIL TESTING OF APPLICANTS IN A PROCTORED SETTING: ARE PERSONALITY,BIODATA, AND SITUATIONAL JUDGMENT TESTS COMPARABLE?

This quasi-experimental study compares the equivalence of proctored Web-based tests to paper-and-pencil tests in a selection setting. The predictor battery was composed of measures of Conscientiousness, Agreeableness, Emotional Stability, a biodata form, and a situational judgment test. Three samples were administered the same test items, but differed with respect to context and test format: 425 incumbents completed a paper-and-pencil version, 2,544 applicants completed a paper-and-pencil version, and 2,356 applicants completed an identical Web-based version (although these individuals were not randomly assigned to test format, the job and organization were the same for all samples). The results found effects for both context and test format. Relative to the applicants completing the paper-and-pencil measures, the Web-based measures showed (a) better distributional properties, (b) lower means, (c) more variance, (d) higher internal consistency reliabilities, and (e) stronger intercorrelations. Although the applicant context resulted in higher mean responses than the incumbent context, this difference was about twice as large for the paper-and-pencil test than for the Web-based test. All such differences were greater for personality measures than for biodata and situational judgment tests. Overall, these results suggest that proctored Web-based testing has some positive benefits relative to paper-and-pencil measures, and we identify several implications of these findings for research and practice.     

Incremental Validity of Situational Judgment Tests for Task and Contextual Job Performance

This paper has three goals. First, it responds to calls for additional research on subgroup differences in situational judgment tests. Second, it expands the cumulative knowledge on the incremental validity of situational judgment tests beyond cognitive ability and personality. Third, it examines the validity and incremental validity of various predictors for both task and contextual performance.     

Establishing the Measurement Equivalence of Online Selection Assessments Delivered on Mobile versus Nonmobile Devices

Recent usage data suggest job applicants are completing online selection assessments using mobile devices (e.g., smartphones) in greater numbers. To determine the appropriateness of this new technology, this study examined the measurement equivalence of selection assessments delivered on mobile and nonmobile devices (e.g., personal computers). Measurement invariance tests conducted with multigroup confirmatory factor analysis suggest mobile versions of a cognitive ability‐type assessment, two biodata assessments, a multimedia work simulation, and a text‐based situational judgment test appear to be equivalent to nonmobile versions. However, mobile device user latent means were half a standard deviation lower than their nonmobile counterparts for the situational judgment test. Implications for mobile device usage within selection and assessment are discussed.     

Situational judgment tests in high-stakes settings: issues and strategies with generating alternate forms

This study used principles underlying item generation theory to posit competing perspectives about which features of situational judgment tests might enhance or impede consistent measurement across repeat test administrations. This led to 3 alternate-form development approaches (random assignment, incident isomorphism, and item isomorphism). The effects of these approaches on alternate-form consistency, mean score changes, and criterion-related validity were examined in a high-stakes context (N = 3,361). Generally, results revealed that even small changes in the context of the situations presented resulted in significantly lower alternate-form consistency. Conversely, placing more constraints on the alternate-form development process proved beneficial. The contributions, implications, and limitations of these results for the development of situational judgment tests and high-stakes testing are discussed.     

Situational Judgment and Job Performance

Data from 160 civil service employees demonstrate the validity of a situational judgment test in predicting overall job performance as well as three performance dimensions: task performance (core technical proficiency), motivational contextual performance (job dedication), and interpersonal contextual performance (interpersonal facilitation). Situational judgment also provided incremental validity over the prediction provided jointly by cognitive ability, the Big Five personality traits, and job experience. These findings extended the work of Clevenger, Pereira, Wiechmann, Schmitt, and Harvey (2001) on the incremental validity of situational judgment tests as well as the meta-analytic results reported by McDaniel, Morgeson, Finnegan, Campion, and Braverman (2001). Implications are discussed in terms of research on the prediction and understanding of job performance.     

False consensus in situational judgment tests: What would others do?

We introduce an alternative response instruction to reduce the fakability of situational judgment tests. This novel instruction is based on the false consensus effect, a robust social psychological bias whereby people infer that the majority of other people’s thoughts, attitudes, and behaviors are aligned with their own. In four studies, including both field and laboratory data (total N = 882), we demonstrate that participants show a false consensus bias when asked what others would do in situational judgment tests. Furthermore, the situational judgment test based on the false consensus effect turned out to relatively difficult to be fake, and produced scores that were meaningfully correlated with conceptually related traits, as well as both self-reported and behavioral outcomes.     

A SITUATIONAL JUDGMENT TEST OF PERSONAL INITIATIVE AND ITS RELATIONSHIP TO PERFORMANCE

Although situational judgment tests have been found to be valid predictors of performance, they have rarely been used to measure particular constructs. In this study, we apply the situational judgment test method to the measurement of personal initiative, a construct defined as situated action. We used respondents' situated preferences in mental simulations of work scenarios as formative indicators of their overall level of personal initiative at work. Results from a validation study showed that the situational judgment test of personal initiative (SJT-PI) had adequate validity and complemented a Likert-type self-report measure of personal initiative in predicting behavioral criteria. Situated preferences for personal initiative were hypothesized to be proximal predictors of actual behavior and were accordingly found to mediate the relationship between generalized self-efficacy, felt responsibility, and actual behavior. Furthermore, situated preferences for personal initiative could be differentiated empirically from organizational citizenship behavior. We conclude that situational judgment tests are a promising method for measuring personal initiative and may be a general means of improving the validity of measurement in organizations.     

Validity evidence for the situational judgment test paradigm in emotional intelligence measurement

To date, various measurement approaches have been proposed to assess emotional intelligence (EI). Recently, two new EI tests have been developed based on the situational judgment test (SJT) paradigm: the Situational Test of Emotional Understanding (STEU) and the Situational Test of Emotion Management (STEM). Initial attempts have been made to examine the construct‐related validity of these new tests; we extend these findings by placing the tests in a broad nomological network. To this end, 850 undergraduate students completed a personality inventory, a cognitive ability test, a self‐report EI test, a performance‐based EI measure, the STEU, and the STEM. The SJT‐based EI tests were not strongly correlated with personality and fluid cognitive ability. Regarding their relation with existing EI measures, the tests did not capture the same construct as self‐report EI measures, but corresponded rather to performance‐based EI measures. Overall, these results lend support for the SJT paradigm for measuring EI as an ability.     

Practice effects on visuomotor and problem-solving tests by children

Practice effects on a visuomotor test (the Developmental Test of Visuo-Motor Integration), a timed visual discrimination test (the Underlining Test), and two problem-solving tests (the Porteus Mazes Test and the Tower of Hanoi Test) were analyzed. Children of two age groups (Ms: 7.7 and 11.6 yr.) were chosen to study the effect of age on practice effects. The tests were repeated nine times with test-retest intervals of 2 mo. The Developmental Test of Visuo-Motor Integration showed no practice effects, while the Porteus Mazes Test, the Underlining Test, and the Tower of Hanoi Test showed significant practice effects. Practice effects were larger for the older age group on all the tests, except the Developmental Test of Visuo-Motor Integration. The Developmental Test of Visuo-Motor Integration and the Underlining Test showed good reliability, but those of the problem-solving tasks were less satisfactory. The stability of all the tests, except the Tower of Hanoi Test, was good.     

The Role of Cognitive Ability in Self‐Efficacy and Self‐Assessed Test Performance1

Research has shown that test takers are often unable to assess their own test performance accurately. However, the role of cognitive ability in assessing one's test performance has not been explored. We examined whether high cognitive ability participants were better than low cognitive ability participants in assessing their performance on a video‐based situational judgment test (SJT) of customer‐service skills. Results indicated a strong relationship between actual and perceived SJT performance for high cognitive ability participants, but no relationship for those low in cognitive ability. The discussion focuses on implications for metacognitive theory, test perceptions, and providing feedback to applicants.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.     

Situational Tests in Student Selection: An Examination of Predictive Validity,Adverse Impact,and Construct Validity

The Flemish Admission Exam ‘Medical and Dental Studies’ is comprised of four cognitive ability tests and four situational tests, namely two work samples (i.e., a lecture and a medical text) and two video‐based situational judgement tests (i.e., a physician–patient interaction and a medical expert discussion). On the basis of the Admission Exam scores of 941 candidates (359 men, 582 women) this study shows that situational tests significantly can predict better than cognitive ability tests, with lecture and text emerging as significant predictors. When situational tests are combined with cognitive ability tests, there are no mean gender differences. Situational tests also enable us to measure a broader range of constructs. For example, in this study, the personality factor Openness is related to better situational test performance. Overall, this study demonstrates that situational tests may be a useful complement to traditional student selection procedures.     

Emotion Regulation and the Quality of Social Interaction: Does the Ability to Evaluate Emotional Situations and Identify Effective Responses Matter?

We examined self and friends' ratings of social relationship quality and everyday social interactions in 3 studies involving 544 college students in Germany, Spain, and the United States. Scores on a situational judgment test measuring strategic emotion regulation ability (SERA) were negatively related to conflict with others. SERA was more consistently and strongly related to conflict with others than to the positive dimension of relationship quality (support, companionship, and nurturance). The relationship between SERA and conflict was generally not mediated by trait positive or negative affect, and it remained significant or marginally significant controlling for the Big Five personality traits. These findings highlight the importance of the ability to evaluate emotional situations and identify effective responses to these in interpersonal emotion regulation. Furthermore, they suggest that situational judgment and flexible response selection may help people to manage conflicts more than to bond with others.