The Magnitude and Extent of Cheating and Response Distortion Effects on Unproctored Internet‐Based Tests of Cognitive Ability and Personality

The use of unproctored internet‐based testing (UIT) for employee selection is quite widespread. Although this mode of testing has advantages over onsite testing, researchers and practitioners continue to be concerned about potential malfeasance (e.g., cheating and response distortion) under high‐stakes conditions. Therefore, the primary objective of the present study was to investigate the magnitude and extent of high‐ and low‐stakes retest effects on the scores of a UIT speeded cognitive ability test and two UIT personality measures. These data permitted inferences about the magnitude and extent of malfeasant responding. The study objectives were accomplished by implementing two within‐subjects design studies ( Study 1 N=296; Study 2 N=318) in which test takers first completed the tests as job applicants (high‐stakes) or incumbents (low‐stakes) then as research participants (low‐stakes). For the speeded cognitive ability measure, the pattern of test score differences was more consonant with a psychometric practice effect than a malfeasance explanation. This result is likely due to the speeded nature of the test. And for the UIT personality measures, the pattern of higher high‐stakes scores compared with lower low‐stakes scores is similar to those reported for proctored tests in the extant literature. Thus, our results indicate that the use of a UIT administration does not uniquely threaten personality measures in terms of elevated scores under high‐stakes testing that are higher than those observed for proctored tests in the extant literature.     

The testing environment as an explanation for unproctored internet‐based testing device‐type effects

Device‐type effects on cognitive tests appear to covary with whether unproctored internet‐based test (UIT) scores were obtained operationally or nonoperationally. The present study examined whether the testing environment and distractions therein—one of three contextual factors identified as plausible explanations—accounts for this covariation. Four‐hundred and twenty‐five college students were randomly assigned to one of three conditions which differed by the testing environment and device used to complete a cognitive test. Test scores did not differ between conditions despite greater distractions reported by those who tested in the distracting environment. Results were replicated within‐study, and suggest that the self‐selection and testing‐stakes hypotheses warrant subsequent empirical examination as explanations for the differences in device‐type effects observed in operational and nonoperational settings.     

UNPROCTORED INTERNET TESTING IN EMPLOYMENT SETTINGS

As the Internet has become more accessible to individuals and organizations, the use of computerized testing has become more feasible. Computerized testing has brought with it a demand for unproctored testing that allows test takers to take employment tests at times and places convenient to them. However despite the advantages of costs and convenience, unproctored Internet testing (UIT) introduces a number of issues, many of which have not yet been resolved. These problems range from hardware and software issues to concerns about the security of the test content, the identity of candidates, and cheating. This article explores the pros and cons of unproctored, Internet testing. Six panelists share their opinions and experiences regarding issues around UIT and offer suggestions for appropriate use and future research.     

Offsetting Performance Losses Due to Cheating in Unproctored Internet‐based Testing by Increasing the Applicant Pool

A concern about unproctored Internet‐based testing (UIT) without follow‐up verification is that because applicants are able to cheat when completing tests, test validity decreases to an unknown extent. However, this view ignores a potential advantage to UIT: Should UIT increase applicant pool size, an organization with a fixed goal in terms of the number of candidates to advance to the next stage of the hiring process can increase the cutoff score and thus be more selective. In this study, a simulation was conducted to determine if this advantage outweighs the cheating disadvantage in the prediction of job performance. When the applicant pool is increased substantially through UIT, job performance outcomes are often higher even when many applicants increase their test scores through cheating.     

Computer Adaptive Testing (CAT): A Faster,Smarter, and More Secure Approach to Pre-Employment Testing

As the usage of unproctored Internet testing (UIT) rises, new methods of mitigating challenges associated with UIT have been proposed. We suggest that one of the most promising methods is computer adaptive testing (CAT), and is a major advancement in pre-employment testing. CAT combines science and technology to help deliver a targeted and secure testing experience. In this article, we describe the use of CAT in organizations and highlight examples of how CAT has been applied to the measurement of cognitive ability, knowledge, and personality traits. We also set out a research agenda that will advance the development and implementation of future CATs.     

Proctored Versus Unproctored Internet Tests: Are unproctored noncognitive tests as predictive of job performance?

As the usage of unproctored Internet testing (UIT) increases in selection settings, concerns about the validity of such practices must be addressed. While recent examinations of the issue have focused primarily on the practicality, ethics, and potential legal ramifications of UIT, this paper provides an examination of the criterion‐related validity of unproctored assessments. Using a database of validity evidence, we examine the predictive validity of several noncognitive assessments (i.e., personality and biodata) with respect to a number of subjective and objective job performance metrics. Results generally indicate that assessments administered in proctored and unproctored settings have similar validities. Limitations of this research and implications for practice are discussed.     

An Information-Processing-Based Conceptual Framework of the Effects of Unproctored Internet-Based Testing Devices on Scores on Employment-Related Assessments and Tests

The widespread use of unproctored Internet-based testing (UIT) in employment-related assessments has resulted in a burgeoning use of mobile devices to complete these assessments. Coupled with this is a concomitant interest in UIT-device-type effects, specifically, differences or lack thereof between assessments completed on “mobile” versus “nonmobile” devices. However, there is limited theoretical or conceptual work that seeks to explain the observed effects of UIT device type on test scores. Consequently, this article first presents a review of the extant empirical literature and then, on the basis of that, presents a framework—the structural characteristics/information processing framework—for psychologically conceptualizing the effect of UIT device types on test and assessment scores. The framework is used to explain previous findings and advance testable propositions for future research.     

How Secure are Unproctored Pre-Employment Tests? Analysis of Inconsistent Test Scores

Employment tests have long been scrutinized for psychometric considerations such as validity and reliability, but the extent to which cheating may occur on pre-employment tests has generally been overlooked. With the rise of unproctored, on-demand, online testing, the need has never been greater to focus on design and process considerations that can help mitigate the potential for cheating on employment tests. This paper builds on the limited existing research on the detection of inconsistent test scores on unproctored Internet testing. Job candidates (n = 4,026) completed a computer adaptive cognitive ability test under two conditions: an unproctored screening test followed by a proctored confirmation test. Analyses focused on detecting instances of inconsistent test scores based on comparison of standard errors of measure for the unproctored and proctored test scores (Guo and Drasgow, Int J Sel Assess 18:351–364, 2010). Results revealed a relatively low number of inconsistent scores and are discussed in the context of future research, application, and theory building on the nature of cheating on pre-employment tests.     

Two‐Step Testing in Employee Selection: Is score inflation a problem?

Unproctored Internet testing in employee selection has become increasingly popular over the past few years. However, there is a concern that cheating during unproctored administrations may influence the test results in terms of score inflation. The current research attempts to determine the extent of cheating on an unproctored Internet test of perceptual speed by analyzing data from a parallel version of the test administered in a proctored setting. Among a sample of 856 job applicants from nine European countries, there was no evidence that cheating occurred. In fact, evidence of an effect operating in the opposite direction was obtained. These results indicate that the use of unproctored Internet testing is appropriate for this type of test.     

Identifying Cheating on Unproctored Internet Tests: The Z‐test and the likelihood ratio test

Unproctored Internet testing (UIT) is becoming more popular in employment settings due to its cost effectiveness and efficiency. However, one of the major concerns with UIT is the possibility of cheating behaviors: a more capable conspirator can sit beside the real applicant and answer test items, or the applicant may use unauthorized materials. The present study examined the effectiveness of using a proctored verification test following the UIT to identify cheating in UIT, where 2 test statistics, a Z‐test and a likelihood ratio (LR) test, compare the consistency of test performance across the testing conditions. A simulation study was conducted to test the effectiveness of the two test statistics for a computerized adaptive test format. Results indicate that both test statistics have high power to detect dishonest job applicants at low Type I error rates. Compared with the LR test, the Z‐test was more efficient and effective and is therefore recommended for practical applications. The theoretical and practical implications are discussed.     

Using Invariance to Examine Cheating in Unproctored Ability Tests

Despite their widespread use in personnel selection, there is concern that cheating could undermine the validity of unproctored Internet‐based tests. This study examined the presence of cheating in a speeded ability test used for personnel selection. The same test was administered to applicants in either proctored or unproctored conditions. Item response theory differential functioning analyses were used to evaluate the equivalence of the psychometric properties of test items across proctored and unproctored conditions. A few items displayed different psychometric properties, and the nature of these differences was not uniform. Theta scores were not reflective of widespread cheating among unproctored examinees. Thus, results were not consistent with what would be expected if cheating on unproctored tests was pervasive.     

Equivalence of unproctored internet testing and proctored paper‐and‐pencil testing of the Big Five

The use of unproctored Internet‐based testing (IBT) for personality assessment is increasingly popular, especially in personnel selection. Previous studies on its equivalence to traditional proctored paper‐and‐pencil testing (PPT) have used between‐subjects designs, which makes it difficult to separate intergroup effects from format effects, and have shown mixed results. The aim of the present study was to assess the quantitative, qualitative, and auxiliary equivalence of unproctored IBT and proctored PPT of personality, using a within‐sample design. Undergraduate students (n = 407) completed both an Internet and a paper‐and‐pencil version of a measure of the Big Five, with a 1 to 3 week interval. The proctored paper‐and‐pencil assessment was completed in class and the unproctored Internet assessment was completed at the time and place chosen by the participants. Results showed effect sizes for mean differences to vary from null to small. Skewness and kurtosis indices, reliability coefficients, intercorrelations magnitudes, as well as factor solutions, were highly similar across formats. Respondents did not prefer IBT over PPT on a series of statement about their perceptions and reactions to IBT. In conclusion, results suggest that IBT and PPT of the Big Five personality traits are equivalent, and that the data obtained are comparable across formats.     

The Use of Mobile Devices in High‐stakes Remotely Delivered Assessments and Testing

With Internet access no longer restricted to desktop and laptop computers, job applicants now have the opportunity to complete remotely delivered assessments on mobile, handheld small screen devices such as smartphones, and personal digital assistants. In this study, a large dataset is used to investigate demographic and score differences between job applicants who completed a remotely delivered high‐stakes assessment on a mobile device and those who completed it on a nonmobile device. Based on a sample of 3,575,207 job applicants who completed an unproctored Internet‐based assessment between January 2011 and April 2012, the percentage of applicants completing the assessment on a mobile device was small, 1.93%, but nevertheless represented more than 69,000 people. Overall, there were small test‐taker demographic differences in the use of mobile devices versus nonmobile devices in that mobile devices were slightly more likely to be used by women, African‐Americans and Hispanics, and younger applicants. Scores on a personality measure were similar for mobile and nonmobile devices but scores on a general mental ability test were substantially lower for mobile devices. Tests of measurement invariance also indicated equivalence across the mobile and nonmobile samples. Test taker and organizational implications for completing remotely delivered high‐stakes noncognitive and cognitive assessments on mobile versus nonmobile devices are discussed.     

An Investigation of Stereotype Threat in Employment Tests1

This investigation extends stereotype threat research by examining its effect on sex and ethnicity in employment testing. Study 1 used archival data from a company‐specific cognitive ability test, demonstrating score increases (greater for some minorities) when applicants completed demographic items after a cognitive skills test, rather than before the test. The first experiment (Study 2) used college students taking the same company‐specific test with inconclusive results. Possible reasons dictated the design of a second experiment with a college population (Study 3). This study used the Wonderlic Personnel Test (WPT) and investigated priming through positioning of demographic items, as well as high/low threat. Positioning of the demographic items after the WPT was associated with an increase in test scores.     

Effects of Individual Differences on the Perceived Job Relatedness of a Cognitive Ability Test and a Multimedia Situational Judgment Test

Although there is a growing number of publications concerning applicant reactions to different selection instruments, the relationships between individual differences and applicant reactions have largely remained unexplored. The aim of the present study was to examine the effects of several testing‐related and general individual differences (anxiety, self‐evaluations, and personality) on the most commonly studied dimension of applicant reactions, namely the perceived job relatedness of selection instruments. Participants were 153 psychology students, who completed a cognitive ability test and a multimedia situational judgment test as part of their educational program. Our results indicated that computer anxiety negatively affected perceived job relatedness and core self‐evaluations, subjective well‐being, agreeableness, emotional stability, and openness to experience positively affected perceived job relatedness. Openness to experience was the most consistent predictor of perceived job relatedness. The results of our study suggest that certain individuals may be more predisposed to react positively to selection instruments. Therefore, we concluded that the nature of the applicant pool should be carefully considered when designing interventions to improve applicant reactions.     

Predicting Stereotype Threat,Test Anxiety,and Cognitive Ability Test Performance: An Examination of Three Models

As the classroom and workplace, among other contexts, become more diverse in their population characteristics, the need to be aware of specific factors impacting testing outcome issues correspondingly increases. The focus in this study, among other purposes, was to identify possible interactions between examinee's individual-difference characteristics (e.g., personality) and characteristics of the testing environment on test anxiety reactions and cognitive ability test performance. This study assessed reactions and performance in 4 different testing conditions through a path-analytic testing of 3 well-established theoretical models of stress appraisal and coping (i.e., cognitive appraisal, personality trait, and transactional). The transactional model, incorporating both personality and testing condition factors, was best in predicting the variance associated with the cognitive ability test scores, the 3 test anxiety score indexes (i.e., social derogation, physical tenseness, and cognitive obstruction), and the stereotype threat scores. With personality traits, for example, agreeableness was a relatively strong and consistent predictor of all study measures. Regarding testing condition factors, only the manipulation of stereotype threat level (low or high) produced some significant accounting of variance. Higher perceived stereotype threat levels were generally associated with heightened stereotype threat belief perceptions, lower cognitive ability test scores, and, interestingly, reduced feelings of cognitive obstruction in test anxiety.     

Dynamic testing of learning potential in adults with cognitive impairments: A systematic review of methodology and predictive value

Dynamic testing includes procedures that examine the effects of brief training on test performance where pre‐ to post‐training change reflects patients’ learning potential. The objective of this systematic review was to provide clinicians and researchers insight into the concept and methodology of dynamic testing and to explore its predictive validity in adult patients with cognitive impairments. The following electronic databases were searched: PubMed, PsychINFO, and Embase/Medline. Of 1141 potentially relevant articles, 24 studies met the inclusion criteria. The mean methodological quality score was 4.6 of 8. Eleven different dynamic tests were used. The majority of studies used dynamic versions of the Wisconsin Card Sorting Test. The training mostly consisted of a combination of performance feedback, reinforcement, expanded instruction, or strategy training. Learning potential was quantified using numerical (post‐test score, difference score, gain score, regression residuals) and categorical (groups) indices. In five of six longitudinal studies, learning potential significantly predicted rehabilitation outcome. Three of four studies supported the added value of dynamic testing over conventional testing in predicting rehabilitation outcome. This review provides preliminary support that dynamic tests can provide a valuable addition to conventional tests to assess patients’ abilities. Although promising, there was a large variability in methods used for dynamic testing and, therefore, it remains unclear which dynamic testing methods are most appropriate for patients with cognitive impairments. More research is warranted to further evaluate and refine dynamic testing methodology and to further elucidate its predictive validity concerning rehabilitation outcomes relative to other cognitive and functional status indices.     

The predictive validity of cognitive ability tests: A UK meta‐analysis

A meta‐analysis on the validity of tests of general mental ability (GMA) and specific cognitive abilities for predicting job performance and training success in the UK was conducted. An extensive literature search resulted in a database of 283 independent samples with job performance as the criterion (N=13,262), and 223 with training success as the criterion (N=75,311). Primary studies were also coded by occupational group, resulting in seven main groups (clerical, engineer, professional, driver, operator, manager, and sales), and by type of specific ability test (verbal, numerical, perceptual, and spatial). Results indicate that GMA and specific ability tests are valid predictors of both job performance and training success, with operational validities in the magnitude of .5–.6. Minor differences between these UK findings and previous US meta‐analyses are reported. As expected, operational validities were moderated by occupational group, with occupational families possessing greater job complexity demonstrating higher operational validities between cognitive tests and job performance and training success. Implications for the practical use of tests of GMA and specific cognitive abilities in the context of UK selection practices are discussed in conclusion.     

Managing Self-Uncertainty Through Group Identification

Hogg's uncertainty-identity theory (UIT) is briefly described to identify similarities and differences to Van den Bos's uncertainty management model (UMM). Against a background of significant overlap in scope, mission and concepts, four differences are identified: First, UMM is primarily a theory of motivation for ideological conviction; UIT is a theory of motivation for group identification. Second, UMM talks about personal uncertainty; UIT talks about self-uncertainty—the implications of this difference in terminology are discussed. Third, both theories focus on uncertainty about self; but UIT also focuses on an array of moderating variables that affect the experience of uncertainty and the way in which self-uncertainty is reduced. Finally, and most significantly, UMM does not detail the process of uncertainty reduction; UIT does—it specifies social cognitive processes that reduce self-uncertainty and contexts that direct these processes toward “normal” group phenomena or toward more extreme group phenomena.     

Self‐ambivalence and reactions to success versus failure

People differ in the extent to which their self‐evaluations fluctuate in response to positive and negative events. This research tests whether self‐ambivalence predicts this self‐evaluative reactivity. Participants first completed measures of self‐ambivalence and baseline self‐esteem. Next, they were induced a success or failure experience in a cognitive task and finally rated their cognitive self‐evaluations (task‐specific ability, state self‐esteem) and affective reactions (self‐feelings, mood). Self‐ambivalence was associated with stronger effects of the success/failure manipulation on cognitive self‐evaluations but not on affective reactions, with baseline self‐esteem controlled. Possible underlying mechanisms are discussed. Copyright © 2006 John Wiley & Sons, Ltd.