Identifying Cheating on Unproctored Internet Tests: The Z‐test and the likelihood ratio test

Unproctored Internet testing (UIT) is becoming more popular in employment settings due to its cost effectiveness and efficiency. However, one of the major concerns with UIT is the possibility of cheating behaviors: a more capable conspirator can sit beside the real applicant and answer test items, or the applicant may use unauthorized materials. The present study examined the effectiveness of using a proctored verification test following the UIT to identify cheating in UIT, where 2 test statistics, a Z‐test and a likelihood ratio (LR) test, compare the consistency of test performance across the testing conditions. A simulation study was conducted to test the effectiveness of the two test statistics for a computerized adaptive test format. Results indicate that both test statistics have high power to detect dishonest job applicants at low Type I error rates. Compared with the LR test, the Z‐test was more efficient and effective and is therefore recommended for practical applications. The theoretical and practical implications are discussed.     

The testing environment as an explanation for unproctored internet‐based testing device‐type effects

Device‐type effects on cognitive tests appear to covary with whether unproctored internet‐based test (UIT) scores were obtained operationally or nonoperationally. The present study examined whether the testing environment and distractions therein—one of three contextual factors identified as plausible explanations—accounts for this covariation. Four‐hundred and twenty‐five college students were randomly assigned to one of three conditions which differed by the testing environment and device used to complete a cognitive test. Test scores did not differ between conditions despite greater distractions reported by those who tested in the distracting environment. Results were replicated within‐study, and suggest that the self‐selection and testing‐stakes hypotheses warrant subsequent empirical examination as explanations for the differences in device‐type effects observed in operational and nonoperational settings.     

Individual Differences in Social Dominance Orientation Predict Support for the Use of Cognitive Ability Tests

This study investigates the personality processes involved in the debate surrounding the use of cognitive ability tests in college admissions. In Study 1, 108 undergraduates (M_age = 18.88 years, 60 women, 80 Whites) completed measures of social dominance orientation (SDO), testing self‐efficacy, and attitudes regarding the use of cognitive ability tests in college admissions; SAT/ACT scores were collected from the registrar. Sixty‐seven undergraduates (M_age = 19.06 years, 39 women, 49 Whites) completed the same measures in Study 2, along with measures of endorsement of commonly presented arguments about test use. In Study 3, 321 American adults (M_age = 35.58 years, 180 women, 251 Whites) completed the same measures used in Study 2; half were provided with facts about race and validity issues surrounding cognitive ability tests. Individual differences in SDO significantly predicted support for the use of cognitive ability tests in all samples, after controlling for SAT/ACT scores and test self‐efficacy and also among participants who read facts about cognitive ability tests. Moreover, arguments for and against test use mediated this effect. The present study sheds new light on an old debate by demonstrating that individual differences in beliefs about hierarchy play a key role in attitudes toward cognitive ability test use.     

Big Five traits: Predictors of retesting propensity and score improvement

Although most high‐stakes admissions, credentialing, and pre‐employment tests allow candidates to retest, relatively little is known about the personal traits of candidates who persist in retesting upon initial failure. In this study we investigated whether Big Five traits may predispose initially unsuccessful applicants to retest and subsequently improve on high‐stakes cognitive ability and knowledge tests required for personnel selection. In this study personality measures (unlike the cognitive tests) did not affect selection outcomes and hence did not provide applicants incentive to distort their personality responses to gain entry into the organization. Applicants higher in conscientiousness were more likely to retest, and emotional stability positively predicted cognitive test score improvement. We discuss implications of these results for organizations considering retesting policies.     

Incremental Validity of Leaderless Group Discussion Ratings Over and Above General Mental Ability and Personality in Predicting Promotion

Leaderless group discussions (LGDs) constitute one of the oldest assessment center exercises. In recent times, their added value has sometimes been questioned in light of trends to streamline assessment centers. The purpose of the present study is to examine the incremental validity of LGD ratings over cognitive ability scores and personality ratings for the prediction of extrinsic career success (i.e., promotion speed and number of promotions). We investigated this issue in the context of the promotion of French naval officers (N = 93) in an academy for high‐level executive positions over a 10‐year period. Results indicated that LGD ratings accounted for incremental variance in the prediction of promotion criterion measures, beyond cognitive ability and personality test scores. These results confirm that LGD ratings provide a unique contribution to the prediction of extrinsic career success in high‐level executive positions.     

Proctored Versus Unproctored Internet Tests: Are unproctored noncognitive tests as predictive of job performance?

As the usage of unproctored Internet testing (UIT) increases in selection settings, concerns about the validity of such practices must be addressed. While recent examinations of the issue have focused primarily on the practicality, ethics, and potential legal ramifications of UIT, this paper provides an examination of the criterion‐related validity of unproctored assessments. Using a database of validity evidence, we examine the predictive validity of several noncognitive assessments (i.e., personality and biodata) with respect to a number of subjective and objective job performance metrics. Results generally indicate that assessments administered in proctored and unproctored settings have similar validities. Limitations of this research and implications for practice are discussed.     

Context Matters,but How Much? Latent state–trait analysis of cognitive ability assessments

Cognitive abilities are thought to represent temporally stable constructs, however, accumulating evidence suggests that effects of the measurement situation could affect its measurement (e.g., effects of test motivation, stress level). The present study modeled these effects explicitly in a latent variables approach. In contrast to previous studies, we investigated participants (job candidates) in repeated high‐stakes settings (N = 188). We found that cognitive ability measurements in high‐stakes settings not only reflect a stable latent trait and random measurement error, but also systematic effects of the test setting. Our results support the application of cognitive ability tests in organizational contexts but have implications for its use in applied settings such as personnel selection.     

How Secure are Unproctored Pre-Employment Tests? Analysis of Inconsistent Test Scores

Employment tests have long been scrutinized for psychometric considerations such as validity and reliability, but the extent to which cheating may occur on pre-employment tests has generally been overlooked. With the rise of unproctored, on-demand, online testing, the need has never been greater to focus on design and process considerations that can help mitigate the potential for cheating on employment tests. This paper builds on the limited existing research on the detection of inconsistent test scores on unproctored Internet testing. Job candidates (n = 4,026) completed a computer adaptive cognitive ability test under two conditions: an unproctored screening test followed by a proctored confirmation test. Analyses focused on detecting instances of inconsistent test scores based on comparison of standard errors of measure for the unproctored and proctored test scores (Guo and Drasgow, Int J Sel Assess 18:351–364, 2010). Results revealed a relatively low number of inconsistent scores and are discussed in the context of future research, application, and theory building on the nature of cheating on pre-employment tests.     

Comparing Personality Test Formats and Warnings: Effects on criterion‐related validity and test‐taker reactions

The potential for applicant response distortion on personality measures remains a major concern in high‐stakes testing situations. Many approaches to understanding response distortion are too transparent (e.g., instructed faking studies) – or are too subtle (e.g., correlations with social desirability measures as indices of faking). Recent research reveals more promising approaches in two methods: using forced‐choice (FC) personality test items and warning against faking. The present study examined effects of these two methods on criterion‐related validity and test‐taker reactions. Results supported incremental validity for an FC and Likert‐scale measure in warning and no‐warning conditions, above and beyond cognitive ability. No clear differences emerged between the FC vs Likert measures or warning vs no‐warning conditions in terms of validity. However, some evidence suggested that FC measures and warnings may produce negative test‐taker reactions. We conclude with implications for implementation in selection settings.     

The relationship between cognitive ability and personality scores in selection situations: A meta‐analysis

Several faking theories have identified applicants’ cognitive ability (CA) as a determinant of faking—the intentional distortion of answers by candidates—but the corresponding empirical findings in the area of personality tests are often ambiguous. Following the assumption that CA is important for faking, we expected applicants with high CA to show higher personality scores in selection situations, leading in this case to significant correlations between CA and personality scores, but not in nonselection situations. This meta‐analysis (66 studies, k = 115 individual samples, N = 46,265) showed this pattern of results as well as moderation effects for the study design (laboratory vs. field), the response format of the personality test, and the type of CA test.     

Emotional Intelligence in Selection Contexts: Measurement method,criterion‐related validity,and vulnerability to response distortion

Two studies examined aspects of the validity of self‐report and performance‐based measures of emotional intelligence (EI) relevant to their use in personnel selection. In Study 1, structural equation modeling indicated that a two‐factor model with separate factors for the two types of EI measures fit better than a one‐factor solution. The performance‐based EI factor was more related to cognitive ability (R=.38) than personality (R=.26), whereas the self‐report EI factor was more related to personality (R=.85) than cognitive ability (R=.09). Although the performance‐based EI factor correlated more strongly with job performance (ρ=.24) than did that of the self‐report (ρ=.05), it provided little incremental validity beyond cognitive ability and conscientiousness. In Study 2, participants were asked to complete the measures as if applying for a job, and mean scores were then compared with those of Study 1. Results indicated that self‐report EI measures were more vulnerable to distortion than were the performance‐based measures. Implications for the assessment of EI in personnel selection contexts are discussed.     

Using Invariance to Examine Cheating in Unproctored Ability Tests

Despite their widespread use in personnel selection, there is concern that cheating could undermine the validity of unproctored Internet‐based tests. This study examined the presence of cheating in a speeded ability test used for personnel selection. The same test was administered to applicants in either proctored or unproctored conditions. Item response theory differential functioning analyses were used to evaluate the equivalence of the psychometric properties of test items across proctored and unproctored conditions. A few items displayed different psychometric properties, and the nature of these differences was not uniform. Theta scores were not reflective of widespread cheating among unproctored examinees. Thus, results were not consistent with what would be expected if cheating on unproctored tests was pervasive.     

Predicting Stereotype Threat,Test Anxiety,and Cognitive Ability Test Performance: An Examination of Three Models

As the classroom and workplace, among other contexts, become more diverse in their population characteristics, the need to be aware of specific factors impacting testing outcome issues correspondingly increases. The focus in this study, among other purposes, was to identify possible interactions between examinee's individual-difference characteristics (e.g., personality) and characteristics of the testing environment on test anxiety reactions and cognitive ability test performance. This study assessed reactions and performance in 4 different testing conditions through a path-analytic testing of 3 well-established theoretical models of stress appraisal and coping (i.e., cognitive appraisal, personality trait, and transactional). The transactional model, incorporating both personality and testing condition factors, was best in predicting the variance associated with the cognitive ability test scores, the 3 test anxiety score indexes (i.e., social derogation, physical tenseness, and cognitive obstruction), and the stereotype threat scores. With personality traits, for example, agreeableness was a relatively strong and consistent predictor of all study measures. Regarding testing condition factors, only the manipulation of stereotype threat level (low or high) produced some significant accounting of variance. Higher perceived stereotype threat levels were generally associated with heightened stereotype threat belief perceptions, lower cognitive ability test scores, and, interestingly, reduced feelings of cognitive obstruction in test anxiety.     

Selection Testing via the Internet: Practical Considerations and Exploratory Empirical Findings*

Despite a growing body of applied research on using the Internet for some human resource management practices, few studies have provided equivalence information or practical lessons concerning selection testing via the Internet. We identify several issues associated with measurement and validity, the role of several individual characteristics, respondents' reactions and behaviors, and other considerations concerning Internet test administration. We also report results from an exploratory study of the correlation between paper‐and‐pencil and Internet‐administered cognitively oriented selection tests (including timed and untimed, proctored tests). Our empirical results suggest modest degrees of cross‐mode equivalence for an untimed situational judgment test (r= .84) and for a timed cognitive ability test (r= .60). Further, some types of items (math, verbal, spatial) in the timed cognitive ability test seem to play a differential role in the reduced cross‐mode equivalence. New issues regarding the perception of, and reaction to, items presented via the Internet are presented, and a variety of practical issues are derived and discussed.     

Retrieval (Sometimes) Enhances Learning: Performance Pressure Reduces the Benefits of Retrieval Practice

Academic testing has received substantial support as a useful educational activity with robust retention benefits, given that tests can promote retrieval practice. However, testing can also instantiate performance‐related pressure and anxiety that may misappropriate the resources responsible for producing learning benefits. The current project examined the effects of performance pressure on retrieval practice. In two experiments, we instantiated performance pressure with either high‐stakes or low‐stakes quizzes, and assessed memory and comprehension of content on both quizzes and final tests. Quiz performance was equivalent under high‐stakes and low‐stakes conditions, demonstrating that learners adapted to high‐pressure quizzes. However, final test performance was better after low‐stakes versus high‐stakes quizzes, and only low‐stakes quizzes led to a performance advantage over a rereading control group. Participants additionally exhibited some sensitivity to the difficulty of retrieving under pressure. These data highlight the benefits of retrieval practice but indicate that they can be disrupted under pressure‐driven conditions. Copyright © 2014 John Wiley & Sons, Ltd.     

The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection

Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.     

Anger,personality, and optimistic stress appraisals

Two studies examined the impacts of experimentally induced anger and personality on appraisals of hypothetical stressors. Consistent with predictions Study 1 (N = 62) revealed that anger (compared to a neutral state) led to more optimistic appraisals of the stressors, and that those impacts were limited to appraisal dimensions known to evoke anger. Study 2 (N = 116) partially replicated these findings, and also revealed that personality moderated these effects such that the most optimistic appraisals emerged among the most extraverted and emotionally stable participants. These findings add to the extant work on the cognitive consequences of discrete emotions, integrate the affect, judgement, and personality literatures with appraisal theories of emotion, and suggest future research & directions.     

The Role of Individual Differences in the Accuracy of Confidence Judgments

Generally, self-assessment of accuracy in the cognitive domain produces overconfidence, whereas self-assessment of visual perceptual judgments results in under-confidence. Despite contrary empirical evidence, in models attempting to explain those phenomena, individual differences have often been disregarded. The authors report on 2 studies in which that shortcoming was addressed. In Experiment 1, participants (N = 520) completed a large number of cognitive-ability tests. Results indicated that individual differences provide a meaningful source of overconfidence and that a metacognitive trait might mediate that effect. In further analysis, there was only a relatively small correlation between test accuracy and confidence bias. In Experiment 2 (N = 107 participants), both perceptual and cognitive ability tests were included, along with measures of personality. Results again indicated the presence of a confidence factor that transcended the nature of the testing vehicle. Furthermore, a small relationship was found between that factor and some self-reported personality measures. Thus, personality traits and cognitive ability appeared to play only a small role in determining the accuracy of self-assessment. Collectively, the present results suggest that there are multiple causes of miscalibration, which current models of over- and underconfidence fail to encompass.