The Use of Mobile Devices in High‐stakes Remotely Delivered Assessments and Testing

With Internet access no longer restricted to desktop and laptop computers, job applicants now have the opportunity to complete remotely delivered assessments on mobile, handheld small screen devices such as smartphones, and personal digital assistants. In this study, a large dataset is used to investigate demographic and score differences between job applicants who completed a remotely delivered high‐stakes assessment on a mobile device and those who completed it on a nonmobile device. Based on a sample of 3,575,207 job applicants who completed an unproctored Internet‐based assessment between January 2011 and April 2012, the percentage of applicants completing the assessment on a mobile device was small, 1.93%, but nevertheless represented more than 69,000 people. Overall, there were small test‐taker demographic differences in the use of mobile devices versus nonmobile devices in that mobile devices were slightly more likely to be used by women, African‐Americans and Hispanics, and younger applicants. Scores on a personality measure were similar for mobile and nonmobile devices but scores on a general mental ability test were substantially lower for mobile devices. Tests of measurement invariance also indicated equivalence across the mobile and nonmobile samples. Test taker and organizational implications for completing remotely delivered high‐stakes noncognitive and cognitive assessments on mobile versus nonmobile devices are discussed.     

Pre‐employment risk screening: Comparability of integrity assessment technology platforms

The prevalence of mobile devices (e.g., smartphones and tablets) has led to job applicants increasingly choosing to use these mobile devices to complete online pre‐employment assessments in the application process. This study identifies increasing mobile device usage trends for applicants completing an unproctored integrity assessment across 13 quarterly time periods. Further, the comparability of the integrity assessment results when completed on mobile devices compared to when it is completed on computers was examined. Results revealed that the integrity assessment had comparable mean scores, number of admissions, reliability estimates, and validity coefficients across mobile devices and computers. Pass rates for the assessment across the delivery modes showed small differences though some were statistically significant. Applicants using smartphones to complete the assessment took significantly more time than applicants using computers or tablets.     

Internet-Based,Unproctored Assessments on Mobile and Non-Mobile Devices: Usage,Measurement Equivalence,and Outcomes

Purpose

The purpose of this study was to determine the usage rates, measurement equivalence, and potential outcome differences between mobile and non-mobile device-based deliveries of an unproctored, non-cognitive assessment.

Design/Methodology/Approach

This study utilized a quasi-experimental design based on archival data obtained from applicants who completed a non-cognitive assessment on a mobile (n = 7,743; e.g., smartphones, tablet computers) or non-mobile (n = 929,341; e.g., desktop computers) device as part of an operational, high-stakes pre-employment selection process.

Findings

One percent of applicants used mobile devices to complete the assessment. Multiple-group confirmatory factor analysis indicated the assessment was equivalent across mobile and non-mobile devices at the configural, metric, scalar, and latent mean levels. A comparison of observed score means using one-way and factorial ANOVAs demonstrated that the use of mobile and non-mobile devices did not produce any practically significant score differences on the assessment across devices or applicant demographic subgroups.

Implications

Industry and technological trends suggest mobile device usage will only increase. Thus, demonstrating that mobile device functionality and hardware characteristics do not change the psychometric functioning or applicant outcomes for a non-cognitive, text-based selection assessment is critical to talent assessment.

Originality/Value

This study provides the first empirical examination of the usage of mobile devices to complete talent assessments and their impact on assessment properties and applicant outcomes, and serves as the foundation for future research and application of this growing technological trend in pre-employment assessment.

     

An Information-Processing-Based Conceptual Framework of the Effects of Unproctored Internet-Based Testing Devices on Scores on Employment-Related Assessments and Tests

The widespread use of unproctored Internet-based testing (UIT) in employment-related assessments has resulted in a burgeoning use of mobile devices to complete these assessments. Coupled with this is a concomitant interest in UIT-device-type effects, specifically, differences or lack thereof between assessments completed on “mobile” versus “nonmobile” devices. However, there is limited theoretical or conceptual work that seeks to explain the observed effects of UIT device type on test scores. Consequently, this article first presents a review of the extant empirical literature and then, on the basis of that, presents a framework—the structural characteristics/information processing framework—for psychologically conceptualizing the effect of UIT device types on test and assessment scores. The framework is used to explain previous findings and advance testable propositions for future research.     

Mobile Internet Testing: An analysis of equivalence,individual differences,and reactions

Mobile internet testing (MIT) is the latest cost effective technological push in employment testing. The ability to access assessments via the internet using mobile devices provides increased speed and convenience for both administrators and respondents. In this article, we examine the equivalence of MIT compared with testing on personal computers (PCs) and whether attitudes and other individual differences influence responses and reactions to MIT. Results demonstrated equivalence for a supervisory situational judgment test across testing modes, but not for a cognitive ability test. Significant relationships were found between anxiety and both performance and reactions to mobile assessment. Respondents also reported significantly more positive reactions when tested on a PC versus a mobile device. Future research and practical implications are discussed.     

Can you test me now? Equivalence of GMA tests on mobile and non‐mobile devices

As technology continues to evolve, organizations seek to use personal electronics like smartphones for selection and assessment. While this promises to increase access to a more diverse applicant pool, research is needed to examine whether commonly used assessments function similarly on these devices as on a conventional computer. Contrary to past research, we did not find meaningful differences in general mental ability (GMA) test scores between device groups. We also observed few differences in item functioning between devices. Screen size had a positive, but marginal effect on test scores. These results are optimistic for the use of mobile devices in GMA testing, but additional research is needed to examine the functioning of alternative GMA tests administered on mobile devices.     

Scoring Situational Judgment Tests: Once You Get the Data,Your Troubles Begin

Although situational judgment tests (SJTs) have been in use for decades, consensus has not been reached on the best way to score these assessments or others (e.g., biodata) whose items do not have a single demonstrably correct answer. The purpose of this paper is to review and to demonstrate the scoring strategies that have been described in the literature. Implementation and relative merits of these strategies are described. Then, several of these methods are applied to create 11 different keys for a video‐based SJT in order to demonstrate how to evaluate the quality of keys. Implications of scoring SJTs for theory and practice are discussed.     

Designing Selection Systems for Medicine: The importance of balancing predictive and political validity in high‐stakes selection contexts

Selection into medical education and training is a high‐stakes process. A key unanswered issue is the effectiveness of measuring noncognitive predictors via both low‐fidelity and high‐fidelity selection approaches in this high‐stakes context. We review studies investigating the effectiveness of multiple selection instruments in terms of predictive validity, incremental validity, and applicant reactions in both entry‐level and advanced‐level medical selection. Our results show that the situational judgment test (SJT) is the best single predictor of performance, operationalized in multiple ways. In addition, the low‐fidelity SJT has incremental predictive power over cognitively oriented tests, and high‐fidelity assessment center (AC) exercises add incremental validity over the low‐fidelity (and less costly) selection methods. Concerning applicant reactions, results show that overall, the selection system is positively received. However, the method with the highest predictive validity – the SJT – received comparatively lower face validity ratings which may present a ‘justice dilemma’ for employers. Furthermore, various other stakeholders have a political interest in the selection methods used (e.g., government, the regulators and trade unions).     

Consciousness and Self‐Regulation in Mobile Communication

The everyday use of mobile devices is sometimes performed in a minimally conscious manner (e.g., automaticity, habits, impulses), whereas other times it is performed in a highly conscious manner (e.g., immersion, presence, absorption). In Study 1, we surveyed individuals (n = 250) to evaluate the seemingly oppositional relationship between automatic (less conscious) and immersive (more conscious) tendencies toward texting. Despite their standard separation, confirmatory factor analyses revealed that automaticity and immersion were actually positively related independent of usage frequency. In Study 2 (n = 526), these consciousness tendencies were related to select facets of trait self‐control and mindfulness. Together, these studies underline the importance of media cognition in combination with personality factors for understanding the psychology of mobile device use.     

SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIES

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct-level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007 ; McDaniel & Ngyuen, 2001 ; Schmitt & Chan, 2006 ) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content-based typology. We then conducted a meta-analysis to determine the criterion-related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion-related validity; and (b) video-based situational judgment tests tended to have stronger criterion-related validity than pencil-and-paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.     

Development and validation of a HEXACO situational judgment test

The purpose of this study was to develop and validate a construct-based situational judgment test of the HEXACO personality dimensions. In four studies, among applicants, employees, and Amazon Mechanical Turk participants (Ns = 72–305), we showed that it is possible to assess the six personality dimensions with a situational judgment test and that the criterion-related validity of the situational judgment test is comparable to the criterion-related validity of traditional self-reports but lower than the criterion-related validity of other-reports of personality. Test–retest coefficients (with a time interval of 2 weeks) varied between .55 and .74. Considering personality is the most commonly assessed construct in employee selection contexts (Ryan et al., 2015), this situational judgment test may provide human resources professionals with an alternative assessment tool.     

Repeated Behavior and Environmental Psychology: The Role of Personal Involvement and Habit Formation in Explaining Water Consumption1

Extending existing theory in social and environmental psychology, we develop a model to study important predictors of water consumption behavior. Overall results provide support for the predictive ability of stimuli (e.g., environmental awareness), reasoned processes (e.g., personal involvement), unreasoned processes (e.g., habits), and situational factors (e.g., income) on water consumption behavior. Findings indicate that households with lower water usage display greater awareness of water conservation issues, are more highly involved in the decision to use water, and tend to form habits associated with lower usage levels. Furthermore, the results are consistent with past research that attitudes toward water usage appear to be poor predictors of water consumption behavior. After controlling for situational factors (e.g., household size), the findings substantiate the role of personal involvement and habit formation in explaining water consumption, lending further support to the adaptation and development of repeated behavior models in environmental psychology.     

Situational Tests in Student Selection: An Examination of Predictive Validity,Adverse Impact,and Construct Validity

The Flemish Admission Exam ‘Medical and Dental Studies’ is comprised of four cognitive ability tests and four situational tests, namely two work samples (i.e., a lecture and a medical text) and two video‐based situational judgement tests (i.e., a physician–patient interaction and a medical expert discussion). On the basis of the Admission Exam scores of 941 candidates (359 men, 582 women) this study shows that situational tests significantly can predict better than cognitive ability tests, with lecture and text emerging as significant predictors. When situational tests are combined with cognitive ability tests, there are no mean gender differences. Situational tests also enable us to measure a broader range of constructs. For example, in this study, the personality factor Openness is related to better situational test performance. Overall, this study demonstrates that situational tests may be a useful complement to traditional student selection procedures.     

Incremental Validity of Interactive Multimedia Simulations in Two Organizations

Interactive multimedia simulations are conceptually distinct from other simulations commonly used as selection tools, such as assessment centers and situational judgment tests, and represent a potentially cost‐effective and job‐related candidate assessment tool. This study investigated the predictive validity of these simulations with a combined sample of call center employees from two organizations. Results indicate that customized interactive multimedia simulations demonstrate substantial criterion‐related validity and significant incremental validity over other noncognitive measures such as biodata and personality.     

LinkedIn as a new selection method: Psychometric properties and assessment approach

Various surveys suggest LinkedIn is used as a screening and selection tool by many hiring managers. Despite this widespread use, fairly little is known about whether LinkedIn meets established selection criteria, such as reliability, validity, and legality (i.e., no adverse impact). We examine the properties of LinkedIn-based assessments in two studies. Study 1 shows that raters reach acceptable levels of consistency in their assessments of applicant skills, personality, and cognitive ability. Initial ratings also correlate with subsequent ratings done 1-year later (i.e., demonstrating temporal stability), with slightly higher correlations when profile updates are taken into account. Initial LinkedIn-based ratings correlate with self-reports for more visible skills (leadership, communication, and planning) and personality traits (Extraversion), and for cognitive ability. LinkedIn-based hiring recommendations are positively associated with indicators of career success. Potential adverse impact is also limited. Profiles that are longer, include a picture, and have more connections are rated more positively. Some of those features are valid cues to applicants’ characteristics (e.g., applicants high on Conscientiousness have longer profiles). In Study 2, we show that an itemized LinkedIn assessment is more effective than a global assessment. Implications of these findings for selection and future research are discussed.     

Cross‐Situational Learning of Minimal Word Pairs

Cross‐situational statistical learning of words involves tracking co‐occurrences of auditory words and objects across time to infer word‐referent mappings. Previous research has demonstrated that learners can infer referents across sets of very phonologically distinct words (e.g., WUG, DAX), but it remains unknown whether learners can encode fine phonological differences during cross‐situational statistical learning. This study examined learners’ cross‐situational statistical learning of minimal pairs that differed on one consonant segment (e.g., BON–TON), minimal pairs that differed on one vowel segment (e.g., DEET–DIT), and non‐minimal pairs that differed on two or three segments (e.g., BON–DEET). Learners performed above chance for all pairs, but performed worse on vowel minimal pairs than on consonant minimal pairs or non‐minimal pairs. These findings demonstrate that learners can encode fine phonetic detail while tracking word‐referent co‐occurrence probabilities, but they suggest that phonological encoding may be weaker for vowels than for consonants.     

A Framework for Developing Comparable Multilingual Assessments for Minority Populations: Why Context Matters

The assessment of linguistic minorities often involves using multiple language versions of assessments. In these assessments, comparability of scores across language groups is central to valid comparative interpretations. Various frameworks and guidelines describe factors that need to be considered when developing comparable assessments. These frameworks provide limited information in relation to the development of multiple language versions of assessments for assessing linguistic minorities within countries. To this end, we make various suggestions for the types of factors that should be considered when assessing linguistic minorities. Our recommendations are tailored to the particular constraints potentially faced by various jurisdictions tasked with developing multiple language versions of assessments for linguistic minorities. These challenges include having limited financial and staffing resources to develop comparable assessments and having insufficient sample sizes to perform psychometric analyses (e.g., item response theory) to examine comparability. Although we contextualize our study by focusing on linguistic minorities within Canada due to its bilingual status, our findings may also apply to other bilingual and multilingual countries with similar minority/majority contexts.     

Proctored Versus Unproctored Internet Tests: Are unproctored noncognitive tests as predictive of job performance?

As the usage of unproctored Internet testing (UIT) increases in selection settings, concerns about the validity of such practices must be addressed. While recent examinations of the issue have focused primarily on the practicality, ethics, and potential legal ramifications of UIT, this paper provides an examination of the criterion‐related validity of unproctored assessments. Using a database of validity evidence, we examine the predictive validity of several noncognitive assessments (i.e., personality and biodata) with respect to a number of subjective and objective job performance metrics. Results generally indicate that assessments administered in proctored and unproctored settings have similar validities. Limitations of this research and implications for practice are discussed.     

VIDEO-BASED SITUATIONAL TESTING

In two separate studies, video-based tests of situational judgment were developed and validated against measures of performance for hourly service workers. In the first study, 684 employees were used to develop a test of retail associate judgment and 787 newly hired employees were used to cross-validate the instrument. In the second study, 412 current employees were used to develop a test of nursing home caregiver judgment and 148 newly hired caregivers were used to cross-validate this video-based test. In both studies, responses to video-based situational vignettes were empirically keyed against supervisory ratings of performance. The resulting keys produced uncorrected cross-validities in the low .20s. The video-based test scores were also found to be related to measures of cognitive ability and, to a lesser extent, experience. These results suggest that cognitive ability and possibly experience account for some but not all of the predictiveness of video-based situational tests. Video-based situational tests demonstrated score differences between Whites and non-Whites of roughly one-half a standard deviation, indicating that the use of such tests could produce adverse impact against non-Whites. In the second study, customer preferences regarding desired behavior were also used to develop a rational scoring key. This customer driven key was significantly related to performance ( r = .33) in the cross-validation sample, demonstrating that customers' preferences can provide the basis for developing valid predictors. Potential benefits of video-based situational tests in selection and directions for future research are discussed.