The Use of Mobile Devices in High‐stakes Remotely Delivered Assessments and Testing

With Internet access no longer restricted to desktop and laptop computers, job applicants now have the opportunity to complete remotely delivered assessments on mobile, handheld small screen devices such as smartphones, and personal digital assistants. In this study, a large dataset is used to investigate demographic and score differences between job applicants who completed a remotely delivered high‐stakes assessment on a mobile device and those who completed it on a nonmobile device. Based on a sample of 3,575,207 job applicants who completed an unproctored Internet‐based assessment between January 2011 and April 2012, the percentage of applicants completing the assessment on a mobile device was small, 1.93%, but nevertheless represented more than 69,000 people. Overall, there were small test‐taker demographic differences in the use of mobile devices versus nonmobile devices in that mobile devices were slightly more likely to be used by women, African‐Americans and Hispanics, and younger applicants. Scores on a personality measure were similar for mobile and nonmobile devices but scores on a general mental ability test were substantially lower for mobile devices. Tests of measurement invariance also indicated equivalence across the mobile and nonmobile samples. Test taker and organizational implications for completing remotely delivered high‐stakes noncognitive and cognitive assessments on mobile versus nonmobile devices are discussed.     

Internet-Based,Unproctored Assessments on Mobile and Non-Mobile Devices: Usage,Measurement Equivalence,and Outcomes

Purpose

The purpose of this study was to determine the usage rates, measurement equivalence, and potential outcome differences between mobile and non-mobile device-based deliveries of an unproctored, non-cognitive assessment.

Design/Methodology/Approach

This study utilized a quasi-experimental design based on archival data obtained from applicants who completed a non-cognitive assessment on a mobile (n = 7,743; e.g., smartphones, tablet computers) or non-mobile (n = 929,341; e.g., desktop computers) device as part of an operational, high-stakes pre-employment selection process.

Findings

One percent of applicants used mobile devices to complete the assessment. Multiple-group confirmatory factor analysis indicated the assessment was equivalent across mobile and non-mobile devices at the configural, metric, scalar, and latent mean levels. A comparison of observed score means using one-way and factorial ANOVAs demonstrated that the use of mobile and non-mobile devices did not produce any practically significant score differences on the assessment across devices or applicant demographic subgroups.

Implications

Industry and technological trends suggest mobile device usage will only increase. Thus, demonstrating that mobile device functionality and hardware characteristics do not change the psychometric functioning or applicant outcomes for a non-cognitive, text-based selection assessment is critical to talent assessment.

Originality/Value

This study provides the first empirical examination of the usage of mobile devices to complete talent assessments and their impact on assessment properties and applicant outcomes, and serves as the foundation for future research and application of this growing technological trend in pre-employment assessment.

     

Establishing the Measurement Equivalence of Online Selection Assessments Delivered on Mobile versus Nonmobile Devices

Recent usage data suggest job applicants are completing online selection assessments using mobile devices (e.g., smartphones) in greater numbers. To determine the appropriateness of this new technology, this study examined the measurement equivalence of selection assessments delivered on mobile and nonmobile devices (e.g., personal computers). Measurement invariance tests conducted with multigroup confirmatory factor analysis suggest mobile versions of a cognitive ability‐type assessment, two biodata assessments, a multimedia work simulation, and a text‐based situational judgment test appear to be equivalent to nonmobile versions. However, mobile device user latent means were half a standard deviation lower than their nonmobile counterparts for the situational judgment test. Implications for mobile device usage within selection and assessment are discussed.     

Mobile Internet Testing: An analysis of equivalence,individual differences,and reactions

Mobile internet testing (MIT) is the latest cost effective technological push in employment testing. The ability to access assessments via the internet using mobile devices provides increased speed and convenience for both administrators and respondents. In this article, we examine the equivalence of MIT compared with testing on personal computers (PCs) and whether attitudes and other individual differences influence responses and reactions to MIT. Results demonstrated equivalence for a supervisory situational judgment test across testing modes, but not for a cognitive ability test. Significant relationships were found between anxiety and both performance and reactions to mobile assessment. Respondents also reported significantly more positive reactions when tested on a PC versus a mobile device. Future research and practical implications are discussed.     

Paper-and-pencil and web-based testing: the measurement invariance of the Big Five personality tests in applied settings

This study investigates the measurement equivalence of a five-factor measure of personality across two groups applying for jobs, who completed the same questionnaire using either a paper-and-pencil (n = 429) or a web online answer format (n = 651). The data were collected using the Big Five Questionnaire-2 (BFQ-2; which is a measure of the Five Factor Model) of personality traits. Multiple-group confirmatory factor analysis was used to test for the equivalence of factor covariance and mean structures of the BFQ-2. Findings suggested that the Five Factor Model scales have the same measurement unit and origin across applicants using different administration modes. However, latent means were slightly higher for applicants who responded in a web and unproctored condition than for applicants who completed a paper-and-pencil version of the same test. Practical implications for personality assessment are discussed.     

移动购物更快吗?决策场景与思维模式的相容性

既有文献对消费者延迟选择影响因素的研究, 主要集中于决策复杂度、决策者特质或情绪, 少有对决策场景进行深入探讨。互联网购物时代, PC端与手机端已成为重要的消费场景。针对手机端是否会加快消费者决策过程的问题目前也存在不一致的结论。本文引入决策双系统理论, 力图解释以往看似冲突的结论。消费者的购买决策模式, 不仅会受到决策任务(产品价格)的影响, 还会受到决策场景(购买终端)影响。当场景与决策任务使消费者启动的思维模式相一致时, 会提升决策流畅度, 降低延迟选择。本文发现, 移动端(PC端)与低(高)价格产品更容易激发相容的经验性(理性)思维, 降低延迟选择倾向。反之, 则会同时启动两种相左的决策思维, 增加延迟选择倾向。本文从场景启动与价格启动视角, 将以往消费者单一的决策思维模式, 拓展到多因素启动的思维模式相容性问题; 同时本文通过探究延迟选择问题, 对价格策略与场景化营销带来启示。     

Can you test me now? Equivalence of GMA tests on mobile and non‐mobile devices

As technology continues to evolve, organizations seek to use personal electronics like smartphones for selection and assessment. While this promises to increase access to a more diverse applicant pool, research is needed to examine whether commonly used assessments function similarly on these devices as on a conventional computer. Contrary to past research, we did not find meaningful differences in general mental ability (GMA) test scores between device groups. We also observed few differences in item functioning between devices. Screen size had a positive, but marginal effect on test scores. These results are optimistic for the use of mobile devices in GMA testing, but additional research is needed to examine the functioning of alternative GMA tests administered on mobile devices.     

Personality measurement, faking, and employment selection

Real job applicants completed a 5-factor model personality measure as part of the job application process. They were rejected; 6 months later they (n = 5,266) reapplied for the same job and completed the same personality measure. Results indicated that 5.2% or fewer improved their scores on any scale on the 2nd occasion; moreover, scale scores were as likely to change in the negative direction as the positive. Only 3 applicants changed scores on all 5 scales beyond a 95% confidence threshold. Construct validity of the personality scales remained intact across the 2 administrations, and the same structural model provided an acceptable fit to the scale score matrix on both occasions. For the small number of applicants whose scores changed beyond the standard error of measurement, the authors found the changes were systematic and predictable using measures of social skill, social desirability, and integrity. Results suggest that faking on personality measures is not a significant problem in real-world selection settings.     

Evaluation of tablet computers for visual function assessment

Recent advances in technology and the increased use of tablet computers for mobile health applications such as vision testing necessitate an understanding of the behavior of the displays of such devices, to facilitate the reproduction of existing or the development of new vision assessment tests. The purpose of this study was to investigate the physical characteristics of one model of tablet computer (iPad mini Retina display) with regard to display consistency across a set of devices (15) and their potential application as clinical vision assessment tools. Once the tablet computer was switched on, it required about 13 min to reach luminance stability, while chromaticity remained constant. The luminance output of the device remained stable until a battery level of 5%. Luminance varied from center to peripheral locations of the display and with viewing angle, whereas the chromaticity did not vary. A minimal (1%) variation in luminance was observed due to temperature, and once again chromaticity remained constant. Also, these devices showed good temporal stability of luminance and chromaticity. All 15 tablet computers showed gamma functions approximating the standard gamma (2.20) and showed similar color gamut sizes, except for the blue primary, which displayed minimal variations. The physical characteristics across the 15 devices were similar and are known, thereby facilitating the use of this model of tablet computer as visual stimulus displays.     

nodeGame: Real-time,synchronous, online experiments in the browser

nodeGame is a free, open-source JavaScript/ HTML5 framework for conducting synchronous experiments online and in the lab directly in the browser window. It is specifically designed to support behavioral research along three dimensions: (i) larger group sizes, (ii) real-time (but also discrete time) experiments, and (iii) batches of simultaneous experiments. nodeGame has a modular source code, and defines an API (application programming interface) through which experimenters can create new strategic environments and configure the platform. With zero-install, nodeGame can run on a great variety of devices, from desktop computers to laptops, smartphones, and tablets. The current version of the software is 3.0, and extensive documentation is available on the wiki pages at http://nodegame.org.     

THE EFFECTS OF APPLICANTs' REACTIONS TO COGNITIVE ABILITY TESTS AND AN ASSESSMENT CENTER

Manufacturing applicants' perceptions of two selection devices were examined. In Study 1, applicants ( n = 3,984) completed cognitive ability tests and a survey of reactions. In Study 2, a subset of applicants from Study 1 ( n = 194) participated in an assessment center and completed the survey. Applicants reacted favorably to the procedures but viewed the assessment center as more face valid than the cognitive tests. Applicants who perceived the selection techniques more favorably were also more satisfied with the selection process, the job, and the organization. Although applicants' perceptions of the procedures were related to job acceptance intentions, applicants' liking of the job and organization explained the largest unique variance. In future studies, applicants' job acceptance intentions and attitudes toward the job and organization should be assessed before and after administration of selection devices; not controlling for prior impressions resulted in overestimation of the contribution of applicants' perceptions of selection procedures.     

Cross-cultural validity of integrity assessments for lower-level and higher-level jobs

This research brief examines the cross-cultural validity of two multi-dimensional integrity assessments, one for lower-level jobs and one for higher-level jobs. Job applicants for a global company completed the assessments as a part of the hiring process in three countries: the United States, South Africa, and Mexico. U.S. applicants tended to score higher (i.e., higher integrity) on these assessments and make fewer admissions of counterproductive work behaviors than applicants from South Africa and Mexico. The overall scores for these assessments (and most of the individual scales) were correlated significantly with, for each country separately and globally, three composites of self-reported counterproductive work behavior criterion measures (Theft, Counterproductivity, and Overall Risky Behavior). Construct validity evidence was examined with a measure of Collusion Avoidance (tolerance for colluding with external organized criminals). Overall, the results of this research highlight the importance of including an integrity assessment in the hiring processes for lower-level and higher-level (i.e., professional and managerial) jobs globally. Integrity assessments appear to have practical value in countries with higher prevalence of crime and corruption. The results also bolster previous research indicating that integrity assessments are valid outside of the U.S. context in which they were initially developed over 60 years ago.     

Portable Digital Devices: Meeting Challenges to Psychotherapeutic Privacy

Portable digital devices, such as smartphones, tablets, and netbooks, now permeate our society. These devices allow substantial efficiencies in access to and communication of information by mental health professionals. They also bring with them risks to psychotherapist–client privacy. This article reviews these threats and offers suggestions as to their mitigation.     

An Information-Processing-Based Conceptual Framework of the Effects of Unproctored Internet-Based Testing Devices on Scores on Employment-Related Assessments and Tests

The widespread use of unproctored Internet-based testing (UIT) in employment-related assessments has resulted in a burgeoning use of mobile devices to complete these assessments. Coupled with this is a concomitant interest in UIT-device-type effects, specifically, differences or lack thereof between assessments completed on “mobile” versus “nonmobile” devices. However, there is limited theoretical or conceptual work that seeks to explain the observed effects of UIT device type on test scores. Consequently, this article first presents a review of the extant empirical literature and then, on the basis of that, presents a framework—the structural characteristics/information processing framework—for psychologically conceptualizing the effect of UIT device types on test and assessment scores. The framework is used to explain previous findings and advance testable propositions for future research.     

Tablets,touchscreens, and touchpads: How varying touch interfaces trigger psychological ownership and endowment

As mouse-driven desktop computers give way to touchpad laptops and touchscreen tablets, the role of touch in online consumer behavior has become increasingly important. This work presents initial explorations into the effects of varying touch-based interfaces on consumers, and argues that research into the interfaces used to access content can be as important as research into the content itself. Two laboratory studies using a variety of touch technologies explore how touchscreen interfaces can increase perceived psychological ownership, and this in turn magnifies the endowment effect. Touch interfaces also interact with importance of product haptics and actual interface ownership in their effects on perceived product ownership, with stronger effects for products high in haptic importance and interfaces that are owned. Results highlight that perceptions of online products and marketing activities are filtered through the lens of the interfaces used to explore them, and touch-based devices like tablets can lead to higher product valuations when compared to traditional computers.     

Predicting counterproductivity among drug store applicants

The ability of a preemployment integrity test to predict future admissions of counterproductive behavior by employees was investigated. Job applicants completed the London House Personnel Selection Inventory (PSI) as part of an overall selection process. Approximately one year later, a survey was sent to applicants at their home addresses. The survey asked for the frequency with which these applicants engaged in a variety of counterproductive behaviors at work. The PSI accurately predicted admissions of cash and merchandise theft, theft of time, behaviors associated with poor job performance, and the frequency with which the theft of other employees was observed.     

移动社交媒介使用行为对青少年友谊质量的影响:网络自我表露和网络社会支持的中介作用

随着移动智能设备和移动互联网的出现,移动社交媒介已日益成为人们网络活动和人际交往的新平台。为考察移动社交媒介使用行为、网络自我表露、网络社会支持和友谊质量之间的关系,本研究采用问卷法对473名青少年进行了调查。结果发现:(1)移动社交媒介使用行为会对网络自我表露和网络社会支持产生显著的直接正向影响,网络自我表露会对网络社会支持产生显著的直接正向影响,网络社会支持会对友谊质量产生显著的直接正向影响。(2)移动社交媒介使用行为可以通过网络社会支持的中介作用对友谊质量产生显著的间接影响,还可以通过网络自我表露和网络社会支持的链式中介作用对友谊质量产生显著的间接影响。     

Can smartphones be used to bring computer-based tasks from the lab to the field? A mobile experience-sampling method study about the pace of life

Researchers are increasingly using smartphones to collect scientific data. To date, most smartphone studies have collected questionnaire data or data from the built-in sensors. So far, few studies have analyzed whether smartphones can also be used to conduct computer-based tasks (CBTs). Using a mobile experience-sampling method study and a computer-based tapping task as examples (N = 246; twice a day for three weeks, 6,000+ measurements), we analyzed how well smartphones can be used to conduct a CBT. We assessed methodological aspects such as potential technologically induced problems, dropout, task noncompliance, and the accuracy of millisecond measurements. Overall, we found few problems: Dropout rate was low, and the time measurements were very accurate. Nevertheless, particularly at the beginning of the study, some participants did not comply with the task instructions, probably because they did not read the instructions before beginning the task. To summarize, the results suggest that smartphones can be used to transfer CBTs from the lab to the field, and that real-world variations across device manufacturers, OS types, and CPU load conditions did not substantially distort the results.     

The effects of operating a touch screen smartphone and other common activities performed while bicycling on cycling behaviour

Although it has been shown that making phone calls or sending text messages while riding a bicycle can have a negative impact on bicyclist’s behaviour, in countries such as the Netherlands the operation of a mobile phone while cycling on a bicycle is not illegal and is actually quite common. In recent years conventional mobile phones with a physical keypad are increasingly being replaced by smartphones with a touch screen. The operation of a touch screen phone ironically cannot be done purely ‘by touch’ due to the lack of tactile feedback, and instead requires fixations on a relatively small screen. The question therefore can be asked whether the operation of touch screen telephones deteriorates cycling behaviour more than operation of a conventional mobile phone.Twenty-four participants completed a track on their own bicycle while sending a text message from a conventional and a touch screen mobile phone. In addition the effects of other common activities that can accompany bicycling were studied, including texting at the same time as listening to music, talking on a mobile phone or cycling next to someone and speaking with this companion, and playing a game on a touch screen phone while bicycling. The impacts of all the above conditions on cycling performance and visual detection performance were compared with control conditions in which participants cycled with either one or two hands on the handlebars and were not required to perform any secondary tasks.Bicycle speed was reduced in all telephone conditions and in the condition when cycling next to someone. Lateral position variation increased in all telephone conditions. Use of the touch screen led to a more central position in the cycle lane and resulted in worse visual detection performance compared with the operation of a conventional mobile phone. The main effect of listening to music was that an auditory signal to stop cycling was missed by 83% of the participants. In conclusion, while all investigated types of phone deteriorated cycling performance, the use of a touch phone has a larger negative effect on cycling performance than a conventional mobile phone. With touch screen smartphones taking the place of conventional mobile phones and being used for other purposes than verbal communication, these effects on cycling performance pose a threat to traffic safety.