Assessing and overcoming participant dishonesty in online data collection

Crowdsourcing services, such as MTurk, have opened a large pool of participants to researchers. Unfortunately, it can be difficult to confidently acquire a sample that matches a given demographic, psychographic, or behavioral dimension. This problem exists because little information is known about individual participants and because some participants are motivated to misrepresent their identity with the goal of financial reward. Despite the fact that online workers do not typically display a greater than average level of dishonesty, when researchers overtly request that only a certain population take part in an online study, a nontrivial portion misrepresent their identity. In this study, a proposed system is tested that researchers can use to quickly, fairly, and easily screen participants on any dimension. In contrast to an overt request, the reported system results in significantly fewer (near zero) instances of participant misrepresentation. Tests for misrepresentations were conducted by using a large database of past participant records (~45,000 unique workers). This research presents and tests an important tool for the increasingly prevalent practice of online data collection.     

Open-source software to conduct online rating studies

Several software programs exist to assist researchers in setting up online questionnaires. Existing tools are of little help for delivering online rating studies, for which it is often desirable to collect data from participants for only a subset of a stimulus set. OR-Vis enables researchers to quickly set up online rating studies by supplying the set of items to be rated, the number of stimuli an individual participant responds to, the number of participants an item is shown to, and the rating questions. The software then generates and delivers unique questionnaires for each participant, while managing the data collection process. The present article describes OR-Vis, its installation process, and how to use it to gather data. OR-Vis is open-source software and can be downloaded from www.orvis.uni-muenster.de.     

Improving online panel data usage in sales research

The exponential expansion of the Internet has fundamentally altered the nature of personal selling and sales management. The Internet has also created significant change for sales researchers and proffered a bevy of new means of conducting research. Online panel data collection is one such means that has recently proliferated. Online panel data collection imbues a host of advantages sales researchers may reap. However, there are a proportionate number of disadvantages and concerns with collecting and using this type of data. This article delineates said benefits and drawbacks and also provides a review of recent sales-related examples employing this method of data collection. Furthermore, key considerations in utilizing online panel data are advanced. Importantly, several ways to maximize the benefits and minimize the drawbacks inherent to online panel data are provided. This article aims to provide a resource for sales scholars in optimizing their usage of online panel data.     

Comparing In-Person,Sona, and Mechanical Turk Measurements of Three Prejudice-Relevant Constructs

Electronic data collection and participant pool management tools give researchers new ways to conduct research. The current study investigated the equivalency of in-person and online administrations of the Right-Wing Authoritarianism, Social Dominance Orientation, and Modern Racism scales across three modalities (administration in person, online through Sona Systems, and online through Mechanical Turk). Results indicate that in-person administration was largely equivalent to the randomly assigned online sample (Sona Systems) in terms of their intercorrelations, mean scores, variability, and reliability. However, the Sona sample consistently responded with strong attitudes for all measures, suggesting that social desirability may be decreased when completing these measures online. In addition, the Mechanical Turk sample differed in many ways from both in-person and Sona administration; although this nonequivalence is at least partially explained by sample demographic differences, other considerations may have exacerbated nonequivalence, including prior exposure to the measures and a desire to respond correctly.     

Ethical and Regulatory Considerations for Using Social Media Platforms to Locate and Track Research Participants

As social media becomes increasingly popular, human subjects researchers are able to use these platforms to locate, track, and communicate with study participants, thereby increasing participant retention and the generalizability and validity of research. The use of social media; however, raises novel ethical and regulatory issues that have received limited attention in the literature and federal regulations. We review research ethics and regulations and outline the implications for maintaining participant privacy, respecting participant autonomy, and promoting researcher transparency when using social media to locate and track participants. We offer a rubric that can be used in future studies to determine ethical and regulation-consistent use of social media platforms and illustrate the rubric using our study team’s experience with Facebook. We also offer recommendations for both researchers and institutional review boards that emphasize the importance of well-described procedures for social media use as part of informed consent.     

Virtual reality as an immersive projective and autodriving advancement technique

The COVID-19 lockdown posed challenges for traditional qualitative researchers, as well as opportunities for innovative qualitative research. While online interviews and focus groups via video conferencing platforms were the most widely used methods, Zoom Fatigue also occurred, hampering some participants' experiences and eventually impacting data quality. Incorporating virtual reality (VR) was one possible solution. This study explores the experiences and involvement of family participants in a qualitative research study using VR during the COVID-19 pandemic lockdown. The research aims to assess the impact of VR on data quantity and quality, as well as the depth of insights compared to traditional methods without VR. The study employs VR roleplay and autodriving methods to elicit family participant engagement. The findings suggest that the application of VR in autodriving improves data quality and quantity while also making the research process more enjoyable for participating parents and children. The results indicate that VR holds significant potential for advancing qualitative research methodology, especially for previously underrepresented groups such as children.     

Practical advice for conducting ethical online experiments and questionnaires for United States psychologists

It is increasingly easy and, therefore, increasingly common to conduct experiments and questionnaire studies in online environments. However, the online environment is not a data collection medium that is familiar to many researchers or to many research methods instructors. Because of this, researchers have received little information about how to address ethical issues when conducting online research. Researchers need practical suggestions on how to translate federal and professional ethics codes into this new data collection medium. This article assists United States psychologists in designing online studies that meet accepted standards for informed consent, deception, debriefing, the right to withdraw, security of test materials, copyright of participants’ materials, confidentiality and anonymity, and avoiding harm.     

Online Data Collection: Strategies for Research

Online data collection, through e‐mail and Web‐based surveys, is becoming an increasingly popular research methodology. In this article, the authors outline the benefits and limitations of this type of data collection to help researchers determine whether their data could be collected online in a way that retains the integrity of the data. A detailed procedure, including strategies to manage limitations, is given for researchers wishing to conduct their own online surveys.     

Studying sense of agency online: Can intentional binding be observed in uncontrolled online settings?

Intentional binding is often used as an implicit index of the sense of agency. However, intentional binding research has primarily been conducted in controlled lab environments. During the COVID-19 global pandemic, there has been a shift to implementing studies using online platforms and it is an open question whether the intentional binding effect can be found using an online experimental set-up and participant sample. Here, we address this question by asking online participants to complete the Libet clock version of the intentional binding task, which we make freely available to researchers as a jsPsych (De Leeuw, 2014) plugin. Intentional binding was observed in the form of later keypress estimates and earlier auditory tone estimates, when the auditory tone followed the keypress. These findings confirm that intentional binding can be assessed in online contexts. We discuss these findings in relation to the broader intentional binding literature.     

Hopes and cautions in implementing Bayesian structural equation modeling

Muthén and Asparouhov (2012) have proposed and demonstrated an approach to model specification and estimation in structural equation modeling (SEM) using Bayesian methods. Their contribution builds on previous work in this area by (a) focusing on the translation of conventional SEM models into a Bayesian framework wherein parameters fixed at zero in a conventional model can be respecified using small-variance priors and (b) implementing their approach in software that is widely accessible. We recognize potential benefits for applied researchers as discussed by Muthén and Asparouhov, and we also see a tradeoff in that effective use of the proposed approach introduces increased demands in terms of expertise of users to navigate new complexities in model specification, parameter estimation, and evaluation of results. We also raise cautions regarding the issues of model modification and model fit. Although we see significant potential value in the use of Bayesian SEM, we also believe that effective use will require an awareness of these complexities. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

Online data collection from video game players: methodological issues.

The paper outlines the advantages and disadvantages of using the Internet to collect data concerning both online and offline gamers. Drawing from experience of a number of studies carried out online by the authors and by reviewing the available literature, the authors discuss the main issues concerning data collected from video game players. The paper examines a number of areas, including recruiting and utilizing participants, validity, suitable methods of data collection (i.e., questionnaire studies, online tests, participant observation, online interviews), and ethical issues. It is concluded that online research methods can be a useful way of examining the psychosocial aspects of video game playing.     

The state of web-based research: A survey and call for inclusion in curricula

The first papers that reported on conducting psychological research on the web were presented at the Society for Computers in Psychology conference 20 years ago, in 1996. Since that time, there has been an explosive increase in the number of studies that use the web for data collection. As such, it seems a good time, 20 years on, to examine the health and adoption of sound practices of research on the web. The number of studies conducted online has increased dramatically. Overall, it seems that the web can be a method for conducting valid psychological studies. However, it is less clear that students and researchers are aware of the nature of web research. While many studies are well conducted, there is also a certain laxness appearing regarding the design and conduct of online studies. This laxness appears both anecdotally to the authors as managers of large sites for posting links to online studies, and in a survey of current researchers. One of the deficiencies discovered is that there is no coherent approach to educating researchers as to the unique features of web research.     

Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk

Psychological researchers have begun to utilize Amazon’s Mechanical Turk (MTurk) marketplace as a participant pool. Although past work has established that MTurk is well suited to examining individual behavior, pseudo-dyadic interactions, in which participants falsely believe they are interacting with a partner, are a key element of social and cognitive psychology. The ability to conduct such interdependent research on MTurk would increase the utility of this online population for a broad range of psychologists. The present research therefore attempts to qualitatively replicate well-established pseudo-dyadic tasks on MTurk in order to establish the utility of this platform as a tool for researchers. We find that participants do behave as if a partner is real, even when doing so incurs a financial cost, and that they are sensitive to subtle information about the partner in a minimal-groups paradigm, supporting the use of MTurk for pseudo-dyadic research.     

Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English

In the search to find cheaper, faster approaches for data collection, crowdsourcing methods (i.e., online labor portals that allow independent workers to complete surveys for compensation) have risen in popularity as a tool for personality researchers, despite a lack of evidence regarding the equivalence of crowdsourcing with traditional data collection methods. The purpose of this study was to evaluate crowdsourcing as a data collection tool by examining the measurement equivalence of crowdsourced data (i.e., from Amazon.com’s MTurk) with more traditional samples (i.e., an undergraduate sample and a sample of organizational employees). Our results (using a popular measure of Big Five personality) provided evidence of measurement equivalence across all three samples, with one important exception: crowdsourced data (from MTurk) only exhibited measurement invariance with traditional data collection methods when responses were restricted to participants from native-English speaking countries. Although MTurk appears to be an easy, cost-effective data collection tool, our results suggest that MTurk data are similar to traditionally-collected data only when the MTurk sample is restricted to IP addresses from English-speaking countries.     

Using the Delphi Technique: Experience from the world of counselling and psychotherapy

Background: Delphi studies have been popular in the health and social care domain for some time, particularly within nursing. They are used to elicit views and opinions from experts in the field on specific topics, without the need for face‐to‐face interviews. Focus: This paper brings Delphi methodology to the attention of potential researchers in the counselling and psychotherapy field, who may not have previously considered Delphi methodology, describing the researcher's experience of using a Delphi study through the media of electronic mail, and incorporating participant feedback. Aims: To discuss the advantages and disadvantages, as well as the complexities and limitations of the Delphi methodology. I will outline previous recommendations from Delphi researchers and offer further suggestions based on my experience of completing a Delphi study.     

Online implementation of an event history calendar with formr: A tutorial

Event history calendars (EHCs) are popular tools for retrospective data collection. Originally conceptualized as face‐to‐face interviews, EHCs contain various questions about the respondents' autobiography in order to use their experiences as cues to facilitate remembering. For relationship researchers, EHCs are particularly valuable when trying to reconstruct the relational past of individuals. However, although many studies are conducted online nowadays, no freely available online adaptation of the EHC is available yet. In this tutorial, detailed instructions are provided on how to implement an online EHC for the reconstruction of romantic relationship histories within the open‐source framework formr. Ways to customize the online EHC and provide a template for researchers to adapt the tool for their own purposes are showcased.     

psiTurk: An open-source framework for conducting replicable behavioral experiments online

Online data collection has begun to revolutionize the behavioral sciences. However, conducting carefully controlled behavioral experiments online introduces a number of new of technical and scientific challenges. The project described in this paper, psiTurk, is an open-source platform which helps researchers develop experiment designs which can be conducted over the Internet. The tool primarily interfaces with Amazon’s Mechanical Turk, a popular crowd-sourcing labor market. This paper describes the basic architecture of the system and introduces new users to the overall goals. psiTurk aims to reduce the technical hurdles for researchers developing online experiments while improving the transparency and collaborative nature of the behavioral sciences.     

心理治疗结果的临床显著性分析方法

心理治疗结果研究中,如何对结果测量数据进行统计分析,是非常重要的环节之一。研究者发现,不能简单使用统计显著性来考察心理治疗的结果,因而引入了临床显著性的概念以便更准确地从临床角度考察治疗的情况。在验证临床显著性改变中,西方学者提出了JT法、GLN法、EN法和HA法等统计分析方法,这些方法都试图更准确地分析临床显著性,但在心理治疗结果研究中,这些方法仍然存在概念明确性、标准和数据收集完善性等方面的问题。因此今后的研究应该关注方法的创新或修订,还是关注数据解释是目前西方关于临床显著性分析方法的两大争论。国内也应在心理治疗结果的研究中引入临床显著性分析方法,同时还应考虑效果评价指标的设立和有效测量工具编制的问题,为正确评价心理治疗结果服务,并最终为提高心理治疗效果服务。