Online Data Collection: Strategies for Research

Online data collection, through e‐mail and Web‐based surveys, is becoming an increasingly popular research methodology. In this article, the authors outline the benefits and limitations of this type of data collection to help researchers determine whether their data could be collected online in a way that retains the integrity of the data. A detailed procedure, including strategies to manage limitations, is given for researchers wishing to conduct their own online surveys.     

Data Collection via Online Platforms: Challenges and Recommendations for Future Research

Online platforms such as Amazon's Mechanical Turk (MTurk) are increasingly used by researchers to collect survey and experimental data. Yet, such platforms often represent a tumultuous terrain for both researchers and reviewers. Researchers have to navigate the complexities of obtaining representative samples from online participant cohorts, ensuring data quality, ethically incentivizing participant engagement, and maintaining transparency. Reviewers, on the other hand, have to navigate the complexities of evaluating the efficacy of such data collection and execution efforts in answering important research questions. In order to provide clarity to these issues, this article provides researchers and reviewers with a series of recommendations for effectively executing and evaluating data collection via online platforms, respectively.     

The state of web-based research: A survey and call for inclusion in curricula

The first papers that reported on conducting psychological research on the web were presented at the Society for Computers in Psychology conference 20 years ago, in 1996. Since that time, there has been an explosive increase in the number of studies that use the web for data collection. As such, it seems a good time, 20 years on, to examine the health and adoption of sound practices of research on the web. The number of studies conducted online has increased dramatically. Overall, it seems that the web can be a method for conducting valid psychological studies. However, it is less clear that students and researchers are aware of the nature of web research. While many studies are well conducted, there is also a certain laxness appearing regarding the design and conduct of online studies. This laxness appears both anecdotally to the authors as managers of large sites for posting links to online studies, and in a survey of current researchers. One of the deficiencies discovered is that there is no coherent approach to educating researchers as to the unique features of web research.     

Building and managing an online panel with phpPanelAdmin

Online data collection within social and behavioral research is burgeoning. For surveying individuals online, researchers are increasingly relying on online panels and similar subject pools. phpPanelAdmin is a Web-based platform that is intended to assist researchers in quickly setting up and managing an online panel. phpPanelAdmin solves many problems that can arise when one is collecting data in online panels. As an open-source program placed under GNU General Public License, phpPanelAdmin (www.goeritz.net/panelware) is not only free of charge but free of restrictions on its use, modification, and distribution. Building a new online panel—which before required weeks and special skills—is now possible within minutes and at no cost. Furthermore, phpPanelAdmin can be connected to an existing online panel or even be used for managing offline subject pools. Functions include creating a panel sign-up form; automatically unfolding the panel database; searching for panelists; viewing, modifying, and deleting panelists’ data; deleting and cloning variables; drawing samples; sending e-mail to panelists; exporting data; displaying panel statistics at a glance; identifying duplicates; creating and managing e-mail templates; executing user-defined SQL statements; and obtaining help.     

Online implementation of an event history calendar with formr: A tutorial

Event history calendars (EHCs) are popular tools for retrospective data collection. Originally conceptualized as face‐to‐face interviews, EHCs contain various questions about the respondents' autobiography in order to use their experiences as cues to facilitate remembering. For relationship researchers, EHCs are particularly valuable when trying to reconstruct the relational past of individuals. However, although many studies are conducted online nowadays, no freely available online adaptation of the EHC is available yet. In this tutorial, detailed instructions are provided on how to implement an online EHC for the reconstruction of romantic relationship histories within the open‐source framework formr. Ways to customize the online EHC and provide a template for researchers to adapt the tool for their own purposes are showcased.     

Levels of analysis and sources of data in sales research: a multilevel-multisource review

Sales researchers are increasingly adopting a multilevel-multisource (MLMS) approach to answer many key questions involving sales managers, salespeople and customers. MLMS research involves the acquisition and analysis of data collected from two or more sources pertaining to multiple hierarchical levels and presents a number of opportunities and challenges for sales researchers to consider. The authors highlight the benefits and the drawbacks of MLMS research and describe various approaches for implementing an MLMS collection and analysis. Additionally, a review of the MLMS studies conducted in the sales domain is provided which summarizes and delineates relationships tested in the extant literature. Based on this review, the authors advance a number of underdeveloped areas of research where MLMS approaches can be applied to further the understanding of the dynamic conditions that typify sales research.     

THE EVOLUTION OF A RESEARCH DESIGN: A CASE STUDY IN CONTENT READING

In recent years, educators and researchers have become increasingly interested in naturalistic or responsive research. This approach to inquiry seems to be uniquely suited to research in the complex educational setting found in classrooms. This paper discusses one aspect of the application of a responsive paradigm to research in reading in the content area ‐‐ the evolution of the research design. In this study of four secondary school social studies classes major modifications in design were made at several points. These modifications are described and the reasons for the changes are discussed. In summary, allowing the design to emerge as the study progressed resulted in the collection of data that provided a complete view of reading in these classes. In addition, on a personal level, the researchers found that engaging in this type of inquiry influenced their views about how to collect data and what data to collect while, at the same time, enlarging their perspective of reading occurring in content area classrooms.     

Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.     

Assessing and overcoming participant dishonesty in online data collection

Crowdsourcing services, such as MTurk, have opened a large pool of participants to researchers. Unfortunately, it can be difficult to confidently acquire a sample that matches a given demographic, psychographic, or behavioral dimension. This problem exists because little information is known about individual participants and because some participants are motivated to misrepresent their identity with the goal of financial reward. Despite the fact that online workers do not typically display a greater than average level of dishonesty, when researchers overtly request that only a certain population take part in an online study, a nontrivial portion misrepresent their identity. In this study, a proposed system is tested that researchers can use to quickly, fairly, and easily screen participants on any dimension. In contrast to an overt request, the reported system results in significantly fewer (near zero) instances of participant misrepresentation. Tests for misrepresentations were conducted by using a large database of past participant records (~45,000 unique workers). This research presents and tests an important tool for the increasingly prevalent practice of online data collection.     

Improving online panel data usage in sales research

The exponential expansion of the Internet has fundamentally altered the nature of personal selling and sales management. The Internet has also created significant change for sales researchers and proffered a bevy of new means of conducting research. Online panel data collection is one such means that has recently proliferated. Online panel data collection imbues a host of advantages sales researchers may reap. However, there are a proportionate number of disadvantages and concerns with collecting and using this type of data. This article delineates said benefits and drawbacks and also provides a review of recent sales-related examples employing this method of data collection. Furthermore, key considerations in utilizing online panel data are advanced. Importantly, several ways to maximize the benefits and minimize the drawbacks inherent to online panel data are provided. This article aims to provide a resource for sales scholars in optimizing their usage of online panel data.     

Open-source software to conduct online rating studies

Several software programs exist to assist researchers in setting up online questionnaires. Existing tools are of little help for delivering online rating studies, for which it is often desirable to collect data from participants for only a subset of a stimulus set. OR-Vis enables researchers to quickly set up online rating studies by supplying the set of items to be rated, the number of stimuli an individual participant responds to, the number of participants an item is shown to, and the rating questions. The software then generates and delivers unique questionnaires for each participant, while managing the data collection process. The present article describes OR-Vis, its installation process, and how to use it to gather data. OR-Vis is open-source software and can be downloaded from www.orvis.uni-muenster.de.     

TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences

In recent years, Mechanical Turk (MTurk) has revolutionized social science by providing a way to collect behavioral data with unprecedented speed and efficiency. However, MTurk was not intended to be a research tool, and many common research tasks are difficult and time-consuming to implement as a result. TurkPrime was designed as a research platform that integrates with MTurk and supports tasks that are common to the social and behavioral sciences. Like MTurk, TurkPrime is an Internet-based platform that runs on any browser and does not require any downloads or installation. Tasks that can be implemented with TurkPrime include: excluding participants on the basis of previous participation, longitudinal studies, making changes to a study while it is running, automating the approval process, increasing the speed of data collection, sending bulk e-mails and bonuses, enhancing communication with participants, monitoring dropout and engagement rates, providing enhanced sampling options, and many others. This article describes how TurkPrime saves time and resources, improves data quality, and allows researchers to design and implement studies that were previously very difficult or impossible to carry out on MTurk. TurkPrime is designed as a research tool whose aim is to improve the quality of the crowdsourcing data collection process. Various features have been and continue to be implemented on the basis of feedback from the research community. TurkPrime is a free research platform.     

Principles for Effective Asynchronous Online Instruction in Religious Studies

Asynchronous online instruction has become increasingly popular in the field of religious studies. However, despite voluminous research on online learning in general and numerous articles on online theological instruction, there has been little discussion of how to effectively design and deliver online undergraduate courses in religious studies. Drawing on recent research, experiences teaching and learning online, and interviews with colleagues, this paper discusses key principles of effective online instruction. It recommends instructors focus on humanizing their course website, “chunking” their course content, making their approach to the study of religion clear, structuring and monitoring online discussions, prioritizing prompt and constructive feedback, and making course material relevant to learners.     

A Systematic Review of Online Response Rates in Four Counseling Journals

This systematic review of articles published in 4 counseling journals found 207 articles that used online recruitment, of which 98 (47.3%) reported a response rate. These 98 studies collectively contained 22,466 individual participants. Results showed that nearly one third of all empirical articles used online recruitment, and the average response rate was 34.2% (SD = 22.6). Response rates are further described by journal, population, research methodology, and recruitment strategy. Researcher‐reported issues with online recruitment and data collection were also analyzed qualitatively, and recommendations for researchers are discussed.     

Comparing In-Person,Sona, and Mechanical Turk Measurements of Three Prejudice-Relevant Constructs

Electronic data collection and participant pool management tools give researchers new ways to conduct research. The current study investigated the equivalency of in-person and online administrations of the Right-Wing Authoritarianism, Social Dominance Orientation, and Modern Racism scales across three modalities (administration in person, online through Sona Systems, and online through Mechanical Turk). Results indicate that in-person administration was largely equivalent to the randomly assigned online sample (Sona Systems) in terms of their intercorrelations, mean scores, variability, and reliability. However, the Sona sample consistently responded with strong attitudes for all measures, suggesting that social desirability may be decreased when completing these measures online. In addition, the Mechanical Turk sample differed in many ways from both in-person and Sona administration; although this nonequivalence is at least partially explained by sample demographic differences, other considerations may have exacerbated nonequivalence, including prior exposure to the measures and a desire to respond correctly.     

Theoretically informed qualitative psychotherapy research: A primer

This paper presents an introduction to theoretically informed qualitative psychotherapy research (QPR). Although QPR researchers have traditionally remained silent on theory, we suggest this has resulted in an implicit and unacknowledged use of theory. We argue instead for a clear articulation of qualitative researchers' theory and outline how theory can be incorporated to inform the entire qualitative research process. This approach assumes the research problem is embedded in a clearly defined and articulated theoretical framework, which also informs data collection and data analysis. We outline how researchers can use explicit theoretical frameworks to inform research question formulation, data collection and data analysis and illustrate this with specific applications of the method in practice. We believe that starting from a declared theoretical framework sets up a dialogue between the research problem, the type of data required and their meaningful analysis and interpretation. This aims not only to achieve greater depth in the final product of research, but also to enhance its utility in terms of practice; it contributes to building, altering and differentiating theory; and it allows for greater transparency by openly articulating the theoretical framework that scaffolds the entirety of the research process.     

An overview of issues in infant and developmental research for the creation of robust and replicable science

In the last few years, the field of psychology has been challenged with a crisis in the rigor and reproducibility of the science. The focus of these issues has primarily been in social, cognitive, and cognitive neuroscience psychology, however, the area of developmental research is not immune to these issues. This paper provides an overview of the “replication crisis” and the choices made by researchers that are often not noted in methods, thus making the replication of studies more difficult. In this review we discuss issues of researcher flexibility in the data design and selection of sample size, collection, and analysis stages of research. In each of these areas we address examples of bias and how developmental researchers can address these issues in their own research.     

Dyadic Models for the Study of Health

Dyadic designs have been used in health research to investigate intra‐ and inter‐personal mechanisms of health and well‐being in various types of dyads, including parent–child dyads, siblings, friends, and romantic partners. Although a growing number of researchers are designing studies that capture the interdependent complexities of relationships, many still need more information on how to analyze the data in a way that maximizes its value. Therefore, the purpose of this review paper is twofold: (1) to address some of the ways in which dyadic data analysis is being used in current health research, with an emphasis on research that has employed the Actor‐Partner Interdependence Model, and (2) to propose and explain various methodological and substantive considerations that researchers should consider when using dyadic data analysis in their own research.