Online implementation of an event history calendar with formr: A tutorial

Event history calendars (EHCs) are popular tools for retrospective data collection. Originally conceptualized as face‐to‐face interviews, EHCs contain various questions about the respondents' autobiography in order to use their experiences as cues to facilitate remembering. For relationship researchers, EHCs are particularly valuable when trying to reconstruct the relational past of individuals. However, although many studies are conducted online nowadays, no freely available online adaptation of the EHC is available yet. In this tutorial, detailed instructions are provided on how to implement an online EHC for the reconstruction of romantic relationship histories within the open‐source framework formr. Ways to customize the online EHC and provide a template for researchers to adapt the tool for their own purposes are showcased.     

Improving online panel data usage in sales research

The exponential expansion of the Internet has fundamentally altered the nature of personal selling and sales management. The Internet has also created significant change for sales researchers and proffered a bevy of new means of conducting research. Online panel data collection is one such means that has recently proliferated. Online panel data collection imbues a host of advantages sales researchers may reap. However, there are a proportionate number of disadvantages and concerns with collecting and using this type of data. This article delineates said benefits and drawbacks and also provides a review of recent sales-related examples employing this method of data collection. Furthermore, key considerations in utilizing online panel data are advanced. Importantly, several ways to maximize the benefits and minimize the drawbacks inherent to online panel data are provided. This article aims to provide a resource for sales scholars in optimizing their usage of online panel data.     

Practical advice for conducting ethical online experiments and questionnaires for United States psychologists

It is increasingly easy and, therefore, increasingly common to conduct experiments and questionnaire studies in online environments. However, the online environment is not a data collection medium that is familiar to many researchers or to many research methods instructors. Because of this, researchers have received little information about how to address ethical issues when conducting online research. Researchers need practical suggestions on how to translate federal and professional ethics codes into this new data collection medium. This article assists United States psychologists in designing online studies that meet accepted standards for informed consent, deception, debriefing, the right to withdraw, security of test materials, copyright of participants’ materials, confidentiality and anonymity, and avoiding harm.     

Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English

In the search to find cheaper, faster approaches for data collection, crowdsourcing methods (i.e., online labor portals that allow independent workers to complete surveys for compensation) have risen in popularity as a tool for personality researchers, despite a lack of evidence regarding the equivalence of crowdsourcing with traditional data collection methods. The purpose of this study was to evaluate crowdsourcing as a data collection tool by examining the measurement equivalence of crowdsourced data (i.e., from Amazon.com’s MTurk) with more traditional samples (i.e., an undergraduate sample and a sample of organizational employees). Our results (using a popular measure of Big Five personality) provided evidence of measurement equivalence across all three samples, with one important exception: crowdsourced data (from MTurk) only exhibited measurement invariance with traditional data collection methods when responses were restricted to participants from native-English speaking countries. Although MTurk appears to be an easy, cost-effective data collection tool, our results suggest that MTurk data are similar to traditionally-collected data only when the MTurk sample is restricted to IP addresses from English-speaking countries.     

Data Collection via Online Platforms: Challenges and Recommendations for Future Research

Online platforms such as Amazon's Mechanical Turk (MTurk) are increasingly used by researchers to collect survey and experimental data. Yet, such platforms often represent a tumultuous terrain for both researchers and reviewers. Researchers have to navigate the complexities of obtaining representative samples from online participant cohorts, ensuring data quality, ethically incentivizing participant engagement, and maintaining transparency. Reviewers, on the other hand, have to navigate the complexities of evaluating the efficacy of such data collection and execution efforts in answering important research questions. In order to provide clarity to these issues, this article provides researchers and reviewers with a series of recommendations for effectively executing and evaluating data collection via online platforms, respectively.     

Understanding the Complexity of Everyday Interethnic Contact: Recommendations for Researchers

Interethnic contact has long been a focus of social psychological research, and in this article, we provide some recommendations about how to study interethnic social contact. Our primary recommendation is that researchers should study interethnic contact more in vivo to complement single assessment surveys and laboratory research, sometimes of arbitrarily defined groups, with more naturalistic research and with research involving groups that exist in the real world. This recommendation is based in part on the limitations of surveys and laboratory methods and in part on the advantages provided by what are referred to as intensive repeated measures designs, methods that have proven to be useful in studying other topics. We discuss these limitations and the relative advantages of different types of intensive repeated measures methods for the study of interethnic contact and provide some recommendations for their use. Finally, regardless of how they study interethnic contact, researchers need to expand the focus of their questions, increase the sensitivity of the data they collect, and increase the ethnic diversity of the samples they study.     

Assessing and overcoming participant dishonesty in online data collection

Crowdsourcing services, such as MTurk, have opened a large pool of participants to researchers. Unfortunately, it can be difficult to confidently acquire a sample that matches a given demographic, psychographic, or behavioral dimension. This problem exists because little information is known about individual participants and because some participants are motivated to misrepresent their identity with the goal of financial reward. Despite the fact that online workers do not typically display a greater than average level of dishonesty, when researchers overtly request that only a certain population take part in an online study, a nontrivial portion misrepresent their identity. In this study, a proposed system is tested that researchers can use to quickly, fairly, and easily screen participants on any dimension. In contrast to an overt request, the reported system results in significantly fewer (near zero) instances of participant misrepresentation. Tests for misrepresentations were conducted by using a large database of past participant records (~45,000 unique workers). This research presents and tests an important tool for the increasingly prevalent practice of online data collection.     

A Systematic Review of Online Response Rates in Four Counseling Journals

This systematic review of articles published in 4 counseling journals found 207 articles that used online recruitment, of which 98 (47.3%) reported a response rate. These 98 studies collectively contained 22,466 individual participants. Results showed that nearly one third of all empirical articles used online recruitment, and the average response rate was 34.2% (SD = 22.6). Response rates are further described by journal, population, research methodology, and recruitment strategy. Researcher‐reported issues with online recruitment and data collection were also analyzed qualitatively, and recommendations for researchers are discussed.     

Using personal digital assistants (PDAs) for the collection of safety belt use data in the field

From 1975 through 2003, because of the use of safety belts, an estimated 180,000 deaths from traffic crashes have been prevented. In order to assess the gains that have been made in safety belt use across the U.S., in individual states, and in communities, belt use surveys are conducted at regular intervals to determine use rates. The most valid method for surveying safety belt use is through direct observation. Direct observation surveys are conducted along roadways by trained researchers looking into passing vehicles and recording safety belt use. This method of data collection has been effective in the past through the use of paper-and-pencil data recording, yet it could be improved through the use of electronic and communication technology. Reported here is a study designed to compare electronic data collection, using personal digital assistants (PDAs), with collection using the traditional paperand-pencil method during the annual statewide survey of safety belt use in Michigan that we have been conducting since 1984. The goals of the study were to develop a PDA database program for data entry in the field, to directly compare the PDA data collection process with the paper-and-pencil method on both accuracy and speed, and to assess mechanical and environmental factors, such as battery life, screen visibility, and reaction to adverse weather, that may act as limitations to the PDA method, in comparison with the paper-and-pencil method. In a direct comparison of methods, two observers collected data at the same roadway intersections, one using paper and a pencil and one using a PDA equipped with our custom software. The study showed that the PDA method was as fast and as accurate as the paper-and-pencil method. There were no adverse effects on the PDA caused by environmental conditions. The PDA was superior to the paper-and-pencil method in rainy weather and for data collector supervision. In addition, the use of the PDA obviated the need for entry of paper-recorded data into an electronic format. We conclude that the use of PDAs for safety belt field data collection is superior to the paper-and-pencil method. A Web address is given where the custom PDA safety belt data collection software can be obtained free of charge.     

psiTurk: An open-source framework for conducting replicable behavioral experiments online

Online data collection has begun to revolutionize the behavioral sciences. However, conducting carefully controlled behavioral experiments online introduces a number of new of technical and scientific challenges. The project described in this paper, psiTurk, is an open-source platform which helps researchers develop experiment designs which can be conducted over the Internet. The tool primarily interfaces with Amazon’s Mechanical Turk, a popular crowd-sourcing labor market. This paper describes the basic architecture of the system and introduces new users to the overall goals. psiTurk aims to reduce the technical hurdles for researchers developing online experiments while improving the transparency and collaborative nature of the behavioral sciences.     

Perceptions of adolescent bullying: Attributions of blame and responsibility in cases of cyber‐bullying

Cyber‐bullying (where victims are targeted via online social networking or other electronic means) has gained increased attention in research and the broadcast media, but previous research has not investigated attribution of blame in such cyber‐bullying events. This experiment hypothesized that participants would assign higher ratings of blame to bullying perpetrators when the bullying situations were depicted as having highly foreseeable outcomes (vs. unforeseeable outcomes), and as occurring in school (vs. online). In addition, a significant interaction was predicted between outcome foreseeability and bullying situation, with highly foreseeable in‐school events being rated as the most predictable and attributable to the bully's actions. One‐hundred sixty‐three participants completed surveys containing demographic items, items regarding their past experiences of victimization, and one of four randomly‐assigned vignettes detailing a bullying situation (which participants rated). While hypotheses regarding outcome foreseeability were supported, no cyber‐bullying vs. in‐school main effects (or corresponding interaction effects) were detected. Implications for future research and practice, as well as study limitations, are discussed.     

Open-source software to conduct online rating studies

Several software programs exist to assist researchers in setting up online questionnaires. Existing tools are of little help for delivering online rating studies, for which it is often desirable to collect data from participants for only a subset of a stimulus set. OR-Vis enables researchers to quickly set up online rating studies by supplying the set of items to be rated, the number of stimuli an individual participant responds to, the number of participants an item is shown to, and the rating questions. The software then generates and delivers unique questionnaires for each participant, while managing the data collection process. The present article describes OR-Vis, its installation process, and how to use it to gather data. OR-Vis is open-source software and can be downloaded from www.orvis.uni-muenster.de.     

Comparing In-Person,Sona, and Mechanical Turk Measurements of Three Prejudice-Relevant Constructs

Electronic data collection and participant pool management tools give researchers new ways to conduct research. The current study investigated the equivalency of in-person and online administrations of the Right-Wing Authoritarianism, Social Dominance Orientation, and Modern Racism scales across three modalities (administration in person, online through Sona Systems, and online through Mechanical Turk). Results indicate that in-person administration was largely equivalent to the randomly assigned online sample (Sona Systems) in terms of their intercorrelations, mean scores, variability, and reliability. However, the Sona sample consistently responded with strong attitudes for all measures, suggesting that social desirability may be decreased when completing these measures online. In addition, the Mechanical Turk sample differed in many ways from both in-person and Sona administration; although this nonequivalence is at least partially explained by sample demographic differences, other considerations may have exacerbated nonequivalence, including prior exposure to the measures and a desire to respond correctly.     

Building and managing an online panel with phpPanelAdmin

Online data collection within social and behavioral research is burgeoning. For surveying individuals online, researchers are increasingly relying on online panels and similar subject pools. phpPanelAdmin is a Web-based platform that is intended to assist researchers in quickly setting up and managing an online panel. phpPanelAdmin solves many problems that can arise when one is collecting data in online panels. As an open-source program placed under GNU General Public License, phpPanelAdmin (www.goeritz.net/panelware) is not only free of charge but free of restrictions on its use, modification, and distribution. Building a new online panel—which before required weeks and special skills—is now possible within minutes and at no cost. Furthermore, phpPanelAdmin can be connected to an existing online panel or even be used for managing offline subject pools. Functions include creating a panel sign-up form; automatically unfolding the panel database; searching for panelists; viewing, modifying, and deleting panelists’ data; deleting and cloning variables; drawing samples; sending e-mail to panelists; exporting data; displaying panel statistics at a glance; identifying duplicates; creating and managing e-mail templates; executing user-defined SQL statements; and obtaining help.     

Assessing the Validity of Data Synthesis Methods to Estimate Religious Populations

The present study tests the validity of a data synthesis approach to population estimates of religiously defined groups. This is particularly important in places like the United States, where there is no definitive source of official data on its population's religious composition, and researchers must rely on costly, large‐scale surveys, or congregational membership studies. Each approach has limitations, especially for estimation of small religious groups and for estimation within small geographic areas. Without official statistics, the degree of bias in estimates is unknown. Data synthesis, specifically Bayesian multilevel estimation with poststratification, offers a useful alternative that maximizes the utility of data across all sources to estimate multiple groups from the same sources of data. This method also facilitates comparison of groups. This study provides evidence of the validity of the approach by synthesizing data from Canada, a country that includes questions about religious identification in its national census.     

Performing high‐powered studies efficiently with sequential analyses

Running studies with high statistical power, while effect size estimates in psychology are often inaccurate, leads to a practical challenge when designing an experiment. This challenge can be addressed by performing sequential analyses while the data collection is still in progress. At an interim analysis, data collection can be stopped whenever the results are convincing enough to conclude that an effect is present, more data can be collected, or the study can be terminated whenever it is extremely unlikely that the predicted effect will be observed if data collection would be continued. Such interim analyses can be performed while controlling the Type 1 error rate. Sequential analyses can greatly improve the efficiency with which data are collected. Additional flexibility is provided by adaptive designs where sample sizes are increased on the basis of the observed effect size. The need for pre‐registration, ways to prevent experimenter bias, and a comparison between Bayesian approaches and null‐hypothesis significance testing (NHST) are discussed. Sequential analyses, which are widely used in large‐scale medical trials, provide an efficient way to perform high‐powered informative experiments. I hope this introduction will provide a practical primer that allows researchers to incorporate sequential analyses in their research. Copyright © 2014 John Wiley & Sons, Ltd.     

World-Wide Web survey research: Benefits, potential problems, and solutions

The World-Wide Web presents survey researchers with an unprecedented tool for the collection of data. The costs in terms of both time and money for publishing a survey on the Web are low compared with costs associated with conventional surveying methods. The data entry stage is eliminated for the survey administrator, and software can ensure that the data acquired from participants is free from common entry errors. Importantly, Web surveys can interactively provide participants with customized feedback. These features come at a price—ensuring that appropriately written software manages the data collection process. Although the potential for missing data, unacceptable responses, duplicate submissions, and Web abuse exist, one can take measures when creating the survey software to minimize the frequency and negative consequences of such incidents.     

Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.     

The state of web-based research: A survey and call for inclusion in curricula

The first papers that reported on conducting psychological research on the web were presented at the Society for Computers in Psychology conference 20 years ago, in 1996. Since that time, there has been an explosive increase in the number of studies that use the web for data collection. As such, it seems a good time, 20 years on, to examine the health and adoption of sound practices of research on the web. The number of studies conducted online has increased dramatically. Overall, it seems that the web can be a method for conducting valid psychological studies. However, it is less clear that students and researchers are aware of the nature of web research. While many studies are well conducted, there is also a certain laxness appearing regarding the design and conduct of online studies. This laxness appears both anecdotally to the authors as managers of large sites for posting links to online studies, and in a survey of current researchers. One of the deficiencies discovered is that there is no coherent approach to educating researchers as to the unique features of web research.