Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk

Psychological researchers have begun to utilize Amazon’s Mechanical Turk (MTurk) marketplace as a participant pool. Although past work has established that MTurk is well suited to examining individual behavior, pseudo-dyadic interactions, in which participants falsely believe they are interacting with a partner, are a key element of social and cognitive psychology. The ability to conduct such interdependent research on MTurk would increase the utility of this online population for a broad range of psychologists. The present research therefore attempts to qualitatively replicate well-established pseudo-dyadic tasks on MTurk in order to establish the utility of this platform as a tool for researchers. We find that participants do behave as if a partner is real, even when doing so incurs a financial cost, and that they are sensitive to subtle information about the partner in a minimal-groups paradigm, supporting the use of MTurk for pseudo-dyadic research.     

Renewal and resurgence phenomena generalize to Amazon's Mechanical Turk

Amazon's Mechanical Turk (MTurk) is a crowdsourcing platform that provides researchers with the potential for obtaining behavioral data for very little cost. However, the extent to which the results of common behavioral phenomena found in basic, translational, and applied laboratories may be reproduced (as a first step towards prospective research) via MTurk remains relatively unexplored. We evaluated renewal and resurgence arrangements using MTurk as the subject recruitment platform as a first step to determining the generality of the obtained data. Results suggested that MTurk participants produced renewal and resurgence data similar to those reported in basic, translational, and applied studies.     

Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.     

Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English

In the search to find cheaper, faster approaches for data collection, crowdsourcing methods (i.e., online labor portals that allow independent workers to complete surveys for compensation) have risen in popularity as a tool for personality researchers, despite a lack of evidence regarding the equivalence of crowdsourcing with traditional data collection methods. The purpose of this study was to evaluate crowdsourcing as a data collection tool by examining the measurement equivalence of crowdsourced data (i.e., from Amazon.com’s MTurk) with more traditional samples (i.e., an undergraduate sample and a sample of organizational employees). Our results (using a popular measure of Big Five personality) provided evidence of measurement equivalence across all three samples, with one important exception: crowdsourced data (from MTurk) only exhibited measurement invariance with traditional data collection methods when responses were restricted to participants from native-English speaking countries. Although MTurk appears to be an easy, cost-effective data collection tool, our results suggest that MTurk data are similar to traditionally-collected data only when the MTurk sample is restricted to IP addresses from English-speaking countries.     

Assessing and overcoming participant dishonesty in online data collection

Crowdsourcing services, such as MTurk, have opened a large pool of participants to researchers. Unfortunately, it can be difficult to confidently acquire a sample that matches a given demographic, psychographic, or behavioral dimension. This problem exists because little information is known about individual participants and because some participants are motivated to misrepresent their identity with the goal of financial reward. Despite the fact that online workers do not typically display a greater than average level of dishonesty, when researchers overtly request that only a certain population take part in an online study, a nontrivial portion misrepresent their identity. In this study, a proposed system is tested that researchers can use to quickly, fairly, and easily screen participants on any dimension. In contrast to an overt request, the reported system results in significantly fewer (near zero) instances of participant misrepresentation. Tests for misrepresentations were conducted by using a large database of past participant records (~45,000 unique workers). This research presents and tests an important tool for the increasingly prevalent practice of online data collection.     

The pyeTribe: Simultaneous eyetracking for economic games

The recent introduction of inexpensive eyetrackers has opened up a wealth of opportunities for researchers to study attention in interactive tasks. No software package has previously been available to help researchers exploit those opportunities. We created “the pyeTribe,” a software package that offers, among others, the following features: first, a communication platform between many eyetrackers to allow for simultaneous recording of multiple participants; second, the simultaneous calibration of multiple eyetrackers without the experimenter’s supervision; third, data collection restricted to periods of interest, thus reducing the volume of data and easing analysis. We used a standard economic game (the public goods game) to examine the data quality and demonstrate the potential of our software package. Moreover, we conducted a modeling analysis, which illustrates how combining process and behavioral data can improve models of human decision-making behavior in social situations. Our software is open source.     

Using Amazon's Mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction

This study investigated Amazon.com's website Mechanical Turk (MTurk) as a research tool for measuring body size estimation and dissatisfaction. 160 U.S. participants completed the BIAS-BD figural drawing scale and demographic questions posted on the MTurk website. The BIAS-BD consists of 17 drawings of various male and female body sizes based on anthropometric data corresponding to a range of 60% below to 140% above the average U.S. adult. Respondents selected a drawing that best reflected their current size and ideal size. Results revealed that respondents overestimated their body size by 6% and desired an ideal size 9.2% smaller than their perceived size. Findings are compared with three previous studies using the BIAS-BD scale. A general correspondence in findings between the four studies was found. We conclude that the MTurk can serve as a viable method for collecting data on the perceptual and attitudinal aspects of body image quickly and inexpensively.     

Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations

Purpose

Amazon Mechanical Turk is an increasingly popular data source in the organizational psychology research community. This paper presents an evaluation of MTurk and provides a set of practical recommendations for researchers using MTurk.

Design/Methodology/Approach

We present an evaluation of methodological concerns related to the use of MTurk and potential threats to validity inferences. Based on our evaluation, we also provide a set of recommendations to strengthen validity inferences using MTurk samples.

Findings

Although MTurk samples can overcome some important validity concerns, there are other limitations researchers must consider in light of their research objectives. Researchers should carefully evaluate the appropriateness and quality of MTurk samples based on the different issues we discuss in our evaluation.

Implications

There is not a one-size-fits-all answer to whether MTurk is appropriate for a research study. The answer depends on the research questions and the data collection and analytic procedures adopted. The quality of the data is not defined by the data source per se, but rather the decisions researchers make during the stages of study design, data collection, and data analysis.

Originality/Value

The current paper extends the literature by evaluating MTurk in a more comprehensive manner than in prior reviews. Past review papers focused primarily on internal and external validity, with less attention paid to statistical conclusion and construct validity—which are equally important in making accurate inferences about research findings. This paper also provides a set of practical recommendations in addressing validity concerns when using MTurk.

     

A procedure for studying online conversational processing using a chat bot

This article reports the development of a tool for examining the social and cognitive processes of people involved in a conversational interaction. Research on how people process utterances while they are actually engaged in an interaction has been extremely rare. To that end, we have developed a conversational bot (computer program designed to mimic human communication) with which participants can chat in a format similar to instant messaging. This program allows for the recording of comprehension speed, and it can be interfaced with secondary tasks (e.g., lexical decisions) in order to examine online conversational processing. Additional research possibilities with this program are discussed.     

An evaluation of a caregiver training protocol on the generalization and maintenance of successful pediatric feeding interventions

Previous researchers have documented the positive effects achieved by trained clinicians providing behavioral interventions for pediatric feeding disorders; however, few have evaluated the maintenance of those treatments when subsequently implemented by primary caregivers. Further, the majority of previous caregiver training research has relied on the use of multicomponent training packages making it difficult to determine which components are necessary for success. The purpose of the current investigation was to evaluate the effects of instructions and feedback on caregivers' implementation of feeding protocols in a home setting. Results of the current study suggest that feedback may be the most effective training component in training packages designed to teach caregivers to implement an effective feeding protocol. Potential concomitant changes in child behavior are also discussed.     

Associations Between Social Anxiety and Affective and Empathic Forecasts: A Replication and Extension in a Mechanical Turk Sample

People often overestimate the intensity and duration of their future emotions, referred to as an impact bias. Impact biases have been documented in predictions people make about their own emotions, as well as the others’ emotions (i.e., affective and empathic forecasting, respectively). Recent studies have shown that negative impact biases may be stronger, and positive impact biases may be attenuated, in individuals with symptoms of social anxiety. The current study sought to replicate and extend these findings in a Mechanical Turk (MTurk) sample. MTurk is a particularly interesting online platform for such research because of the unusually high prevalence of social anxiety among MTurk users. Within a computer-based survey, 93 MTurk users read vignettes in which a second-person narrator elicited either disgust, anger, or happiness from another person. After each vignette, participants predicted how the narrator (i.e., affective forecasts) and the other person (i.e., empathic forecasts) would feel. Overall, results confirmed the existence of associations between social anxiety symptoms and negative affective and empathic forecasting biases, though no significant relations were found between social anxiety symptoms and positive forecasting biases. Negative affective and empathic forecasting biases were significantly correlated. Age and gender were also examined as potential predictors and moderators of hypothesized effects. Though younger age and female gender were associated with specific forecast ratings, controlling for these variables did not alter the associations between social anxiety and affective or empathic forecasts and no moderation effects were found. Overall, results provide additional support for the relevance of impact biases to social anxiety and suggest that they may be useful targets of intervention.     

Incorporating good programs into the larger laboratory context

The Scheduled Measurement System (SMS) is a collection of related programs for administering performance and rating tasks to human subjects. It was designed to provide a common task scheduling and data management environment for a diversity of measures. SMS has greatly reduced the effort required to analyze and archive data after experiments, but it has been less successful in reducing some of the effort required to implement experiments. While programming is not difficult, documentation and user training remain the most primitive parts of the system.     

Intentional memory and online data collection: A test of the effects of animacy and threat on episodic memory

As research in the behavioural sciences increasingly relies on Amazon’s Mechanical Turk (MTurk) for recruiting participants, it is an empirical question whether an intentional memory experiment could be conducted online and whether performance of MTurk participants and college students would be comparable. College students and MTurk participants viewed threatening and nonthreatening items that were animate or inanimate and completed a recall test. Recent research indicates that human memory systems evolved to prioritise survival-relevant information, including animacy of items. The results of both studies indicated that participants recalled more animate than inanimate items and recalled more threatening items than nonthreatening items. The results indicate that it is possible to conduct memory experiments online and that results match patterns found in laboratory studies of the animacy effect (Nairne, VanArsdall, Pandeirada, Cogdill, & LeBreton, 2013), but that researchers should include safeguards to ensure the online participants are attending to and following directions.     

Effects of Public Posting and Goal Setting on Team Performance in a Residential Setting

Treatment integrity has been shown to be of critical importance in behavioral programming. The extent to which tasks are completed as designed often has a direct effect on the quality of care provided to clients. Most research on treatment integrity has focused on implementation of interventions. The current study used a multiple baseline design across behaviors to evaluate the effects of public posting of team performance data and goal setting on data collection by direct‐care staff. Results of the current study suggest that public posting and goal setting may be effective strategies to improve staff data collection. Copyright © 2016 John Wiley & Sons, Ltd.     

Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

Crowdsourcing services—particularly Amazon Mechanical Turk—have made it easy for behavioral scientists to recruit research participants. However, researchers have overlooked crucial differences between crowdsourcing and traditional recruitment methods that provide unique opportunities and challenges. We show that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants. We describe how both of these problems can be avoided using advanced interface features that also allow prescreening and longitudinal data collection. Using these techniques can minimize the effects of previously ignored drawbacks and expand the scope of crowdsourcing as a tool for psychological research.     

Effects of the coronavirus pandemic on perceived capitalization support provision and receipt

The coronavirus pandemic has influenced relationship dynamics due to heightened stress, increased isolation, and the need for social distancing. Prior work suggests receiving support for positive events (“capitalization support”) during this time could boost well-being, but only if it is provided in a manner that is active (involved) and constructive (supportive), as opposed to passive (uninvolved) and/or destructive (discouraging). Given the need for reduced social contact, we predicted the pandemic would make providing active, constructive support more challenging and might increase the frequency of passive support. We conducted two studies to explore how perceptions of capitalization methods may have shifted during the pandemic using an online participant pool (MTurk). Overall, we found that participants reported receiving significantly less active-constructive capitalization support and receiving more passive-constructive support during the pandemic (compared to pre-pandemic perceptions). In addition, we found that those who reported receiving and providing more active-constructive support tended to report more positive well-being outcomes. Interestingly, passive-constructive support was also associated with greater well-being for recipients (counter to prior research that has shown it associated with only costs). Overall, our data suggests that capitalization support perceptions are associated with well-being even during the pandemic, but methods of capitalization support perceived may have shifted during the pandemic.     

Construal‐level Mindsets Enhance Behavioral Persistence in Response to Incentive Valence

In this research, we find that incentive valence and construal‐level mindsets can interact to influence behavioral persistence on challenging tasks. An abstract mindset improves persistence in response to positively framed incentives whereas a concrete mindset improves persistence in response to negatively framed incentives. This interaction effect can be observed even when the cues inducing construal‐level mindsets are not related to the incentives or the incentivized tasks. Participants in our studies were either positively or negatively incentivized to solve a set of difficult anagrams, and were primed with an abstract or a concrete mindset using spatial (Study 1) and social (Study 2) cues. The participants persisted longer in response to the positively framed incentive when primed with spatially or socially remote cues. In contrast, for the negatively framed incentive, participants persisted longer when primed with spatially or socially proximal cues. Copyright © 2017 John Wiley & Sons, Ltd.     

Reaping the benefits of research: Technology transfer

This paper presents the results of a technology transfer project designed to transfer knowledge about model behavioral intervention projects that significantly decreased HIV-related risk behaviors. The National AIDS Demonstration Research Program Technology Transfer (NADR TT) Project encompassed: (1) preparation of manuals based on successful intervention research; (2) convening of 7 Regional meetings on Behavior Change Strategies for Injection Drug Users (IDUs); and (3) conduct of twenty-three state implementation training events. An evaluation of the NADR TT project showed that it was successful in its efforts to transfer knowledge regarding model intervention programs. Participants expressed an increased level of confidence in their ability to implement the model and individual model elements were perceived to be useful to—and implemented by—participants. Results also pointed out that significant barriers to implementation of model elements of the behavioral interventions exist.     

The regulatory function of self-conscious emotion: insights from patients with orbitofrontal damage

Although once considered disruptive, self-conscious emotions are now theorized to be fundamentally involved in the regulation of social behavior. The present study examined the social regulation function of self-conscious emotions by comparing healthy participants with a neuropsychological population--patients with orbitofrontal lesions--characterized by selective regulatory deficits. Orbitofrontal patients and healthy controls participated in a series of tasks designed to assess their social regulation and self-conscious emotions. Another task assessed the ability to infer others' emotional states, an appraisal process involved in self-conscious emotion. Consistent with the theory that self-conscious emotions are important for regulating social behavior, the findings show that deficient behavioral regulation is associated with inappropriate self-conscious emotions that reinforce maladaptive behavior. Additionally, deficient behavioral regulation is associated with impairments in interpreting the self-conscious emotions of others.     

社会排斥类型对求助行为意愿的影响

从心理需求威胁的视角切入,探讨不同类型的社会排斥对求助行为意愿的影响及其作用机理。两个实验研究的结果表明,相较于被忽略,被拒绝会促使人们在后续困境中产生更高的求助行为意愿。该效应产生的心理机制在于被拒绝会威胁归属需求,被忽略会威胁控制需求,而求助行为本身能够满足人们的归属需求,但同时会降低控制感,因此归属需求和控制需求在社会排斥类型对求助行为意愿的影响中均起着显著的中介作用。