Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations

Purpose

Amazon Mechanical Turk is an increasingly popular data source in the organizational psychology research community. This paper presents an evaluation of MTurk and provides a set of practical recommendations for researchers using MTurk.

Design/Methodology/Approach

We present an evaluation of methodological concerns related to the use of MTurk and potential threats to validity inferences. Based on our evaluation, we also provide a set of recommendations to strengthen validity inferences using MTurk samples.

Findings

Although MTurk samples can overcome some important validity concerns, there are other limitations researchers must consider in light of their research objectives. Researchers should carefully evaluate the appropriateness and quality of MTurk samples based on the different issues we discuss in our evaluation.

Implications

There is not a one-size-fits-all answer to whether MTurk is appropriate for a research study. The answer depends on the research questions and the data collection and analytic procedures adopted. The quality of the data is not defined by the data source per se, but rather the decisions researchers make during the stages of study design, data collection, and data analysis.

Originality/Value

The current paper extends the literature by evaluating MTurk in a more comprehensive manner than in prior reviews. Past review papers focused primarily on internal and external validity, with less attention paid to statistical conclusion and construct validity—which are equally important in making accurate inferences about research findings. This paper also provides a set of practical recommendations in addressing validity concerns when using MTurk.

     

A theory of bidirectional judgments

This paper is concerned with a scaling theory for “bidirectional” judgments, for which the order of judgment is reversible, as in fractional and multiple ratio estimation judgments. With the assumption that judgments are mediated by perceived relations of pairs of stimuli, the theory is developed for judgments of comparison stimuli in relation to standards, taking explicit account of the location of the comparison stimulus relative to the standard. The theory of bidirectional judgments, based on a theory of relative judgment by Fagot (1978, 1979), entails a partial nesting of models characterized by a progressive weakening of the constraints placed on the structure of the data. The weakest model, the relative bias/directional standard (RBDS) model, allows each standard to have two biasing effects, depending on the location of the standard above or below the comparison stimulus. Tests of the theory were carried out on the ratio estimation of brightness and weight data of Engen and Levy (1955) and the part-sum estimation data of Goude (1962). Only the RBDS model was found acceptable for all three data sets     

A choice theory analysis of similarity judgments

The selection of one of several stimuli as most similar to a reference stimulus is assumed to satisfy a choice axiom that permits assigning ratio scale values to each variable-reference stimuli pair. The logarithm of this scale is treated as a distance measure, leading to the following testable conclusions about the pairwise choice probabilities as the reference stimulus is varied. First, the plot is a symmetrically truncated ogive with horizontal tails. Second, if two pairs of choice stimuli have the same midpoint, the ogive of one pair is part of the ogive of the other. In terms of this model, the hysteresis and midpoint displacement effects in the method of bisection are discussed, and relations with Coombs' unfolding techniques are explored.This work was supported in part by grant G-8864 from the National Science Foundation to the University of Pennsylvania. I wish to express my appreciation to Professors Robert R. Bush and Eugene Galanter, with whom I have had a number of very helpful discussions of these ideas.     

Cerebral lateralization of deaf and hearing individuals for linguistic comparison judgments.

A group of congenitally deaf adults and a group of hearing adults, both fluent in sign language, were tested to determine cerebral lateralization. In the most revealing task, subjects were given a series of trials in which they were fist presented with a videotaped sign and then with a word exposed tachistoscopically to the right visual field or left visual field, and were required to judge whether the word corresponded to the sign or not. The results suggested that the comparison processes involved in the decision were performed more efficiently by the left hemisphere for hearing subjects and by the right hemisphere for deaf subjects. However, the deaf subjects performed as well as the hearing subjects in the left hemisphere, suggesting that the deaf are not impeded by their auditory-speech handicap from developing the left hemisphere for at least some types of linguistic processing.     

A conceptual replication examining the risk of overtly listing eligibility criteria on Amazon’s Mechanical Turk

Recent scholarship indicates that explicitly listing eligibility requirements on Amazon’s Mechanical Turk can lead to eligibility falsification. Offering a conceptual replication of prior studies, we assessed the prevalence of eligibility falsification and its impact on data integrity. A screener survey collected the summer before the 2016 presidential election assessed political affiliation. Participants were then randomly assigned to be exposed to a second survey link for which they were eligible or ineligible. There was a significant interaction such that the differences between self‐reported Republicans and Democrats on outcome measures (e.g., attitudes toward Hillary Clinton), were smaller among participants that were falsifying eligibility (i.e., imposters) than those that were not (i.e., genuine participants). Moreover, for most outcomes, imposters put forth responses that were significantly different from the responses put forth by those in the political party with which imposters were pretending to be affiliated. Imposters’ responses were also significantly different from participants in the political party with which imposters initially claimed to genuinely belong. For example, those who self‐reported themselves as Democrats on the screener survey but responded to a survey for “only Republicans” (i.e., imposter Republicans), reported more favorable attitudes toward Donald Trump than genuine Democrats, but indicated less favorable attitudes toward Donald Trump than genuine Republicans. These results highlight the potential harms of explicitly listing eligibility requirements and emphasize the importance of minimizing imposter participation.     

The impetus theory in judgments about object motion: A new perspective

Several tendencies found in explicit judgments about object motion have been interpreted as evidence that people possess a naive theory of impetus. The theory states that objects that are caused to move by other objects acquire force that determines the kind of motion exhibited by the object, and that this force gradually dissipates over time. I argue that the findings can better be understood as manifestations of a general understanding of externally caused motion based on experiences of acting on objects. Experiences of acting on objects yield the idea that properties of the cause of motion are transmitted to the effect object. This idea functions as a heuristic for explicit predictions of object motion under conditions of uncertainty. This accounts not only for the findings taken as evidence for the impetus theory, but also for several findings that fall outside the scope of the impetus theory. It has also been claimed that judgments about the location at which a moving object disappeared are influenced by the impetus theory. I argue that these judgments are better explained in a different way, as best-guess extrapolations made by the visual system as a practical guide to interactions with the object, such as interception.     

On the nature of children's judgments of linguistic features: Semantic relations and grammatical morphemes

Children's judgments of sentences were examined in 3-, 4-, and 5-year-olds in an effort to examine the relationship between children's use of various linguistic features and their judgments of these features in formal tasks. The sentences the children were asked to judge differed on the basis of features acquired gradually during the development of children's linguistic usage. The judgments made by the children did not appear related to the course followed in the acquisition of language usage, a finding that suggests that language acquisition and formal linguistic judgments may reflect different processes.     

Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.     

A basis for bias in geographical judgments

To determine why North Americans tend to locate European cities south of North American cities at similar latitudes (Tversky, 1981), we had observers provide bearing estimates between cities in the U.S. and Europe. Earlier research using latitude estimates of these cities has indicated that each continent has several subjective regions (Friedman & Brown, 2000a). Participants judged cities from two subjectively northern regions (Milwaukee-Munich), two subjectively southern regions (Memphis-Lisbon), and the two "crossed" regions (Albuquerque-Geneva; Minneapolis-Rome). Estimates were biased only when cities from the subjectively northern regions of North America were paired with cities from the subjectively southern region of Europe. In contrast to the view that biases are derived from distorted or aligned map-like representations, the data provide evidence that the subjective representation of global geography is principally categorical. Biases in numerical location estimates of individual cities and in bearing estimates between city pairs are derived from plausible reasoning processes operating on the same categorical representations.     

Associations Between Social Anxiety and Affective and Empathic Forecasts: A Replication and Extension in a Mechanical Turk Sample

People often overestimate the intensity and duration of their future emotions, referred to as an impact bias. Impact biases have been documented in predictions people make about their own emotions, as well as the others’ emotions (i.e., affective and empathic forecasting, respectively). Recent studies have shown that negative impact biases may be stronger, and positive impact biases may be attenuated, in individuals with symptoms of social anxiety. The current study sought to replicate and extend these findings in a Mechanical Turk (MTurk) sample. MTurk is a particularly interesting online platform for such research because of the unusually high prevalence of social anxiety among MTurk users. Within a computer-based survey, 93 MTurk users read vignettes in which a second-person narrator elicited either disgust, anger, or happiness from another person. After each vignette, participants predicted how the narrator (i.e., affective forecasts) and the other person (i.e., empathic forecasts) would feel. Overall, results confirmed the existence of associations between social anxiety symptoms and negative affective and empathic forecasting biases, though no significant relations were found between social anxiety symptoms and positive forecasting biases. Negative affective and empathic forecasting biases were significantly correlated. Age and gender were also examined as potential predictors and moderators of hypothesized effects. Though younger age and female gender were associated with specific forecast ratings, controlling for these variables did not alter the associations between social anxiety and affective or empathic forecasts and no moderation effects were found. Overall, results provide additional support for the relevance of impact biases to social anxiety and suggest that they may be useful targets of intervention.     

Extreme judgments depend on the expectation of following judgments: A calibration analysis

Objectives

Judges avoid extreme judgments in the beginning of evaluation sequences. The calibration hypothesis attributes this bias to judges’ need to preserve their judgmental degrees of freedom. It follows that the expectation of a sequence leads to avoiding extreme judgments in the beginning. Thus, judges may make extreme judgments if they expect only one performance but should avoid extreme judgments if they expect a sequence.

Design

A between-group design was used.

Method

One experimental group (n = 21) expected to judge only one gymnastics performance whereas the other group (n = 20) expected to judge a sequence of performances. Both groups then judged only one identical performance.

Results

Groups differed significantly in the frequency of extreme judgments. Participants expecting one performance used extreme judgment categories more often; participants expecting a series avoided extreme judgments.

Conclusion

The results support calibration processes in sequential judgments. The specification of the underlying process will allow testing possible interventions to avoid serial position biases in serial evaluations in the future.     

Further cross-cultural validation of the theory of mental self-government

This study was designed to achieve two objectives. The 1st was to investigate the cross-cultural validity of the Thinking Styles Inventory (TSI; R. J. Sternberg & R. K. Wagner, 1992), which is based on the theory of mental self-government (R. J. Sternberg, 1988, 1990, 1997). The 2nd was to examine the relationships between thinking styles as assessed by the TSI and a number of student characteristics, including age, gender, college class level, work experience, and travel experience. One hundred fifty-one students from the University of Hong Kong participated in the study. Results indicated that the thinking styles evaluated by the TSI could be identified among the participants. Moreover, there were significant relationships between certain thinking styles, especially creativity-relevant styles and 3 student characteristics: age, work experience, and travel experience. Implications of these findings for teaching and learning in and outside the classroom are discussed.