Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.     

Psychometric evaluation and normative data for a Swedish version of the Patient Health Questionnaire 15‐Item Somatic Symptom Severity Scale

The lack of an available Swedish version of the Patient Health Questionnaire 15‐Item Somatic Symptom Severity Scale (PHQ‐15) motivated the present psychometric evaluation of such a version as well as providing normative data for the PHQ‐15. Data from 3,406 individuals who took part in the Västerbotten Environmental Health Study in Sweden were used. The respondents constitute a random sample, aged 18 to 79 years, stratified for age and sex. They responded to a Swedish translation of the PHQ‐15 as well as the Hospital Anxiety and Depression Scale, the Perceived Stress Scale, and the Shirom Melamed Burnout Questionnaire for assessment of convergent validity. The distribution of PHQ‐15 scores was positively skewed and mesokurtic in shape, and the internal consistency of the PHQ‐15 was satisfactory. Correlation coefficients between PHQ‐15 score and the measures of anxiety, depression, stress and mental/physical exhaustion indicate satisfactory validity. Normative data for PHQ‐15 scores as well as for categories of somatic symptom severity are provided. The favorable psychometric properties of the Swedish version of the PHQ‐15 suggest use of this instrument for quantification of somatization in Swedish and similar populations, and has the advantage of available normative data.     

Validity of the Rey Visual Design Learning Test in Primary and Secondary School Children

Studies on adults have revealed a disadvantageous effect of negative emotional stimuli on executive functions (EF), and it is suggested that this effect is amplified in children. The present study’s aim was to assess how emotional facial expressions affected working memory in 9- to 12-year-olds, using a working memory task with emotional facial expressions as stimuli. Additionally, we explored how degree of internalizing and externalizing symptoms in typically developing children was related to performance on the same task. Before employing the working memory task with emotional facial expressions as stimuli, an independent sample of 9- to 12-year-olds was asked to recognize the facial expressions intended to serve as stimuli for the working memory task and to rate the facial expressions on the degree to which the emotion was expressed and for arousal to obtain a baseline for how children during this age recognize and react to facial expressions. The first study revealed that children rated the facial expressions with similar intensity and arousal across age. When employing the working memory task with facial expressions, results revealed that negatively valenced expressions impaired working memory more than neutral and positively valenced expressions. The ability to successfully complete the working memory task increased between 9 to 12 years of age. Children’s total problems were associated with poorer performance on the working memory task with facial expressions. Results on the effect of emotion on working memory are discussed in light of recent models and empirical findings on how emotional information might interact and interfere with cognitive processes such as working memory.     

Development and Initial Validation of a General Psychological Well-being Scale (GPWS) in an African Context

This study aimed to develop and validate the General Psychological Well-being Scale (GPWS) in an African sample, based on the empirical overlap between hedonic and eudaimonic facets of well-being as found in previous research. The quantitative cross-sectional study was conducted in three phases: secondary data analysis (n = 2005), pilot study (n = 296) and main study (n = 459). The pilot and main study included the GPWS as well as other psychological well-being measures for criterion-related validity. The pilot study yielded satisfactory psychometric properties. The main study yielded a high, reliable Cronbach alpha of.89 and evidence of construct validity. The GPWS appears to be a unidimensional scale suitable for research use with Setswana-speaking people.     

Career Counselling: The Usefulness of the Career Interest Profile (CIP)

This study reports the findings from exploratory career counselling with Italian undergraduate university students using the Career Interest Profile (CIP). The participants (n = 178, females = 51.69%; mean age 22.28, SD = 1.59 years) received elective career counselling on campus. The students completed the CIP and the Self-Directed Search (SDS). Data were analyzed using both qualitative and quantitative approaches. The CIP yielded satisfactory results with regard to trustworthiness and intercorrelations with the SDS and provides career counsellors with a useful instrument for the identification of interest profiles of Italian undergraduate university students. Participants reported being enabled to explore prospective career interest areas using career-story narratives for constructing life trajectories and enhanced development of their career stories.     

Psychometric properties of the Norwegian version of the Dysfunctional Attitude Scale (Form A)

The present study reports the reliability and validity of the Norwegian version of the Dysfunctional Attitude Scale in non‐clinical and clinical populations. The participants were 344 young male military recruits, 41 healthy controls and 142 psychiatric outpatients. All the participants completed the Dysfunctional Attitude Scale, the Beck Depression Inventory and the Automatic Thoughts Questionnaire. The analysis of the Dysfunctional Attitude Scale revealed a Cronbach's alpha of 0.85, indicating satisfactory reliability. Evidence for the construct validity was obtained by the correlation between the Dysfunctional Attitude Scale and the Beck Depression Inventory (r?=?0.47) and the Dysfunctional Attitude Scale and the Automatic Thoughts Questionnaire (r?=?0.47). Finally, the Dysfunctional Attitude Scale significantly discriminated between clinically depressed, non‐depressed psychiatric patients and healthy controls. The results showed that the Norwegian version of the Dysfunctional Attitude Scale possess satisfactory psychometric properties suggesting that this instrument is appropriate for use as a cognitive measure in a Norwegian cultural context.     

Assessment of dental anxiety: A facet approach

Dental anxiety is a complicated phenomenon, and its multicomponent nature is often not acknowledged in existing measurement instruments. Using a facet design, a new Dental Anxiety Inventory (DAI) was constructed. Facets chosen were: time (made up of four elements: at home, on your way to the denstist, in the denstist's waiting room, in the dental chair), situation (three elements: introductory aspects, dentist-patient interaction, actual dental treatment), and reaction (three elements: subjective feelings, physical reactions, cognitive reactions). Psychometric qualities of the DAI are good. The internal structure of the DAI was studied by linear and nonlinear techniques, and was partially recovered from the data. The construct and criterion validity of the DAI was supported in several studies.     

Measuring test anxiety in children: Scale development and internal construct validation

Given the increased testing of school-aged children in the United States there is a need for a current and valid scale to measure the effects of test anxiety in children. The domain of children's test anxiety was theorized to be comprised of three dimensions: thoughts, autonomic reactions, and off-task behaviors. Four stages are described in the evolution of the Children's Test Anxiety Scale (CTAS): planning, construction, quantitative evaluation, and validation. A 50-item scale was administered to a development sample (N=230) of children in grades 3–6 to obtain item analysis and reliability estimates which resulted in a refined 30-item scale. The reduced scale was administered to a validation sample (N=261) to obtain construct validity evidence. A three-factor structure fit the data reasonably well. Recommendations for future research with the scale are described.     

The multidimensionality of test anxiety: A test of different models

Abstract

Confirmatory factor analysis was employed to test the multidimensional structure of test anxiety. Starting with the traditional two-factor model, it was of interest whether additional dimensions could still be represented by a single higher-order factor of test anxiety or whether the additional dimensions represented correlates of test anxiety. The Revised Test Anxiety (RTA) scale and the German Test Anxiety Inventory (TAI-G) were combined collecting data from a binational sample of 218 American and 218 German university students. Considering the statistical and theoretical aspects of model fit, they indicated that a model consisting of three primary factors (worry, emotionality, and lack of confidence) fit the data best. While distraction and self-efficacy may be regarded as correlates of test anxiety, lack of confidence was substantiated as a component of test anxiety. Implications for the conceptualization of the content domain and hence area of test anxiety were discussed.