Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations

Purpose

Amazon Mechanical Turk is an increasingly popular data source in the organizational psychology research community. This paper presents an evaluation of MTurk and provides a set of practical recommendations for researchers using MTurk.

Design/Methodology/Approach

We present an evaluation of methodological concerns related to the use of MTurk and potential threats to validity inferences. Based on our evaluation, we also provide a set of recommendations to strengthen validity inferences using MTurk samples.

Findings

Although MTurk samples can overcome some important validity concerns, there are other limitations researchers must consider in light of their research objectives. Researchers should carefully evaluate the appropriateness and quality of MTurk samples based on the different issues we discuss in our evaluation.

Implications

There is not a one-size-fits-all answer to whether MTurk is appropriate for a research study. The answer depends on the research questions and the data collection and analytic procedures adopted. The quality of the data is not defined by the data source per se, but rather the decisions researchers make during the stages of study design, data collection, and data analysis.

Originality/Value

The current paper extends the literature by evaluating MTurk in a more comprehensive manner than in prior reviews. Past review papers focused primarily on internal and external validity, with less attention paid to statistical conclusion and construct validity—which are equally important in making accurate inferences about research findings. This paper also provides a set of practical recommendations in addressing validity concerns when using MTurk.

     

Idiosyncratic Deals and Helping Behavior: The Moderating Role of I-deal Opportunity for Co-workers

Purpose

This study draws from social comparison theory to explore why and under which circumstances individuals receiving idiosyncratic deals (i-deals) are likely to help their co-workers.

Design

Data were collected with an alumni association of engineers. Participants completed two questionnaires (N = 182 at Time 2).

Findings

We find that the relationship between i-deals and helping behavior is not direct, but is mediated by organizational-based self-esteem. This relationship is stronger when i-deal recipients believe that their co-workers do not have the opportunity to get i-deals for themselves.

Implications

I-deal recipients are expected to help their colleagues because helping colleagues is consistent with the positive self developed thanks to i-deals. When co-workers have the opportunity to get i-deals for themselves, social comparison between the i-deal recipient and colleagues is likely to be more salient, which strengthen the indirect relationship between i-deal and helping behavior.

Originality

This study tests i-deals from the vantage point of social comparison theory rather than from the perspective of social exchange. We thereby provide a richer account of the complexities involved in helping behavior. By exploring contextual variables that are likely to trigger social comparisons, we also expect to better understand the circumstances under which i-deals are likely to be associated with helping behavior.

     

Why Do Situational Interviews Predict Performance? Is it Saying How You Would Behave or Knowing How You Should Behave?

Purpose

The present study examined two theoretical explanations for why situational interviews predict work-related performance, namely (a) that they are measures of interviewees’ behavioral intentions or (b) that they are measures of interviewees’ ability to correctly decipher situational demands.

Design/Methodology/Approach

We tested these explanations with 101 students, who participated in a 2-day selection simulation.

Findings

In line with the first explanation, there was considerable similarity between what participants said they would do and their actual behavior in corresponding work-related situations. However, the underlying postulated mechanism was not supported by the data. In line with the second explanation, participants’ ability to correctly decipher situational demands was related to performance in both the interview and work-related situations. Furthermore, the relationship between the interview and performance in the work-related situations was partially explained by this ability to decipher situational demands.

Implications

Assessing interviewees’ ability to identify criteria might be of additional value for making selection decisions, particularly for jobs where it is essential to assess situational demands.

Originality/Value

The present study made an effort to open the ‘black box’ of situational interview validity by examining two explanations for their validity. The results provided only moderate support for the first explanation. However, the second explanation was fully supported by these results.

     

Back to Basics with Mixed-Effects Models: Nine Take-Away Points

Purpose

Multilevel mixed effects models are widely used in organizational behavior and organizational psychology to test and advance theory. At times, however, the complexity of the models leads researchers to draw erroneous inferences or otherwise use the models in less than optimal ways. We present nine take-away points intended to enhance the theoretical precision and utility of the models.

Approach

We demonstrate our points using two types of simulated data: one in which group membership is irrelevant, and the other in which relationships exist only because of group membership. We then demonstrate that the effects we observe in simulated data replicate in organizational data.

Findings

Little that we address will be new to methodology experts; nonetheless, we draw together a variety of points that we believe will help advance both theory and analytic rigor in multilevel analyses.

Implications

We make two points that run somewhat counter to conventional norms. First, we argue that mixed-effects models are appropriate even when ICC(1) values associated with the outcome data are small and non-significant. Second, we show that high ICC(2) values are not a prerequisite for detecting emergent multilevel relationships.

Originality/Value

The article is designed to be a resource for researchers who are learning about and applying mixed-effects (i.e., multilevel) models.

     

Editorial: Evidence on Questionable Research Practices: The Good,the Bad,and the Ugly

Purpose

Questionable research or reporting practices (QRPs) contribute to a growing concern regarding the credibility of research in the organizational sciences and related fields. Such practices include design, analytic, or reporting practices that may introduce biased evidence, which can have harmful implications for evidence-based practice, theory development, and perceptions of the rigor of science.

Design/Methodology/Approach

To assess the extent to which QRPs are actually a concern, we conducted a systematic review to consider the evidence on QRPs. Using a triangulation approach (e.g., by reviewing data from observations, sensitivity analyses, and surveys), we identified the good, the bad, and the ugly.

Findings

Of the 64 studies that fit our criteria, 6 appeared to find little to no evidence of engagement in QRPs and the other 58 found more severe evidence (91 %).

Implications

Drawing upon the findings, we provide recommendations for future research related to publication practices and academic training.

Originality/value

We report findings from studies that suggest that QRPs are not a problem, that QRPs are used at a suboptimal rate, and that QRPs present a threat to the viability of organizational science research.

     

Act Expediently,with Autonomy: Vicarious Learning,Empowered Behaviors,and Performance

Purpose

The purpose of this research is to investigate how organizations can best facilitate an empowered workforce that makes autonomous decisions and acts expediently, which the literature on high performing organizations posits will increase the likelihood of sustained performance and retaining competitive advantages. We introduce a novel mechanism for encouraging such behaviors and pursuant outcomes: vicarious learning from a supervisor who demonstrates autonomy and expediency.

Design/Methodology/Approach

We drew experimental data from a sample of participants who underwent a managerial simulation, and used these data to investigate relationships between the vicarious learning of empowered behaviors and individual task performance (n = 100).

Findings

Results indicate that when supervisors behave with autonomy and expediency this both increases the extent to which individuals behave similarly, and is associated with enhanced individual performance. Further, we find that expedient behavior fully mediates the relationship between empowered supervisor behavior and performance.

Implications

Findings show that supervisors need not necessarily engage directly in empowering others. Rather, by modeling behaviors, supervisors can craft a context where employees may act with autonomy and efficiency. This provides an opportunity for empowerment that is both actionable and cost-effective.

Originality/Value

This is the first study to consider empowerment as a managerial phenomenon that can be vicariously learned, integrating theories of social learning and empowerment, and extending existing empowerment constructs (including psychological and structural) to develop an indirect, yet potent means of encouraging empowered behavior.

     

Supervisor Humor and Employee Outcomes: The Role of Social Distance and Affective Trust in Supervisor

Purpose

To examine how social distance and affective trust in supervisor affect the relationships between supervisor humor and the psychological well-being and job performance of subordinates.

Design/Methodology/Approach

A survey was conducted among 322 matched supervisor–subordinate dyads in 14 South Korean organizations. Multi-level analyses were performed to test the research hypotheses, including the moderating effects.

Findings

Self-enhancing humor of supervisors was positively associated with the psychological well-being and job performance of subordinates. Affiliative humor was positively associated with psychological well-being, whereas aggressive humor was negatively associated with psychological well-being. In addition, supervisor humor was indirectly related to the psychological well-being of subordinates via social distance. Moreover, affective trust in supervisor significantly moderated the relationship between supervisor humor and social distance, such that the relationship between affiliative humor and social distance was stronger when affective trust in supervisor was high rather than low.

Implications

These findings are important in developing and refining humor theory on the responses of employees to various types of supervisor humor. Moreover, they provide practical implications for organizations. For example, organizations should note that supervisor humor may not always produce good results, and thus should encourage managers to use constructive humor. Similarly, supervisors should build a high-trust relationship with their subordinates to increase the effectiveness of their constructive humor.

Originality/Value

This study is one of the few studies that has examined the mechanism and boundary conditions of the effects of supervisor humor on employee outcomes.

     

Development and Validation of the Workplace Age Discrimination Scale

Purpose

Workplace age discrimination research is proliferating, but researchers lack a valid measure with which to capture targets’ discriminatory experiences. We developed a measure of perceived workplace age discrimination that assesses overt and covert forms of discrimination and then compared older, middle-aged, and younger workers’ experiences.

Design/Methodology

In Study 1, we developed the Workplace Age Discrimination Scale (WADS) based on older workers’ experiences using a deductive approach, a qualitative study, and two quantitative surveys. In Study 2, we validated the measure among young employees using a qualitative and two quantitative surveys. In Study 3, we tested the WADS among middle-aged workers and tested models of invariance between age groups.

Findings

Participants frequently endorsed covert discriminatory experiences, which the WADS reflects. The WADS contains convergent and discriminant validity, high reliability, and a unidimensional structure across age groups. It demonstrates criterion-related validity among older and younger workers but not middle-aged workers, given their low experiences of age discrimination. Age discrimination frequency follows a U-shaped pattern across age groups.

Implications

Researchers can use the WADS to identify long-term outcomes of age discrimination and to further compare workers’ discriminatory experiences. Practitioners and policymakers can use the measure to develop interventions to ameliorate workplace age discrimination and inform policymaking.

Originality/Value

The WADS is the first validated measure of targets’ perspectives of workplace age discrimination. Our results challenge assumptions that only older workers experience age discrimination (younger workers’ means were highest) and that age discrimination is usually overt in nature (it is often covert).

     

Humor in Teams: Multilevel Relationships Between Humor Climate,Inclusion, Trust,and Citizenship Behaviors

Purpose

This study examines the cross-level influence of positive and offensive leader humor climates on employee inclusion and citizenship behaviors, and the moderating effect of trust in such relationships.

Design/Methodology/Approach

We collected data from a sample of 225 respondents nested within 23 teams from a Canadian financial organization. A multilevel confirmatory analysis was used to provide evidence that variables of this study are distinct and a HLM analysis to test the hypotheses.

Findings

We find that employees’ perception of inclusion is influenced much more by an offensive humor climate than by a positive one. The results also suggest that the perception of inclusion plays a significant intermediary role in the influence of humor climates on citizenship behavior. Finally, trust in leaders acts as an important contingent condition in the effectiveness of a humor climate.

Implications

Use of humor does not always pay. Offensive humor by supervisor is a risky strategy that may undermine the beneficial effects of positive humor climate, increase employee exclusion and weaker individual performance.

Originality/Value

Our study shows the utility of using micro- and macro-approaches, and more specifically, the relevance of adopting an integrative multilevel view of the effect of a humor environment in predicting individual inclusion and citizenship behaviors.

     

The Moderating Role of Intergroup Contact in Race Composition,Perceived Similarity,and Applicant Attraction Relationships

Purpose

This study investigated the moderating effect of intergroup contact on the relationship between the race composition of organizational representatives, perceived similarity, and minority applicant attraction.

Design/Methodology/Approach

344 minority Malaysian-Chinese university students read a job advertisement that varied the racial composition of organizational representatives (100 % Malay or 50 % Malay–50 % Chinese or 100 % Chinese). Of these participants, 161 were Malaysian-Chinese in Malaysia (high intergroup contact location) and 183 were Malaysian-Chinese in Australia (low intergroup contact location). After reading the advertisement, participants responded to a series of scale items (e.g., perceived surface-level similarity, perceived deep-level similarity, and applicant attraction).

Findings

Results showed that the effect of race composition on attraction was stronger for minority participants in Australia than for minority participants in Malaysia. Perceived deep-level similarity mediated this moderated relationship.

Implications

The study findings suggest that organizations should include minority representatives in their recruitment advertising to attract minority applicants, particularly to attract minorities in locations with few opportunities for intergroup contact.

Originality/Value

By testing the mediating effects of perceived surface-level and deep-level similarity, this study contributes to our understanding of the mechanism linking the interaction between race composition and location with applicant attraction.

     

Linking Workplace Aggression to Employee Well-Being and Work: The Moderating Role of Family-Supportive Supervisor Behaviors (FSSB)

Purpose

The present study examined the moderating effects of family-supportive supervisor behaviors (FSSB) on the relationship between two types of workplace aggression (i.e., patient-initiated physical aggression and coworker-initiated psychological aggression) and employee well-being and work outcomes.

Methodology

Data were obtained from a field sample of 417 healthcare workers in two psychiatric hospitals. Hypotheses were tested using moderated multiple regression analyses.

Findings

Psychiatric care providers’ perceptions of FSSB moderated the relationship between patient-initiated physical aggression and physical symptoms, exhaustion and cynicism. In addition, FSSB moderated the relationship between coworker-initiated psychological aggression and physical symptoms and turnover intentions.

Implications

Based on our findings, family-supportive supervision is a plausible boundary condition for the relationship between workplace aggression and well-being and work outcomes. This study suggests that, in addition to directly addressing aggression prevention and reduction, family-supportive supervision is a trainable resource that healthcare organizations should facilitate to improve employee work and well-being in settings with high workplace aggression.

Originality

This is the first study to examine the role of FSSB in influencing the relationship between two forms of workplace aggression: patient-initiated physical and coworker-initiated psychological aggression and employee outcomes.

     

Parent Work Conditions and Adolescent Core Self-Evaluations: Examining the Effects of Work Resource Drain and Parent Gender

Purpose

Using resource drain and social cognitive theory frameworks, this study investigates the process through which parent time and energy resource drain relates to adolescent core self-evaluation. Parent–child interactive and routine activities and parental social support are tested as mediators.

Methodology

Data are reported by mothers, fathers, and adolescent children in 151 middle-class, dual-earner families from the 500 Family Study. Hypotheses are tested using path analysis and moderated mediation.

Findings

Parent work hours negatively relate to adolescent reports of received social support, partially due to decreased time spent engaged in parent–child interactive activities. Fathers’ work hours indirectly relate to adolescent core self-evaluations through parent–child interactive activities and social support from fathers; this indirect relationship was not found for mothers. Exploratory analyses examined differences by parent and child gender. Fathers’ work hours are more likely to detract from parent–child routine activities relative to mothers’ work hours. The relationship between father work hours and parent–child interactive activities is buffered for sons compared to daughters.

Implications

Results help to explain why parent work-related resource drain relates to adolescent core self-evaluations. Parent gender, type of resource drain, and type of activity are identified as potential boundary conditions. Interactive activities and social support are key mediators warranting attention.

Originality/Value

Social cognitive theory is used as a framework for linking parent work-related resource drain and child outcomes. Mechanisms are identified that explain adolescent core self-evaluations development. Use of multisource data helps to mitigate problems with single-source data used in previous studies.

     

Behavioral Expression of Job Interview Anxiety

Purpose

The aim of this study was to investigate (a) the behavioral cues that are displayed by, and trait judgments formed about, anxious interviewees, and (b) why anxious interviewees receive lower interview performance ratings. The Behavioral Expression of Interview Anxiety Model was created as a conceptual framework to explore these relations.

Design/Methodology/Approach

We videotaped and transcribed mock job interviews, obtained ratings of interview anxiety and interview performance, and trained raters to assess several verbal and nonverbal cues and trait judgments.

Findings

The results indicated that few behavioral cues, but several traits were related to interviewee and interviewer ratings of interview anxiety. Two factors emerged from our factor analysis on the trait judgments—Assertiveness and Interpersonal Warmth. Mediation analyses were performed and indicated that Assertiveness and Interpersonal Warmth mediated the relation between interview anxiety and interview performance. Speech rate (words spoken per minute) and Assertiveness were found to mediate the relation between interviewee and interviewer ratings of interview anxiety.

Implications

Overall, the results indicated that interviewees should focus less on their nervous tics and more on the broader impressions that they convey. Our findings indicate that anxious interviewees may want to focus on how assertive and interpersonally warm they appear to interviewers.

Originality/Value

To our knowledge, this is the first study to use a validated interview anxiety measure to examine behavioral cues and traits exhibited by anxious interviewees. We offer new insight into why anxious interviewees receive lower interview performance ratings.

     

The Effect of Method Characteristics on Retest Score Gains and Criterion-Related Validity

Purpose

We sought to empirically assess the effect of predictor method characteristics (test form, item-type, and test-type) on retest score change associated with an invariant construct—general mental ability (GMA)—and to evaluate the effect of retesting on the criterion-related validity of assessments that vary in their susceptibility to retest effects.

Design

Three hundred seven individuals completed a battery of GMA assessments. After a 6-week interval, participants returned to the testing site to retest using both alternate and identical forms of the initial assessments.

Findings

Greater score gains were observed on assessments comprising heterogeneous item-types than homogeneous item-types, and on performance-based assessments than self-report assessments. However, despite variations in score gains, the relationships between the initial test scores and criterion scores were no different than the relationships between retest scores and criterion scores for all assessments.

Implications

Tests and procedures that reduce reliance on test- or item-specific knowledge and skill may help minimize score changes due to retesting across multiple administrations. Moreover, under the boundary conditions present in this study, the criterion-related validity of ability assessments may not be affected by increases in test-specific knowledge and skills.

Originality/Value

Despite the prevalence and industry support of retesting, a comprehensive understanding of retest score change still eludes researchers and practitioners. This ambiguity may be due in part to neglecting the method-construct distinctions in the retest literature. This is the first report to explicitly utilize the method-construct distinction in an effort to examine the causes and consequences of retest effects.

     

Differential Effects of Task Performance,Organizational Citizenship Behavior,and Job Complexity on Voluntary Turnover

Purpose

The purpose of this study was to investigate the performance–turnover relationship by considering the effects of task performance and OCBs simultaneously while also examining the moderating effect job complexity has on the relationship between voluntary turnover and each type of performance.

Design/Methodology/Approach

Data were obtained as part of a larger study to validate an employment test, in which actual turnover data and supervisory ratings of job performance were collected for employees in two hospitals (n = 782).

Findings

Task performance exhibited a curvilinear relationship with turnover, while OCB exhibited a negative linear relationship with turnover. Job complexity moderated both of these relationships. For task performance, turnover in high-complexity jobs was greater for low performers but lower for high performers relative to that of employees in low-complexity jobs. For OCB, the negative relationship with turnover was more pronounced in high-complexity jobs.

Implications

Both low- and high-task performers are more likely to turnover, while employees exhibiting high OCBs are less likely to turnover. These results imply that retention strategies are critical for top performers, but especially in high-complexity jobs. Organizations may be able to discourage voluntary turnover by creating conditions that stimulate OCB, particularly in highly complex jobs.

Originality/Value

Most prior performance–turnover relationship research used unidimensional measures of performance, whereas this study included two dimensions of performance and examined this relationship while controlling for one-performance dimension when predicting the other. Furthermore, this study is one of the first studies to suggest that job complexity moderates the performance–turnover relationship.

     

The Manager as Coach: The Role of Feedback Orientation

Purpose

This study investigated the consequences of manager feedback orientation in the manager-as-coach process. Integrating theories of feedback and coaching, we examined the extent to which manager feedback orientation was related to indicators of effective coaching and subordinate feedback orientation.

Design/methodology/approach

One hundred three manager–subordinate dyads participated in this study.

Findings

Managers who value feedback for themselves (high feedback orientation) were viewed as better coaches as assessed through employee perceptions of coaching behaviors, the coaching relationship, and the feedback environment. Manager feedback orientation was also related to subordinate feedback orientation, and this relationship was mediated by the coaching effectiveness indicators.

Implications

This study demonstrated that the coaching manager with higher feedback orientation is viewed as more effective than the coaching manager with lower feedback orientation. This study assesses previously untested theories of coaching and demonstrates the value of manager feedback orientation in the coaching process.

Originality/value

This is the first study to integrate the feedback and coaching literatures to test derived hypotheses regarding feedback orientation in the manager-as-coach framework.

     

How Abusive Supervision Affects Workplace Deviance: A Moderated-Mediation Examination of Aggressiveness and Work-Related Negative Affect

Purpose

This research examines the linking mechanisms and conditional processes underlying the abusive supervision and workplace deviance relationship. Based primarily on Affective Events Theory, it was hypothesized that work-related negative affect would mediate the relationship between abusive supervision and workplace deviance, and that this indirect effect would be moderated by employee-based and organization-based aggressiveness.

Design/Methodology/Approach

Two independent studies were conducted, including diverse working samples and multi-wave data, to test these relationships through mediation and moderated-mediation bootstrapping procedures.

Findings

Both studies suggest that work-related negative affect mediates the abusive supervision and workplace deviance relationship. Mixed findings were found for the moderating effect of employee-based and organization-based aggressiveness. In Study 1 higher levels of employee-based aggressive beliefs and attitudes increased the magnitude of the indirect effect; however, in Study 2 when taking into account organization-based aggressive norms only the facet of social discounting bias increased this relationship. In Study 2 higher levels of organization-based aggressive norms also increased the magnitude of the indirect effect for supervisor-directed deviance.

Implications

Theoretical and practical implications of these findings suggest a movement toward an emotion-centered process-based theory of workplace deviance.

Originality/Value

A central question in organizational behavior research revolves around what drives employees to engage in various workplace behaviors. Replicating research that suggests abusive supervision is an important factor in this question, this research helps illuminate the processes underlying this perception-to-behavior link, as well as the boundary conditions of these processes.

     

Impression Management Use in Resumes and Cover Letters

Purpose

This study contributes to the ecological validity of resume research by systematically examining the impression management (IM) content of actual resumes and cover letters and empirically testing the effect on applicant evaluation.

Design/methodology/approach

A content analysis of the frequency and intensity of IM tactic use in 60 resumes and cover letters was completed (Study 1). Next, an experiment was conducted in which IM tactic use was manipulated and the effect on applicant evaluation examined, using a sample of MTurk workers as evaluators (Study 2).

Findings

In Study 1, four self-promotion categories, three ingratiation categories, and one hybrid category were delineated. In Study 2, ingratiation and lower intensity self-promotion were found to increase perceptions of job and organization fit.

Implications

Employers should be aware that resumes and cover letters contain IM tactics that may influence applicant evaluation. In addition, employment training programs might communicate the benefits of using ingratiation and lower intensity self-promotion, while emphasizing the importance of accurately conveying one’s qualifications. Furthermore, the present taxonomy of IM resume content might be applied to resume database search engines to identify and index IM tactic use.

Originality/value

This research is the first to develop a taxonomy of IM tactics based on actual resumes and cover letters and may facilitate more comprehensive manipulations of IM tactic use and better integration of IM research across the selection process.

     

Internet-Based,Unproctored Assessments on Mobile and Non-Mobile Devices: Usage,Measurement Equivalence,and Outcomes

Purpose

The purpose of this study was to determine the usage rates, measurement equivalence, and potential outcome differences between mobile and non-mobile device-based deliveries of an unproctored, non-cognitive assessment.

Design/Methodology/Approach

This study utilized a quasi-experimental design based on archival data obtained from applicants who completed a non-cognitive assessment on a mobile (n = 7,743; e.g., smartphones, tablet computers) or non-mobile (n = 929,341; e.g., desktop computers) device as part of an operational, high-stakes pre-employment selection process.

Findings

One percent of applicants used mobile devices to complete the assessment. Multiple-group confirmatory factor analysis indicated the assessment was equivalent across mobile and non-mobile devices at the configural, metric, scalar, and latent mean levels. A comparison of observed score means using one-way and factorial ANOVAs demonstrated that the use of mobile and non-mobile devices did not produce any practically significant score differences on the assessment across devices or applicant demographic subgroups.

Implications

Industry and technological trends suggest mobile device usage will only increase. Thus, demonstrating that mobile device functionality and hardware characteristics do not change the psychometric functioning or applicant outcomes for a non-cognitive, text-based selection assessment is critical to talent assessment.

Originality/Value

This study provides the first empirical examination of the usage of mobile devices to complete talent assessments and their impact on assessment properties and applicant outcomes, and serves as the foundation for future research and application of this growing technological trend in pre-employment assessment.