Meta-analyses in sales research

Meta-analytic inquiries enable researchers to synthesize empirical findings obtained over the evolution of a topic area and identify boundary conditions affecting the associations of key variables. Sales researchers can employ meta-analytic techniques to amalgamate empirical work conducted in a given topic area, and several sales researchers have effectively used meta-analyses to advance our understanding of the field. This article provides an exposition of this research design by analyzing its application in sales research. The authors review the meta-analytic studies in sales research and advance key considerations in topical foci, article selection, data coding and evaluation, and analytic approaches. An empirical example is provided to illustrate the power of meta-analysis in substantiating or refuting findings that diverge from accumulated insight in sales research. Results provide support for a positive effort–job satisfaction association in contrast to findings evidencing a negative association between these variables. A second empirical example is used to evaluate the nomological and discriminant validity of two related constructs: job involvement and organizational commitment. Results suggest that these two constructs may be empirically redundant. The authors also provide guidance for the future in regard to new substantive research areas, construct assessment, and additional types of meta-analytic approaches.     

REVISITING INTERVIEW–COGNITIVE ABILITY RELATIONSHIPS: ATTENDING TO SPECIFIC RANGE RESTRICTION MECHANISMS IN META-ANALYSIS

This study revisits the relationship between interviews and cognitive ability tests, finding lower magnitudes of correlation than have previous meta-analyses; a finding that has implications for both the construct and incremental validity of the interview. Our lower estimates of this relationship than previous meta-analyses were mainly due to (a) an updated set of studies, (b) exclusion of samples in which interviewers potentially had access to applicants' cognitive test scores, and (c) attention to specific range restriction mechanisms that allowed us to identify a sizable subset of studies for which range restriction could be accurately accounted. Moderator analysis results were similar to previous meta-analyses, but magnitudes of correlation were generally lower than in previous meta-analyses. Findings have implications for the construct and incremental validity of interviews, and meta-analytic methodology in general.     

Publication bias in psychological science: comment on Ferguson and Brannick (2012)

It is well documented that studies reporting statistically significant results are more likely to be published than are studies reporting nonsignificant results--a phenomenon called publication bias. Publication bias in meta-analytic reviews should be identified and reduced when possible. Ferguson and Brannick (2012) argued that the inclusion of unpublished articles is ineffective and possibly counterproductive as a means of reducing publication bias in meta-analyses. We show how idiosyncratic choices on the part of Ferguson and Brannick led to an erroneous conclusion. We demonstrate that their key finding--that publication bias was more likely when unpublished studies were included--may be an artifact of the way they assessed publication bias. We also point out how the lack of transparency about key choices and the absence of information about critical features of Ferguson and Brannick's sample and procedures might have obscured readers' ability to assess the validity of their claims. Furthermore, we demonstrate that many of the claims they made are without empirical support, even though they could have tested these claims empirically, and that these claims may be misleading. With their claim that addressing publication bias introduces subjectivity and bias into meta-analysis, they ignored a large body of evidence showing that including unpublished studies that meet the inclusion criteria of a meta-analysis decreases (rather than increases) publication bias. Rather than exclude unpublished studies, we recommend that meta-analysts code study characteristics related to methodological quality (e.g., experimental vs. nonexperimental design) and test whether these factors influence the meta-analytic results.     

Meta-Analysis and Construct Validity

It is argued that meta-analysis is a fundamental tool in any construct valida- tion effort. Meta-analytic resuits provide the stable and precise estimates of relationships between variables that are needed for causal modeling and the- ory development. A review of the application of meta-analysis to the area of employment validities is used to show the theoretical and applied benefits this tool can bring to construct validation research. The case is also made that meta-analyses focused on the criterion side of prediction equations would lead to enhanced knowledge of the construct validity of criteria.     

The Influence of Judgment Calls on Meta-Analytic Findings

Previous research has suggested that judgment calls (i.e., methodological choices made in the process of conducting a meta-analysis) have a strong influence on meta-analytic findings and question their robustness. However, prior research applies case study comparison or reanalysis of a few meta-analyses with a focus on a few selected judgment calls. These studies neglect the fact that different judgment calls are related to each other and simultaneously influence the outcomes of a meta-analysis, and that meta-analytic findings can vary due to non–judgment call differences between meta-analyses (e.g., variations of effects over time). The current study analyzes the influence of 13 judgment calls in 176 meta-analyses in marketing research by applying a multivariate, multilevel meta-meta-analysis. The analysis considers simultaneous influences from different judgment calls on meta-analytic effect sizes and controls for alternative explanations based on non–judgment call differences between meta-analyses. The findings suggest that judgment calls have only a minor influence on meta-analytic findings, whereas non–judgment call differences between meta-analyses are more likely to explain differences in meta-analytic findings. The findings support the robustness of meta-analytic results and conclusions.     

Personality and job performance: the Big Five revisited

Prior meta-analyses investigating the relation between the Big 5 personality dimensions and job performance have all contained a threat to construct validity, in that much of the data included within these analyses was not derived from actual Big 5 measures. In addition, these reviews did not address the relations between the Big 5 and contextual performance. Therefore, the present study sought to provide a meta-analytic estimate of the criterion-related validity of explicit Big 5 measures for predicting job performance and contextual performance. The results for job performance closely paralleled 2 of the previous meta-analyses, whereas analyses with contextual performance showed more complex relations among the Big 5 and performance. A more critical interpretation of the Big 5-performance relationship is presented, and suggestions for future research aimed at enhancing the validity of personality predictors are provided.     

Expanding the criterion domain? A quantitative review of the OCB literature

The authors investigate the construct validity of the organizational citizenship behavior (OCB)-task performance distinction by providing a quantitative review of the OCB literature. The authors extend previous meta-analytic reviews of the OCB literature by (a) using confirmatory factor analysis (CFA) to investigate the dimensionality of OCB, (b) using CFA to examine the distinction between OCB and task performance, and (c) examining the relationship between a latent OCB factor and task performance and attitudinal variables. Results support a single factor model of OCB that is distinct from, albeit strongly related to, task performance. In addition, results show that OCB consistently relates more strongly to attitudes than does task performance and shares a modest amount of variance with attitudinal correlates beyond task performance.     

Using theory to evaluate personality and job-performance relations: a socioanalytic perspective

The authors used socioanalytic theory to understand individual differences in people's performance at work. Specifically, if predictors and criteria are aligned by using theory, then the meta-analytic validity of personality measures exceeds that of atheoretical approaches. As performance assessment moved from general to specific job criteria, all Big Five personality dimensions more precisely predicted relevant criterion variables, with estimated true validities of .43 (Emotional Stability), .35 (Extraversion-Ambition), .34 (Agreeableness), .36 (Conscientiousness), and .34 (Intellect-Openness to Experience).     

Sociotropy and autonomy: preliminary evidence for construct validity using TAT narratives

In this investigation, we evaluated the construct validity of sociotropy and autonomy as assessed by the revised Personal Style Inventory (PSI; Robins et al., 1994). Stories given to 6 cards of the Thematic Apperception Test (Murray, 1943) were coded for need for Achievement (McClelland, Atkinson, Clark, & Lowell, 1953) and need for Affiliation (Heyns, Veroff, & Atkinson, 1958). These scores were correlated with PSI Sociotropy and Autonomy, along with their component subscales. The construct validity of Sociotropy, Autonomy, and 5 of 6 component subscales were supported as hypoth sized. Consistent with past research, there was no support for the construct validity of the Perfectionism/Self-Criticism subscale of Autonomy. In addition, separate analyses by gender suggested that the construct validity of sociotropy may be greater for women than for men. The results represent an important finding in that nonquestionnaire measures of interpersonal and achievement-related concerns were found to support the validity of the PSI, a need identified by the questionnaire's authors.     

The impact of methodological factors on child psychotherapy outcome research: A meta-analysis for researchers

Two recent meta-analyses have generated evidence for child and adolescent psychotherapy effects. However, critics note that such meta-analyses often include studies with methodological shortcomings which might invalidate their results. In the present study, we explored whether the results of the most extensive child/adolescent meta-analysis might have been influenced by such methodological variables, focusing on internal validity and external validity factors. Together, these factors accounted for two-thirds as much variance as the substantive factors (e.g., type of therapy, age) in the original metaanalysis. This suggests that relative to these therapy and child-characteristic variables, methodological factors have a substantial, though smaller, impact on metaanalysis results. In general, increased experimental rigor was related to larger effect sizes; this argues against the hypothesis that methodologically weak studies have led to an overestimate of therapy effects. No significant interactive relations were found between validity factors and predictors of outcome; this suggests that the relations noted in previous metaanalyses between outcome and various variables were notdistorted by the validity factors tested here.The research described here was supported in part by grants from the North Carolina Department of Human Resources, Division of Mental Health, Mental Retardation, and Substance Abuse Services (41626), and from the National Institute of Mental Health (1 RO3 MH38450). The authors gratefully acknowledge the assistance of Todd Morton, who provided reliability coding.     

Cohesion and performance in groups: a meta-analytic clarification of construct relations

Previous meta-analytic examinations of group cohesion and performance have focused primarily on contextual factors. This study examined issues relevant to applied researchers by providing a more detailed analysis of the criterion domain. In addition, the authors reinvestigated the role of components of cohesion using more modern meta-analytic methods and in light of different types of performance criteria. The results of the authors' meta-analyses revealed stronger correlations between cohesion and performance when performance was defined as behavior (as opposed to outcome), when it was assessed with efficiency measures (as opposed to effectiveness measures), and as patterns of team workflow became more intensive. In addition, and in contrast to B. Mullen and C. Copper's (1994) meta-analysis, the 3 main components of cohesion were independently related to the various performance domains. Implications for organizations and future research on cohesion and performance are discussed.     

Sociotropy and Autonomy: Preliminary Evidence for Construct Validity Using TAT Narratives

In this investigation, we evaluated the construct validity of sociotropy and autonomy as assessed by the revised Personal Style Inventory (PSI; Robins et al., 1994). Stories given to 6 cards of the Thematic Apperception Test (Murray, 1943) were coded for need for Achievement (McClelland, Atkinson, Clark, & Lowell, 1953) and need for Affiliation (Heyns, Veroff, & Atkinson, 1958). These scores were correlated with PSI Sociotropy and Autonomy, along with their component subscales. The construct validity of Sociotropy, Autonomy, and 5 of 6 component subscales were supported as hypothesized. Consistent with past research, there was no support for the construct validity of the Perfectionism/Self-Criticism subscale of Autonomy. In addition, separate analyses by gender suggested that the construct validity of sociotropy may be greater for women than for men. The results represent an important finding in that nonquestionnaire measures of interpersonal and achievement-related concerns were found to support the validity of the PSI, a need identified by the questionnaire's authors.     

Structural equation models and personality research

This article is a nontechnical introduction to the use of structural equation models in personality research Although such models can be fruitfully used to address a variety of important theoretical issues, the substantive focus in this article is on the use of such models for elucidating the construct validity of personality measures We include numerous more specific topics under our treatment of construct validity First of all, we show how structural equation models can be applied to the issues of convergent and discriminant validity Do our variables measure the constructs we want them to measure and not other constructs that we would prefer not to measure? Second, we show the utility of structural equation models for predictive validity Do our variables reliably predict other constructs with which they are theoretically linked? Finally, we examine the stability of personality constructs through structural equation models Through-out, our emphasis is on the particular advantages that structural equation models bring to these analytic tasks Ultimately, such models must be used in the service of theory, and when used appropriately, they can help us to refine both our measures and our theories of individual differences     

Gamification in employee selection: The development of a gamified assessment

Gamification has attracted increased attention among organizations and human resource professionals recently, as a novel and promising concept for attracting and selecting prospective employees. In the current study, we explore the construct validity of a new gamified assessment method in employee selection that we developed following the situational judgement test (SJT) methodology. Our findings support the applicability of game elements into a traditional form of assessment built to assess candidates' soft skills. Specifically, our study contributes to research on gamification and employee selection exploring the construct validity of a gamified assessment method indicating that the psychometric properties of SJTs and their transformation into a gamified assessment are a suitable avenue for future research and practice in this field.     

Examining the “Pros” and “Cons” of Team Conflict: A Team-Level Meta-Analysis of Task,Relationship, and Process Conflict

Jehn (e.g., 1997) offered three distinct types of team conflict, namely, task conflict, relationship conflict, and process conflict. Despite existing meta-analyses, there remain important and ongoing issues that warrant further meta-analytic investigation. Our contribution is threefold. First, we report novel meta-analytic findings involving moderators of the conflict–team performance relationship. Second, we report on meta-analytic correlations involving all three conflict types and team innovation. Third, we report on the relations involving task conflict and relationship conflict with previously unexamined, but critical, teamwork variables: team potency, cooperative behaviors, competitive behaviors, and avoidance behaviors. Input for the current meta-analysis included 89 independent samples, 6,122 teams, and approximately 28,000 team members.     

Associations Between Attachment Narratives and Self-Report Measures of Attachment in Middle Childhood: Extending Evidence for the Validity of the Child Attachment Interview

The assessment of attachment security in infancy and adulthood is well-studied, but middle childhood has been relatively neglected in the literature. The Child Attachment Interview (CAI) represents a promising assessment of attachment in middle childhood; recent evidence supports its reliability and validity in restricted samples. We assessed the construct validity of the CAI among a diverse sample of 104 8–12 year-old children (M _age  = 9.80 years; 50 boys; 49 % family income <$40,000; 38 % Latino/a). We evaluated associations among CAI variables and self-report measures of attachment (convergent validity). Further, we evaluated whether CAI variables uniquely relate to theoretically-relevant constructs (internalizing symptoms) above and beyond self-reported attachment (incremental validity) and a theoretically-distinct construct, child temperament (discriminant validity). Our findings support the CAI’s validity, and have important implications for assessing and understanding attachment processes in middle childhood, as well as their development across the lifespan.     

Critical ingredients of career choice interventions: More analyses and new hypotheses

Recent meta-analyses of the career choice intervention literature (Brown & Ryan Krane, 2000; Ryan, 1999) identified five intervention ingredients that were individually associated with career choice outcome, and collectively were linearly related to increases in career choice effect sizes. In this article, we summarize these meta-analytic findings, present analyses that further confirm these ingredients’ importance, and present some hypotheses to define each of the critical ingredients and suggest how they might be implemented to maximize their effectiveness.     

Confirmatory Factor Analysis of the Structure of Adult ADHD Symptoms

Previous research has examined the factor structure of ADHD symptoms in adults using confirmatory factor analysis and there is mixed support for two-factor, three-factor, or more complex bifactor models. The current study re-examines these factor structures on the basis of not only overall model fit, but also criteria such as chi-square difference tests, invariance, reliability, and validity. Three datasets were pooled to form a total sample of 430 adults (50% female). Participants completed questionnaires assessing their ADHD symptoms, education, psychopathology, and parenting (the latter as validity variables). Although the bifactor models demonstrated the best fit, the reliability and construct replicability of specific factors in these models was poor. The three-factor model had good fit and demonstrated good reliability, validity, construct replicability, and some invariance across gender and dataset. Our results replicate previous studies that find superior fit for bifactor models. However, the superior statistical fit may reflect only the increased complexity of these models. The more parsimonious three-factor model may demonstrate stronger validity and clinical utility.     

The critical role of the research question, inclusion criteria, and transparency in meta-analyses of integrity test research: a reply to Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012)

We clear up a number of misconceptions from the critiques of our meta-analysis (Van Iddekinge, Roth, Raymark, & Odle-Dusseau, 2012). We reiterate that our research question focused on the criterion-related validity of integrity tests for predicting individual work behavior and that our inclusion criteria flowed from this question. We also reviewed the primary studies we could access from Ones, Viswesvaran, and Schmidt's (1993) meta-analysis of integrity tests and found that only about 30% of the studies met our inclusion criteria. Further, analyses of some of the types of studies we had to exclude revealed potentially inflated validity estimates (e.g., corrected validities as high as .80 for polygraph studies). We also discuss our experience trying to obtain primary studies and other information from authors of Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012). In addition, we address concerns raised about certain decisions we made and values we used, and we demonstrate how such concerns would have little or no effect on our results or conclusions. Finally, we discuss some other misconceptions about our meta-analysis, as well as some divergent views about the integrity test literature in general. Overall, we stand by our research question, methods, and results, which suggest that the validity of integrity tests for criteria such as job performance and counterproductive work behavior is weaker than the authors of the critiques appear to believe.