Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications

This article presents a d-statistic for single-case designs that is in the same metric as the d-statistic used in between-subjects designs such as randomized experiments and offers some reasons why such a statistic would be useful in SCD research. The d has a formal statistical development, is accompanied by appropriate power analyses, and can be estimated using user-friendly SPSS macros. We discuss both advantages and disadvantages of d compared to other approaches such as previous d-statistics, overlap statistics, and multilevel modeling. It requires at least three cases for computation and assumes normally distributed outcomes and stationarity, assumptions that are discussed in some detail. We also show how to test these assumptions. The core of the article then demonstrates in depth how to compute d for one study, including estimation of the autocorrelation and the ratio of between case variance to total variance (between case plus within case variance), how to compute power using a macro, and how to use the d to conduct a meta-analysis of studies using single-case designs in the free program R, including syntax in an appendix. This syntax includes how to read data, compute fixed and random effect average effect sizes, prepare a forest plot and a cumulative meta-analysis, estimate various influence statistics to identify studies contributing to heterogeneity and effect size, and do various kinds of publication bias analyses. This d may prove useful for both the analysis and meta-analysis of data from SCDs.     

Methods for second order meta-analysis and illustrative applications

This paper presents methods for second order meta-analysis along with several illustrative applications. A second order meta-analysis is a meta-analysis of a number of statistically independent and methodologically comparable first order meta-analyses examining ostensibly the same relationship in different contexts. First order meta-analysis greatly reduces sampling error variance but does not eliminate it. The residual sampling error is called second order sampling error. The purpose of a second order meta-analysis is to estimate the proportion of the variance in mean meta-analytic effect sizes across multiple first order meta-analyses attributable to second order sampling error and to use this information to improve accuracy of estimation for each first order meta-analytic estimate. We present equations and methods based on the random effects model for second order meta-analysis for three situations and three empirical applications of second order meta-analysis to illustrate the potential value of these methods to the pursuit of cumulative knowledge.     

Five-factor personality domains and job performance: A second order meta-analysis

Several meta-analyses have investigated the job-related validities of the traits associated with the Five-Factor Model (FFM). The presence of second-order sampling error, however, might complicate the interpretation of these meta-analyses (i.e., random error across different meta-analyses). The current paper therefore evaluates variability across different meta-analyses and tests for key moderators (i.e., performance criteria, sources of ratings, and context). Results suggested that the variation in the predictive validity of several FFM traits and facets across multiple performance criteria was attributable to second-order sampling error. However, true variation existed for some moderators such as sources of personality information (self-reports versus informant reports). The current analyses therefore enhance knowledge about the usefulness and generalizability of personality traits in work contexts.     

Differentiated associations between childhood maltreatment experiences and social understanding: A meta-analysis and systematic review

The extreme parenting experiences encountered by children who are physically abused or neglected place them at increased risk for impaired socio-emotional development. There is growing evidence that maltreated children may apprehend interpersonal encounters in different ways from children without such traumatic histories. This systematic review examines the links between childhood physical abuse and neglect and various constituent skills of social understanding (including emotion recognition and understanding, perspective taking, false belief understanding, and attributional biases) in 51 empirical studies. The review incorporates a meta-analysis of 19 studies on emotion recognition and understanding in this population. This showed an overall negative effect of maltreatment, but moderation analyses revealed that significantly stronger effects were found for measures of emotion understanding rather than recognition, and for younger rather than older age groups. The broader review also reveals a complex and differentiated profile of social understanding among maltreated children. Directions for future research that addresses individual differences in children’s experiences, both within and outside the maltreatment context, are discussed.     

Shedding light on the psychological and behavioral determinants of travel mode choice: A meta-analysis

Mobility represents a relevant topic from the standpoint of environmental degradation, health-related consequences and social inclusion. Since private mobility is responsible for the greatest share of polluting emissions, it is necessary to gain deeper understanding of the mechanisms underpinning the choice of individuals to use either cars or alternative, environment-friendly transport modes. A meta-analysis on 58 primary studies is conducted to synthesize evidence on the determinants of travel mode choice, as regards both behavioral intentions and actual behaviors. Results suggest that, besides intentions, habits and past use represent the most relevant predictor, followed by constructs referring to the Theory of Planned Behavior framework. Environmental variables, on the other hand, play a relevant role in shaping behavioral intentions while their effect on actual behaviors is negligible, so that a deep intention behavior gap emerges. A moderator analysis is performed to explain the high heterogeneity in the results. Behaviors’ operationalization and measurement emerges as the moderator affecting heterogeneity of outcomes the most; trip purpose, sample type and year of the study also show a moderate effect on heterogeneity, while location does not appear to be a relevant moderator.     

Creativity and the Dark Triad: A meta-analysis

This paper presents a meta-analysis of the relationships between creativity (creative potential, activities, and achievement) and the Dark Triad of personality: narcissism, Machiavellianism, and psychopathy. Multilevel meta-analytic models demonstrated a small but significant positive association between creativity and narcissism (r = 0.15 [0.10, 0.29]) and Machiavellianism (r = 0.06 [0.02, 0.09]), but not with psychopathy (r = 0.03 [-0.02, 0.07]). Creativity measures (self-report-vs.-performance), aspects (self-perception, creative activity, creative abilities, creative achievements), and domains (general, art, science, and everyday creativity) moderated the links with the Dark Triad. We discuss the possible mechanism of associations between Dark Triad traits and creativity and highlight future research directions.     

Organizational identification: A meta-analysis

The last two decades have witnessed a surge in interest in research on organizational identification (OI). This paper presents a comprehensive meta-analysis of this research (k = 96). Results indicate that (a) OI is correlated with a wide range of work-related attitudes, behaviors, and context variables, (b) OI is empirically distinct from its closest conceptual neighbor, attitudinal organizational commitment (AOC), and (c) the two most common OI measures (the Mael scale and the Organizational Identification Questionnaire) produce very different results. It is argued that OI scales, especially the Mael scale, may be preferable over AOC scales for studies aimed at explaining, and partly also for studies aimed at predicting, work behavior.     

Intelligence and interpersonal sensitivity: A meta-analysis

A meta-analytic review investigated the association between general intelligence and interpersonal sensitivity. The review involved 38 independent samples with 2988 total participants. There was a highly significant small-to-medium effect for intelligence measures to be correlated with decoding accuracy (r = .19, p < .001). Significant moderators included the type of decoding judgment (emotion vs. intended meaning judgments), decoding channel (audio-only vs. audio-plus-video channel), and target gender (both male-and-female targets vs. female-only targets). Interpersonal decoding accuracy requires some level of social sophistication and results of this meta-analysis suggest that part of that social sophistication involves the cognitive abilities comprising general intelligence.     

Goal conflict and psychological well-being: A meta-analysis

Goal conflict has long been an important aspect of motivation theories, but the results of research on the relationship between goal conflict and psychological well-being have been inconsistent. A meta-analytic review of the literature (k = 54) was conducted to examine this association. Higher levels of goal conflict are related to lower levels of positive psychological outcomes and greater psychological distress, though this relationship is stronger for distress outcomes. Other moderators that produced significant differences in effect sizes were whether a goal matrix was used to assess goal conflict, whether unipolar or bipolar assessment of goal conflict was used, and whether adult or student samples were studied. This meta-analysis provides evidence that goal conflict has a negative association with psychological well-being.     

On the relative effectiveness of affect regulation strategies: A meta-analysis

To examine the effectiveness of various affect regulation strategies and categories of affect regulation strategies, a meta-analysis was conducted. Results generally indicate that reappraisal (d=0.65) and distraction (d=0.46 for all studies; d=0.95 for studies with a negative or no affect induction) are the most effective regulation/repair strategies, producing the largest hedonic shift in affect. The effectiveness of different categories of regulation/repair strategies depended on the valence of the preceding affect induction. Results also indicate that stronger affect inductions and the use of bivariate affect measures will provide a richer understanding of affect regulation. Additionally, not all specific strategies or categories of strategies have been researched and the impact of individual differences on affect regulation has received relatively little attention. Finally, results indicate that control conditions in affect regulation research may not provide a valid point for comparison, as they facilitate effective affect repair.     

Sex differences in the ability to recognise non-verbal displays of emotion: A meta-analysis

The present study aimed to quantify the magnitude of sex differences in humans' ability to accurately recognise non-verbal emotional displays. Studies of relevance were those that required explicit labelling of discrete emotions presented in the visual and/or auditory modality. A final set of 551 effect sizes from 215 samples was included in a multilevel meta-analysis. The results showed a small overall advantage in favour of females on emotion recognition tasks (d = 0.19). However, the magnitude of that sex difference was moderated by several factors, namely specific emotion, emotion type (negative, positive), sex of the actor, sensory modality (visual, audio, audio-visual) and age of the participants. Method of presentation (computer, slides, print, etc.), type of measurement (response time, accuracy) and year of publication did not significantly contribute to variance in effect sizes. These findings are discussed in the context of social and biological explanations of sex differences in emotion recognition.     

A meta-analysis of burnout with job demands, resources, and attitudes

A meta-analysis was conducted on job demands, resources, and attitudes and their relation with burnout in regard to the COR theory. The version of the Maslach Burnout Inventory used was explored as a moderator of the aforementioned variables. Results suggest that higher demands, lower resources, and lower adaptive organizational attitudes are associated with burnout. In particular, results of the current study show stronger relations than previous meta-analysis (Lee & Ashforth, 1996) have suggested. The scale type also provided some evidence of moderation, with stronger effects found in samples that utilized the MBI-HSS. Implications of the findings in relation to the COR theory and future research directions to clarify the relation between job demands, job resources, organizational attitudes and burnout are discussed.     

Effects of psychological interventions on competitive anxiety in sport: A meta-analysis

ObjectivesThis study reviewed the effects of psychological interventions on competitive anxiety in sport.DesignMeta-analysis and systematic review.MethodPsycINFO, PsycARTICLES, SPORTDiscus, Web of Science, ProQuest, and Sage databases were searched for experimental studies that fulfilled the inclusion criteria. Risk of bias was assessed using the 12 criteria Cochrane Review Book Group tool. Hedge's g and 95% confidence intervals (CIs) were calculated and pooled using a random effects model employing the Hartung-Knapp-Sidik-Jonkman (HKSJ) method.ResultsThe search strategy identified 37 studies which fulfilled the inclusion criteria. The meta-analysis was conducted on 34 studies after removal of outliers. The results showed an overall small to medium-sized effect for psychological interventions on competitive anxiety in athletes (g = −0.42; 95% CI, −0.58 to −0.25). Subsequent subgroup analyses showed that this finding was robust regardless of experimental design, anxiety measure, anxiety type, gender, country, sport, intervention component, intervention delivery method, and intervention duration. The results indicated that the effects might be greater for athletes of higher levels of competition as compared to those from lower levels of competition. Separate meta-analyses also suggested that there were medium to large-sized effects for cognitive anxiety (g = −0.54) and self-confidence (g = 0.55) intensity, and a small to medium-sized effect for somatic anxiety (g = −0.36) intensity.ConclusionThe findings from this review study provide a robust evidence base for the use of psychological interventions to help reduce competitive anxiety in athletes. Future studies need to investigate how psychological interventions might affect the directional interpretation of anxiety symptoms.     

Motivational climate interventions in physical education: A meta-analysis

Objective

The purpose of this study was to synthesize findings from motivational climate interventions employing [Ames, 1992a] and [Ames, 1992b] and [Epstein, 1988] and [Epstein, 1989] TARGET framework within school-based physical education contexts.

Design

The present study employed a quantitative research synthesis design. Meta-analysis uses empirical studies to summarize past research by drawing overall conclusions from separate investigations. This research design highlights important and unsolved issues related to motivational climate interventions within physical education.

Methods

Standard meta-analytic procedures incorporating inclusion and exclusion criteria, literature search, coding procedures, and statistical methods were used to identify and synthesize 22 studies with 24 independent samples. Cohen’s (1988) criteria for effect sizes were used to interpret and evaluate results.

Results

There was an overall small positive treatment effect (g = 0.103) for groups exposed to mastery motivational climates. Outcome analyses identified the most consistent and largest overall treatment effects for behavioral outcomes (g = 0.39–0.49) followed by affective outcomes (g = −0.27 to 0.59) and cognitive outcomes (g = −0.25 to 0.32). Moderator analyses were directed by study heterogeneity and identified several trends in intervention features and study features with the most substantial trend for participant features as elementary students had the largest overall treatment effect (g = 0.41).

Conclusions

Outcome and moderator analyses identified several trends in methodological features, participant features, and study features that should be addressed in future physical education motivational climate interventions.     

A meta-analysis of the effectiveness of the low-ball compliance-gaining procedure

IntroductionThe low-ball (Cialdini et al., 1978)¹ is a compliance-gaining technique consisting of making an attractive initial offer to get a person to agree to the request and then making the terms less favorable (target request).Objective and methodsThe effectiveness of this technique was evaluated in a meta-analysis using 17 references, 23 studies, 44 subgroups and a combined sample size of 4733.ResultsAnalysis reported a weighted mean correlation coefficient of r = .16 and a weighted mean odds ratio of OR = 2.47. Moderator analysis reported that the low-ball was more efficient with a high cost request in terms of effort for the participant and when the solicitation of the target request is deferred.ConclusionThese findings appeared congruent with commitment theoretical explanation of the low-ball.     

Psychological correlates of car use: A meta-analysis

This meta-analysis synthesised quantitative research into potentially modifiable psychological correlates of car use and intentions to drive. Online psychology and transportation databases were searched, and inclusion criteria applied to potentially relevant records. An ancestry approach was also employed to search selected publications. Meta-analyses of effect size rs were performed on 23 unique study datasets. Results generally supported the predictive utility of variables derived from the Theory of Planned Behaviour, though cognitions towards not driving displayed uniformly larger effects than were observed for car use cognitions. There was also a strong effect of habit on behaviour. Support for effects of pro-environment cognitions on driving was weak. However, a dearth of available evidence limited our findings and precludes development of clear evidence-based recommendations for intervention design. Directions for future research are discussed.     

How to protect eyewitness memory against the misinformation effect: A meta-analysis of post-warning studies

Four decades of research and hundreds of studies speak to the power of post-event misinformation to bias eyewitness accounts of events (see e.g., Loftus’ summary, 2005). A subset of this research has explored if the adverse influence of misinformation on remembering can be undone or at least reduced through a later warning about its presence. We meta-analyzed 25 such post-warning studies (including 155 effect sizes) to determine the effectiveness of different types of warnings and to explore moderator effects. Key findings were that (1) post-warnings are surprisingly effective, reducing the misinformation effect to less than half of its size on average. (2) Some types of post-warning (following a theoretical classification) seem to be more effective than others, particularly studies using an enlightenment procedure (Blank, 1998). (3) The post-warning reduction in the misinformation effect reflects a specific increase in misled performance (relative to no warning), at negligible cost for control performance. We conclude with a discussion of theoretical and practical implications.     

Moral responsibility and free will: A meta-analysis

Fundamental beliefs about free will and moral responsibility are often thought to shape our ability to have healthy relationships with others and ourselves. Emotional reactions have also been shown to have an important and pervasive impact on judgments and behaviors. Recent research suggests that emotional reactions play a prominent role in judgments about free will, influencing judgments about determinism’s relation to free will and moral responsibility. However, the extent to which affect influences these judgments is unclear. We conducted a meta-analysis to estimate the impact of affect. Our meta-analysis indicates that beliefs in free will are largely robust to emotional reactions.     

The many faces of the Still-Face Paradigm: A review and meta-analysis

The Still-Face Paradigm (SFP) designed by Tronick, Als, Adamson, Wise, and Brazelton (Tronick, E., Als, H., Adamson, L., Wise, S., & Brazelton, T. B. (1978). Infants response to entrapment between contradictory messages in face-to-face interaction. Journal of the American Academy of Child and Adolescent Psychiatry, 17, 1–13) has been used for many different purposes in over 80 empirical studies. In the current paper, the nature and correlates of infant behavior in the SFP were examined in a systematic narrative review and a series of meta-analyses. The results of the meta-analyses confirmed the classic still-face effect of reduced positive affect and gaze, and increased negative affect, as well as a partial carry-over effect into the reunion episode consisting of lower positive and higher negative affect compared to baseline. The still-face effect is very robust as it was found regardless of most sample variations such as infant gender and risk status, and regardless of most procedural variations, such as the length of the SFP episodes and the use of intervals between episodes. The few moderator effects that were found in the meta-analyses tended to put findings from the narrative review in a new perspective. Additional meta-analyses confirmed the narrative review in finding that higher maternal sensitivity predicted more infant positive affect during the still-face. Infants’ higher positive affect and lower negative affect during the still-face were predictive of secure attachment at age 1 year. The meta-analytic results for maternal depression were equivocal. Implications for future research include a need for studies testing the role of the adults’ identity (parent versus stranger, mother versus father) to elucidate the relationship-specificity of the still-face effect. Also, the role of maternal sensitivity and temperament as potential moderators of the still-face effect need to be examined further. On a procedural level, the effects of the timing of the still-face and of the duration of the reunion on infant responses deserve future research attention.     

Changing the logic of replication: A case from infant studies

Among infant researchers there is growing concern regarding the widespread practice of undertaking studies that have small sample sizes and employ tests with low statistical power (to detect a wide range of possible effects). For many researchers, issues of confidence may be partially resolved by relying on replications. Here, we bring further evidence that the classical logic of confirmation, according to which the result of a replication study confirms the original finding when it reaches statistical significance, could be usefully abandoned. With real examples taken from the infant literature and Monte Carlo simulations, we show that a very wide range of possible replication results would in a formal statistical sense constitute confirmation as they can be explained simply due to sampling error. Thus, often no useful conclusion can be derived from a single or small number of replication studies. We suggest that, in order to accumulate and generate new knowledge, the dichotomous view of replication as confirmatory/disconfirmatory can be replaced by an approach that emphasizes the estimation of effect sizes via meta-analysis. Moreover, we discuss possible solutions for reducing problems affecting the validity of conclusions drawn from meta-analyses in infant research.