Time-Varying Treatments in Observational Studies: Marginal Structural Models of the Effects of Early Grade Retention on Math Achievement

One of the main objectives of many empirical studies in the social and behavioral sciences is to assess the causal effect of a treatment or intervention on the occurrence of a certain event. The randomized controlled trial is generally considered the gold standard to evaluate such causal effects. However, for ethical or practical reasons, social scientists are often bound to the use of nonexperimental, observational designs. When the treatment and control group are different with regard to variables that are related to the outcome, this may induce the problem of confounding. A variety of statistical techniques, such as regression, matching, and subclassification, is now available and routinely used to adjust for confounding due to measured variables. However, these techniques are not appropriate for dealing with time-varying confounding, which arises in situations where the treatment or intervention can be received at multiple timepoints. In this article, we explain the use of marginal structural models and inverse probability weighting to control for time-varying confounding in observational studies. We illustrate the approach with an empirical example of grade retention effects on mathematics development throughout primary school.     

From correlates to causes: can quasi-experimental studies and statistical innovations bring us closer to identifying the causes of antisocial behavior?

Longitudinal, epidemiological studies have identified robust risk factors for youth antisocial behavior, including harsh and coercive discipline, maltreatment, smoking during pregnancy, divorce, teen parenthood, peer deviance, parental psychopathology, and social disadvantage. Nevertheless, because this literature is largely based on observational studies, it remains unclear whether these risk factors have truly causal effects. Identifying causal risk factors for antisocial behavior would be informative for intervention efforts and for studies that test whether individuals are differentially susceptible to risk exposures. In this article, we identify the challenges to causal inference posed by observational studies and describe quasi-experimental methods and statistical innovations that may move researchers beyond discussions of risk factors to allow for stronger causal inference. We then review studies that used these methods, and we evaluate whether robust risk factors identified from observational studies are likely to play a causal role in the emergence and development of youth antisocial behavior. There is evidence of causal effects for most of the risk factors we review. However, these effects are typically smaller than those reported in observational studies, suggesting that familial confounding, social selection, and misidentification might also explain some of the association between risk exposures and antisocial behavior. For some risk factors (e.g., smoking during pregnancy, parent alcohol problems), the evidence is weak that they have environmentally mediated effects on youth antisocial behavior. We discuss the implications of these findings for intervention efforts to reduce antisocial behavior and for basic research on the etiology and course of antisocial behavior.     

Causal graphical views of fixed effects and random effects models

Despite the long-standing discussion on fixed effects (FE) and random effects (RE) models, how and under what conditions both methods can eliminate unmeasured confounding bias has not yet been widely understood in practice. Using a simple pretest–posttest design in a linear setting, this paper translates the conventional algebraic formalization of FE and RE models into causal graphs and provides intuitively accessible graphical explanations about their data-generating and bias-removing processes. The proposed causal graphs highlight that FE and RE models consider different data-generating models. RE models presume a data-generating model that is identical to a randomized controlled trial, while FE models allow for unobserved time-invariant treatment–outcome confounding. Augmenting regular causal graphs that describe data-generating processes by adding the computational structures of FE and RE estimators, the paper visualizes how FE estimators (gain score and deviation score estimators) and RE estimators (quasi-deviation score estimators) offset unmeasured confounding bias. In contrast to standard regression or matching estimators that reduce confounding bias by blocking non-causal paths via conditioning, FE and RE estimators offset confounding bias by deliberately creating new non-causal paths and associations of opposite sign. Though FE and RE estimators are similar in their bias-offsetting mechanisms, the augmented graphs reveal their subtle differences that can result in different biases in observational studies.     

Using Ensemble-Based Methods for Directly Estimating Causal Effects: An Investigation of Tree-Based G-Computation

Researchers are increasingly using observational or nonrandomized data to estimate causal treatment effects. Essential to the production of high-quality evidence is the ability to reduce or minimize the confounding that frequently occurs in observational studies. When using the potential outcome framework to define causal treatment effects, one requires the potential outcome under each possible treatment. However, only the outcome under the actual treatment received is observed, whereas the potential outcomes under the other treatments are considered missing data. Some authors have proposed that parametric regression models be used to estimate potential outcomes. In this study, we examined the use of ensemble-based methods (bagged regression trees, random forests, and boosted regression trees) to directly estimate average treatment effects by imputing potential outcomes. We used an extensive series of Monte Carlo simulations to estimate bias, variance, and mean squared error of treatment effects estimated using different ensemble methods. For comparative purposes, we compared the performance of these methods with inverse probability of treatment weighting using the propensity score when logistic regression or ensemble methods were used to estimate the propensity score. Using boosted regression trees of depth 3 or 4 to impute potential outcomes tended to result in estimates with bias equivalent to that of the best performing methods. Using an empirical case study, we compared inferences on the effect of in-hospital smoking cessation counseling on subsequent mortality in patients hospitalized with an acute myocardial infarction.     

Addressing confounding errors when using non-experimental, observational data to make causal claims

In their recent book, Is Inequality Bad for Our Health?, Daniels, Kennedy, and Kawachi claim that to “act justly in health policy, we must have knowledge about the causal pathways through which socioeconomic (and other) inequalities work to produce differential health outcomes.” One of the central problems with this approach is its dependency on “knowledge about the causal pathways.” A widely held belief is that the randomized clinical trial (RCT) is, and ought to be the “gold standard” of evaluating the causal efficacy of interventions. However, often the only data available are non-experimental, observational data. For such data, the necessary randomization is missing. Because the randomization is missing, it seems to follow that it is not possible to make epistemically warranted claims about the causal pathways. Although we are not sanguine about the difficulty in using observational data to make warranted causal claims, we are not as pessimistic as those who believe that the only warranted causal claims are claims based on data from (idealized) RCTs. We argue that careful, thoughtful study design, informed by expert knowledge, that incorporates propensity score matching methods in conjunction with instrumental variable analyses, provides the possibility of warranted causal claims using observational data.     

Drawing Causal Inferences Using Propensity Scores: A Practical Guide for Community Psychologists

Confounding present in observational data impede community psychologists’ ability to draw causal inferences. This paper describes propensity score methods as a conceptually straightforward approach to drawing causal inferences from observational data. A step-by-step demonstration of three propensity score methods—weighting, matching, and subclassification—is presented in the context of an empirical examination of the causal effect of preschool experiences (Head Start vs. parental care) on reading development in kindergarten. Although the unadjusted population estimate indicated that children with parental care had substantially higher reading scores than children who attended Head Start, all propensity score adjustments reduce the size of this overall causal effect by more than half. The causal effect was also defined and estimated among children who attended Head Start. Results provide no evidence for improved reading if those children had instead received parental care. We carefully define different causal effects and discuss their respective policy implications, summarize advantages and limitations of each propensity score method, and provide SAS and R syntax so that community psychologists may conduct causal inference in their own research.     

Beyond the Mean: A Flexible Framework for Studying Causal Effects Using Linear Models

Psychometrika - Graph-based causal models are a flexible tool for causal inference from observational data. In this paper, we develop a comprehensive framework to define, identify, and estimate a...     

Combining group-based trajectory modeling and propensity score matching for causal inferences in nonexperimental longitudinal data

A central theme of research on human development and psychopathology is whether a therapeutic intervention or a turning-point event, such as a family break-up, alters the trajectory of the behavior under study. This article describes and applies a method for using observational longitudinal data to make more transparent causal inferences about the impact of such events on developmental trajectories. The method combines 2 distinct lines of research: work on the use of finite mixture modeling to analyze developmental trajectories and work on propensity score matching. The propensity scores are used to balance observed covariates and the trajectory groups are used to control pretreatment measures of response. The trajectory groups also aid in characterizing classes of subjects for which no good matches are available. The approach is demonstrated with an analysis of the impact of gang membership on violent delinquency based on data from a large longitudinal study conducted in Montréal, Canada.     

Balancing Treatment and Control Groups in Quasi‐Experiments: An Introduction to Propensity Scoring

Organizational and applied sciences have long struggled with improving causal inference in quasi‐experiments. We introduce organizational researchers to propensity scoring, a statistical technique that has become popular in other applied sciences as a means for improving internal validity. Propensity scoring statistically models how individuals in a quasi‐experiment have been assigned to conditions in order to estimate treatment effects among individuals with approximately equal probabilities of receiving the treatment. If propensity scores are created from relevant covariates, matching on the propensity score makes treatment assignment ignorable and approximates a true experimental design. We illustrate how matching on the propensity score can be applied by using 2 examples: examining the effects of online instruction and estimating the benefits of preparatory coaching for the SAT. In both cases, propensity‐scoring methods effectively reduced inequivalence between treatment and control groups on many variables. Propensity scoring stands out as a valuable technique capable of improving causal inference from many of organizational research's quasi‐experiments.     

Inferring causal networks from observations and interventions

Information about the structure of a causal system can come in the form of observational data—random samples of the system's autonomous behavior—or interventional data—samples conditioned on the particular values of one or more variables that have been experimentally manipulated. Here we study people's ability to infer causal structure from both observation and intervention, and to choose informative interventions on the basis of observational data. In three causal inference tasks, participants were to some degree capable of distinguishing between competing causal hypotheses on the basis of purely observational data. Performance improved substantially when participants were allowed to observe the effects of interventions that they performed on the systems. We develop computational models of how people infer causal structure from data and how they plan intervention experiments, based on the representational framework of causal graphical models and the inferential principles of optimal Bayesian decision‐making and maximizing expected information gain. These analyses suggest that people can make rational causal inferences, subject to psychologically reasonable representational assumptions and computationally reasonable processing constraints.     

Assessing moderated mediation in linear models requires fewer confounding assumptions than assessing mediation

It is well known from the mediation analysis literature that the identification of direct and indirect effects relies on strong no unmeasured confounding assumptions of no unmeasured confounding. Even in randomized studies the mediator may still be correlated with unobserved prognostic variables that affect the outcome, in which case the mediator's role in the causal process may not be inferred without bias. In the behavioural and social science literature very little attention has been given so far to the causal assumptions required for moderated mediation analysis. In this paper we focus on the index for moderated mediation, which measures by how much the mediated effect is larger or smaller for varying levels of the moderator. We show that in linear models this index can be estimated without bias in the presence of unmeasured common causes of the moderator, mediator and outcome under certain conditions. Importantly, one can thus use the test for moderated mediation to support evidence for mediation under less stringent confounding conditions. We illustrate our findings with data from a randomized experiment assessing the impact of being primed with social deception upon observer responses to others’ pain, and from an observational study of individuals who ended a romantic relationship assessing the effect of attachment anxiety during the relationship on mental distress 2 years after the break‐up.     

Weekly Cycles in Daily Report Data: An Overlooked Issue

Daily diaries and other everyday experience methods are increasingly used to study relationships between two time‐varying variables X and Y. Although daily data potentially often have weekly cyclical patterns (e.g., stress may be higher on weekdays and lower on weekends), the majority of daily diary studies have ignored this possibility. In this study, we investigated the effect of ignoring existing weekly cycles. We reanalyzed an empirical dataset (stress and alcohol consumption) and performed Monte Carlo simulations to investigate the impact of omitting weekly cycles. In the empirical dataset, ignoring cycles led to the inference of a significant within‐person X–Y relation whereas modeling cycles suggested that this relationship did not exist. Simulation results indicated that ignoring cycles that existed in both X and Y led to bias in the estimated within‐person X–Y relationship. The amount and direction of bias depended on the magnitude of the cycles, magnitude of the true within‐person X–Y relation, and synchronization of the cycles. We encourage researchers conducting daily diary studies to address potential weekly cycles in their data. We provide guidelines for detecting and modeling cycles to remove their influence and discuss challenges of causal inference in daily experience studies.     

Rethinking the Control of Nuisance Variables in Theory Testing

In testing hypotheses, researchers frequently have used multiple regression analysis to control for nuisance variables (i.e., potential confounding variables that are correlated with hypothesized causal variables). In this paper, we highlight limitations of this control strategy, and we discuss fundamental issues that should be considered in deciding whether to use it. Ultimately, we suggest the use of multiple regression analysis sometimes may not improve causal understanding and may actually limit the generalizability of results. Instead of the common practice of controlling for nuisance variables, we suggest that looking at shared variance frequently is a more appropriate test of a theoretical hypothesis.     

AN EMPIRICAL TEST OF CAUSAL INFERENCE BETWEEN ROLE PERCEPTIONS, SATISFACTION WITH WORK, PERFORMANCE AND ORGANIZATIONAL LEVEL

Previous research has suggested that organizational level may explain to a significant extent the differential impact of role perceptions (i.e., role ambiguity and role conflict) on employee satisfaction and performance. Causal inferences could not be drawn from these studies because of the predominant use of static correlational methods. In this study, in a hospital setting, a six-month time-lag between data collection periods was used to develop causal inferences. The results supported the hypothesis that role ambiguity was a source of causal inference with satisfaction with work at the higher organizational level, while role conflict was a source of causal inference with satisfaction with work at the lower organizational level. The source and direction of causal influence with respect to role perceptions and performance was supported only at the higher organizational level.     

Using full matching to estimate causal effects in nonexperimental studies: examining the relationship between adolescent marijuana use and adult outcomes

Matching methods such as nearest neighbor propensity score matching are increasingly popular techniques for controlling confounding in nonexperimental studies. However, simple k:1 matching methods, which select k well-matched comparison individuals for each treated individual, are sometimes criticized for being overly restrictive and discarding data (the unmatched comparison individuals). The authors illustrate the use of a more flexible method called full matching. Full matching makes use of all individuals in the data by forming a series of matched sets in which each set has either 1 treated individual and multiple comparison individuals or 1 comparison individual and multiple treated individuals. Full matching has been shown to be particularly effective at reducing bias due to observed confounding variables. The authors illustrate this approach using data from the Woodlawn Study, examining the relationship between adolescent marijuana use and adult outcomes.     

Hunting Side Effects and Explaining Them: Should We Reverse Evidence Hierarchies Upside Down?

Philosophical discussions have critically analysed the methodological pitfalls and epistemological implications of evidence assessment in medicine, however they have mainly focused on evidence of treatment efficacy. Most of this work is devoted to statistical methods of causal inference with a special attention to the privileged role assigned to randomized controlled trials (RCTs) in evidence based medicine. Regardless of whether the RCT’s privilege holds for efficacy assessment, it is nevertheless important to make a distinction between causal inference of intended and unintended effects, in that the unknowns at stake are heterogonous in the two contexts. However, although “lower level” evidence is increasingly acknowledged to be a valid source of information contributory to assessing the risk profile of medications on theoretical or empirical grounds, current practices have difficulty in assigning a precise epistemic status to this kind of evidence because they are more or less implicitly parasitic on the (statistical) methods developed to test drug efficacy. My thesis is that (1) “lower level” evidence is justified on distinct grounds and at different conditions depending on the different epistemologies which one wishes to endorse, in that each impose different constraints on the methods we adopt to collect and evaluate evidence; (2) such constraints ought to be understood to be different in the case of evidence for risk versus benefit assessment for a series of reasons which I will illustrate on the basis of the recent debate on the causal association between acetaminophen (a.k.a. paracetamol) and asthma.     

Does EMDR reduce post‐traumatic stress disorder symptomatology in combat veterans?

Prior meta‐analyses have suggested that eye‐movement desensitization and reprocessing (EMDR) may be effective in alleviating the symptoms of post‐traumatic stress disorder (PTSD). EMDR is now being recommended as a treatment for military combat veterans who suffer from PTSD. We provide a review of published outcome studies that appeared in print from 1987 – April, 2008 which examined the specific effects of EMDR on PTSD among military combat veterans. Studies were identified through electronic bibliographic databases, web sites, and manual searches of article reference lists. A total of six randomized controlled trials (RCTs) and three quasi‐experimental studies met our inclusionary criteria and are reviewed. The evidence supporting the use of EMDR to treat combat veterans suffering from PTSD is sparse and equivocal, and does not rise to the threshold of labeling the therapy as an empirically supported treatment. It is premature to incorporate EMDR into routine care for veterans to alleviate combat‐related PTSD. EMDR needs a considerably stronger evidentiary foundation which includes large‐scale RCTs involving credible placebo controlled treatment conditions. Copyright © 2009 John Wiley & Sons, Ltd.     

Time as a guide to cause

How do people learn causal structure? In 2 studies, the authors investigated the interplay between temporal-order, intervention, and covariational cues. In Study 1, temporal order overrode covariation information, leading to spurious causal inferences when the temporal cues were misleading. In Study 2, both temporal order and intervention contributed to accurate causal inference well beyond that achievable through covariational data alone. Together, the studies show that people use both temporal-order and interventional cues to infer causal structure and that these cues dominate the available statistical information. A hypothesis-driven account of learning is endorsed, whereby people use cues such as temporal order to generate initial models and then test these models against the incoming covariational data.     

Is the association between childhood maltreatment and aggressive behavior mediated by hostile attribution bias in women? A discordant twin and sibling study

Understanding the mechanisms behind aggressive behavior (AGG) is vital so that effective prevention and intervention strategies can be developed. Maltreated children are hypothesized to be prone to social information processing biases, such as hostile attribution bias (HAB), which, in turn, may increase the likelihood of behaving aggressively. The first aim of the present study was to replicate findings regarding associations between childhood maltreatment (CM), HAB, and aggression in a population‐based sample of Finnish female twins and their sisters (N = 2,167). However, these associations might not be causal but instead confounded by familial factors, shared between the variables. The second aim was, thus, to test the associations when potential confounding by familial (genetic or common environmental) effects were controlled for using a multilevel discordant twin and sibling design within (a) 379 pairs of twins (n_pairs = 239) or siblings (n_pairs = 140), and (b) within the 131 monozygotic (MZ) twin pairs. Consistent with previous studies, HAB mediated the association between CM and AGG when familial confounding was uncontrolled. No support was found for the mediation when controlling for familial confounding. Between‐pair associations were found between CM and AGG, and between CM and HAB. In addition, within‐pair associations were found between HAB and AGG, and between CM and AGG, however, these were nonsignificant in the discordant MZ analysis, offering the most stringent control of familial confounding. The results indicate the necessity of taking familial confounding into account when investigating the development of AGG.