A Tutorial and Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In-Hospital Smoking Cessation Counseling on Mortality

Propensity score methods allow investigators to estimate causal treatment effects using observational or nonrandomized data. In this article we provide a practical illustration of the appropriate steps in conducting propensity score analyses. For illustrative purposes, we use a sample of current smokers who were discharged alive after being hospitalized with a diagnosis of acute myocardial infarction. The exposure of interest was receipt of smoking cessation counseling prior to hospital discharge and the outcome was mortality with 3 years of hospital discharge. We illustrate the following concepts: first, how to specify the propensity score model; second, how to match treated and untreated participants on the propensity score; third, how to compare the similarity of baseline characteristics between treated and untreated participants after stratifying on the propensity score, in a sample matched on the propensity score, or in a sample weighted by the inverse probability of treatment; fourth, how to estimate the effect of treatment on outcomes when using propensity score matching, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, or covariate adjustment using the propensity score. Finally, we compare the results of the propensity score analyses with those obtained using conventional regression adjustment.     

A Tutorial and Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In-Hospital Smoking Cessation Counseling on Mortality

Propensity score methods allow investigators to estimate causal treatment effects using observational or nonrandomized data. In this article we provide a practical illustration of the appropriate steps in conducting propensity score analyses. For illustrative purposes, we use a sample of current smokers who were discharged alive after being hospitalized with a diagnosis of acute myocardial infarction. The exposure of interest was receipt of smoking cessation counseling prior to hospital discharge and the outcome was mortality with 3 years of hospital discharge. We illustrate the following concepts: first, how to specify the propensity score model; second, how to match treated and untreated participants on the propensity score; third, how to compare the similarity of baseline characteristics between treated and untreated participants after stratifying on the propensity score, in a sample matched on the propensity score, or in a sample weighted by the inverse probability of treatment; fourth, how to estimate the effect of treatment on outcomes when using propensity score matching, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, or covariate adjustment using the propensity score. Finally, we compare the results of the propensity score analyses with those obtained using conventional regression adjustment.     

Using Ensemble-Based Methods for Directly Estimating Causal Effects: An Investigation of Tree-Based G-Computation

Researchers are increasingly using observational or nonrandomized data to estimate causal treatment effects. Essential to the production of high-quality evidence is the ability to reduce or minimize the confounding that frequently occurs in observational studies. When using the potential outcome framework to define causal treatment effects, one requires the potential outcome under each possible treatment. However, only the outcome under the actual treatment received is observed, whereas the potential outcomes under the other treatments are considered missing data. Some authors have proposed that parametric regression models be used to estimate potential outcomes. In this study, we examined the use of ensemble-based methods (bagged regression trees, random forests, and boosted regression trees) to directly estimate average treatment effects by imputing potential outcomes. We used an extensive series of Monte Carlo simulations to estimate bias, variance, and mean squared error of treatment effects estimated using different ensemble methods. For comparative purposes, we compared the performance of these methods with inverse probability of treatment weighting using the propensity score when logistic regression or ensemble methods were used to estimate the propensity score. Using boosted regression trees of depth 3 or 4 to impute potential outcomes tended to result in estimates with bias equivalent to that of the best performing methods. Using an empirical case study, we compared inferences on the effect of in-hospital smoking cessation counseling on subsequent mortality in patients hospitalized with an acute myocardial infarction.     

Propensity score estimation with boosted regression for evaluating causal effects in observational studies

Causal effect modeling with naturalistic rather than experimental data is challenging. In observational studies participants in different treatment conditions may also differ on pretreatment characteristics that influence outcomes. Propensity score methods can theoretically eliminate these confounds for all observed covariates, but accurate estimation of propensity scores is impeded by large numbers of covariates, uncertain functional forms for their associations with treatment selection, and other problems. This article demonstrates that boosting, a modern statistical technique, can overcome many of these obstacles. The authors illustrate this approach with a study of adolescent probationers in substance abuse treatment programs. Propensity score weights estimated using boosting eliminate most pretreatment group differences and substantially alter the apparent relative effects of adolescent substance abuse treatment.     

Combining propensity score matching and group-based trajectory analysis in an observational study

In a nonrandomized or observational study, propensity scores may be used to balance observed covariates and trajectory groups may be used to control baseline or pretreatment measures of outcome. The trajectory groups also aid in characterizing classes of subjects for whom no good matches are available and to define substantively interesting groups between which treatment effects may vary. These and related methods are illustrated using data from a Montreal-based study. The effects on subsequent violence of gang joining at age 14 are studied while controlling for measured characteristics of boys prior to age 14. The boys are divided into trajectory groups based on violence from ages 11 to 13. Within trajectory group, joiners are optimally matched to a variable number of controls using propensity scores, Mahalanobis distances, and a combinatorial optimization algorithm. Use of variable ratio matching results in greater efficiency than pair matching and also greater bias reduction than matching at a fixed ratio. The possible impact of failing to adjust for an important but unmeasured covariate is examined using sensitivity analysis.     

A Propensity Score Adjustment for Multiple Group Structural Equation Modeling

In the behavioral and social sciences, quasi-experimental and observational studies are used due to the difficulty achieving a random assignment. However, the estimation of differences between groups in observational studies frequently suffers from bias due to differences in the distributions of covariates. To estimate average treatment effects when the treatment variable is binary, Rosenbaum and Rubin (1983a) proposed adjustment methods for pretreatment variables using the propensity score. However, these studies were interested only in estimating the average causal effect and/or marginal means. In the behavioral and social sciences, a general estimation method is required to estimate parameters in multiple group structural equation modeling where the differences of covariates are adjusted. We show that a Horvitz–Thompson-type estimator, propensity score weighted M estimator (PWME) is consistent, even when we use estimated propensity scores, and the asymptotic variance of the PWME is shown to be less than that with true propensity scores. Furthermore, we show that the asymptotic distribution of the propensity score weighted statistic under a null hypothesis is a weighted sum of independent χ² ₁ variables. We show the method can compare latent variable means with covariates adjusted using propensity scores, which was not feasible by previous methods. We also apply the proposed method for correlated longitudinal binary responses with informative dropout using data from the Longitudinal Study of Aging (LSOA). The results of a simulation study indicate that the proposed estimation method is more robust than the maximum likelihood (ML) estimation method, in that PWME does not require the knowledge of the relationships among dependent variables and covariates.     

Humphrey's paradox and the interpretation of inverse conditional propensities

The aim of this paper is to distinguish between, and examine, three issues surrounding Humphreys's paradox and interpretation of conditional propensities. The first issue involves the controversy over the interpretation of inverse conditional propensities — conditional propensities in which the conditioned event occurs before the conditioning event. The second issue is the consistency of the dispositional nature of the propensity interpretation and the inversion theorems of the probability calculus, where an inversion theorem is any theorem of probability that makes explicit (or implicit) appeal to a conditional probability and its corresponding inverse conditional probability. The third issue concerns the relationship between the notion of stochastic independence which is supported by the propensity interpretation, and various notions of causal independence. In examining each of these issues, it is argued that the dispositional character of the propensity interpretation provides a consistent and useful interpretation of the probability calculus.I would like to thank William L. Harper, Paul Humphreys, John Nicholas and Kathleen Okruhlik for helpful comments and advice on earlier drafts of this paper. Research for this paper was supported by a fellowship from the Social Sciences and Humanities Research Council of Canada (award number 452-90-2513).     

Drawing Causal Inferences Using Propensity Scores: A Practical Guide for Community Psychologists

Confounding present in observational data impede community psychologists’ ability to draw causal inferences. This paper describes propensity score methods as a conceptually straightforward approach to drawing causal inferences from observational data. A step-by-step demonstration of three propensity score methods—weighting, matching, and subclassification—is presented in the context of an empirical examination of the causal effect of preschool experiences (Head Start vs. parental care) on reading development in kindergarten. Although the unadjusted population estimate indicated that children with parental care had substantially higher reading scores than children who attended Head Start, all propensity score adjustments reduce the size of this overall causal effect by more than half. The causal effect was also defined and estimated among children who attended Head Start. Results provide no evidence for improved reading if those children had instead received parental care. We carefully define different causal effects and discuss their respective policy implications, summarize advantages and limitations of each propensity score method, and provide SAS and R syntax so that community psychologists may conduct causal inference in their own research.     

Balancing Treatment and Control Groups in Quasi‐Experiments: An Introduction to Propensity Scoring

Organizational and applied sciences have long struggled with improving causal inference in quasi‐experiments. We introduce organizational researchers to propensity scoring, a statistical technique that has become popular in other applied sciences as a means for improving internal validity. Propensity scoring statistically models how individuals in a quasi‐experiment have been assigned to conditions in order to estimate treatment effects among individuals with approximately equal probabilities of receiving the treatment. If propensity scores are created from relevant covariates, matching on the propensity score makes treatment assignment ignorable and approximates a true experimental design. We illustrate how matching on the propensity score can be applied by using 2 examples: examining the effects of online instruction and estimating the benefits of preparatory coaching for the SAT. In both cases, propensity‐scoring methods effectively reduced inequivalence between treatment and control groups on many variables. Propensity scoring stands out as a valuable technique capable of improving causal inference from many of organizational research's quasi‐experiments.     

The reference class problem is your problem too

The reference class problem arises when we want to assign a probability to a proposition (or sentence, or event) X, which may be classified in various ways, yet its probability can change depending on how it is classified. The problem is usually regarded as one specifically for the frequentist interpretation of probability and is often considered fatal to it. I argue that versions of the classical, logical, propensity and subjectivist interpretations also fall prey to their own variants of the reference class problem. Other versions of these interpretations apparently evade the problem. But I contend that they are all “no-theory” theories of probability - accounts that leave quite obscure why probability should function as a guide to life, a suitable basis for rational inference and action. The reference class problem besets those theories that are genuinely informative and that plausibly constrain our inductive reasonings and decisions. I distinguish a “metaphysical” and an “epistemological” reference class problem. I submit that we can dissolve the former problem by recognizing that probability is fundamentally a two-place notion: conditional probability is the proper primitive of probability theory. However, I concede that the epistemological problem remains.     

A Two-Step Bayesian Approach for Propensity Score Analysis: Simulations and Case Study

A two-step Bayesian propensity score approach is introduced that incorporates prior information in the propensity score equation and outcome equation without the problems associated with simultaneous Bayesian propensity score approaches. The corresponding variance estimators are also provided. The two-step Bayesian propensity score is provided for three methods of implementation: propensity score stratification, weighting, and optimal full matching. Three simulation studies and one case study are presented to elaborate the proposed two-step Bayesian propensity score approach. Results of the simulation studies reveal that greater precision in the propensity score equation yields better recovery of the frequentist-based treatment effect. A slight advantage is shown for the Bayesian approach in small samples. Results also reveal that greater precision around the wrong treatment effect can lead to seriously distorted results. However, greater precision around the correct treatment effect parameter yields quite good results, with slight improvement seen with greater precision in the propensity score equation. A comparison of coverage rates for the conventional frequentist approach and proposed Bayesian approach is also provided. The case study reveals that credible intervals are wider than frequentist confidence intervals when priors are non-informative.     

Combining group-based trajectory modeling and propensity score matching for causal inferences in nonexperimental longitudinal data

A central theme of research on human development and psychopathology is whether a therapeutic intervention or a turning-point event, such as a family break-up, alters the trajectory of the behavior under study. This article describes and applies a method for using observational longitudinal data to make more transparent causal inferences about the impact of such events on developmental trajectories. The method combines 2 distinct lines of research: work on the use of finite mixture modeling to analyze developmental trajectories and work on propensity score matching. The propensity scores are used to balance observed covariates and the trajectory groups are used to control pretreatment measures of response. The trajectory groups also aid in characterizing classes of subjects for which no good matches are available. The approach is demonstrated with an analysis of the impact of gang membership on violent delinquency based on data from a large longitudinal study conducted in Montréal, Canada.     

The Use of Propensity Scores in Mediation Analysis

Mediation analysis uses measures of hypothesized mediating variables to test theory for how a treatment achieves effects on outcomes and to improve subsequent treatments by identifying the most efficient treatment components. Most current mediation analysis methods rely on untested distributional and functional form assumptions for valid conclusions, especially regarding the relation between the mediator and outcome variables. Propensity score methods offer an alternative whereby the propensity score is used to compare individuals in the treatment and control groups who would have had the same value of the mediator had they been assigned to the same treatment condition. This article describes the use of propensity score weighting for mediation with a focus on explicating the underlying assumptions. Propensity scores have the potential to offer an alternative estimation procedure for mediation analysis with alternative assumptions from those of standard mediation analysis. The methods are illustrated investigating the mediational effects of an intervention to improve sense of mastery to reduce depression using data from the Job Search Intervention Study (JOBS II). We find significant treatment effects for those individuals who would have improved sense of mastery when in the treatment condition but no effects for those who would not have improved sense of mastery under treatment.     

Bayesian Model Averaging for Propensity Score Analysis

This article considers Bayesian model averaging as a means of addressing uncertainty in the selection of variables in the propensity score equation. We investigate an approximate Bayesian model averaging approach based on the model-averaged propensity score estimates produced by the R package BMA but that ignores uncertainty in the propensity score. We also provide a fully Bayesian model averaging approach via Markov chain Monte Carlo sampling (MCMC) to account for uncertainty in both parameters and models. A detailed study of our approach examines the differences in the causal estimate when incorporating noninformative versus informative priors in the model averaging stage. We examine these approaches under common methods of propensity score implementation. In addition, we evaluate the impact of changing the size of Occam’s window used to narrow down the range of possible models. We also assess the predictive performance of both Bayesian model averaging propensity score approaches and compare it with the case without Bayesian model averaging. Overall, results show that both Bayesian model averaging propensity score approaches recover the treatment effect estimates well and generally provide larger uncertainty estimates, as expected. Both Bayesian model averaging approaches offer slightly better prediction of the propensity score compared with the Bayesian approach with a single propensity score equation. Covariate balance checks for the case study show that both Bayesian model averaging approaches offer good balance. The fully Bayesian model averaging approach also provides posterior probability intervals of the balance indices.     

How Do Propensity Score Methods Measure Up in the Presence of Measurement Error? A Monte Carlo Study

Considering that the absence of measurement error in research is a rare phenomenon and its effects can be dramatic, we examine the impact of measurement error on propensity score (PS) analysis used to minimize selection bias in behavioral and social observational studies. A Monte Carlo study was conducted to explore the effects of measurement error on the treatment effect and balance estimates in PS analysis across seven different PS conditioning methods. In general, the results indicate that even low levels of measurement error in the covariates lead to substantial bias in estimates of treatment effects and concomitant reduction in confidence interval coverage across all methods of conditioning on the PS.     

Prenatal Exposure to Maternal Smoking and Childhood Behavioural Problems: A Quasi-experimental Approach

This retrospective cross-sectional paper examines the relationship between maternal smoking during pregnancy and children’s behavioural problems at 9?years of age independent of a wide range of possible confounders. The final sample comprised 7,505 nine-year-old school children participating in the first wave of the Growing Up in Ireland study. The children were selected through the Irish national school system using a 2-stage sampling method and were representative of the nine-year population. Information on maternal smoking during pregnancy was obtained retrospectively at 9?years of age via parental recall and children’s behavioural problems were assessed using the Strengths and Difficulties Questionnaire across separate parent and teacher-report instruments. A quasi-experimental approach using propensity score matching was used to create treatment (smoking) and control (non-smoking) groups which did not differ significantly in their propensity to smoke in terms of 16 observed characteristics. After matching on the propensity score, children whose mothers smoked during pregnancy were 3.5?% (p?<?0.001) and 3.4?% (p?<?0.001) more likely to score in the problematic range on the SDQ total difficulties index according to parent and teacher-report respectively. Maternal smoking during pregnancy was more strongly associated with externalising than internalising behavioural problems. Analysis of the dose–response relationship showed that the differential between matched treatment and control groups increased with level of maternal smoking. Given that smoking is a modifiable risk factor, the promotion of successful cessation in pregnancy may prevent potentially adverse long-term consequences.     

Abstract: An International Comparison of Private and Public Schools Using Multilevel Propensity Score Methods and Graphics

As can be seen from the recent Special Issue of MBR on propensity score analysis (PSA) methods, the use of PSA has gained increasing popularity for estimating causal effects in observational studies. However, PSA use with multilevel or clustered data has been limited, and to date there seems to have been no development of specialized graphics for such data. This paper introduces the multilevelPSA (http://multilevelPSA.r-forge.r-project.org) package for R that provides cluster-based functions for estimating propensity scores as well as graphics to exhibit results for multilevel data. This work extends to the multilevel case the framework for visualizing propensity score analysis introduced by Helmreich and Pruzek (2009). International data from the Programme for International Student Assessment (Organization for Economic Co-operation and Development, 2009) are comprehensively examined to compare private with public schools on reading, mathematics, and science outcomes after adjusting for covariate differences in the multilevel context.

Particularly for analyses of large data sets, focusing on statistical significance is limiting. As can readily be seen, overall results favor “private” over “public” schools, at least for end of secondary school math achievement. But the graphics provide a more nuanced understanding of the nature and magnitude of adjusted differences for countries. Furthermore, the graphics are readily interpreted by a nontechnical audience. Broadly speaking, it is seen that modern graphics can enhance and extend conventional numerical summaries by focusing on details of what data have to say for multilevel comparisons of many countries based on propensity score methods.     

Counterfactual Triviality: A Lewis‐Impossibility Argument for Counterfactuals

I formulate a counterfactual version of the notorious ‘Ramsey Test’. Whereas the Ramsey Test for indicative conditionals links credence in indicatives to conditional credences, the counterfactual version links credence in counterfactuals to expected conditional chance. I outline two forms: a Ramsey Identity on which the probability of the conditional should be identical to the corresponding conditional probability/expectation of chance; and a Ramsey Bound on which credence in the conditional should never exceed the latter. Even in the weaker, bound, form, the counterfactual Ramsey Test makes counterfactuals subject to the very argument that Lewis used to argue against the indicative version of the Ramsey Test. I compare the assumptions needed to run each, pointing to assumptions about the time‐evolution of chances that can replace the appeal to Bayesian assumptions about credence update in motivating the assumptions of the argument. I finish by outlining two reactions to the discussion: to indicativize the debate on counterfactuals; or to counterfactualize the debate on indicatives.     

Betting on conditionals

A study is reported testing two hypotheses about a close parallel relation between indicative conditionals, if A then B, and conditional bets, I bet you that if A then B. The first is that both the indicative conditional and the conditional bet are related to the conditional probability, P(B|A). The second is that de Finetti's three-valued truth table has psychological reality for both types of conditional—true, false, or void for indicative conditionals and win, lose, or void for conditional bets. The participants were presented with an array of chips in two different colours and two different shapes, and an indicative conditional or a conditional bet about a random chip. They had to make judgements in two conditions: either about the chances of making the indicative conditional true or false or about the chances of winning or losing the conditional bet. The observed distributions of responses in the two conditions were generally related to the conditional probability, supporting the first hypothesis. In addition, a majority of participants in further conditions chose the third option, “void”, when the antecedent of the conditional was false, supporting the second hypothesis.