Systems change as an outcome and a process in the work of community collaboratives for health

The widespread development of comprehensive community initiatives that aim to improve community health is driven by the need to change the systems charged with delivering the services and creating the policies related to a variety of health outcomes. Georgia's Family Connection initiative is the nation's largest statewide network of community collaboratives for health, with collaboratives operating in 159 counties. Data on community context, collaborative processes, engagement in systems change, and changes in programs and activities implemented, gathered consistently at the collaborative level over 3 years, will be used to answer the following questions. How do community contexts and the structure and processes of collaboratives affect implementation of systems change? How do systems changes affect intermediate outcomes such as the type of programs offered in a community? Longitudinal change in systems change and program implementation is described and significant predictors of between‐collaborative variation in longitudinal change for each outcome are identified.     

Using full matching to estimate causal effects in nonexperimental studies: examining the relationship between adolescent marijuana use and adult outcomes

Matching methods such as nearest neighbor propensity score matching are increasingly popular techniques for controlling confounding in nonexperimental studies. However, simple k:1 matching methods, which select k well-matched comparison individuals for each treated individual, are sometimes criticized for being overly restrictive and discarding data (the unmatched comparison individuals). The authors illustrate the use of a more flexible method called full matching. Full matching makes use of all individuals in the data by forming a series of matched sets in which each set has either 1 treated individual and multiple comparison individuals or 1 comparison individual and multiple treated individuals. Full matching has been shown to be particularly effective at reducing bias due to observed confounding variables. The authors illustrate this approach using data from the Woodlawn Study, examining the relationship between adolescent marijuana use and adult outcomes.     

A Two-Step Bayesian Approach for Propensity Score Analysis: Simulations and Case Study

A two-step Bayesian propensity score approach is introduced that incorporates prior information in the propensity score equation and outcome equation without the problems associated with simultaneous Bayesian propensity score approaches. The corresponding variance estimators are also provided. The two-step Bayesian propensity score is provided for three methods of implementation: propensity score stratification, weighting, and optimal full matching. Three simulation studies and one case study are presented to elaborate the proposed two-step Bayesian propensity score approach. Results of the simulation studies reveal that greater precision in the propensity score equation yields better recovery of the frequentist-based treatment effect. A slight advantage is shown for the Bayesian approach in small samples. Results also reveal that greater precision around the wrong treatment effect can lead to seriously distorted results. However, greater precision around the correct treatment effect parameter yields quite good results, with slight improvement seen with greater precision in the propensity score equation. A comparison of coverage rates for the conventional frequentist approach and proposed Bayesian approach is also provided. The case study reveals that credible intervals are wider than frequentist confidence intervals when priors are non-informative.     

Evaluation of a cognitive‐behavioral intervention for high‐ and medium‐risk probationers

Reducing recidivism is a central goal of treatment programs for offenders. Preliminary evidence suggests that cognitive‐behavioral group interventions based on the National Institute of Corrections curriculum (Bush, Glick, & Taymans, 1997) may be effective in reducing recidivism rates among adult probationers. We evaluated the effectiveness of a program based on this curriculum among 167 high‐ and medium‐risk probationers assigned to this program and a comparison group of 120 high‐ and medium‐risk probationers matched on age and number of prior criminal charges. Improvements over prior studies included use of survival analytic methods and propensity score matching, a longer follow‐up interval, and examination of treatment effectiveness within ethnic groups. Relative to the comparison group, treatment group probationers were more likely to complete probation satisfactorily and survive longer before rearrest. Moreover, supplementary analyses suggested that ethnicity was associated with differences in intervention effectiveness. Treatment was predictive of lower recidivism rates among European Americans and African Americans but was less effective among Latino American probationers.     

Profiling Capacity for Coordination and Systems Change: The Relative Contribution of Stakeholder Relationships in Interorganizational Collaboratives

In response to increasing demands for greater coordination and collaboration among community institutions, interorganizational collaboratives (i.e., coalitions, partnerships, coordinating councils) have emerged as a popular mechanism for strengthening the capacity of a community system to respond to public and social issues. This study adopts a network approach to explore the relative importance of dense networks of cooperative relationships among members of interorganizational collaboratives for two outcomes of effectiveness: improving interorganizational coordination and fostering systems change. Based on survey and social network data collected from 48 different collaboratives, findings indicate that, relative to other key characteristics of collaboratives identified in previous literature, cooperative stakeholder relationships were the strongest predictor of systems change outcomes. However, for coordination outcomes, stakeholder relationships were overshadowed in importance by the leadership and decision making capacity of the collaborative. Collectively, findings suggest key differences in the requisite capacity profiles for coordination and systems change outcomes.     

An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies

The propensity score is the probability of treatment assignment conditional on observed baseline characteristics. The propensity score allows one to design and analyze an observational (nonrandomized) study so that it mimics some of the particular characteristics of a randomized controlled trial. In particular, the propensity score is a balancing score: conditional on the propensity score, the distribution of observed baseline covariates will be similar between treated and untreated subjects. I describe 4 different propensity score methods: matching on the propensity score, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, and covariate adjustment using the propensity score. I describe balance diagnostics for examining whether the propensity score model has been adequately specified. Furthermore, I discuss differences between regression-based methods and propensity score-based methods for the analysis of observational data. I describe different causal average treatment effects and their relationship with propensity score analyses.     

Going Home Again: Transitioning Youth to Families After Group Care Placement

This paper presents results from the transitioning youth to families intervention, which aimed to promote family care for youth served in group care programs in the child welfare system. The intervention was conducted in two counties in a Mid-Atlantic state. The effort encompassed administrative case review and family involvement meetings. We assessed the effect of the intervention in promoting placement in family settings within 12 months. We also explored other effects of the intervention identified by participants. Using propensity score matching with administrative data to compare one-year placement settings for the intervention counties and non-intervention counties, a higher rate of family reunification was identified for youth in the treatment counties. To provide a richer contextual understanding of the effects of the intervention, thematic analysis of open-ended comments from youth and caregiver participants was conducted. These resulting themes provided further understanding of the value of the intervention especially in the areas of planning for the transition, improving youth insight about placement options, and the importance of family involvement. Participants also made recommendations for how to enhance the intervention and promote the transition of youth from group care to family settings. Overall, the effect of the intervention in transitioning youth to family settings was nominal; however unanticipated benefits of engaging youth and family in the transition process were noted. From this evaluation, we provide suggestions for future research and the development of effort to transition youth from group care settings.     

Parenting of low birth weight infants: A review of issues and interventions

Factors that may adversely affect parenting of low birth weight infants, including infant characteristics, parental emotional responses to premature birth, and patterns of parent-infant interaction, are reviewed. In addition, intervention studies designed to improve infant developmental outcomes through influencing parenting behaviors are examined. There was great diversity in theoretical frameworks, in timing of onset and frequency of interventions, and in measurement of outcomes. The interventions that were most effective in influencing behaviors and infant developmental outcomes were those in which there were multiple long-term contacts with parents and/or those that actively involved them in the intervention. The specific theoretical framework upon which the interventions were based appeared to be of less importance in influencing results. In order to increase our knowledge and ability to provide costeffective programs, there is a need to focus attention and data collection efforts on the process and intermediary steps of intervention as well as on outcome measures. There is also a need to improve our ability to recruit and retain those families who are most at risk for parenting and infant developmental problems including those with very low birth weight infants, those whose infants have serious perinatal complications, and those with indications of high social risk such as poverty-level incomes and problems of substance abuse.     

Balancing Treatment and Control Groups in Quasi‐Experiments: An Introduction to Propensity Scoring

Organizational and applied sciences have long struggled with improving causal inference in quasi‐experiments. We introduce organizational researchers to propensity scoring, a statistical technique that has become popular in other applied sciences as a means for improving internal validity. Propensity scoring statistically models how individuals in a quasi‐experiment have been assigned to conditions in order to estimate treatment effects among individuals with approximately equal probabilities of receiving the treatment. If propensity scores are created from relevant covariates, matching on the propensity score makes treatment assignment ignorable and approximates a true experimental design. We illustrate how matching on the propensity score can be applied by using 2 examples: examining the effects of online instruction and estimating the benefits of preparatory coaching for the SAT. In both cases, propensity‐scoring methods effectively reduced inequivalence between treatment and control groups on many variables. Propensity scoring stands out as a valuable technique capable of improving causal inference from many of organizational research's quasi‐experiments.     

An analysis of the effectiveness of university counselling services

It is important to demonstrate replicable evidence of the effectiveness of counselling procedures. The study aimed to contribute to the currently limited evidence base examining the effectiveness of university student counselling in the UK. Information on therapeutic outcome [based on Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM) scores] for 305 individuals attending a large UK university counselling service was analysed. Following counselling intervention: there was a statistically significant improvement in CORE-OM scores with 63% of individuals showing a reliable improvement and only 2% showing a reliable deterioration. Of those who began with a score in the clinical range, 49% showed a clinically significant change. These results provide additional evidence for the effectiveness of university counselling interventions. Future research should aim to build on this preliminary research using randomised controlled trial designs.     

A Tutorial and Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In-Hospital Smoking Cessation Counseling on Mortality

Propensity score methods allow investigators to estimate causal treatment effects using observational or nonrandomized data. In this article we provide a practical illustration of the appropriate steps in conducting propensity score analyses. For illustrative purposes, we use a sample of current smokers who were discharged alive after being hospitalized with a diagnosis of acute myocardial infarction. The exposure of interest was receipt of smoking cessation counseling prior to hospital discharge and the outcome was mortality with 3 years of hospital discharge. We illustrate the following concepts: first, how to specify the propensity score model; second, how to match treated and untreated participants on the propensity score; third, how to compare the similarity of baseline characteristics between treated and untreated participants after stratifying on the propensity score, in a sample matched on the propensity score, or in a sample weighted by the inverse probability of treatment; fourth, how to estimate the effect of treatment on outcomes when using propensity score matching, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, or covariate adjustment using the propensity score. Finally, we compare the results of the propensity score analyses with those obtained using conventional regression adjustment.     

A Tutorial and Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In-Hospital Smoking Cessation Counseling on Mortality

Propensity score methods allow investigators to estimate causal treatment effects using observational or nonrandomized data. In this article we provide a practical illustration of the appropriate steps in conducting propensity score analyses. For illustrative purposes, we use a sample of current smokers who were discharged alive after being hospitalized with a diagnosis of acute myocardial infarction. The exposure of interest was receipt of smoking cessation counseling prior to hospital discharge and the outcome was mortality with 3 years of hospital discharge. We illustrate the following concepts: first, how to specify the propensity score model; second, how to match treated and untreated participants on the propensity score; third, how to compare the similarity of baseline characteristics between treated and untreated participants after stratifying on the propensity score, in a sample matched on the propensity score, or in a sample weighted by the inverse probability of treatment; fourth, how to estimate the effect of treatment on outcomes when using propensity score matching, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, or covariate adjustment using the propensity score. Finally, we compare the results of the propensity score analyses with those obtained using conventional regression adjustment.     

Measurement of the effectiveness of sex offender treatment

The aim of this paper is to review various methods of examining treatment effectiveness. A number of research methods used to examine the overall effectiveness of sex offender treatment, such as random assignment, risk band analysis, and matched control groups are discussed. Each of these designs offer advantages, but also have methodological shortcomings. While there are those who feel that only the most scientifically rigorous methodology must be employed if one hopes to draw meaningful conclusions, others feel that less stringent criteria in terms of comparison groups can yield meaningful inferential results. As a means of overcoming some of the shortcomings of recidivism outcome studies discussed, we suggest that the examination of more proximate outcomes, such as change within treatment, provide a useful addition to studies of treatment effectiveness.     

Assessing the Impact of a Media‐based Intervention to Prevent Intergroup Violence and Promote Positive Intergroup Relations in Burundi

The present study (N = 1074) examined the impact of a theory‐driven media intervention aimed at violence prevention and intergroup reconciliation in Burundi. We used a novel methodology utilizing audio‐based surveys to assess attitudes related to intergroup conflict and reconciliation among community members. We conducted a propensity score analysis to estimate the causal effects of the intervention by examining differences between listeners and non‐listeners of the radio dramas. The results indicated a positive effect of the intervention on several social psychological outcomes (tolerance, in‐group superiority, social distance, intergroup trust, responsibility attributions, trauma disclosure and competitive victimhood). However, listeners and non‐listeners did not differ in obedience toward leaders or historical perspective taking; and the results for active bystandership, one of the main foci of the intervention, were mixed. Furthermore, the results show that the impact of the intervention sometimes depends on listeners' personal experiences of victimization. Copyright © 2015 John Wiley & Sons, Ltd.     

Religion and Infant Mortality in the United States: A Community‐Level Investigation of Denominational Variations in Postneonatal Deaths

Although a growing body of research has detected the effects of community‐level religiosity on various health outcomes, very little scholarship has examined the influence of religious ecology on infant mortality rates (IMRs). We conduct ordinary least squares (OLS) regression analyses on postneonatal IMRs (PNIMRs) using county‐level data from the National Center for Health Statistics Linked Birth and Infant Death Data (1990, 2000, and 2006–2010), churches and church membership data, and the Area Health Resource File. We find that while overall rates of postneonatal deaths have decreased over time, the effects of religion on this outcome have become more pronounced. Specifically, we find that counties with greater proportions of mainline Protestant and Catholic adherents exhibit significantly lower PNIMRs. We further find that a greater proportion of conservative Protestants, and especially fundamentalists, increases postneonatal infant mortality. Our findings lend additional support to cultural explanations of U.S. infant mortality.     

Family‐based Intervention for Legal System‐involved Girls: A Mixed Methods Evaluation

The increased proportion of juvenile court‐involved girls has spurred interest to implement and evaluate services to reduce girls’ system involvement. The purpose of this study was to examine the effectiveness of a family‐based intervention by using a dominant sequential mixed methods evaluation approach. First, we examined quantitative data using a quasi‐experimental design to determine whether the family‐based intervention reduced recidivism among court‐involved girls. Propensity score matching (PSM) was used to construct statistically equivalent groups to compare one‐year recidivism outcomes for girls who received the court‐run family‐based intervention (n = 181) to a group of girls on probation who did not receive the intervention (n = 803). Qualitative interviews (n = 39) were conducted to contextualize the quantitative findings and highlighted the circumstances that family‐focused interventions for court‐involved girls. Girls who received the program had slightly lower recidivism rates following the intervention. The qualitative findings contextualized the quasi‐experimental results by providing an explanation as to the girls’ family circumstances and insights into the mechanisms of the intervention. Results highlighted the importance of family‐focused interventions for juvenile justice‐involved girls. These findings have practical and policy implications for the use interventions—beyond the individual level—with adjudicated girls and offer suggestions for ways to improve their effectiveness using a community psychology lens. In addition, this paper includes a discussion of evaluating of juvenile court programming from a community psychology perspective including strengths, challenges, and considerations for future work in this area.     

The Academic Advantage of Devotion: Measuring Variation in the Value of Weekly Worship in Late Adolescence on Educational Attainment Using Propensity Score Matching

This study measures the effect of regular worship attendance at age 17 on total years of schooling by age 25, using data from the National Longitudinal Survey of Youth 1997. Expanding on previous work, this study estimates differences in the impact of worship attendance by race and family income status using propensity score matching. Individuals who frequently attend religious services complete .69 more years of schooling than similar individuals who do not frequently attend services. There are significantly greater returns to attendance for low‐income youth and no significant difference in returns by religious affiliation. These findings suggest that religious observance provides greater benefits for low‐income individuals or perhaps provides resources high‐income individuals have access to elsewhere. Moreover, this study extends previous work by examining a more recent and nationally representative sample of youth and by using methods that allow for greater causal inference.     

The Head,the Heart,and the Community: Piloting a Brief Home-Based Mindfulness Intervention for Couples With Low Income

Although mindfulness is documented to reduce both individual and relationship stress and has the flexibility to be taught anywhere (e.g., at home, clinic setting, etc.), research examining mindfulness interventions among individuals with low income and economic marginalization (LIEM; APA, 2019), or persons whose economic position negatively impacts their health or well-being due to factors such as access to healthcare, is limited. To address this gap, the author and colleagues used Community Based Participatory Research methods to develop a brief, couple-based, mindfulness intervention tailored for communities with LIEM (see Lenger et al., 2022). The present study tested this newly developed brief, couple-based, mindfulness intervention's effectiveness in improving individual and relationship health through 1- and 2-months post-intervention. The intervention was piloted on a sample of 39 couples with an overrepresentation of couples with LIEM. To improve access to care, couples could participate in their home or a variety of local clinics. Thirty-nine couples received the intervention and completed assessments on mindfulness, depression, anxiety, stress, relationship satisfaction, and communication at baseline, 1-month, and 2-months post-intervention. Results revealed that mindfulness, depression, and relationship satisfaction improved from baseline to 1-month post-intervention. Mindfulness improved at a greater rate for couples with LIEM relative to couples with higher incomes. From baseline to 2-months post-intervention, depression and stress significantly improved at similar rates for couples with higher income and couples with LIEM. Thus, this study indicates that mindfulness can be taught in a brief, two-session format, and can have favorable outcomes on individual and relationship functioning.     

Lessons learned in systems change initiatives: benchmarks and indicators

Measuring progress toward systems change, sustainable efforts that address root causes of an issue by changing policies and practices, is a difficult task for communities, evaluators, and foundations. Tracking and documenting changes in resources, power, policy, sustainable funding, structured relationships and roles, and underlying values require multi-level analyses. Systems change analysts must consider at least four “strata” at once: (1) events and trends, (2) patterns of interaction, (3) context and cultural or social models, and (4) the systems themselves. In this paper we provide a brief overview of systems change; a discussion of collaboratives as one “engine” of social change; a discussion of benchmarks and indicators of collaboratives focused on systems change; and suggestions for further research. The analysis draws upon several analytic frameworks described in the literature. We illustrate these concepts with examples from six systems change initiatives funded by The California Endowment. The need for further research is outlined.