Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation

The first purpose of this review was to assess the impact of implementation on program outcomes, and the second purpose was to identify factors affecting the implementation process. Results from over quantitative 500 studies offered strong empirical support to the conclusion that the level of implementation affects the outcomes obtained in promotion and prevention programs. Findings from 81 additional reports indicate there are at least 23 contextual factors that influence implementation. The implementation process is affected by variables related to communities, providers and innovations, and aspects of the prevention delivery system (i.e., organizational functioning) and the prevention support system (i.e., training and technical assistance). The collection of implementation data is an essential feature of program evaluations, and more information is needed on which and how various factors influence implementation in different community settings.     

Effects of positive youth development programs on school, family, and community systems

A review of efforts at social system change in 526 universal competence-promotion outcome studies indicated that 64% of the interventions attempted some type of microsystemic or mesosystemic change involving schools, families, or community-based organizations in an attempt to foster developmental competencies in children and adolescents. Only 24% of the reports provided quantitative data on the change that occurred in targeted systems. However, studies containing the necessary information produced several mean effect sizes that were statistically significant, and ranged from modest to large in magnitude. These data indicate that attempts to change social systems affecting children and adolescents can be successful. Future work should measure more thoroughly the extent to which the systemic changes that are targeted through intervention are achieved, and investigate how such changes contribute to the development and sustainability of the outcomes that might be demonstrated by participants of competence-promotion programs. The authors wish to thank David Wilson for supplying the macros used to calculate study level effect sizes and Mark Lipsey for supplying the SPSS macros used to weight effect sizes and calculate mean effects across studies. This study was supported in part by a grant from the William T. Grant Foundation (#2212) awarded to the first and last authors.