Opening the Black Box: Using Process Evaluation Measures to Assess Implementation and Theory Building

The past decade has seen increasing recognition in prevention science of the need to move away from a black box approach to intervention evaluation and toward an approach that can elaborate on the mechanisms through which changes in the outcomes operate (Chen & Rossi, 1989; Durlak & Wells, 1997; Spoth et al., 1995). An approach that examines issues of program implementation is particularly critical in the design of efficacy studies of school-based preventive interventions. Numerous preventive intervention strategies are now delivered within the schools, often by regular classroom teachers. The extent to which teachers faithfully deliver a particular curriculum or incorporate instructional strategies emphasized by an intervention is a critical question for the overall project evaluation. This article illustrates the utilization of process measures from a multicomponent school-based prevention program to examine implementation of a teaching staff development intervention, and the program's underlying theoretical basis. Given the nested study design, the analyses utilize hierarchical linear models (Bryk & Raudenbush, 1992) to examine changes in teaching strategies by condition and investigate the hypothesized relationships between teaching practices and student behaviors based on the program's theoretical framework. Results suggest that teaching practices in two of the six intervention focus areas were positively impacted in the first 18 months of the project. Findings also support th relationships between teachers' instructional practices and students' behaviour.     

Preventing Depression Among Adolescent Girls: Pathways Toward Effective and Sustainable Programs

Up to 25% of adolescent girls experience an episode of major depression, at least twice the rate found with adolescent boys. In addition to reducing the suffering associated with depression, prevention efforts with this high-risk population have the potential to avert short- and long-term functional impairment, reduce the risk of associated mental and physical health problems, and provide mental health services to teens who may not otherwise receive help. The effectiveness of such programs depends upon the ability to reach at-risk girls and provide effective intervention, and to accomplish these goals in ways that are sustainable in community settings such as schools. This article discusses the advantages and drawbacks of several methods of delivering prevention programs for adolescent depression, and highlights strategies to enhance the effectiveness and sustainability of these programs. Lessons learned from the implementation of a universal prevention program for Grade 9 girls (Resourceful Adolescent Program-RAP; Shochet et al., 2001) are discussed, and practical recommendations are provided with regard to systems, program, implementation, and evaluation factors. Session-by-session details of RAP are highlighted, focusing on cognitive-behavioral and interpersonal therapy concepts and methods, adapted for use in a classroom setting.     

Sustainable antibullying program implementation: School profiles and predictors

We examined the sustainability of the KiVa antibullying program in Finland from its nationwide roll-out in 2009 to 2016. Using latent class analyses, we identified four different patterns of implementation. The persistent schools (43%) maintained a high likelihood of participation throughout the study period. The awakened (14%) had a decreasing trend during the first years, but then increased the likelihood of program participation. The tail-offs (20%) decreased in the likelihood of participating after the third year, and the drop-offs (23%) already after the first year. The findings suggest that many schools need support during the initial years to launch and maintain the implementation of evidence-based programs; yet a large proportion of schools manage to sustain the program implementation for several years. The logistic regression analyses showed that large schools persisted more likely than small schools. Lower initial level of victimization was also related to the sustainability of the program. Finally, persistent program participation was predicted by several school-level actions during the initial years of implementing the program. These results imply that the sustainability of evidence-based programs could be enhanced by supporting and guiding schools when setting up the program during the initial implementation.     

School Climate and Implementation of a Preventive Intervention

Although there has been wide dissemination of research-based psychosocial prevention programs, a similarly strong research base to guide program implementation has been lacking. Program implementation has been particularly difficult for schools, due partly to insufficient understanding of how school ecologies interact with these programs. This study examined the effects of multiple dimensions of school climate on level and rate of change in implementation of a violence prevention intervention across three school years. Using multi-level modeling, the study found that teacher-reported support between staff and among teachers and students predicted higher average levels of implementation. Teacher-reported administrative leadership predicted greater growth in implementation across 3 years. Findings offer implications for an ecological model of program implementation that considers school-level contextual effects on adoption and sustainability of new programs in schools. The Metropolitan Area Child Study Research Group is a collaboration of (in alphabetical order) Leonard Eron, University of Michigan; Nancy Guerra, University of California, Riverside; David Henry, University of Illinois at Chicago; L. Rowell Huesmann, University of Michigan; Patrick Tolan, University of Illinois at Chicago; and Richard VanAcker, University of Illinois at Chicago.     

Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation

The first purpose of this review was to assess the impact of implementation on program outcomes, and the second purpose was to identify factors affecting the implementation process. Results from over quantitative 500 studies offered strong empirical support to the conclusion that the level of implementation affects the outcomes obtained in promotion and prevention programs. Findings from 81 additional reports indicate there are at least 23 contextual factors that influence implementation. The implementation process is affected by variables related to communities, providers and innovations, and aspects of the prevention delivery system (i.e., organizational functioning) and the prevention support system (i.e., training and technical assistance). The collection of implementation data is an essential feature of program evaluations, and more information is needed on which and how various factors influence implementation in different community settings.     

Sustained Implementation of Evidence‐based Programs in Disadvantaged Communities: A Conceptual Framework of Supporting Factors

This paper presents a review of the empirical literature for studies evaluating factors that facilitate and create barriers to sustained program implementation in disadvantaged communities. It outlines study methodology and sustainment outcomes and proposes a conceptual model that involves implementation sustainment support for providers delivering evidence‐based health and family services in disadvantaged communities. Sustained program implementation in the community setting is a significant issue as only 43% of studies reported successfully sustained programs. The review identified 18 factors that facilitate success and create barriers to program sustainment. The factors are synthesized into three themes; program characteristics, workplace capacity, and process and interaction factors. The majority of factors map onto commonly cited sustainability influences in implementation science. However, there was an additional focus for studies included in this review on the importance of factors such as program burden, program familiarity and perceived competence in program skills, workplace support for the program, staff mobility and turnover, supervision and peer support, and ongoing technical assistance. The need to use a conceptual framework and develop measures to guide and evaluate capacity building in EBP implementation and sustainment in low‐resource community settings is highlighted.     

The KiVa antibullying curriculum and outcome: Does fidelity matter?

Research on school-based prevention suggests that the success of prevention programs depends on whether they are implemented as intended. In antibullying program evaluations, however, limited attention has been paid to implementation fidelity. The present study fills in this gap by examining the link between the implementation of the KiVa antibullying program and outcome. With a large sample of 7413 students (7–12 years) from 417 classrooms within 76 elementary schools, we tested whether the degree of implementation of the student lessons in the KiVa curriculum was related to the effectiveness of the program in reducing bullying problems in classrooms. Results of multilevel structural equation modeling revealed that after nine months of implementation, lesson adherence as well as lesson preparation time (but not duration of lessons) were associated with reductions in victimization at the classroom level. No statistically significant effects, however, were found for classroom-level bullying. The different outcomes for victimization and bullying as well as the importance of documenting program fidelity are discussed.     

What We Know—and Don't Know—About Preventing Child Maltreatment

Abstract

Although child maltreatment is routinely acknowledged as a serious social problem, it remains widespread, raising questions about how prevention efforts can be improved. Following a review of theory and research relevant to prevention programming, the two dominant models of child abuse prevention-child empowerment and parent education-are examined, emphasizing both those factors that appear to contribute to program success (e.g., early intervention, sufficient time commitments, cultural sensitivity) and weaknesses in current efforts (e.g., poor implementation). Although underutilized, broader social and system-level reforms, including social policy and media initiatives, hold promise for prevention. Ways in which to leverage these tools are explored. Additional research is urged to explore assumptions on which programs are based, as well as to conduct more rigorous evaluation of current programs. The article concludes that a sustained commitment to prevention efforts aimed at child abuse and neglect is required.     

The Role of Lay Health Advisors in Cardiovascular Risk Reduction: A Review

Interventions are needed to reduce the negative impact of cardiovascular disease. The combination of health risks for disease, disability, and mortality, particularly among underserved populations, might be best addressed with programs designed to enhance awareness and development of resources within a context of community support. The objectives of this review were to: (1) provide a comprehensive review and evaluation of the roles, evaluation, and effectiveness of LHA in community-based programs with an emphasis on cardiovascular risk reduction; and (2) provide recommendations for future research involving LHA in such programs. Computer and manual searches were conducted of articles in the English-language literature from 1980 to 2007. Twenty articles were evaluated, which emphasized the role of the LHA in cardiovascular risk reduction. A review of research literature provides a starting point for determining salient approaches for intervention and evaluation, issues related to program implementation and sustainability, and strengths and limitations of existing approaches.     

Enhancing school-based prevention and youth development through coordinated social,emotional, and academic learning

A comprehensive mission for schools is to educate students to be knowledgeable, responsible, socially skilled, healthy, caring, and contributing citizens. This mission is supported by the growing number of school-based prevention and youth development programs. Yet, the current impact of these programs is limited because of insufficient coordination with other components of school operations and inattention to implementation and evaluation factors necessary for strong program impact and sustainability. Widespread implementation of beneficial prevention programming requires further development of research-based, comprehensive school reform models that improve social, health, and academic outcomes; educational policies that demand accountability for fostering children's full development; professional development that prepares and supports educators to implement programs effectively; and systematic monitoring and evaluation to guide school improvement.     

Effectiveness of two smoking prevention programs according to type of applicator

Although research on the prevention of drug use has shown that the effectiveness of programs depends partly on who applies them, the results of studies designed to clarify this issue are contradictory. This study compared the effects on smoking, in the short and medium term, of the implementation of two school-based prevention programs, depending on the type of applicator. The sample was made up of 200 seventh-grade students distributed among five experimental conditions: four of intervention, in which the programs were implemented by teachers from the school or by external psychologists, and one of control. The results show that teachers applying the Barbacana program and psychologists applying the Construyendo Salud (Building Health) program succeeded in reducing the incidence of cigarette smoking (BR-teachers 15%; C.S-psychologists 5%) and strengthening attitudes against smoking. An interaction effect between type of program and type of applicators was clearly observed.     

Training Community Members to Serve as Paraprofessionals in an Evidence-Based,Prevention Program for Parents of Preschoolers

Widespread dissemination of evidence-based programs for underserved populations may require non-traditional means of service provision. Collaboration with paraprofessionals from communities that are targeted for intervention holds promise as a delivery strategy that may make programs more accessible and acceptable, especially to parents living in low-income, urban neighborhoods. We describe a paraprofessional training program for individuals living in a community targeted for preventive intervention based on high levels of poverty and community violence. The design and implementation of the training program are described in the context of issues related to the use of paraprofessionals in community-based, preventive interventions with parents of young children. We also provide insight into lessons learned from a feasibility study as well as general guidelines for the development of paraprofessional training programs for delivery of evidence-based programs.     

Primary Prevention Mental Health Programs for Children and Adolescents: A Meta-Analytic Review

Used meta-analysis to review 177 primary prevention programs designed to prevent behavioral and social problems in children and adolescents. Findings provide empirical support for further research and practice in primary prevention. Most categories of programs produced outcomes similar to or higher in magnitude than those obtained by many other established preventive and treatment interventions in the social sciences and medicine. Programs modifying the school environment, individually focused mental health promotion efforts, and attempts to help children negotiate stressful transitions yield significant mean effects ranging from 0.24 to 0.93. In practical terms, the average participant in a primary prevention program surpasses the performance of between 59% to 82% of those in a control group, and outcomes reflect an 8% to 46% difference in success rates favoring prevention groups. Most categories of programs had the dual benefit of significantly reducing problems and significantly increasing competencies. Priorities for future research include clearer specification of intervention procedures and program goals, assessment of program implementation, more follow-up studies, and determining how characteristics of the intervention and participants relate to different outcomes.     

School-Based Mental Health Services: A Research Review

This paper provides a synthetic review of research on school-based mental health services. Schools play an increasingly important role in providing mental health services to children, yet most school-based programs being provided have no evidence to support their impact. A computerized search of references published between 1985 and 1999 was used to identify studies of school-based mental health services for children. Study inclusion was determined by (i) use of randomized, quasi-experimental, or multiple baseline research design; (ii) inclusion of a control group; (iii) use of standardized outcome measures; and (iv) baseline and postintervention outcome assessment. The application of these criteria yielded a final sample of 47 studies on which this review is based. Results suggest that there are a strong group of school-based mental health programs that have evidence of impact across a range of emotional and behavioral problems. However, there were no programs that specifically targeted particular clinical syndromes. Important features of the implementation process that increase the probability of service sustainability and maintenance were identified. These include (i) consistent program implementation; (ii) inclusion of parents, teachers, or peers; (iii) use of multiple modalities; (iv) integration of program content into general classroom curriculum; and (v) developmentally appropriate program components. Implications of these findings and directions for future research are discussed.     

Enhancing Social Responsibility and Prosocial Leadership to Prevent Aggression,Peer Victimization,and Emotional Problems in Elementary School Children

Testing the theories that form the basis of prevention programs can enhance our understanding of behavioral change and inform the development, coordination, and adaptation of prevention programs. However, theories of change showing the linkages from intervention program components to risk or protective factors to desired outcomes across time are rarely specified or tested. In this 2‐year longitudinal study, we test the theory that increases in two protective factors (i.e., children's prosocial leadership and their teachers’ expectations of social responsibility) targeted by the WITS Programs (Walk Away, Ignore, Talk it Out, and Seek Help) would be associated with declines in peer victimization, aggression, and emotional problems. Participants included Canadian students, in grades 1–4 at baseline (n = 1329) and their parents and teachers. Consistent with our theory of change, variability in program implementation (adherence and integration) and in children's use of program skills (child responsiveness) are related to increases in both protective factors. Increases in these protective factors are associated with subsequent declines in children's aggression, victimization, and emotional problems. We discuss how enhancement of these protective factors may operate to improve child outcomes and the need for theory‐based research to refine and improve the effectiveness of intervention strategies and to improve program scale‐up.     

Training of interventionists and cultural adaptation procedures: A systematic review of culturally adapted evidence-based parenting programs in Africa

There has been an increase in the implementation of evidence-based parenting programs from high-income countries to several African countries. In this review, we systematically evaluated intervention studies of culturally adapted parenting programs in nine African countries with the objective of examining the quality of training for interventionists and cultural adaptation procedures. A total of 18 studies, obtained from an electronic search of 6 databases, met the inclusion criteria and were evaluated following PRISMA guidelines. The Ecological Validity Model was adopted to organize data on cultural adaptation procedures. Sixteen of the 18 studies reported information regarding the clinical training of interventionists and the cultural adaptations undertaken. Live and interactive workshops were the most common format used to train interventionists in the focal intervention. Overall, cultural adaptations in most studies included translation of intervention protocols into the local language. However, studies varied in the way cultural adaptation procedures were reported with some studies failing to report on cultural adaptation procedures. Concurring with previous literature, attending to issues of culture, power, privilege, access, sustainability, and other relevant concepts to increase the cultural relevance is highly encouraged in parent intervention studies in Africa. This review provides a baseline upon which future training and adaptation procedures can be built.     

青少年吸毒行为预防干预研究述评及展望

青少年吸毒问题已令人触目惊心,应引起研究者高度的关注。该文对国外18岁以下青少年吸毒预防干预的方法、模式和影响干预效果有效性的重要要素进行了述评,介绍了一些国外著名的青少年吸毒预防干预方案,在此基础上结合当前最新的研究进展对该领域未来的发展方向和趋势进行了展望。     

Predictors of champion behaviors in an evidence-based parenting program: A structural equation modeling approach

The importance of champions in the implementation of evidence-based public health programs has been well established. Champions perform a range of behaviors which can have significant influence on the adoption and sustainability of interventions. This study investigates champion behaviors in parenting practitioners with the aim to provide insight into the range and extent that practitioners engage in champion behaviors and to examine predictors of practitioners' champion behaviors using structural equation modeling. Data were collected during a large international implementation survey of 1606 practitioners trained to deliver an evidence-based parenting program, the triple P—Positive Parenting Program. We developed a 13-item Champion Behaviors Scale which was administered alongside other measures of predictors of implementation outcomes. Perceived usefulness of the program was the most important facilitator of both public and personal champion behaviors. Certain desirable features of the program tended to not only be positively associated with the perceived usefulness of the program, but also had a unique impact on practitioners' personal champion behaviors. Higher positions within organizations were linked with more public champion behaviors. Although organizational support was found to be unrelated to champion behaviors in the structural model, it facilitated other predictors of champion behaviors.     

Modeling the Bullying Prevention Program Preferences of Educators: A Discrete Choice Conjoint Experiment

We used discrete choice conjoint analysis to model the bullying prevention program preferences of educators. Using themes from computerized decision support lab focus groups (n = 45 educators), we composed 20 three-level bullying prevention program design attributes. Each of 1,176 educators completed 25 choice tasks presenting experimentally varied combinations of the study’s attribute levels. Latent class analysis yielded three segments with different preferences. Decision Sensitive educators (31%) preferred that individual schools select bullying prevention programs. In contrast, Support Sensitive educators (51%) preferred that local school boards chose bullying prevention programs. This segment preferred more logistical and social support at every stage of the adoption, training, implementation, and long term maintenance processes. Cost Sensitive educators (16%) showed a stronger preference for programs minimizing costs, training, and implementation time demands. They felt prevention programs were less effective and that the time and space in the curriculum for bullying prevention was less adequate. They were less likely to believe that bullying prevention was their responsibility and more likely to agree that prevention was the responsibility of parents. All segments preferred programs supported by the anecdotal reports of colleagues from other schools rather than those based on scientific evidence. To ensure that the bullying prevention options available reflect the complex combination of attributes influencing real world adoption decisions, program developers need to accommodate the differing views of the Decision, Support, and Cost Sensitive segments while maximizing the support of parents and students.