Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation

The first purpose of this review was to assess the impact of implementation on program outcomes, and the second purpose was to identify factors affecting the implementation process. Results from over quantitative 500 studies offered strong empirical support to the conclusion that the level of implementation affects the outcomes obtained in promotion and prevention programs. Findings from 81 additional reports indicate there are at least 23 contextual factors that influence implementation. The implementation process is affected by variables related to communities, providers and innovations, and aspects of the prevention delivery system (i.e., organizational functioning) and the prevention support system (i.e., training and technical assistance). The collection of implementation data is an essential feature of program evaluations, and more information is needed on which and how various factors influence implementation in different community settings.     

Developing Measures of Community-Relevant Outcomes for Violence Prevention Programs: A Community-Based Participatory Research Approach to Measurement

Community-Based Participatory Research is a research paradigm that encourages community participation in designing and implementing evaluation research, though the actual outcome measures usually reflect the “external” academic researchers’ view of program effect and the policy-makers’ needs for decision-making. This paper describes a replicable process by which existing standardized psychometric scales commonly used in youth-related intervention programs were modified to measure indicators of program success defined by community partners. This study utilizes a secondary analysis of data gathered in the context of a community-based youth violence prevention program. Data were retooled into new measures developed using items from the Alabama Parenting Questionnaire, the Hare Area Specific Self-Esteem Scale, and the Youth Asset Survey. These measures evaluated two community-defined outcome indicators, “More Parental Involvement” and “Showing Kids Love.” Results showed that existing scale items can be re-organized to create measures of community-defined outcomes that are psychometrically reliable and valid. Results also show that the community definitions of parent or parenting caregivers exemplified by the two indicators are similar to how these constructs have been defined in previous research, but they are not synonymous. There are nuanced differences that are important and worthy of better understanding, in part through better measurement.     

Inside the Black Box: Assessing and Improving Quality in Youth Programs

Over the past decade, structured programming for children and youth during the non-school hours has expanded exponentially. A confluence of recent research studies and program evaluations backs the publicly perceived notion that after-school programs can positively influence important developmental and learning outcomes. The rapid expansion of the field and the potential of programs to contribute to child and youth development have made defining what high quality programs look like and learning how to improve program quality key challenges facing the field. This paper describes what is known about the relation between youth program quality and youth developmental outcomes, summarizes different quality assessment tools being used in the field, and discusses how such tools are being used to drive systemic quality improvement efforts.     

Dilemmas of Practice: Challenges to Program Quality Encountered by Youth Program Leaders

To create and sustain high quality youth development programs it is important to understand the challenging situations and dilemmas that emerge in program leaders’ daily work with youth. In this research the experiences of leaders in 12 programs were followed over a 2–9 month period, which led to the identification of 250 dilemma situations. Qualitative analyses identified 5 categories and 12 subcategories of dilemmas that reflected distinct types of considerations (e.g., youth’s personalities, relationships with the community). The analyses also found that the experienced leaders in the study typically responded to these dilemmas in ways that were youth-centered and that balanced multiple considerations. It is argued that researchers need to go beyond identifying features of high quality programs, and more fully examine how effective leaders create and sustain high quality in response to the challenging situations of practice.     

Youth Risk and Mentoring Relationship Quality: The Moderating Effect of Program Experiences

Because program experiences are more amendable than mentor or mentee characteristics, they may be important factors to consider in buffering the negative impact of youth risk on the quality of the mentor–mentee bond. Data from 455 mentees (ages 11–18; 57% male) and their undergraduate student mentors (82.3% female) from the Campus Connections mentoring program were used to assess whether youth risk and mentors’ program experiences (i.e., program structure, supportive relationships with staff, opportunities for skill building, support for efficacy and mattering, and opportunities to belong) were associated with mentoring relationship quality and whether mentors’ experience within the program moderated the association between youth risk and mentoring relationship quality. Results indicated that environmental, but not individual, risk was negatively associated with relationship quality. Mentors’ experiences with the program were positively associated with mentoring relationship quality, and in many cases, above and beyond youth level of risk. Finally, mentors’ perception of program structure, supportive relationships, and opportunities for skill building attenuated the negative relationship between environmental, but not individual, risk and relationship quality. Mentors’ experiences of program support for efficacy and mattering and opportunities to belong were not significant moderators in any model. Implications for programs and future research directions are presented.     

Mobilizing Communities and Building Capacity for Youth Violence Prevention: The National Academic Centers of Excellence for Youth Violence Prevention

Violence, including its occurrence among youth, results in considerable physical, emotional, social, and economic consequences in the US. Youth violence prevention work at the Division of Violence Prevention (DVP) at the Centers for Disease Control and Prevention (CDC) emphasizes preventing youth violence-related behaviors, injuries, and deaths by collaborating with academic and community partners and stakeholders. In 2000 and 2005, DVP funded the National Academic Centers of Excellence (ACE) for Youth Violence Prevention. Most ACE Centers focus on building community capacity and competence so that evidence-based programs for youth violence prevention can be successfully implemented through effective and supportive research-community partnerships. This commentary provides historical information about the ACE Program, including the development, goals, accomplishments of the Centers, and the utilization of a community-based participatory research approach to prevent youth violence.     

Outcomes and Findings of Program Evaluation for the Intensive Mental Health Program

We describe changes in adaptive functioning for children enrolled in the Intensive Mental Health Program (IMHP), an innovative model for serving children with the most severe forms of SED. Most children show marked improvements in multiple indicators of role performance and symptomatology. Although a high risk, high need group, most children attended school regularly and lived with a stable caregiver at discharge.     

Demonstration of an Integrated Treatment for Smoking Cessation and Anxiety Symptoms in People With HIV: A Clinical Case Study

Despite high rates of co-occurring tobacco use and anxiety symptoms and disorders among persons with HIV, evidence-based interventions for these individuals are not yet available. The present study sought to evaluate an integrated treatment model addressing smoking cessation and anxiety sypmtoms among HIV-positive smokers. Treatment was an 8-week intervention integrating a standard smoking cessation protocol (i.e., cognitive-behavioral therapy [CBT], nicotine replacement therapy) with CBT for anxiety. Inclusion criteria were 18–65 years of age, ≥ 10 cigarettes/day, State–Trait Anxiety Inventory [STAI-T] score of > 39, and moderate motivation (i.e., ≥ 5 out of 10 on a 10-point Likert scale) to quit smoking. Primary outcomes included scores on the Anxiety Sensitivity Index (ASI) and cigarettes smoked per day. Self-reported abstinence was biochemically verified by carbon monoxide breath analysis. Three male participants (mean age 49.3, SD = 9.1) completed through 2-month follow-up. At baseline all participants reported smoking an average of 20 cigarettes per day. Two participants quit smoking and maintained abstinence by the 2-month follow-up, and demonstrated a reduction in ASI scores. Participant 3 continued to smoke but at a reduced rate. Participants’ response to cognitive and behavioral strategies (e.g., creating balanced thoughts, interoceptive exposures) will be discussed. Clinical lessons learned include use of a flexible approach to cognitive restructuring, use of imaginal and in vivo exposures in session to better prepare patients for homework practice, and flexibility in delivering the treatment in an individual or group format. This clinical presentation provides preliminary support for the feasibility and initial effectiveness of an integrated treatment to reduce anxiety symptoms and aid in smoking cessation in anxious, HIV-positive smokers.     

The Program Environment Scale: Assessing Client Perceptions of Community-Based Programs for the Severely Mentally Ill

The Program Environment Scale (PES) was developed for use with clients of community-based programs for the severely mentally ill. It is intended to fill the gap in available tools for assessing clients' perceptions of program functioning as it affects their quality of life in a program. Formal pretests were conducted with 121 clients at 12 randomly selected programs near Washington, DC. The final field test used a revised form (29 domains; 129 items) with 221 clients in 22 programs selected randomly throughout the U.S., including Clubhouse, day treatment, psychosocial rehabilitation, and social club programs. Twenty-three subscales met at least five of eight psychometric criteria for internal consistency and discriminant validity. A 24th subscale was retained because of its substantive importance. Successful subscales cover program atmosphere and interactions (program cares about me, energy level, friendliness, openness, staff-client and client-client respect, reasonable rules, availability of positive physical contact, protection from bad touch, staff investment in their jobs, and confidentiality), client empowerment/staff-client equality (program and treatment empowerment, egalitarian space use), and service components (support for paid work, work importance, emergency access, family activities, housing, public benefits, community activities, medications, substance abuse, and continuity). Subscale validity is indicated by associations of specific service offerings with scores on scales measuring client perceptions of those services, and by an ability to differentiate among program models (i.e., Clubhouses, day treatment programs, and psychosocial rehabilitation programs look different from each other). Subscale scores were not influenced by client characteristics (gender, race, age, diagnosis, number of hospitalizations, length of time in program). The final scale has 97 items and takes about 25 minutes to complete. The PES succeeds in measuring different aspects of programs as clients perceive them. In the programs we visited, directors felt the PES covers the important things they want to know about how clients perceive their program. The PES should become a useful tool both for researchers interested in how client responses to programs may influence their therapeutic outcomes, and for practitioners interested in improving their clients' program experiences and/or increasing convergence of staff and client views of their program.     

Building Opportunity: Developing City Systems to Expand and Improve After School Programs

A growing body of research is examining the strengths and weaknesses of specific after school programs and their effect on youth outcomes. Few reviews, however, have sought to examine the components of citywide system-building—to understand intentional efforts to develop, support and sustain high quality after school programming across a community. Beginning in the mid-1990s and continuing through the present, private funders, public officials and program practitioners in cities across America have joined together to build systems to support the expansion and improvement of after school programs at the city-level. This paper presents the community context and underlying principles that drove the development of Baltimore’s After-School Strategy; articulates a set of system components derived from this experience and the available literature; and lays out future work to expand high quality after-school opportunities for youth in Baltimore and in other distressed urban environments.     

The Youth Public Arts Program: Interpersonal and Intrapersonal Outcomes for At-Risk Youth

This article provides the findings of a program evaluation of the Youth Public Arts Program, a long-standing community-based art program that serves at-risk youth in eastern North Carolina. The program evaluation was undertaken to examine the art skills, social and interpersonal skills, and community involvement of the participants. The study included four focus groups and telephone surveys with a total of 38 stakeholders (program participants, parents of participants, program staff, and referral sources). Results indicated that the at-risk youth had gains in both interpersonal and intrapersonal skills, including family relationships, peer relationships, community interaction, increased art skills, confidence, and self-regulation. This article adds to the very limited literature on community-based art programs for youth.     

Stages of Change in Smoking Cessation: A Comparison of Expectancies Among Precontemplators and Contemplators

Previous research has shown that smokers progress through a series of stages of change as they attempt to quit their habit. This study evaluated smokers in the precontemplation and contemplation stages in order to explore how we can effectively facilitate positive stage transition. Precontemplators (n = 71) and contemplators (n = 30) were compared in terms of two types of expectancies: self-efficacy beliefs and response outcome expectancies. Stage of change was defined according to both categorical and continuous dimensions, each of which led to different results. The findings are discussed in terms of their potential applications to smoking cessation programs and their implications regarding the stage model of change.     

Challenges in Disseminating Model Programs: A Qualitative Analysis of the Strengthening Washington DC Families Program

This article discusses the challenges faced when a popular model program, the Strengthening Families Program, which in the past has been implemented on a smaller scale in single organizations, moves to a larger, multiorganization endeavor. On the basis of 42 interviews conducted with program staff, the results highlight two main themes that address program and organizational characteristics. The themes relate to the cultural relevance of the program, the accessibility of program partners, the organization of program services, and the coordination of program activities. The article also offers a discussion of how to better prepare sites for program delivery. This discussion focuses explicitly on the need for good communication among program partners, extensive community outreach, and the ability to secure additional resources.     

Youth belonging and cognitive engagement in organized activities: A large-scale field study

Numerous studies of organized activities have found that participation is associated with a range of positive outcomes; however, findings from recent randomized trials have been more mixed. Understanding youth's psychological experiences of program involvement – their cognitive and emotional reaction to and participation in activities – may be key to understanding the influence of organized activities. Hierarchical linear modeling was used to investigate correlates of youth belonging and cognitive engagement in a sample of 1160 youth in 123 program offerings in 66 sites. Results revealed that intensity (frequency) of exposure positively predicted belonging and cognitive engagement; however, duration was negatively associated with cognitive engagement. The staff practice of providing a welcoming atmosphere predicted belonging; whereas provision of active skill-building predicted cognitive engagement. These relations were found to vary across content type.     

After-School Youth Development Programs: A Developmental-Ecological Model of Current Research

Although there has been a rapid increase in funding and attention to after-school programs, there is little understanding of how after-school programs impact children's developmental trajectories. The heterogeneity of American children makes it very unlikely that all children need after-school programming or that there is but one brand of after-school programming suitable for all youth. We discuss the numerous developmental and contextual factors that may influence which children benefit most from after-school programs as well as the nature of the after-school programs most beneficial to children's needs. The value of utilizing an ecological and developmental perspective to after-school program evaluation are presented, including the need for improved research designs and more detailed analyses of program type and services as well as a more complete determination of which children benefit the most from after-school participation.     

Process Evaluation of A Physical Activity Self-Management Program For Children: Spark

Abstract

SPARK is a controlled field study of a multi-component elementary school program to promote physical activity. One component is a weekly classroom-based self-management program, with the goal of increasing physical activity outside of school. The curriculum included goal setting, behavioral skills training, a reward system, and parent involvement. An evaluation of curriculum implementation and association between process and outcome was conducted using direct observation of lessons, subjective ratings by 24 teachers and 391 parents, and participation records of 549 students. Teachers viewed the self-management curriculum less positively than the physical education curriculum. Teachers were observed implementing an average of 65% of curriculum elements, which may have contributed to the limited effects of the self-management program. Student participation in the program, measured by points earned for doing out of school physical activity, was correlated significantly with changes in body mass index and multiple psychosocial variables among boys but not girls. Barriers to full program implementation were identified, and a remaining challenge is to improve self-management curriculum implementation by teachers.     

A Typology of Youth Participation and Empowerment for Child and Adolescent Health Promotion

Research suggests that increasing egalitarian relations between young people and adults is optimal for healthy development; however, the empirical assessment of shared control in youth–adult partnerships is emerging, and the field still requires careful observation, identification, categorization and labeling. Thus, our objective is to offer a conceptual typology that identifies degrees of youth–adult participation while considering the development potential within each type. We use an empowerment framework, rooted in evidence-based findings, to identify five types of youth participation: (1) Vessel, (2) Symbolic, (3) Pluralistic, (4) Independent and (5) Autonomous. The typology is constructed as a heuristic device to provide researchers, practitioners and policy-makers with a common language for articulating degrees of youth participation for optimal child and adolescent health promotion.     

Challenges in Supporting Community Implementation of Science-based Programs: A Critical Review of Local Partnerships for Success Plans

This paper provides the opportunity to consider local challenges to implementing science-based programs. Wandersman et al. (American Journal of Community Psychology 2008) define three systems necessary to effectively implement science-based programming. These systems include the Prevention Support System, which is responsible for assisting communities in implementing effective practices; the Prevention Research System, which is responsible for developing and packaging prevention programming; and the Prevention Delivery System, which is responsible for delivering services. The authors conduct a content analysis related to six county level efforts to implement science-based programs in Ohio. These counties are part of a larger Ohio initiative referred to as Partnerships for Success. The authors suggest that local officials are quite skilled at accessing the Prevention Research System and providers at the local level represent an effective service delivery system. The authors contend that the Prevention Support System in these counties is capable of enhancing several functions related to local infrastructure but is deficient in other important elements.     

Care Like Kin: Community Based Youth Serving Organizations and the Social Reproduction of Disconnected Youth in Rural America

Increasing economic insecurity faced by older youth in rural America presents a crisis of social reproduction for disconnected youth in these areas. Increasingly community based youth serving organizations (CBYSOs) are recognizing and responding to the social reproduction needs of this particularly vulnerable youth population. Such responses are often hidden from funders, government agencies, and community residents. Yet these institutions play an important substitution function for disconnected youth and provide critical social support and social leverage for this population. Based on case studies of three CBYSOs in the San Joaquin Valley of California, this article explores how and why CBYSOs play a substitution function for disconnected youth in rural communities. It is the argument of this article that the social reproduction work of CBYSOs is undertaken with a ethic of care that may have the capacity to transform the political, social, and economic contexts that face this marginalized youth population.