Using the Performance Diagnostic Checklist to evaluate and promote paraeducators' treatment fidelity

Paraeducators, who increasingly serve as implementers for students in special education, can struggle with intervention fidelity. Special educators, often responsible for providing paraeducators with support, receive limited supervision training. From organizational behavior management, the Performance Diagnostic Checklist-Human Services (PDC-HS) is a checklist to detect reasons for poor workplace performance and identify aligned support. Utilizing a multiple baseline design, this study evaluated the impact of PDC-HS indicated implementation support on the intervention fidelity of five paraeducators implementing six behavior intervention plans for students receiving special education services. The PDC-HS indicated that all paraeducators would benefit from behavior skills training and visuals of the plans, in addition to other specific supports for individual paraeducators (e.g., prompts, feedback). These implementation supports were effective in improving intervention fidelity for four paraeducators, whereas one paraeducator (across two students) did not respond to these supports. Changes in student outcomes (i.e., academic engagement, disruptive behavior) were inconsistent, suggesting that the behavior intervention plans may have needed to be revised. Usability data suggested that the PDC-HS and indicated supports were generally acceptable, but revisions may be warranted. The implications of this study for school-based implementation support research are discussed.     

Training and Supervising Lay Providers in Kenya: Strategies and Mixed-Methods Outcomes☆☆☆

ObjectiveTraining lay providers to deliver mental health interventions is both effective and cost-effective. However, more research is needed to document training and supervision procedures and to collect lay providers’ feedback.MethodsThis study documents training and supervision from a randomized controlled trial of the Shamiri intervention, a four-session, school-based intervention that significantly reduced symptoms of anxiety and depression in Kenyan adolescents. We delivered a 10-hour training to 13 lay providers, M (SD)_age = 21.00 (1.95), %_female = 61.54. We also hosted 30-minute supervision meetings twice weekly. Training and supervision were delivered primarily by undergraduates. Independent raters coded session recordings for fidelity and quality of services, and we collected quantitative and qualitative feedback from lay providers.ResultsReliability and mean ratings for all six of our fidelity and quality measures (delivering required content, adhering to specified details, thoroughness, skillfulness, clarity, and purity) were very good to excellent. Lay provider quantitative ratings of training were also overwhelmingly positive, with an overall satisfaction rating of 6.46/7.00. We identified central qualitative themes in lay provider comments: Generally, comments about training style, content, and personal interactions were overwhelmingly positive, and many lay providers reported personal growth. Comments about timing and location of training were mixed.ConclusionsThis study provides preliminary evidence that a very brief training delivered primarily by undergraduates can teach high school-graduate lay providers to deliver effective mental health interventions. Additionally, we discuss lessons learned and implications for future research, including the importance of considering local context when planning and of continuously collecting and addressing lay provider feedback.     

The Effectiveness of Cultural Adjustment and Trauma Services (CATS): Generating Practice-Based Evidence on a Comprehensive, School-Based Mental Health Intervention for Immigrant Youth

A collaborative study of Cultural Adjustment and Trauma Services (CATS), a comprehensive, school-based mental health program for traumatized immigrant children and adolescents, was conducted to generate practice-based evidence on the service delivery model across two school districts. Program effectiveness was assessed by testing whether client functioning and PTSD symptoms improved as a result of 7 separate service elements. An array of clinical services including CBT, supportive therapy, and coordinating services were provided to all students, and an evidence-based intervention for trauma, TF-CBT, was implemented with a subset of students. Greater quantities of CBT and supportive therapy increased functioning, while greater quantities of coordinating services decreased symptoms of PTSD. TF-CBT services were associated with both improved functioning and PTSD symptoms, although TF-CBT was implemented with fidelity to the overall comprehensive service model rather than the structured intervention model. Results suggest the comprehensive school-based model was effective, though different service components affected different student outcomes. Implications of these findings for immigrant mental health interventions and implementing structured evidence-based practices into community mental health programs are discussed. Suggestions are made for future research on existing mental health practices with immigrants.     

Treatment integrity of school-based interventions with children in the journal of applied behavior analysis 1991-2005

We reviewed all school-based experimental studies with individuals 0 to 18 years published in the Journal of Applied Behavior Analysis (JABA) between 1991 and 2005. A total of 142 articles (152 studies) that met review criteria were included. Nearly all (95%) of these experiments provided an operational definition of the independent variable, but only 30% of the studies provided treatment integrity data. Nearly half of studies (45%) were judged to be at high risk for treatment inaccuracies. Treatment integrity data were more likely to be included in studies that used teachers, multiple treatment agents, or both. Although there was a substantial increase in reporting operational definitions of independent variables, results suggest that there was only a modest improvement in reported integrity over the past 30 years of JABA studies. Recommendations for research and practice are discussed.     

Increasing the Rigor of Procedural Fidelity Assessment: An Empirical Comparison of Direct Observation and Permanent Product Review Methods

Although it is widely accepted that procedural fidelity data are important for making valid decisions about intervention effectiveness, there is little empirical guidance for researchers and practitioners regarding how to assess procedural fidelity. A first step in moving procedural fidelity assessment research forward is to develop a comprehensive understanding of the relation between currently recommended assessment methods, systematic direct observation (SDO) and permanent product review (PPR). To this end, we first discuss methodological issues related to procedural fidelity assessment via SDO and PPR, and how they are evidenced in the behavioral intervention outcome literature. Second, we present a single-case design study in which teachers’ implementation of behavior support plans was assessed via SDO and PPR. Third, we analyze the SDO and PPR data with regard to (a) relation between the methods; (b) methodological issues including definitions of procedural fidelity and intervention steps, representativeness of intervention steps, and collection of an adequate sample; and (c) association with student outcomes. Finally, we discuss implications for research and practice.     

Informant discrepancies in assessments of psychosocial functioning in school-based services and research: Review and directions for future research

Psychosocial functioning plays a key role in students' wellbeing and performance inside and outside of school. As such, techniques designed to measure and improve psychosocial functioning factor prominently in school-based service delivery and research. Given that the different contexts (e.g., school, home, community) in which students exist vary in the degree to which they influence psychosocial functioning, educators and researchers often rely on multiple informants to characterize intervention targets, monitor intervention progress, and inform the selection of evidence-based services. These informants include teachers, students, and parents. Across research teams, domains, and measurement methodologies, researchers commonly observe discrepancies among informants' reports. We review theory and research—occurring largely outside of school-based service delivery and research—that demonstrates how patterns of informant discrepancies represent meaningful differences that can inform our understanding of psychosocial functioning. In turn, we advance a research agenda to improve use and interpretation of informant discrepancies in school-based services and research.     

Evidence for a Comprehensive Sexuality Education Intervention that Enhances Chinese Adolescents’ Sexual Knowledge and Gender Awareness and Empowers Young Women

While Chinese adolescents become more sexually active, their sexual attitudes are profoundly influenced by traditional gender and sexual norms. The tensions between the rapid transition in youth sexuality and lags in sex education call for a paradigm shift of sexuality education in China. The present study examined the effectiveness of a school-based empowerment comprehensive sexuality education (CSE) intervention in changing adolescents’ sexual knowledge, gender and sexual attitudes, and sexual self-efficacy. We used data from a quasi-experimental intervention study with self-administered pretest and posttest surveys conducted among 1131 10th grade high school students in Lanzhou, northwest China. Whereas 583 participants in two intervention schools attended 12 45-min CSE sessions focusing on gender, rights, and critical thinking skills across 4–7 months, 548 participants in two comparison schools received no or limited conventional sex education. After controlling for differences in sociodemographics and in the influence of family, peers, and internet use between the two groups, female and male adolescents exhibited significantly more accurate sexual knowledge, stronger endorsement of nontraditional gender roles, and greater rejection of sexual double standards immediately after the intervention. Additionally, young women reported significantly higher sexual self-efficacy following the intervention. The findings suggest that the school-based empowerment CSE intervention is helpful in creating an environment in which both young men and young women benefit in sexual knowledge and gender awareness and young women increase their sexual decision-making skills, thereby lending support to the theory and practice of an empowerment CSE approach.

     

Evaluation of a Targeted Intervention Within a Schoolwide System of Behavior Support

A recommended component of comprehensive behavior support in schools is a targeted intervention designed to efficiently and effectively meet the support needs of students who are not responding to preventive schoolwide intervention strategies (Sugai, Horner et al., 2000; Walker et al., 1996). This study provides a fine-grained analysis using a multiple baseline across students design with the targeted intervention as the independent variable, and both classroom problem behavior and academic engagement as dependent variables. The multiple baseline analysis documents a modest functional relationship between implementation of the intervention and reduction in problem behavior. Implications for the design of behavior support in schools and the development of future research are provided.     

School-Based Substance Abuse Prevention Programs

Substance abuse has been identified as a significant problem among adolescents. Personal and social coping skills training programs implemented in school settings have shown evidence of effectiveness as a primary prevention approach. Early intervention programs for students most at risk are also needed, however, few have been adequately evaluated. Special services providers possess training and skills that are well suited for planning and implementing school-based programs that address the substance abuse issue.     

An Evaluation of a Classwide Intervention Package Involving Self-Management and a Group Contingency on Classroom Behavior of Middle School Students

The effectiveness of an intervention package involving self-management and a group contingency at increasing appropriate classroom behaviors was evaluated in a sample of middle school students. Participants included all students in each of the 3 eighth-grade general education classrooms and their teachers. The intervention package included strategies recommended as part of best practice in classroom management to involve both building skill (self-management) and reinforcing appropriate behavior (group contingency). Data sources involved assessment of targeted behaviors using Direct Behavior Rating—single item scales completed by students and systematic direct observations completed by external observers. Outcomes suggested that, on average, student behavior moderately improved during intervention as compared to baseline when examining observational data for off-task behavior. Results for Direct Behavior Rating data were not as pronounced across all targets and classrooms in suggesting improvement for students. Limitations and future directions, along with implications for school-based practitioners working in middle school general education settings, are discussed.     

Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions

Assessing treatment fidelity is a core methodological consideration in the study of treatment outcome; it influences both the degree to which changes can be attributed to the intervention and the ability to replicate and disseminate the intervention. Efforts to increase access to evidence-based psychological treatments are receiving unprecedented support; but pressures exist to adapt treatments to service settings, running the risk of compromising fidelity. However, little evidence is available to inform the necessary conditions for the transportation of interventions to service provision settings, and the degree to which fidelity is even evaluated or emphasized in dissemination and implementation programs varies dramatically. Moreover, adaptation is associated with several benefits for dissemination efforts and may address relevant barriers to adoption. A particularly promising strategy for maximizing the benefits of both fidelity and adaptation is the use of transdiagnostic interventions. Such treatments allow for greater flexibility of the pacing and content of treatment, while still providing structure to facilitate testing and replication. Preliminary evidence supports the efficacy of this strategy, which may be particularly conducive to dissemination into service provision settings. At this time, further research is needed to evaluate the relationships among fidelity, adaptation, and outcome, and to determine the potential for transdiagnostic treatments to facilitate dissemination.     

Moving Forward with Systems of Care: Needs and New Directions

The articles in this Special Issue on system change within systems of care (SOCs) provide guidance regarding strategies for modifying SOCs to address the needs of different populations, and ways for changing systems to support more positive child and family outcomes. This paper frames central needs, unanswered questions, and issues that remain for those working to implement SOCs. Specific needs and new directions considered include: (1) rigorous implementation-focused research to identify the necessary and sufficient elements of SOCs and the primary practice approach currently used in SOCs, wraparound; (2) applied research to assess SOCs and document their effectiveness in non-standard or non-traditional settings (i.e., non-mental health settings, including child welfare, juvenile justice, local housing authorities); (3) controlled outcome studies for school-based wraparound initiatives; (4) research to document the effectiveness of the family support efforts that are part of most SOCs; and (5) attention to context, for families, service providers, and collaborative implementation efforts, by researchers and providers alike. Progress in these areas can inform well-targeted system change efforts in the context of SOCs, a critical need given changes in federal funding for these initiatives.     

Fidelity Failures in Brief Strategic Family Therapy for Adolescent Drug Abuse: A Clinical Analysis

As evidence‐based family treatments for adolescent substance use and conduct problems gain traction, cutting edge research moves beyond randomized efficacy trials to address questions such as how these treatments work and how best to disseminate them to community settings. A key factor in effective dissemination is treatment fidelity, which refers to implementing an intervention in a manner consistent with an established manual. While most fidelity research is quantitative, this study offers a qualitative clinical analysis of fidelity failures in a large, multisite effectiveness trial of Brief Strategic Family Therapy (BSFT) for adolescent drug abuse, where BSFT developers trained community therapists to administer this intervention in their own agencies. Using case notes and video recordings of therapy sessions, an independent expert panel first rated 103 cases on quantitative fidelity scales grounded in the BSFT manual and the broader structural‐strategic framework that informs BSFT intervention. Because fidelity was generally low, the panel reviewed all cases qualitatively to identify emergent types or categories of fidelity failure. Ten categories of failures emerged, characterized by therapist omissions (e.g., failure to engage key family members, failure to think in threes) and commissions (e.g., off‐model, nonsystemic formulations/interventions). Of these, “failure to think in threes” appeared basic and particularly problematic, reflecting the central place of this idea in structural theory and therapy. Although subject to possible bias, our observations highlight likely stumbling blocks in exporting a complex family treatment like BSFT to community settings. These findings also underscore the importance of treatment fidelity in family therapy research.     

A Comparison of Two Flashcard Drill Methods Targeting Word Recognition

Traditional drill and practice (TD) and incremental rehearsal (IR) are two flashcard drill instructional methods previously noted to improve word recognition. The current study sought to compare the effectiveness and efficiency of these two methods, as assessed by next day retention assessments, under 2 conditions (i.e., opportunities to respond held constant across methods, length of instructional session held constant across methods) with 4 first-grade students identified by their teachers as struggling readers. Social validity of the two intervention methods also was evaluated. Results suggested important differences in response to instruction across students. Differences in effectiveness between TD and IR were minimal both when holding opportunities to respond and length of instructional session constant across methods. Minimal differences in efficiency were noted between TD and IR when holding instructional time constant across conditions. However, TD was clearly more efficient than IR for all students when holding opportunities to respond constant across conditions. Social validity data indicated that half of the students preferred the TD method and half preferred the IR method. Limitations and implications of the current study for school-based professionals are discussed.     

Infant mental health home visiting therapists’ fidelity to the Michigan IMH-HV model in community practice settings

Implementation research suggests that fidelity to a therapeutic model is important for enhancing outcomes, yet can be difficult to achieve in community practice settings. Furthermore, few published studies have reported on characteristics of treatment fidelity. The present study examined fidelity to the Infant Mental Health Home Visiting (IMH-HV) model among 51 therapists with a range of experience practicing in community settings across the state of Michigan. IMH therapists completed fidelity checklists after every session with participating families to track use of 15 treatment strategies central to the IMH-HV model across the 12-month study period. Results indicated that the most commonly endorsed components utilized in home visits were developmental guidance and infant–parent psychotherapy, followed by the provision of emotional support. Use of IMH-HV components did not vary over time for the entire sample; however, patterns of strategies used showed somewhat more variability among more experienced therapists and when serving higher risk families. Findings demonstrate that IMH-HV therapists report a range of adherence to the model in community settings, with greatest fidelity to several model core components. Ongoing training in the flexible use of all core strategies may further enhance fidelity and contribute to positive outcomes for caregivers and their children receiving IMH-HV services.     

Measuring Five Dimensions of Religiosity Across Adolescence

This paper theorizes and tests a latent variable model of adolescent religiosity in which five dimensions of religiosity are interrelated: religious beliefs, religious exclusivity, external practice, private practice, and religious salience. Research often theorizes overlapping and independent influences of single items or dimensions of religiosity on outcomes such as adolescent sexual behavior, but rarely operationalizes the dimensions in a measurement model accounting for their associations with each other and across time. We use longitudinal structural equation modeling with latent variables to analyze data from two waves of the National Study of Youth and Religion. We test our hypothesized measurement model as compared to four alternate measurement models and find that our proposed model maintains superior fit. We then discuss the associations between the five dimensions of religiosity we measure and how these change over time. Our findings suggest how future research might better operationalize multiple dimensions of religiosity in studies of the influence of religion in adolescence.     

A Review of Existing Relational Aggression Programs: Strengths, Limitations, and Future Directions

Research suggests that involvement in relational aggression is associated with serious adjustment problems, including concurrent and future social maladjustment (e.g., problematic friendships; rejection), internalizing problems (e.g., depressive symptoms), and school avoidance. Despite the burgeoning literature focusing on the harmful and damaging nature of relationally aggressive behavior, this research has only recently begun to be used to inform school-based prevention and intervention programming. This article reviews the developmental research related to relational aggression and presents a systematic examination of nine published school-based prevention and intervention programs to prevent relational aggression. Programs reviewed target preschool through eighth-grade students. Strengths and limitations of each program are discussed. Recommendations are offered for future research to develop and validate school-based programming for relational aggression, and implications for school psychologists are discussed.