Evaluating the individual counseling of conduct problem adolescents: The goal attainment scaling method

Goal Attainment Scaling (GAS) is described as a potentially useful way of evaluating goal attainment of individual counseling provided to public school adolescents with conduct problems. Utilization of the GAS method for individual counseling involves a counselor (e.g., school psychologist) and a pupil in goal setting, program design, goal attainment scaling, and goal attainment evaluation activities. An application of the GAS method to evaluate goal attainment of individual counseling of conduct problem adolescents in an urban high school is presented, and formative evaluation data pertaining to practical and technical properties of the approach are reported. Advantages and limitations of the GAS method are discussed.     

Implementing a Comprehensive Program for the Prevention of Conduct Problems in Rural Communities: The Fast Track Experience

Childhood conduct problems are predictive of a number of serious long-term difficulties (e.g., school failure, delinquent behavior, and mental health problems), making the design of effective prevention programs a priority. The Fast Track Program is a demonstration project currently underway in four demographically diverse areas of the United States, testing the feasibility and effectiveness of a comprehensive, multicomponent prevention program targeting children at risk for conduct disorders. This paper describes some lessons learned about the implementation of this program in a rural area. Although there are many areas of commonality in terms of program needs, program design, and implementation issues in rural and urban sites, rural areas differ from urban areas along the dimensions of geographical dispersion and regionalism, and community stability and insularity. Rural programs must cover a broad geographical area and must be sensitive to the multiple, small and regional communities that constitute their service area. Small schools, homogeneous populations, traditional values, limited recreational, educational and mental health services, and politically conservative climates are all more likely to emerge as characteristics of rural rather than urban sites (Sherman, 1992). These characteristics may both pose particular challenges to the implementation of prevention programs in rural areas, as well as offer particular benefits. Three aspects of program implementation are described in detail: (a) community entry and program initiation in rural areas, (b) the adaptation of program components and service delivery to meet the needs of rural families and schools, and (c) issues in administrative organization of a broadly dispersed tricounty rural prevention program.     

A Framework for Summative Evaluation in Informal Science Education

ABSTRACT

Summative evaluation plays a critical role in documenting the impacts of informal science education (ISE), potentially contributing to the ISE knowledge base and informing ongoing improvements in practice and decision-making. In response to the growing demand for capacity-building in ISE evaluation, this article presents a framework for summative evaluation based on an extensive review of literature and research-based refinements. The framework synthesizes key elements of high-quality summative evaluation into three dimensions: (a) Intervention Rationale, (b) Methodological Rigor and Appropriateness, and (c) Evaluation Uses. Judgment of the value of the intervention (e.g., program, exhibition) should draw upon all three dimensions.     

Programa para el Bienestar y Aprendizaje Socioemocional en estudiantes de tercero y cuarto grado: descripción y evaluación de impacto

This paper presents a program to foster socioemotional learning and wellbeing in 3rd and 4th graders, and its impact on evaluation. The program is conceptually grounded in the socioemotional learning perspective, which integrates factors both at the individual and contextual levels. Development of quality interpersonal bonding is emphasized, as is the development of the necessary socioemotional skills to achieve this goal. A quasi-experimental design was implemented which included 647 students in five private subsidized schools in Santiago, Chile. Evaluation with Anova analysis of repeated measures showed an impact of the program on students’ self-esteem reported by the teacher (measured with TAE). Seemingly, the program mitigated the decrease in the perception of school social climate (measured with ECLIS), in social integration, and in academic performance, as observed in the control group. Results are discussed in terms of their implications for the development of interventions in this field.     

Prüfungsangst bewältigen

Test anxiety or exam anxiety is a prevalent problem among students. Therefore, a high proportion of students seek help in university counseling centers. Based on a theoretical review of psychological theories and models of test anxiety, a cognitive-behavioral training program for coping with test anxiety as well as the evaluation of the program is presented. The program includes elements of behavior training with video-supported role playing and simulation of (oral) test situations, enhancement of learning and preparation strategies including time management as well as cognitive intervention strategies and relaxation training. Evaluation of the workshop program shows significant reduction of test anxiety and the strengthening of self-efficacy. The group program also lends itself for application with other groups, such as school students or for the preparation of job interviews.     

Implementation Quality and Positive Experiences in After-School Programs

Data collected during an evaluation of a multi-site trial of an enhanced after-school program were used to relate quality of program implementation to student experiences after school. The enhanced after-school program incorporated a drug use and violence prevention component that was shown to be effective in previous research. Building on Durlak and Dupre’s (Am J Community Psychol 41:327–350, 2008) dimensions of implementation, we assessed the level of dosage, quality of management and climate, participant responsiveness, and staffing quality achieved at the five program sites. We evaluated how these characteristics co-varied with self-reported positive experiences after-school. The study illustrates how multiple dimensions of program implementation can be measured, and shows that some but not all dimensions of implementation are related to the quality of student after-school experiences. Measures of quality of management and climate, participant responsiveness, and staffing stability were most clearly associated with youth experiences. The importance of measuring multiple dimensions of program implementation in intervention research is discussed.     

Preventing conduct problems, promoting social competence: a parent and teacher training partnership in head start

Studied the effectiveness of parent and teacher training as a selective prevention program for 272 Head Start mothers and their 4-year-old children and 61 Head Start teachers. Fourteen Head Start centers (34 classrooms) were randomly assigned to (a) an experimental condition in which parents, teachers, and family service workers participated in the prevention program (Incredible Years) or (b) a control condition consisting of the regular Head Start program. Assessments included teacher and parent reports of child behavior and independent observations at home and at school. Construct scores combining observational and report data were calculated for negative and positive parenting style, parent-teacher bonding, child conduct problems at home and at school, and teacher classroom management style. Following the 12-session weekly program, experimental mothers had significantly lower negative parenting and significantly higher positive parenting scores than control mothers. Parent-teacher bonding was significantly higher for experimental than for control mothers. Experimental children showed significantly fewer conduct problems at school than control children. Children of mothers who attended 6 or more intervention sessions showed significantly fewer conduct problems at home than control children. Children who were the "highest risk" at baseline (high rates of noncompliant and aggressive behavior) showed more clinically significant reductions in these behaviors than high-risk control children. After training, experimental teachers showed significantly better classroom management skills than control teachers. One year later the experimental effects were maintained for parents who attended more than 6 groups. The clinically significant reductions in behavior problems for the highest risk experimental children were also maintained. Implications of this prevention program as a strategy for reducing risk factors leading to delinquency by promoting social competence, school readiness, and reducing conduct problems are discussed.     

Integrating consultation and program evaluation in school psychology

Consultation approaches to providing psychological services are widely used by school psychologists. Recently, the need for school psychologists to be proficient in educational program evaluation has been recognized. The present paper discusses how school psychologists can use their existing consultation skills as an approach to program evaluation. The integration of consultation skills with evaluation results in an approach to program evaluation that is particularly useful where program goals and objectives are broad and nonspecific (as in many educational innovations). Also, the consultation approach to evaluation is easier for school psychologists to blend with their other roles in schools than the assessment-oriented evaluator role of some evaluation models. The consultation approach to evaluation is illustrated using a federally sponsored training program for school teachers and administrators.     

Self-esteem, personality and achievement in high school: a prospective longitudinal study in Texas

Abstract This research explored four empirical questions: (1) Is self‐esteem a better predictor of academic success and adjustment than other aspects of personality? (2) How is self‐esteem related to Big‐Five dimensions of personality during the transition from middle school to high school? (3) Do dispositions like Agreeableness or Openness relate to an adolescent's adaptation and affect reactions to the self? and (4) Do sources of information about adolescents (e.g., self‐rating, other rating, objective “life history”) converge? We also explored the general hypothesis that personality, self‐esteem, and teachers' ratings of adjustment during the middle school years predict later life outcomes during high school. Overall, results indicate Big‐Five personality characteristics were more stable than self‐esteem across this transition period. Agreeableness and Openness assessed in middle school are related to later scholastic competence and behavioral conduct, academic success, and adjustment in high school. Results were discussed in terms of personality development and self‐evaluation.     

A workshop model for training the school psychologist in drug education and intervention techniques

Proposed to: (1) design a model drug training program specifically for graduate student school psychologists, (2) implement the program on a pilot basis, and (3) demonstrate an approach to evaluate the program's effectiveness. A 26-hour drug education workshop was presented to 12 graduate student school psychologists as a model training program including a variety of teaching procedures and extensive evaluation. On prepost questionnaires, improved scores reflected increased understanding of drug-related laws and terminology. While participant's ratings of their knowledge about drugs did not increase, they did rate themselves as more competent regarding drug problems. The workshop's affective climate and many of the teaching techniques were rated positively. The contributions of this workshop were its focus on training techniques and the development of one approach for evaluating training projects for school psychologists.     

Challenges of Evaluating Multilevel Interventions

This article uses the Comprehensive Mixed-Methods Participatory Evaluation (CMMPE; Nastasi and Hitchcock Transforming school mental health services: Population-based approaches to promoting the competency and wellness of children, Thousand Oaks, CA: Corwin Press with National Association of School Psychologists 2008; Nastasi et al. School-based mental health services: creating comprehensive and culturally specific programs. Washington, DC: American Psychological Association 2004) model as a framework for addressing the multiplicity of evaluation decisions and complex nature of questions related to program success in multilevel interventions. CMMPE defines program success in terms of acceptability, integrity, social or cultural validity, outcomes (impact), sustainability and institutionalization, thus broadening the traditional notions of program outcomes. The authors use CMMPE and an example of a community-based multilevel sexual risk prevention program with multiple outcomes to discuss challenges of evaluating multilevel interventions. The sexual risk program exemplifies what Schensul and Tricket (this issue) characterize as multilevel intervention–multilevel evaluation (M–M), with both intervention and evaluation at community, health practitioner, and patient levels. The illustration provides the context for considering several challenges related to M–M designs: feasibility of randomized controlled trials within community-based multilevel intervention; acceptability and social or cultural validity of evaluation procedures; implementer, recipient, and contextual variations in program success; interactions among levels of the intervention; unanticipated changes or conditions; multiple indicators of program success; engaging multiple stakeholders in a participatory process; and evaluating sustainability and institutionalization. The complexity of multilevel intervention and evaluation designs challenges traditional notions of evaluation research and experimental designs. Overcoming these challenges is critical to effective translation of research to practice in psychology and related disciplines.     

Obtaining Systematic Teacher Reports of Disruptive Behavior Disorders Utilizing DSM-IV

This study examines the psychometric properties of the Vanderbilt AD/HD Diagnostic Teacher Rating Scale (VADTRS) and provides preliminary normative data from a large, geographically defined population. The VADTRS consists of the complete list of DSM-IV AD/HD symptoms, a screen for other disruptive behavior disorders, anxiety and depression, and ratings of academic and classroom behavior performance. Teachers in one suburban county completed the scale for their students during 2 consecutive years. Statistical methods included (a) exploratory and confirmatory latent variable analyses of item data, (b) evaluation of the internal consistency of the latent dimensions, (c) evaluation of latent structure concordance between school year samples, and (d) preliminary evaluation of criterion-related validity. The instrument comprises four behavioral dimensions and two performance dimensions. The behavioral dimensions were concordant between school years and were consistent with a priori DSM-IV diagnostic criteria. Correlations between latent dimensions and relevant, known disorders or problems varied from .25 to .66.     

School professionals and the evaluation function

Evaluation is assumed to be an integral part of the professional delivery of school services. As such, professionals employed in school systems are called upon to define alternative roles they might play in evaluation, to consider alternative ways to organize for evaluation, and to focus on various objects of evaluation. Listed alternatives were drawn from emerging literature in school evaluation. Standards suggested for judging school evaluation included those addressing accuracy, utility, propriety, and feasibility of the evaluation.     

Utilization of program evaluation information in public schools: Perspectives and guidelines for school psychologists

The ability of program evaluators in public schools to design evaluations the result in useful evaluation information is central to the advancement of the field of program evaluation. Unless evaluation information is utilized by program decision makers, the worth of an evaluation can be questioned. In this paper, the issue of utilization of program evaluation information in public schools is addressed. First, perspectives on utilization of evaluation information are provided that are derived from knowledge utilization theory and research. Second, a set of program evaluation guidelines are described that are intended to help enhance utilization of evaluation information by program decision makers. Third, a meta-evaluation of the application of the guidelines in public school settings is reported.     

Can religious beliefs combat negative peer influence during adolescence?

The purpose of this study was to identify religiosity dimensions that might be efficacious in helping adolescents combat negative peer influence. In past studies, the protective elements of religiosity were small to non-existent once peer influence was considered. However, peer influence can take many forms and the relation between susceptibility to negative peer influence and religious beliefs is not clear. Two hundred and twenty high school students were surveyed on several dimensions of religiosity via the Attitude to Christianity Inventory, different forms of peer influence (i.e., peer conformity and perceptions of peer behaviours) and their own pro- and anti-social conduct. Through hierarchical regression analyses some religiosity dimensions (i.e., the Bible and social justice) were found to be more protective than others in predicting conduct in the context of peer conformity. Implications for intervention were addressed.     

Development of a school-based treatment program for middle school youth with ADHD

The authors conduct an evaluation of a middle school-based treatment program for youth with ADHD during early stages of treatment development. The studies focus on interpreting outcome trends in preliminary data and identifying assessment issues that will be important to consider when conducting a clinical trial. Parent reports indicate that the majority of students benefit from improvements in academic, social, and overall functioning. Although teachers report beneficial effects for the majority of the participants in the program, there is little agreement about individual students. Measurement problems are associated with understanding normal change during a school year, accounting for normal behavior changes in December and May, and considerable disagreement between teachers. Suggestions to guide future work in this area are provided.     

Description and Immediate Impacts of a Preventive Intervention for Conduct Problems

A population-based randomized intervention trial for the prevention of conduct problems (i.e., oppositional defiant disorder and conduct disorder) is described. The LIFT (Linking the Interests of Families and Teachers) intervention was designed for all first- and fifth-grade elementary school boys and girls and their families living in at-risk neighborhoods characterized by high rates of juvenile delinquency. The 10-week intervention strategy was carefully targeted at proximal and malleable antecedents in three social domains that were identified by a developmental model of conduct problems. From 12 elementary schools, 671 first and fifth graders and their families participated either in the theory-based universal preventive intervention or in a control condition. The intervention consisted of parent training, a classroom-based social skills program, a playground behavioral program, and systematic communication between teachers and parents. A multiple measure assessment strategy was used to evaluate participant satisfaction and participation, fidelity of implementation, and the immediate impacts of the program on targeted antecedents.     

Effects of a School‐Based Life Skills and HIV‐Prevention Program for Adolescents in Mexican High Schools1

A school‐based life skills and sexuality education HIV/AIDS program for adolescents was evaluated in Toluca, Mexico, through a quasi‐experimental pre–post design with 1,566 cases, divided over a control group and an experimental group. Based on a previously tested path model, 3 levels of variables were included as a guide for the contents of the program: personal variables (self‐knowledge, self‐efficacy regarding condom use, and decision making), intervening variables (subjective norms about, knowledge about HIV/AIDS, and attitudes toward condoms), and outcome variables (communication on sexuality and intentions to use condoms). Evaluation was conducted following these categories. Significant increases in all of these variables recognized as precursors of safer sex were found in the experimental group.     

On the use of psychology in evaluating software documentation

Recent research evaluating the usefulness of program documentation has borrowed techniques used in psychology to define the variables of interest and to measure those variables. The results of the research suggest that (1) using a branching or sequential program design language (PDL) produces performance that is superior to that found with other formats (such as flowcharts), and (2) using a detailed design written in a PDL leads to superior coding performance over other formats due, in part, to the small amount of translation that is required in mapping from the design to the code. These conclusions are based primarily on time and accuracy data. This paper discusses the research conducted in this area and the conclusions drawn. The influence of the methodology used to conduct these experiments on the evaluation of programmer performance is also discussed.