Power and measures of effect size in analysis of variance with fixed versus random nested factors

Ignoring a nested factor can influence the validity of statistical decisions about treatment effectiveness. Previous discussions have centered on consequences of ignoring nested factors versus treating them as random factors on Type I errors and measures of effect size (B. E. Wampold & R. C. Serlin). The authors (a) discuss circumstances under which the treatment of nested provider effects as fixed as opposed to random is appropriate; (b) present 2 formulas for the correct estimation of effect sizes when nested factors are fixed; (c) present the results of Monte Carlo simulations of the consequences of treating providers as fixed versus random on effect size estimates, Type I error rates, and power; and (d) discuss implications of mistaken considerations of provider effects for the study of differential treatment effects in psychotherapy research.     

Should providers of treatment be regarded as a random factor? If it ain't broke, don't "fix" it: a comment on Siemer and Joormann (2003)

In their criticism of B. E. Wampold and R. C. Serlin's analysis of treatment effects in nested designs, M. Siemer and J. Joormann argued that providers of services should be considered a fixed factor because typically providers are neither randomly selected from a population of providers nor randomly assigned to treatments, and statistical power to detect treatment effects is greater in the fixed than in the mixed model. The authors of the present article argue that if providers are considered fixed, conclusions about the treatment must be conditioned on the specific providers in the study, and they show that in this case generalizing beyond these providers incurs inflated Type I error rates.     

Ignoring Clustering in Confirmatory Factor Analysis: Some Consequences for Model Fit and Standardized Parameter Estimates

In many situations, researchers collect multilevel (clustered or nested) data yet analyze the data either ignoring the clustering (disaggregation) or averaging the micro-level units within each cluster and analyzing the aggregated data at the macro level (aggregation). In this study we investigate the effects of ignoring the nested nature of data in confirmatory factor analysis (CFA). The bias incurred by ignoring clustering is examined in terms of model fit and standardized parameter estimates, which are usually of interest to researchers who use CFA. We find that the disaggregation approach increases model misfit, especially when the intraclass correlation (ICC) is high, whereas the aggregation approach results in accurate detection of model misfit in the macro level. Standardized parameter estimates from the disaggregation and aggregation approaches are deviated toward the values of the macro- and micro-level standardized parameter estimates, respectively. The degree of deviation depends on ICC and cluster size, particularly for the aggregation method. The standard errors of standardized parameter estimates from the disaggregation approach depend on the macro-level item communalities. Those from the aggregation approach underestimate the standard errors in multilevel CFA (MCFA), especially when ICC is low. Thus, we conclude that MCFA or an alternative approach should be used if possible.     

Treatment of Young Children's Bedtime Refusal and Nighttime Wakings: A Comparison of “Standard” and Graduated Ignoring Procedures

Young children with sleep problems received either standard or graduated ignoring treatment. Both brief treatments were superior to a wait-list control condition and resulted in comparable improvements in bedtime and nighttime sleep problems. At bedtime, the treatments did not differ with respect to maternal compliance and stress. For nighttime wakings, mothers in the graduated ignoring group reported higher rates of compliance and less treatment-related stress. Maternal characteristics predicted treatment outcome in the standard ignoring condition. Following treatment, only positive side effects were observed. When compared to the wait-list group, mothers in the standard ignoring group reported less verbose discipline and decreased stress in parenting, while mothers in the graduated ignoring group reported improved parent–child relationships. Treatment gains were maintained over a 2-month follow-up period.     

Computing Power of Tests of the Variance of Treatment Effects in Designs With Two Levels of Nesting

Experiments that involve nested structures may assign treatment conditions either to entire groups (such as classrooms or schools) or individuals within groups (such as students). Although typically the interest in field experiments is in determining the significance of the overall treatment effect, it is equally important to examine the inconsistency of the treatment effect in different groups. This study provides methods for computing power of tests for the variability of treatment effects across level-2 and level-3 units in three-level designs, where, for example, students are nested within classrooms and classrooms are nested within schools and random assignment takes place at the first or the second level. The power computations take into account nesting effects at the second (e.g., classroom) and at the third (e.g., school) level as well as sample size effects (e.g., number of level-1 and level-2 units). The methods can also be applied to quasi-experimental studies that examine the significance of the variation of group differences in an outcome or associations between predictors and outcomes across level-2 and level-3 units.     

Investigating healthcare providers' knowledge of trichotillomania and its treatment

Given the low prevalence rate and rather secretive nature of trichotillomania, it is uncertain how much trichotillomania-related knowledge physicians and psychologists possess, what the perceived role of a psychologist is in the treatment process, what level of familiarity practitioners have with effective treatments and whether providers have resource materials available for patients. In a postal survey of 501 psychologists and physicians in the USA, providers responded correctly to 61% of the general knowledge items about trichotillomania. Most providers believed that psychologists may play a variety of roles in the clinical management of the disorder. Although providers were fairly accurate about the effectiveness of certain treatments for trichotillomania (e.g. medication, habit-reversal), a number of non-empirically supported treatments were endorsed as being effective. Furthermore, an overwhelming majority of healthcare providers did not have resources or referral information available for patients with trichotillomania. Implications of the findings and limitations of the study are discussed.     

Robustness of parameter and standard error estimates against ignoring a contextual effect of a subject-level covariate in cluster-randomized trials

In experimental research, it is not uncommon to assign clusters to conditions. When analysing the data of such cluster-randomized trials, a multilevel analysis should be applied in order to take into account the dependency of first-level units (i.e., subjects) within a second-level unit (i.e., a cluster). Moreover, the multilevel analysis can handle covariates on both levels. If a first-level covariate is involved, usually the within-cluster effect of this covariate will be estimated, implicitly assuming the contextual effect to be equal. However, this assumption may be violated. The focus of the present simulation study is the effects of ignoring the inequality of the within-cluster and contextual covariate effects on parameter and standard error estimates of the treatment effect, which is the parameter of main interest in experimental research. We found that ignoring the inequality of the within-cluster and contextual effects does not affect the estimation of the treatment effect or its standard errors. However, estimates of the variance components, as well as standard errors of the constant, were found to be biased.     

The role of unattended distractors in sustained inattentional blindness

When participants are attending to a subset of visual targets or events and ignoring irrelevant distractors (“selective looking”), they often fail to detect the appearance of an unexpected visual object or event even when the object is visible for several seconds (“sustained inattentional blindness”). An important factor influencing detection rates in selective looking is the attentional set of the participant: the more similar the features of the unexpected object are to the attended ones, the more probably it will be detected. We examined the possible contribution of active ignoring to this similarity effect by studying the role of the distractor objects in sustained inattentional blindness. First we showed the similarity effect for chromatic colors and then we manipulated the similarity of the unexpected object in relation to the distractor objects and did not find any effects. Moreover, we found that inattentional blindness was present even when the displays did not contain any irrelevant to-be-ignored objects. We conclude that attending to target items on the basis of attentional set, but not active ignoring of nontargets items, is sufficient for the occurrence of sustained inattentional blindness.     

Early Intervention of Eating- and Weight-Related Problems

Obesity and other eating-related problems are widespread and are associated with harmful physical, psychological, and social problems. The dramatic increases in rates of pediatric obesity has created a mounting need for psychologists and other mental health care providers to play a significant role in the assessment and treatment of youth with eating- and weight-related problems. Therefore, it is imperative for providers to be aware of the causes and consequences of eating- and weight-related problems and to be familiar with evidence-based assessment and intervention approaches. Currently, the most well-established intervention approaches are family-based behavioral treatments, and weight loss maintenance treatments with a socio-ecological focus are promising. This paper provides a comprehensive review of these topics and highlights the important roles that mental health care providers can have. Medical settings are often the patient’s first point of contact within the healthcare system, making mental health care providers in such settings uniquely suited to assess for a broad range of eating- and weight-related problems and associated comorbidities, to deliver relevant evidence-based interventions, and to make appropriate referrals. Moving forward, providers and researchers must work together to address key questions related to the nature of eating- and weight-related problems in youth and to achieve breakthroughs in the prevention and treatment of such problems in this vulnerable population.     

Attitudes Toward Adoption of Evidence-Based Practices: A comparison of Autism Early Intervention Providers and Children's Mental Health Providers

Across the country, states are reporting increases in the number of children with autistic spectrum disorders (ASD) served each year in the early intervention system. Research examining factors impacting the successful dissemination and implementation of evidence-based practice (EBPs) into service systems for these children is limited. Preliminary information indicates that adoption of EBPs is variable. Provider attitudes toward the adoption of EBPs may be one factor that limits or facilitates implementation of efficacious treatments and these attitudes vary by organizational context and provider individual differences. The current study examines cross-context differences in provider attitudes toward EBPs by comparing the attitudes of 71 education-based early intervention providers working with children with ASD to the attitudes of 238 mental health providers in the public mental health system. This provides the first examination of ASD early intervention provider attitudes toward EBP. Results indicated that early intervention providers reported significantly more favorable attitudes toward adopting EBPs than did mental health providers. Early intervention providers with extended experience in the field perceived less divergence between their current practice and EBPs. Implications are discussed.     

Meta-Analysis of the Effects of Peer-Administered Psychosocial Interventions on Symptoms of Depression

Many community mental health centers have implemented peer treatment models that employ recovered former clients as cost‐efficient adjunct providers. The effectiveness of these and other peer‐administered interventions (PAIs) for treating depression symptoms has not been well‐established. The current study is a meta‐analysis of PAIs’ effects on depression symptoms. Twenty‐three eligible studies were identified. Study characteristics were coded by multiple raters, random‐effects models were used to compare mean effect sizes, and mixed‐effects models were used to test for moderation. PAIs produced significant pre‐post reductions in depression symptoms (d = .5043 [95 % CI .3675–.6412]). In direct comparisons, PAIs performed as well as non‐peer‐administered interventions (.0848 [?.1455–.3151]), and significantly better than no‐treatment conditions (.2011 [.0104–.3918]). PAIs that involved a professional in a secondary treatment role were significantly less effective than those that were purely peer‐administered, and educational/skills‐based PAIs produced better outcomes than those that were mainly supportive. Follow‐up data, when available, indicated that PAIs’ benefits were maintained. PAIs reduce depression symptoms and warrant further study. The clinical significance of PAIs’ benefits, and whether they are better suited as stand‐alone or adjunct treatments, remain to be established. Implications for the roles of mental health professionals are discussed.     

Modeling external events in the three-level analysis of multiple-baseline across-participants designs: A simulation study

In this study, we focus on a three-level meta-analysis for combining data from studies using multiple-baseline across-participants designs. A complicating factor in such designs is that results might be biased if the dependent variable is affected by not explicitly modeled external events, such as the illness of a teacher, an exciting class activity, or the presence of a foreign observer. In multiple-baseline designs, external effects can become apparent if they simultaneously have an effect on the outcome score(s) of the participants within a study. This study presents a method for adjusting the three-level model to external events and evaluates the appropriateness of the modified model. Therefore, we use a simulation study, and we illustrate the new approach with real data sets. The results indicate that ignoring an external event effect results in biased estimates of the treatment effects, especially when there is only a small number of studies and measurement occasions involved. The mean squared error, as well as the standard error and coverage proportion of the effect estimates, is improved with the modified model. Moreover, the adjusted model results in less biased variance estimates. If there is no external event effect, we find no differences in results between the modified and unmodified models.     

The Impacts of Ignoring a Crossed Factor in Analyzing Cross-Classified Data

Cross-classified random-effects models (CCREMs) are used for modeling nonhierarchical multilevel data. Misspecifying CCREMs as hierarchical linear models (i.e., treating the cross-classified data as strictly hierarchical by ignoring one of the crossed factors) causes biases in the variance component estimates, which in turn, results in biased estimation in the standard errors of the regression coefficients. Analytical studies were conducted to provide closed-form expressions for the biases. With balanced design data structure, ignoring a crossed factor causes overestimation of the variance components of adjacent levels and underestimation of the variance component of the remaining crossed factor. Moreover, ignoring a crossed factor at the kth level causes underestimation of the standard error of the regression coefficient of the predictor associated with the ignored factor and overestimation of the standard error of the regression coefficient of the predictor at the (k?1)th level. Simulation studies were also conducted to examine the effect of different structures of cross-classification on the biases. In general, the direction and magnitude of the biases depend on the level of the ignored crossed factor, the level with which the predictor is associated at, the magnitude of the variance component of the ignored crossed factor, the variance components of the predictors, the sample sizes, and the structure of cross-classification. The results were further illustrated using the Early Childhood Longitudinal Study-Kindergarten Cohort data.     

À la Carte or Prix Fixe? Differing Approaches to Addressing the Gap in Dissemination of Evidence-Based Care in Mental Health

We comment on the use of a common elements treatment approach (CETA) to address mental health symptoms in a trauma-exposed population in Iraq and Thailand, with the aims of highlighting several of the strengths of this approach and how this approach may inform treatment, both within and outside of the United States. In particular, we compare the use of CETA as compared to a more focal treatment approach, consider the potential of using paraprofessionals as treatment providers, and highlight some of the challenges in cultural adaptions of psychotherapy protocols. We also identify remaining research questions, including whether a CETA approach is more efficient than focal interventions, whether CETA’s effects are due to one or two very strong elements, and whether it is more difficult to learn CETA than a robust non-CETA intervention. This use of CETA, along with our other adaptation work, also raises broader issues about how our field develops and disseminates psychotherapy protocols. We discuss several of them, including the need to develop treatments for providers with lower levels of formal education and mental health training and to develop treatment materials that are less expensive, use simplified language and terms, and that can be adapted for use with clients with lower levels of formal education and/or literacy.     

A survey of mental health care provider's and managed care organization attitudes toward, familiarity with, and use of group interventions

Managed Care has had a significant impact on delivery systems for mental health services. Direct and indirect persuasion to provide more cost-effective treatments has been one consequence. The cost-saving qualities and the effectiveness of group interventions have produced clear expectations for an increased use of therapy groups. This study compared perceptions and uses of group treatments on a national sample of managed care organizations and mental health providers. Because group psychotherapy encompasses such a broad definition, five specific types of group interventions were defined: problem-focused homogenous, process-oriented heterogeneous, psycho-educational, self-help, and short-term groups. Implications of differences and similarities between directors of managed care organizations and treatment providers are examined and discussed across five response categories (familiarity/training, perceived effectiveness, likelihood of reimbursement/referral, daily use, and expectation for future use).     

Modelling partially cross‐classified multilevel data

This article proposes an approach to modelling partially cross‐classified multilevel data where some of the level‐1 observations are nested in one random factor and some are cross‐classified by two random factors. Comparisons between a proposed approach to two other commonly used approaches which treat the partially cross‐classified data as either fully nested or fully cross‐classified are completed with a simulation study. Results show that the proposed approach demonstrates desirable performance in terms of parameter estimates and statistical inferences. Both the fully nested model and the fully cross‐classified model suffer from biased estimates of some variance components and statistical inferences of some fixed effects. Results also indicate that the proposed model is robust against cluster size imbalance.     

Structural Equation Modeling Approaches for Analyzing Partially Nested Data

Study designs involving clustering in some study arms, but not all study arms, are common in clinical treatment-outcome and educational settings. For instance, in a treatment arm, persons may be nested in therapy groups, whereas in a control arm there are no groups. Methodological approaches for handling such partially nested designs have recently been developed in a multilevel modeling framework (MLM-PN) and have proved very useful. We introduce two alternative structural equation modeling (SEM) approaches for analyzing partially nested data: a multivariate single-level SEM (SSEM-PN) and a multiple-arm multilevel SEM (MSEM-PN). We show how SSEM-PN and MSEM-PN can produce results equivalent to existing MLM-PNs and can be extended to flexibly accommodate several modeling features that are difficult or impossible to handle in MLM-PNs. For instance, using an SSEM-PN or MSEM-PN, it is possible to specify complex structural models involving cluster-level outcomes, obtain absolute model fit, decompose person-level predictor effects in the treatment arm using latent cluster means, and include traditional factors as predictors/outcomes. Importantly, implementation of such features for partially nested designs differs from that for fully nested designs. An empirical example involving a partially nested depression intervention combines several of these features in an analysis of interest for treatment-outcome studies.     

Barriers to the management of patients with surgically remediable intractable epilepsy

It is estimated that only a small proportion of patients with surgically remediable intractable epilepsy receive surgical treatment. There are multiple reasons why this is the case. Patients with intractable epilepsy are sometimes severely disabled and disability can create barriers to getting recommended care. Patients with epilepsy are not well informed about their condition and the available treatments. The incidence of epilepsy is similar in minority populations, and surgically remediable epilepsy frequently presents in adolescence. Nevertheless, these vulnerable populations have specific barriers to receiving epilepsy care, which are often not addressed. In addition, despite scientific evidence for the benefits of the surgical treatment of epilepsy, many healthcare providers do not recommend or adequately discuss surgery with patients. Solutions to these barriers will require interventions that result in informed and capable patients who actively participate in their care and healthcare providers who practice culturally sensitive, recommended care.     

Effect of a Vocationally‐Focused Brief Cognitive Behavioural Intervention on Employment‐Related Outcomes for Individuals with Mood and Anxiety Disorders

Internet-delivered cognitive behavioural therapy (CBT) can be an effective method for treating major depression, but it often works best when therapist support is provided in the form of e-mail support or telephone calls. The authors investigated whether there were any intraclass correlations within therapists when delivering CBT for major depression via the Internet. They included data from two trials involving 10 therapists treating a total of 103 patients. The results of a nested one-way model in which participants were treated as raters for different therapists indicated that measures pertaining to symptom reductions (Beck Depression Inventory, Montgomery-Åsberg Depression Rating Scale–Self Report, and Beck Anxiety Inventory) did not support a clustering of data within therapists. However, the outcome on a secondary measure of life satisfaction (Quality of Life Inventory) yielded a significant intraclass correlation coefficient for therapists (r = .24, p = .001). The authors propose that text-based treatments are less sensitive to therapist effects when it comes to the primary symptom measures, but that treatment effects not directly targeted by the specific treatment program may be more dependent on the way the support is given and by whom (therapist effect). Limitations of the study are discussed.     

Bridging the Theory-Practice Gap by Getting Even Bolder With the Boulder Model

Cognitive behavioral therapy is an effective treatment for virtually all psychiatric disorders. However, very few patients have access to it and few therapists are trained in the theory and practice of cognitive behavioral therapy. Based on the existing evidence and the articles of this series, the following recommendations are made: (a) all mental health care providers (including Psy.D. and social workers) need to be trained in the practice and theories of empirically supported treatments, specifically cognitive behavioral therapy; (b) clinical practice also needs to be based on theory, not just treatment manuals; and (c) psychological treatments have to move beyond the DSM boundaries.