Statistical power in randomized intervention studies with noncompliance

This study examined various factors that affect statistical power in randomized intervention studies with noncompliance. On the basis of Monte Carlo simulations, this study demonstrates how statistical power changes depending on compliance rate, study design, outcome distributions, and covariate information. It also examines how these factors influence power in different methods of estimating intervention effects. Intent-to-treat analysis and complier average causal effect estimation are compared as 2 alternative ways of estimating intervention effects under noncompliance. The results of this investigation provide practical implications in designing and evaluating intervention studies taking into account noncompliance.     

Placebo effect and randomized clinical trials

The achievement of optimal therapeutic results presupposes the use of appropriate treatment combined with maximal utilization of placebo effects. These aims may sometimes be difficult to satisfy in randomized clinical trials (RCTs). The question thus arises whether there is a conflict between the goals of therapy and those of experimental research; and if so, to what extent, and how is it handled in practice by clinicians and researchers. Various ethical problems have been discussed in several reports connected with RCTs. But we have found no discussion concerning the conflict between obtaining informed consent and promoting optimal placebo effects. Information about RCTs can be given in various ways. Sometimes appropriate information about RCTs to patients involves non-optimal utilization of placebo effects. This gives rise to ethical and methodological problems, which are discussed in this article.     

Statistical power for the two-factor repeated measures ANOVA

Determining a priori power for univariate repeated measures (RM) ANOVA designs with two or more within-subjects factors that have different correlational patterns between the factors is currently difficult due to the unavailability of accurate methods to estimate the error variances used in power calculations. The main objective of this study was to determine the effect of the correlation between the levels in one RM factor on the power of the other RM factor. Monte Carlo simulation procedures were used to estimate power for the A, B, and AB tests of a 2×3, a 2×6, a 2×9, a 3×3, a 3×6, and a 3×9 design under varying experimental conditions of effect size (small, medium, and large), average correlation (.4 and .8), alpha (.01 and .05), and sample size (n = 5, 10 ,15, 20, 25, and 30). Results indicated that the greater the magnitude of the differences between the average correlation among the levels of Factor A and the average correlation in the AB matrix, the lower the power for Factor B (and vice versa). Equations for estimating the error variance of each test of the two-way model were constructed by examining power and mean square error trends across different correlation matrices. Support for the accuracy of these formulae is given, thus allowing for direct analytic power calculations in future studies.     

Motivational enhancement treatment in outpatient addiction centers: A multisite randomized trial

Background/Objectives: Motivational Enhancement Treatment in Spanish (METS) is a brief intervention aimed at resolving patient ambivalence towards behavior change that has demonstrated efficacy in substance use disorder treatment to reduce use and increase treatment engagement in different populations. In order to have evidence for its implementation in Mexico, a multi-site, randomized, two-arm, controlled clinical trial was conducted at three outpatient addiction treatment centers in the country to compare the effect of METS with Counseling as Usual (CAU). Method: One hundred and twenty patients were randomized to receive three sessions of METS (n = 54) or CAU (n = 66) during the first four weeks of treatment and were assessed during the following 12 weeks. Primary outcome measures were self-reported days of substance use and of treatment services utilization, which were tested using Generalized Estimating Equations. Results: Results associated both conditions with significant changes in substance use over, whereas there were no differences between conditions in substance use or in service utilization. Conclusions: Findings do not support the hypothesis that METS is more effective than CAU, but suggest that brief interventions at treatment initiation may improve patient outcomes.     

Cluster randomized trials with treatment noncompliance

Cluster randomized trials (CRTs) have been widely used in field experiments treating a cluster of individuals as the unit of randomization. This study focused particularly on situations where CRTs are accompanied by a common complication, namely, treatment noncompliance or, more generally, intervention nonadherence. In CRTs, compliance may be related not only to individual characteristics but also to the environment of clusters individuals belong to. Therefore, analyses ignoring the connection between compliance and clustering may not provide valid results. Although randomized field experiments often suffer from both noncompliance and clustering of the data, these features have been studied as separate rather than concurrent problems. On the basis of Monte Carlo simulations, this study demonstrated how clustering and noncompliance may affect statistical inferences and how these two complications can be accounted for simultaneously. In particular, the effect of the intervention on individuals who not only were assigned to active intervention but also abided by this intervention assignment (complier average causal effect) was the focus. For estimation of intervention effects considering noncompliance and data clustering, an ML-EM estimation method was employed.     

Statistical decisions and the interim analyses of clinical trials

This paper analyzes statistical decisions during the interim analyses of clinical trials. After some general remarks about the ethical and scientific demands of clinical trials, I introduce the notion of a hard-case clinical trial, explain the basic idea behind it, and provide a real example involving the interim analyses of zidovudine in asymptomatic HIV-infected patients. The example leads me to propose a decision analytic framework for handling ethical conflicts that might arise during the monitoring of hard-case clinical trials. I use computer simulations to show how the framework can assist in reconciling certain ethical conflicts. The framework is partial, lacking the precision of a complete systematization of statistical monitoring procedures in practice.     

Statistical power in network neuroscience

Network neuroscience has emerged as a leading method to study brain connectivity. The success of these investigations is dependent not only on approaches to accurately map connectivity but also on the ability to detect real effects in the data – that is, statistical power. We review the state of statistical power in the field and discuss sample size, effect size, measurement error, and network topology as key factors that influence the power of brain connectivity investigations. We use the term 'differential power' to describe how power can vary between nodes, edges, and graph metrics, leaving traces in both positive and negative connectome findings. We conclude with strategies for working with, rather than around, power in connectivity studies.     

RegRand: Statistical software for the multiple-baseline design

RegRand (Version 1.0) is Macintosh-based software that enables a researcher to conduct a nonparametric statistical analysis of the data from Koehler and Levin’s (1998) recently described regulated randomization single-case multiple-baseline design. Regulated randomization design and analysis principles are reviewed in relation to an educational research application and a step-by-step illustration of them in relation to the RegRand program is presented.     

Statistical power analysis for growth curve models using SAS

Power analysis is critical in research designs. This study discusses a simulation-based approach utilizing the likelihood ratio test to estimate the power of growth curve analysis. The power estimation is implemented through a set of SAS macros. The application of the SAS macros is demonstrated through several examples, including missing data and nonlinear growth trajectory situations. The results of the examples indicate that the power of growth curve analysis increases with the increase of sample sizes, effect sizes, and numbers of measurement occasions. In addition, missing data can reduce power. The SAS macros can be modified to accommodate more complex power analysis for both linear and nonlinear growth curve models.     

A note on statistical power in multi‐site randomized trials with multiple treatments at each site

We derive the statistical power functions in multi‐site randomized trials with multiple treatments at each site, using multi‐level modelling. An F statistic is used to test multiple parameters in the multi‐level model instead of the Wald chi square test as suggested in the current literature. The F statistic is shown to be more conservative than the Wald statistic in testing any overall treatment effect among the multiple study conditions. In addition, we improvise an easy way to estimate the non‐centrality parameters for the means comparison t‐tests and the F test, using Helmert contrast coding in the multi‐level model. The variance of treatment means, which is difficult to fathom but necessary for power analysis, is decomposed into intuitive simple effect sizes in the contrast tests. The method is exemplified by a multi‐site evaluation study of the behavioural interventions for cannabis dependence.     

Statistical power in complex experimental designs

We show that power and sample size tables developed by Cohen (1988, pp. 289–354, 381–389) produce incorrect estimates for factorial designs: power is underestimated, and sample size is overestimated. The source of this bias is shrinkage in the implied value of the noncentrality parameter, λ, caused by using Cohen’s adjustment ton for factorial designs (pp. 365 and 396). The adjustment was intended to compensate for differences in the actual versus presumed (by the tables) error degrees of freedom; however, more accurate estimates are obtained if the tables are used without adjustment. The problems with Cohen’s procedure were discovered while testing subroutines in DATASIM 1.2 for computing power and sample size in completely randomized, randomized-blocks, and split-plot factorial designs. The subroutines give the user the ability to generate power and sample size tables that are as easy to use as Cohen’s, but that eliminate the conservative bias of his tables. We also implemented several improvements relative to “manual” use of Cohen’s tables: (1) Since the user can control the specific values of 1- β,n, andf used on the rows and columns of the table, interpolation is never required; (2) exact as opposed to approximate solutions for the noncentralF distribution are employed; (3) solutions for factorial designs, including those with repeated measures factors, take into account the actual error degrees of freedom for the effect being tested; and (4) provision is made for the computation of power for applications involving the doubly noncentralF distribution.     

The optimal experimental design for multiple alternatives perceptual search

Perceptual bias is inherent to all our senses, particularly in the form of visual illusions and aftereffects. However, many experiments measuring perceptual biases may be susceptible to nonperceptual factors, such as response bias and decision criteria. Here, we quantify how robust multiple alternative perceptual search (MAPS) is for disentangling estimates of perceptual biases from these confounding factors. First, our results show that while there are considerable response biases in our four-alternative forced-choice design, these are unrelated to perceptual biases estimates, and these response biases are not produced by the response modality (keyboard vs. mouse). We also show that perceptual bias estimates are reduced when feedback is given on each trial, likely due to feedback enabling observers to partially (and actively) correct for perceptual biases. However, this does not impact the reliability with which MAPS detects the presence of perceptual biases. Finally, our results show that MAPS can detect actual perceptual biases and is not a decisional bias towards choosing the target in the middle of the candidate stimulus distribution. In summary, researchers conducting a MAPS experiment should use a constant reference stimulus, but consider varying the mean of the candidate distribution. Ideally, they should not employ trial-wise feedback if the magnitude of perceptual biases is of interest.     

Statistical criteria for parallel tests: A comparison of accuracy and power

Parallel tests are needed so that alternate forms can be applied to different groups or on different occasions, but also in the context of split-half reliability estimation for a given test. Statistically, parallelism holds beyond reasonable doubt when the null hypotheses of equality of observed means and variances across the two forms (or halves) are not rejected. Several statistical tests have been proposed for this purpose, but their performance has never been compared. This study assessed the relative performance (type I error rate and power) of the Student–Pitman–Morgan, Bradley–Blackwood, and Wilks tests of equality of means and variances in the typical conditions surrounding studies of parallelism—namely, integer-valued and bounded test scores with distributions that may not be bivariate normal. The results advise against the use of the Wilks test and support the use of the Bradley–Blackwood test because of its simplicity and its minimally better performance in comparison with the more cumbersome Student–Pitman–Morgan test.     

Statistical power and the Rorschach: 1975-1991

The Rorschach Inkblot Test has been the source of long-standing controversies as to its nature and its psychometric properties. Consistent with behavioral science research in general, the concept of statistical power has been entirely ignored by Rorschach researchers. The concept of power is introduced and discussed, and a power survey of the Rorschach literature published between 1975 and 1991 in the Journal of Personality Assessment, Journal of Consulting and Clinical Psychology, Journal of Abnormal Psychology, Journal of Clinical Psychology, Journal of Personality, Psychological Bulletin, American Journal of Psychiatry, and Journal of Personality and Social Psychology was undertaken. Power was calculated for 2,300 statistical tests in 158 journal articles. Power to detect small, medium, and large effect sizes was .13, .56, and .85, respectively. Similar to the findings in other power surveys conducted on behavioral science research, we concluded that Rorschach research is underpowered to detect the differences under investigation. This undoubtedly contributes to the inconsistency of research findings which has been a source of controversy and criticism over the decades. It appears that research conducted according to the Comprehensive System for the Rorschach is more powerful. Recommendations are offered for improving power and strengthening the design sensitivity of Rorschach research, including increasing sample sizes, use of parametric statistics, reduction of error variance, more accurate reporting of findings, and editorial policies reflecting concern about the magnitude of relationships beyond an exclusive focus on levels of statistical significance.     

Statistical power analysis for one-way analysis of variance: A computer program

To facilitate the computation of statistical power for analysis of variance, Cohen developed the index of effect sizef, defined as theSD between groups divided by theSD within groups. A microcomputer program for statistical power allows the user to compute the value off in any of several ways: by specifying the mean andSD for every cell in the ANOVA; by specifying the mean value for the two extreme cells and the pattern of dispersion for the remaining cells; by estimating the proportion of variance in the dependent variable that will be explained by group membership; and/or with reference to conventions for small, medium, and large effects. The program will compute power for any single set of parameters; it will also allow the user to generate tables and graphs showing how power will vary as a function of effect size, sample size, andα.     

Non-parametric inference for ordered alternatives in a randomized block design

This paper treats the problem of testing an ordered hypothesis based on the ranks of the data. The statistical procedures for the randomized block design with more than one observation per cell are derived. Multiple comparisons and estimation procedures are also included.     

Evaluation of a preventive intervention for child anxiety in two randomized attention-control school trials

The present research examined the effectiveness of a cognitive-behavioral therapy (CBT) based intervention program, FRIENDS, for children from grades 4 to 6, using random assignment at the school-level and an attention-control design in two longitudinal studies. The first study targeted children with anxiety symptoms (N = 191, mean age = 10.1) as screened with self, parent, and teacher-reports; the second study took a universal approach with full classrooms of children participating (N = 253, mean age = 9.8). The results showed no intervention effect in both studies, with children’s anxiety symptoms decreasing over time regardless of whether they were in the story-reading (attention control) or FRIENDS condition. The findings also indicated that girls reported a higher level of anxiety than boys and children in higher grades reported lower anxiety relative to younger children in both studies. In addition, similar patterns were found using a subgroup of children with high-anxiety symptoms from both studies.