Computing Power of Tests of the Variance of Treatment Effects in Designs With Two Levels of Nesting

Experiments that involve nested structures may assign treatment conditions either to entire groups (such as classrooms or schools) or individuals within groups (such as students). Although typically the interest in field experiments is in determining the significance of the overall treatment effect, it is equally important to examine the inconsistency of the treatment effect in different groups. This study provides methods for computing power of tests for the variability of treatment effects across level-2 and level-3 units in three-level designs, where, for example, students are nested within classrooms and classrooms are nested within schools and random assignment takes place at the first or the second level. The power computations take into account nesting effects at the second (e.g., classroom) and at the third (e.g., school) level as well as sample size effects (e.g., number of level-1 and level-2 units). The methods can also be applied to quasi-experimental studies that examine the significance of the variation of group differences in an outcome or associations between predictors and outcomes across level-2 and level-3 units.