Challenges of Evaluating Multilevel Interventions

This article uses the Comprehensive Mixed-Methods Participatory Evaluation (CMMPE; Nastasi and Hitchcock Transforming school mental health services: Population-based approaches to promoting the competency and wellness of children, Thousand Oaks, CA: Corwin Press with National Association of School Psychologists 2008; Nastasi et al. School-based mental health services: creating comprehensive and culturally specific programs. Washington, DC: American Psychological Association 2004) model as a framework for addressing the multiplicity of evaluation decisions and complex nature of questions related to program success in multilevel interventions. CMMPE defines program success in terms of acceptability, integrity, social or cultural validity, outcomes (impact), sustainability and institutionalization, thus broadening the traditional notions of program outcomes. The authors use CMMPE and an example of a community-based multilevel sexual risk prevention program with multiple outcomes to discuss challenges of evaluating multilevel interventions. The sexual risk program exemplifies what Schensul and Tricket (this issue) characterize as multilevel intervention–multilevel evaluation (M–M), with both intervention and evaluation at community, health practitioner, and patient levels. The illustration provides the context for considering several challenges related to M–M designs: feasibility of randomized controlled trials within community-based multilevel intervention; acceptability and social or cultural validity of evaluation procedures; implementer, recipient, and contextual variations in program success; interactions among levels of the intervention; unanticipated changes or conditions; multiple indicators of program success; engaging multiple stakeholders in a participatory process; and evaluating sustainability and institutionalization. The complexity of multilevel intervention and evaluation designs challenges traditional notions of evaluation research and experimental designs. Overcoming these challenges is critical to effective translation of research to practice in psychology and related disciplines.     

Rank‐based permutation approaches for non‐parametric factorial designs

Inference methods for null hypotheses formulated in terms of distribution functions in general non‐parametric factorial designs are studied. The methods can be applied to continuous, ordinal or even ordered categorical data in a unified way, and are based only on ranks. In this set‐up Wald‐type statistics and ANOVA‐type statistics are the current state of the art. The first method is asymptotically exact but a rather liberal statistical testing procedure for small to moderate sample size, while the latter is only an approximation which does not possess the correct asymptotic α level under the null. To bridge these gaps, a novel permutation approach is proposed which can be seen as a flexible generalization of the Kruskal–Wallis test to all kinds of factorial designs with independent observations. It is proven that the permutation principle is asymptotically correct while keeping its finite exactness property when data are exchangeable. The results of extensive simulation studies foster these theoretical findings. A real data set exemplifies its applicability.     

Should we rely on the Kenward–Roger approximation when using linear mixed models if the groups have different distributions?

The study explores the robustness to violations of normality and sphericity of linear mixed models when they are used with the Kenward–Roger procedure (KR) in split‐plot designs in which the groups have different distributions and sample sizes are small. The focus is on examining the effect of skewness and kurtosis. To this end, a Monte Carlo simulation study was carried out, involving a split‐plot design with three levels of the between‐subjects grouping factor and four levels of the within‐subjects factor. The results show that: (1) the violation of the sphericity assumption did not affect KR robustness when the assumption of normality was not fulfilled; (2) the robustness of the KR procedure decreased as skewness in the distributions increased, there being no strong effect of kurtosis; and (3) the type of pairing between kurtosis and group size was shown to be a relevant variable to consider when using this procedure, especially when pairing is positive (i.e., when the largest group is associated with the largest value of the kurtosis coefficient and the smallest group with its smallest value). The KR procedure can be a good option for analysing repeated‐measures data when the groups have different distributions, provided the total sample sizes are 45 or larger and the data are not highly or extremely skewed.     

Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications

This article presents a d-statistic for single-case designs that is in the same metric as the d-statistic used in between-subjects designs such as randomized experiments and offers some reasons why such a statistic would be useful in SCD research. The d has a formal statistical development, is accompanied by appropriate power analyses, and can be estimated using user-friendly SPSS macros. We discuss both advantages and disadvantages of d compared to other approaches such as previous d-statistics, overlap statistics, and multilevel modeling. It requires at least three cases for computation and assumes normally distributed outcomes and stationarity, assumptions that are discussed in some detail. We also show how to test these assumptions. The core of the article then demonstrates in depth how to compute d for one study, including estimation of the autocorrelation and the ratio of between case variance to total variance (between case plus within case variance), how to compute power using a macro, and how to use the d to conduct a meta-analysis of studies using single-case designs in the free program R, including syntax in an appendix. This syntax includes how to read data, compute fixed and random effect average effect sizes, prepare a forest plot and a cumulative meta-analysis, estimate various influence statistics to identify studies contributing to heterogeneity and effect size, and do various kinds of publication bias analyses. This d may prove useful for both the analysis and meta-analysis of data from SCDs.     

Beyond inattentional blindness and attentional misdirection: From attentional paradigms to attentional mechanisms

Memmert (2010) tried to foster the development of attentional research by discussing four differences between attentional misdirection (AM) and inattentional blindness (IB). Considering this goal, the comment was received in the intended way by the comments of (Most, 2010) and (Moran and Brady, 2010) who make a number of highly valuable suggestions for further progress. As initially suggested by Memmert (2010) this dialog should help unravel the underlying attentional mechanisms of different paradigms. Therefore, we first discuss the suggested distinction between central and spatial IB by Most (2010). Second, we argue that working memory and perceptual load research seem particularly interesting in this regard and should be taken into consideration when conducting future research along the lines of IB and AM. Third, representative task designs can be an important mosaic piece in across-the-board attention theories and highly useful for deriving further testable hypothesis in naturalistic settings. The most important claim of all commentaries in this issue is that the proposed ideas can all be empirically tested and thereby contribute to the advancement of an unified theoretical framework incorporating IB, AM in consideration of overt and covert attention mechanisms.     

Historically recontextualizing Sidman's Tactics: How behavior analysis avoided psychology's methodological Ouroboros

Psychology is undergoing major cultural changes methodologically, with efforts to redefine how psychologists analyze and report their data. Davidson (2018) argued that psychology's methodological crises stem from mechanical objectivity involving the adoption of an analytic tool as source of dependable knowledge. This has led to institutionalization, and eventually uncritical ritualistic use, such as happened with null hypothesis statistical testing. Davidson invoked the mythological symbol of the Ouroboros to represent the endless churning of statistical fads. Sidman (1960), in his Tactics of Scientific Research provided a shield from these problems in terms of the premium he placed on the experience, expertise, judgement, and decision-making of the scientist, that appear to be absent in psychology's ritualized processes.     

Estimating Age-Based Developmental Trajectories Using Latent Change Score Models Based on Measurement Occasion

Abstract

Accelerated longitudinal designs (ALDs) are designs in which participants from different cohorts provide repeated measures covering a fraction of the time range of the study. ALDs allow researchers to study developmental processes spanning long periods within a relatively shorter time framework. The common trajectory is studied by aggregating the information provided by the different cohorts. Latent change score (LCS) models provide a powerful analytical framework to analyze data from ALDs. With developmental data, LCS models can be specified using measurement occasion as the time metric. This provides a number of benefits, but has an important limitation: It makes it not possible to characterize the longitudinal changes as a function of a developmental process such as age or biological maturation. To overcome this limitation, we propose an extension of an occasion-based LCS model that includes age differences at the first measurement occasion. We conducted a Monte Carlo study and compared the results of including different transformations of the age variable. Our results indicate that some of the proposed transformations resulted in accurate expectations for the studied process across all the ages in the study, and excellent model fit. We discuss these results and provide the R code for our analysis.