Should we rely on the Kenward–Roger approximation when using linear mixed models if the groups have different distributions?

The study explores the robustness to violations of normality and sphericity of linear mixed models when they are used with the Kenward–Roger procedure (KR) in split‐plot designs in which the groups have different distributions and sample sizes are small. The focus is on examining the effect of skewness and kurtosis. To this end, a Monte Carlo simulation study was carried out, involving a split‐plot design with three levels of the between‐subjects grouping factor and four levels of the within‐subjects factor. The results show that: (1) the violation of the sphericity assumption did not affect KR robustness when the assumption of normality was not fulfilled; (2) the robustness of the KR procedure decreased as skewness in the distributions increased, there being no strong effect of kurtosis; and (3) the type of pairing between kurtosis and group size was shown to be a relevant variable to consider when using this procedure, especially when pairing is positive (i.e., when the largest group is associated with the largest value of the kurtosis coefficient and the smallest group with its smallest value). The KR procedure can be a good option for analysing repeated‐measures data when the groups have different distributions, provided the total sample sizes are 45 or larger and the data are not highly or extremely skewed.     

The effect of skewness and kurtosis on the robustness of linear mixed models

This study analyzes the robustness of the linear mixed model (LMM) with the Kenward–Roger (KR) procedure to violations of normality and sphericity when used in split-plot designs with small sample sizes. Specifically, it explores the independent effect of skewness and kurtosis on KR robustness for the values of skewness and kurtosis coefficients that are most frequently found in psychological and educational research data. To this end, a Monte Carlo simulation study was designed, considering a split-plot design with three levels of the between-subjects grouping factor and four levels of the within-subjects factor. Robustness is assessed in terms of the probability of type I error. The results showed that (1) the robustness of the KR procedure does not differ as a function of the violation or satisfaction of the sphericity assumption when small samples are used; (2) the LMM with KR can be a good option for analyzing total sample sizes of 45 or larger when their distributions are normal, slightly or moderately skewed, and with different degrees of kurtosis violation; (3) the effect of skewness on the robustness of the LMM with KR is greater than the corresponding effect of kurtosis for common values; and (4) when data are not normal and the total sample size is 30, the procedure is not robust. Alternative analyses should be performed when the total sample size is 30.     

Using the linear mixed model to analyze nonnormal data distributions in longitudinal designs

Using a Monte Carlo simulation and the Kenward–Roger (KR) correction for degrees of freedom, in this article we analyzed the application of the linear mixed model (LMM) to a mixed repeated measures design. The LMM was first used to select the covariance structure with three types of data distribution: normal, exponential, and log-normal. This showed that, with homogeneous between-groups covariance and when the distribution was normal, the covariance structure with the best fit was the unstructured population matrix. However, with heterogeneous between-groups covariance and when the pairing between covariance matrices and group sizes was null, the best fit was shown by the between-subjects heterogeneous unstructured population matrix, which was the case for all of the distributions analyzed. By contrast, with positive or negative pairings, the within-subjects and between-subjects heterogeneous first-order autoregressive structure produced the best fit. In the second stage of the study, the robustness of the LMM was tested. This showed that the KR method provided adequate control of Type I error rates for the time effect with normally distributed data. However, as skewness increased—as occurs, for example, in the log-normal distribution—the robustness of KR was null, especially when the assumption of sphericity was violated. As regards the influence of kurtosis, the analysis showed that the degree of robustness increased in line with the amount of kurtosis.     

Single-case effect size calculation: Comparing regression and non-parametric approaches across previously published reading intervention data sets

Growing from demands for accountability and research-based practice in the field of education, there is recent focus on developing standards for the implementation and analysis of single-case designs. Effect size methods for single-case designs provide a useful way to discuss treatment magnitude in the context of individual intervention. Although a standard effect size methodology does not yet exist within single-case research, panel experts recently recommended pairing regression and non-parametric approaches when analyzing effect size data. This study compared two single-case effect size methods: the regression-based, Allison-MT method and the newer, non-parametric, Tau-U method. Using previously published research that measured the Words read Correct per Minute (WCPM) variable, these two methods were examined by comparing differences in overall effect size scores and rankings of intervention effect. Results indicated that the regression method produced significantly larger effect sizes than the non-parametric method, but the rankings of the effect size scores had a strong, positive relation. Implications of these findings for research and practice are discussed.     

A NOTE CONCERNING THE MATCHED GROUP DESIGN

Despite its strong logical and technical appeal, the employment of the matched group design in applied psychological research has been restricted by practical problems associated with its use. A procedure which circumvents several of the problems of conventional matching was described. Basically, the procedure involves pairing of subjects from two or more treatment pools after the completion of training or other treatment, but without knowledge of the performance of the subjects. Three empirical tryouts of the procedure were summarized. The procedure worked quite well when only two groups were involved and was fairly satisfactory when applied to four groups. Other experimental designs may be better when more than two groups are involved. It was concluded that matching from treatment pools after differential treatment of the pools warrants more extensive use in applied psychological research than it has been accorded in the past.     

Examining Estimates of Effect Using Non-Overlap of All Pairs in Multiple Baseline Studies of Academic Intervention

The inclusion of single-case design (SCD) studies in meta-analytic research is an important consideration in identifying effective evidence-based practices. Various SCD effect sizes have been previously suggested; non-overlap of all pairs (NAP) is a recently introduced effect size. Preliminary field tests investigating the adequacy of NAP are promising, but no analyses have been conducted using only multiple baseline designs (MBDs). This preliminary study investigated typical values of NAP in MBDs, investigated agreement with visual analysis, and suggested cut scores for interpreting a NAP effect size. Typical values of NAP in MBDs were larger compared to a previous meta-analysis of studies using AB, MBD, or ABAB withdrawal designs, and agreement of suggested cut scores and visual analysis was moderate.     

The effect of unequal group size on Tukey's HSD procedure

A limitation of the Tukey HSD procedure for multiple comparison has been the requirement of equal number of observations for each group. Three approximation techniques have been suggested when the group sizes are unequal. Each of these techniques was empirically analyzed to determine its effect on TypeI error. Two of the considered variables, average group size and differences in group size, caused differing actual probabilities of TypeI error. One of the three techniques (Kramer's) consistently provided actual probabilities in close agreement with corresponding nominal probabilities.     

The use of item parcels in structural equation modelling: Non‐normal data and small sample sizes

Maximum likelihood estimation in confirmatory factor analysis requires large sample sizes, normally distributed item responses, and reliable indicators of each latent construct, but these ideals are rarely met. We examine alternative strategies for dealing with non‐normal data, particularly when the sample size is small. In two simulation studies, we systematically varied: the degree of non‐normality; the sample size from 50 to 1000; the way of indicator formation, comparing items versus parcels; the parcelling strategy, evaluating uniformly positively skews and kurtosis parcels versus those with counterbalancing skews and kurtosis; and the estimation procedure, contrasting maximum likelihood and asymptotically distribution‐free methods. We evaluated the convergence behaviour of solutions, as well as the systematic bias and variability of parameter estimates, and goodness of fit.     

A practical method for analyzing factorial designs with heteroscedastic data

The Type I error rates and powers of three recent tests for analyzing nonorthogonal factorial designs under departures from the assumptions of homogeneity and normality were evaluated using Monte Carlo simulation. Specifically, this work compared the performance of the modified Brown-Forsythe procedure, the generalization of Box's method proposed by Brunner, Dette, and Munk, and the mixed-model procedure adjusted by the Kenward-Roger solution available in the SAS statistical package. With regard to robustness, the three approaches adequately controlled Type I error when the data were generated from symmetric distributions; however, this study's results indicate that, when the data were extracted from asymmetric distributions, the modified Brown-Forsythe approach controlled the Type I error slightly better than the other procedures. With regard to sensitivity, the higher power rates were obtained when the analyses were done with the MIXED procedure of the SAS program. Furthermore, results also identified that, when the data were generated from symmetric distributions, little power was sacrificed by using the generalization of Box's method in place of the modified Brown-Forsythe procedure.     

Generating Multivariate Ordinal Data via Entropy Principles

When conducting robustness research where the focus of attention is on the impact of non-normality, the marginal skewness and kurtosis are often used to set the degree of non-normality. Monte Carlo methods are commonly applied to conduct this type of research by simulating data from distributions with skewness and kurtosis constrained to pre-specified values. Although several procedures have been proposed to simulate data from distributions with these constraints, no corresponding procedures have been applied for discrete distributions. In this paper, we present two procedures based on the principles of maximum entropy and minimum cross-entropy to estimate the multivariate observed ordinal distributions with constraints on skewness and kurtosis. For these procedures, the correlation matrix of the observed variables is not specified but depends on the relationships between the latent response variables. With the estimated distributions, researchers can study robustness not only focusing on the levels of non-normality but also on the variations in the distribution shapes. A simulation study demonstrates that these procedures yield excellent agreement between specified parameters and those of estimated distributions. A robustness study concerning the effect of distribution shape in the context of confirmatory factor analysis shows that shape can affect the robust \(\chi ^2\) and robust fit indices, especially when the sample size is small, the data are severely non-normal, and the fitted model is complex.     

Simulating correlated multivariate nonnormal distributions: Extending the fleishman power method

A procedure for generating multivariate nonnormal distributions is proposed. Our procedure generates average values of intercorrelations much closer to population parameters than competing procedures for skewed and/or heavy tailed distributions and for small sample sizes. Also, it eliminates the necessity of conducting a factorization procedure on the population correlation matrix that underlies the random deviates, and it is simpler to code in a programming language (e.g., FORTRAN). Numerical examples demonstrating the procedures are given. Monte Carlo results indicate our procedure yields excellent agreement between population parameters and average values of intercorrelation, skew, and kurtosis.     

Robust step‐down tests for multivariate independent group designs

A composite step‐down procedure, in which a set of step‐down tests are summarized collectively with Fisher's combination statistic, was considered to test for multivariate mean equality in two‐group designs. An approximate degrees of freedom (ADF) composite procedure based on trimmed/Winsorized estimators and a non‐pooled estimate of error variance is proposed, and compared to a composite procedure based on trimmed/Winsorized estimators and a pooled estimate of error variance. The step‐down procedures were also compared to Hotelling's T² and Johansen's ADF global procedure based on trimmed estimators in a simulation study. Type I error rates of the pooled step‐down procedure were sensitive to covariance heterogeneity in unbalanced designs; error rates were similar to those of Hotelling's T² across all of the investigated conditions. Type I error rates of the ADF composite step‐down procedure were insensitive to covariance heterogeneity and less sensitive to the number of dependent variables when sample size was small than error rates of Johansen's test. The ADF composite step‐down procedure is recommended for testing hypotheses of mean equality in two‐group designs except when the data are sampled from populations with different degrees of multivariate skewness.     

Estimating the mean effect size in meta-analysis: Bias,precision, and mean squared error of different weighting methods

Although use of the standardized mean difference in meta-analysis is appealing for several reasons, there are some drawbacks. In this article, we focus on the following problem: that a precision-weighted mean of the observed effect sizes results in a biased estimate of the mean standardized mean difference. This bias is due to the fact that the weight given to an observed effect size depends on this observed effect size. In order to eliminate the bias, Hedges and Olkin (1985) proposed using the mean effect size estimate to calculate the weights. In the article, we propose a third alternative for calculating the weights: using empirical Bayes estimates of the effect sizes. In a simulation study, these three approaches are compared. The mean squared error (MSE) is used as the criterion by which to evaluate the resulting estimates of the mean effect size. For a meta-analytic dataset with a small number of studies, theMSE is usually smallest when the ordinary procedure is used, whereas for a moderate or large number of studies, the procedures yielding the best results are the empirical Bayes procedure and the procedure of Hedges and Olkin, respectively.     

Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review

The authors conducted a 30-year review (1969-1998) of the size of moderating effects of categorical variables as assessed using multiple regression. The median observed effect size (f(2)) is only .002, but 72% of the moderator tests reviewed had power of .80 or greater to detect a targeted effect conventionally defined as small. Results suggest the need to minimize the influence of artifacts that produce a downward bias in the observed effect size and put into question the use of conventional definitions of moderating effect sizes. As long as an effect has a meaningful impact, the authors advise researchers to conduct a power analysis and plan future research designs on the basis of smaller and more realistic targeted effect sizes.     

A meta-analysis of the n-back task while driving and its effects on cognitive workload

This study investigates the effect of the n-back task on cognitive workload while driving. Results of 20 studies with over 800 participants in total show a moderate to high mean effect size. That means the n-back task varies cognitive load while driving in a substantial matter. Further analysis reveals several moderator variables: experiments conducted in a driving simulator showed larger effect sizes than on-road studies. This effect decreases with increasing driving simulator fidelity. Furthermore, the specific driving task assignment moderates the effect: lane change task scenarios result in higher effects than other situations. Regarding the different measurement methods of cognitive workload, subjective questionnaires seem to have very high sensitivity. In contrast, n-back performance measures, detection response task measures, and physiological measurements result in moderate effect sizes, and driving performance measures show reduced sensitivity. Regarding different implementations of the n-back task itself, surprisingly, no moderators are found. Overall, the findings highlight the suitability of the n-back task as a method of inducing cognitive load in transportation research. The moderator analysis gives an overview of different methodological designs and how these designs will affect effect sizes.     

Reconciling the Two Disciplines of Organisational Science: A Comparison of Findings from Lab and Field Research

A strong preference for field research exists in the organisational sciences. However, it is unclear whether or under what conditions this is warranted. To examine this issue we conducted a second‐order meta‐analysis of 203 lab‐field pairs of meta‐analytic effects representing a diverse range of work‐related relationships. As expected, results showed a larger effect for lab (r = .25) than for field research (r = .14). However, the correspondence between the rank‐order of effect sizes for relationships assessed in lab settings and matched effects assessed in field settings was weaker (r = .61) than previous estimates from related areas of research. Moderators of lab‐field effect size magnitude and rank‐order correspondence were tested. Effect size magnitudes from the lab and field were most similar when lab studies used correlational designs, when using psychological state and trait (as opposed to demographic or workplace characteristic) variables as predictors, and when assessing attitudinal outcomes. Lab–field rank‐order correspondence was strongest when testing psychological state and workplace characteristic predictors and when assessing attitudinal and decisional outcomes. Findings offer recommendations for interpreting primary lab and field effects and inform evaluations of “when” findings from lab and field studies are likely to align.     

The use of extreme groups in assessing relationships

The relationship between variables in applied and experimental research is often investigated by the use of extreme (i.e., upper and lower) groups. Earlier analytical work has demonstrated that the extreme groups procedure is more powerful than the standard correlational approach for some values of the correlation and extreme group size. The present article provides methods for using the covariance information that is usually discarded in the classical extreme groups approach. Essentially, then, the new procedure combines the extreme groups approach and the correlational approach. Consequently, it includes the advantages of each and is shown to be more powerful than either approach used alone.     

Effect sizes in ANCOVA and difference-in-differences designs

It is common practice in both randomized and quasi-experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre-test, for example, can reduce both the standard error of this estimate and—in non-randomized designs—its bias. At the same time, it is also standard to report the effect of an intervention in standardized effect size units, thereby making it comparable to other interventions and studies. Curiously, the estimation of this effect size, including covariate adjustment, has received little attention. In this article, we provide a framework for defining effect sizes in designs with a pre-test (e.g., difference-in-differences and analysis of covariance) and propose estimators of those effect sizes. The estimators and approximations to their sampling distributions are evaluated using a simulation study and then demonstrated using an example from published data.     

Confidence intervals and sample size calculations for the weighted eta-squared effect sizes in one-way heteroscedastic ANOVA

Effect size reporting and interpreting practices have been extensively recommended in academic journals when primary outcomes of all empirical studies have been analyzed. This article presents an alternative approach to constructing confidence intervals of the weighted eta-squared effect size within the context of one-way heteroscedastic ANOVA models. It is shown that the proposed interval procedure has advantages over an existing method in its theoretical justification, computational simplicity, and numerical performance. For design planning, the corresponding sample size procedures for precise interval estimation of the weighted eta-squared association measure are also delineated. Specifically, the developed formulas compute the necessary sample sizes with respect to the considerations of expected confidence interval width and tolerance probability of interval width within a designated value. Supplementary computer programs are provided to aid the implementation of the suggested techniques in practical applications of ANOVA designs when the assumption of homogeneous variances is not tenable.     

Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit?

Inconsistencies in the research findings on F-test robustness to variance heterogeneity could be related to the lack of a standard criterion to assess robustness or to the different measures used to quantify heterogeneity. In the present paper we use Monte Carlo simulation to systematically examine the Type I error rate of F-test under heterogeneity. One-way, balanced, and unbalanced designs with monotonic patterns of variance were considered. Variance ratio (VR) was used as a measure of heterogeneity (1.5, 1.6, 1.7, 1.8, 2, 3, 5, and 9), the coefficient of sample size variation as a measure of inequality between group sizes (0.16, 0.33, and 0.50), and the correlation between variance and group size as an indicator of the pairing between them (1, .50, 0, ?.50, and ?1). Overall, the results suggest that in terms of Type I error a VR above 1.5 may be established as a rule of thumb for considering a potential threat to F-test robustness under heterogeneity with unequal sample sizes.