A note on statistical power in multi‐site randomized trials with multiple treatments at each site

We derive the statistical power functions in multi‐site randomized trials with multiple treatments at each site, using multi‐level modelling. An F statistic is used to test multiple parameters in the multi‐level model instead of the Wald chi square test as suggested in the current literature. The F statistic is shown to be more conservative than the Wald statistic in testing any overall treatment effect among the multiple study conditions. In addition, we improvise an easy way to estimate the non‐centrality parameters for the means comparison t‐tests and the F test, using Helmert contrast coding in the multi‐level model. The variance of treatment means, which is difficult to fathom but necessary for power analysis, is decomposed into intuitive simple effect sizes in the contrast tests. The method is exemplified by a multi‐site evaluation study of the behavioural interventions for cannabis dependence.     

The effect of nonindependence on significance testing in dyadic research

Relationship researchers regularly gather data from both members of the dyad, and these two scores are likely to be correlated. This nonindependence of observations can bias p values in significance testing if person is the unit in the statistical analysis. A method for determining how much bias results from dyadic interdependence is presented. Correction factors based on the degree of interdependence, design type, and the number of dyads are used to adjust the F statistic and its degrees of freedom to produce a corrected p value. Bias depends on the type of design and the degree of nonindependence, while the number of dyads in the study ordinarily has only a small effect on bias. Various strategies for controlling for nonindependence are briefly reviewed.     

Empirically Corrected Rescaled Statistics for SEM with Small N and Large p

Survey data often contain many variables. Structural equation modeling (SEM) is commonly used in analyzing such data. With typical nonnormally distributed data in practice, a rescaled statistic T_rml proposed by Satorra and Bentler was recommended in the literature of SEM. However, T_rml has been shown to be problematic when the sample size N is small and/or the number of variables p is large. There does not exist a reliable test statistic for SEM with small N or large p, especially with nonnormally distributed data. Following the principle of Bartlett correction, this article develops empirical corrections to T_rml so that the mean of the empirically corrected statistics approximately equals the degrees of freedom of the nominal chi-square distribution. Results show that empirically corrected statistics control type I errors reasonably well even when N is smaller than 2p, where T_rml may reject the correct model 100% even for normally distributed data. The application of the empirically corrected statistics is illustrated via a real data example.     

Improved statistics for contrasting means of two samples under non‐normality

This paper presents the asymptotic expansions of the distributions of the two‐sample t‐statistic and the Welch statistic, for testing the equality of the means of two independent populations under non‐normality. Unlike other approaches, we obtain the null distributions in terms of the distribution and density functions of the standard normal variable up to n^?1, where n is the pooled sample size. Based on these expansions, monotone transformations are employed to remove the higher‐order cumulant effect. We show that the new statistics can improve the precision of statistical inference to the level of o (n^?1). Numerical studies are carried out to demonstrate the performance of the improved statistics. Some general rules for practitioners are also recommended.     

Exact distributions of intraclass correlation and Cronbach's alpha with Gaussian data and general covariance

Intraclass correlation and Cronbach's alpha are widely used to describe reliability of tests and measurements. Even with Gaussian data, exact distributions are known only for compound symmetric covariance (equal variances and equal correlations). Recently, large sample Gaussian approximations were derived for the distribution functions. New exact results allow calculating the exact distribution function and other properties of intraclass correlation and Cronbach's alpha, for Gaussian data with any covariance pattern, not just compound symmetry. Probabilities are computed in terms of the distribution function of a weighted sum of independent chi-square random variables. NewF approximations for the distribution functions of intraclass correlation and Cronbach's alpha are much simpler and faster to compute than the exact forms. Assuming the covariance matrix is known, the approximations typically provide sufficient accuracy, even with as few as ten observations. Either the exact or approximate distributions may be used to create confidence intervals around an estimate of reliability. Monte Carlo simulations led to a number of conclusions. Correctly assuming that the covariance matrix is compound symmetric leads to accurate confidence intervals, as was expected from previously known results. However, assuming and estimating a general covariance matrix produces somewhat optimistically narrow confidence intervals with 10 observations. Increasing sample size to 100 gives essentially unbiased coverage. Incorrectly assuming compound symmetry leads to pessimistically large confidence intervals, with pessimism increasing with sample size. In contrast, incorrectly assuming general covariance introduces only a modest optimistic bias in small samples. Hence the new methods seem preferable for creating confidence intervals, except when compound symmetry definitely holds. An earlier version of this paper was submitted in partial fulfillment of the requirements for the M.S. in Biostatistics, and also summarized in a presentation at the meetings of the Eastern North American Region of the International Biometric Society in March, 2001. Kistner's work was supported in part by NIEHS training grant ES07018-24 and NCI program project grant P01 CA47 982-04. She gratefully acknowledges the inspiration of A. Calandra's “Scoring formulas and probability considerations” (Psychometrika, 6, 1–9). Muller's work supported in part by NCI program project grant P01 CA47 982-04.     

Stability analysis in K‐means clustering

This paper develops a new procedure, called stability analysis, for K‐means clustering. Instead of ignoring local optima and only considering the best solution found, this procedure takes advantage of additional information from a K‐means cluster analysis. The information from the locally optimal solutions is collected in an object by object co‐occurrence matrix. The co‐occurrence matrix is clustered and subsequently reordered by a steepest ascent quadratic assignment procedure to aid visual interpretation of the multidimensional cluster structure. Subsequently, measures are developed to determine the overall structure of a data set, the number of clusters and the multidimensional relationships between the clusters.     

When can we trust theF-approximation of the box-test?

Consider a multivariate context withp variates andk independent samples, each of sizen. To test equality of thek population covariance matrices, the likelihood ratio test is commonly employed. Box'sF-approximation to the null distribution of the test statistic can be used to computep-values, if sample sizes are not too small. It is suggested to regard theF-approximation as accurate if the sample sizesn are greater than or equal to 1+0.0613p ²+2.7265p-1.4182p ^0.5+0.235p ^1.4* In (k), for 5p30,k20.This research was supported by the Deutsche Forschungsgemeinschaft through Ste 405/2-1.     

Four improved statistics for contrasting means by correcting skewness and kurtosis

This paper is concerned with removing the influence of non‐normality in the classical t‐statistic for contrasting means. Using higher‐order expansion to quantify the effect of non‐normality, four corrected statistics are provided. Two aim to correct the mean bias and two to correct the overall distribution. The classical t‐statistic is also robust against non‐normality when the observed variables satisfy certain structures. A special case is when the marginal distributions of the contrast are independent and identically distributed.     

Imitation of synthesized English and nonEnglish vowels by children and adults

Imitations of 15 synthesized vowels, some like English vowels and some not, were obtained from nine adults and ten 6-year-old children. Estimates of the first three formant frequencies (F ₁ , F ₂ , and F ₃ ) were made from spectrograms of the vowel imitations. The reliability of reproduction was assessed by calculating standard deviations for five imitations each of ten of the synthetic stimuli. Generally, both the intrasubject and intersubject variabilities were greater for the children than for the adults. However, the differences in intrasubject variability between the two groups often were not much greater than the difference in measurement error which would be expected for voices of different fundamental frequencies. Subjects tended to reproduce the nonEnglish vowels less reliably than the English vowels, although the adults were less influenced by phonetic familiarity than were the children. Vowel familiarity appeared to be especially important for reliable reporoduction of the F ₂ frequency. Plotting of the imitation data for English vowels in a F ₁ –F ₂ plane with linear dimensions revealed a fairly systematic clustering for the four age-sex groups of men, women, boys, and girls, but the group clustering was not so systematic for the imitation data for the nonEnglish vowels.This work was supported by Public Health Service Research Grant NS-HD-12281.     

A cautionary note on using internal cross validation to select the number of clusters

A highly popular method for examining the stability of a data clustering is to split the data into two parts, cluster the observations in Part A, assign the objects in Part B to their nearest centroid in Part A, and then independently cluster the Part B objects. One then examines how close the two partitions are (say, by the Rand measure). Another proposal is to split the data into k parts, and see how their centroids cluster. By means of synthetic data analyses, we demonstrate that these approaches fail to identify the appropriate number of clusters, particularly as sample size becomes large and the variables exhibit higher correlations.The authors express their thanks to the Sol C. Snider Entrepreneurial Center, Wharton School, for support of this project.     

Effect of a Vocationally‐Focused Brief Cognitive Behavioural Intervention on Employment‐Related Outcomes for Individuals with Mood and Anxiety Disorders

Internet-delivered cognitive behavioural therapy (CBT) can be an effective method for treating major depression, but it often works best when therapist support is provided in the form of e-mail support or telephone calls. The authors investigated whether there were any intraclass correlations within therapists when delivering CBT for major depression via the Internet. They included data from two trials involving 10 therapists treating a total of 103 patients. The results of a nested one-way model in which participants were treated as raters for different therapists indicated that measures pertaining to symptom reductions (Beck Depression Inventory, Montgomery-Åsberg Depression Rating Scale–Self Report, and Beck Anxiety Inventory) did not support a clustering of data within therapists. However, the outcome on a secondary measure of life satisfaction (Quality of Life Inventory) yielded a significant intraclass correlation coefficient for therapists (r = .24, p = .001). The authors propose that text-based treatments are less sensitive to therapist effects when it comes to the primary symptom measures, but that treatment effects not directly targeted by the specific treatment program may be more dependent on the way the support is given and by whom (therapist effect). Limitations of the study are discussed.     

Constrained k‐means on cluster proportion and distances among clusters for longitudinal data analysis

Clustering individuals by measures of similarity or dissimilarity at trajectories of changes in longitudinal data enables determination of typical patterns of development and growth. The present research proposes a new constrained k‐means method with lower bound constraints on cluster proportions and distances among clusters at focused variables and time points to fulfill various needs in clustering longitudinal data. The method assumes a large number of clusters at the onset and iteratively deletes and combines clusters according to these constraints. An additional property of the proposed constrained k‐means includes direct estimation of the unknown number of clusters. Simulation results clearly show the usefulness of the method for extracting clusters in plausible, real‐life analysis including non‐normality within clusters, and the proposed algorithm works well and convergence of the estimates is satisfactory. An actual example using Japanese longitudinal data regarding sleep habits and mental health is presented to verify the utility of the proposed constrained k‐means.     

Mapping differential responses to cognitive training using machine learning

We used two simple unsupervised machine learning techniques to identify differential trajectories of change in children who undergo intensive working memory (WM) training. We used self‐organizing maps (SOMs)—a type of simple artificial neural network—to represent multivariate cognitive training data, and then tested whether the way tasks are represented changed as a result of training. The patterns of change we observed in the SOM weight matrices implied that the processes drawn upon to perform WM tasks changed following training. This was then combined with K‐means clustering to identify distinct groups of children who respond to the training in different ways. Firstly, the K‐means clustering was applied to an independent large sample (N = 616, M_age = 9.16 years, range = 5.16–17.91 years) to identify subgroups. We then allocated children who had been through cognitive training (N = 179, M_age = 9.00 years, range = 7.08–11.50 years) to these same four subgroups, both before and after their training. In doing so, we were able to map their improvement trajectories. Scores on a separate measure of fluid intelligence were predictive of a child's improvement trajectory. This paper provides an alternative approach to analysing cognitive training data that go beyond considering changes in individual tasks. This proof‐of‐principle demonstrates a potentially powerful way of distinguishing task‐specific from domain‐general changes following training and of establishing different profiles of response to training.     

Semantic clustering as a neuropsychological predictor for amnestic-MCI

ABSTRACT

Recent research has demonstrated that patients with Alzheimer's disease (AD) show deficits in semantic processing when compared to cognitively healthy individuals. This difference is thought to be attributed to losses in higher cortical systems that are predominantly associated with executive functioning. The first aim of the study will be to determine if differences in semantic clustering can accurately differentiate patients with amnestic mild cognitive impairment (aMCI) from cognitively normal (CN) individuals. The second aim will be to determine the extent to which semantic processing might be associated with executive functions. Data from 202 (134 CN, 68 aMCI) participants were analyzed to quantify differences in semantic clustering ratios on the HVLT-R. Study participants ages ranged from 51 to 87 with education ranging from 6 to 20 years. ANCOVA revealed statistically significant differences on semantic clustering ratios (p < .001). Moderate correlations between semantic clustering Category Fluency Test (r = .45) were also found. Statistically significant group differences were also present on Trails-B and WAIS-R Digit Symbol performance (p < .001). Overall, these data indicate that deficits in semantic clustering are present in aMCI patients.     

Multiple Imputation of Item Scores in Test and Questionnaire Data,and Influence on Psychometric Results

The performance of five simple multiple imputation methods for dealing with missing data were compared. In addition, random imputation and multivariate normal imputation were used as lower and upper benchmark, respectively. Test data were simulated and item scores were deleted such that they were either missing completely at random, missing at random, or not missing at random. Cronbach's alpha, Loevinger's scalability coefficient H, and the item cluster solution from Mokken scale analysis of the complete data were compared with the corresponding results based on the data including imputed scores. The multiple-imputation methods, two-way with normally distributed errors, corrected item-mean substitution with normally distributed errors, and response function, produced discrepancies in Cronbach's coefficient alpha, Loevinger's coefficient H, and the cluster solution from Mokken scale analysis, that were smaller than the discrepancies in upper benchmark multivariate normal imputation.     

On the Computation of the RMSEA and CFI from the Mean-And-Variance Corrected Test Statistic with Nonnormal Data in SEM

A new type of nonnormality correction to the RMSEA has recently been developed, which has several advantages over existing corrections. In particular, the new correction adjusts the sample estimate of the RMSEA for the inflation due to nonnormality, while leaving its population value unchanged, so that established cutoff criteria can still be used to judge the degree of approximate fit. A confidence interval (CI) for the new robust RMSEA based on the mean-corrected (“Satorra-Bentler”) test statistic has also been proposed. Follow up work has provided the same type of nonnormality correction for the CFI (Brosseau-Liard & Savalei, 2014). These developments have recently been implemented in lavaan. This note has three goals: a) to show how to compute the new robust RMSEA and CFI from the mean-and-variance corrected test statistic; b) to offer a new CI for the robust RMSEA based on the mean-and-variance corrected test statistic; and c) to caution that the logic of the new nonnormality corrections to RMSEA and CFI is most appropriate for the maximum likelihood (ML) estimator, and cannot easily be generalized to the most commonly used categorical data estimators.     

Meta-analysis of the relationship between intelligence and inspection time

This study replicated and extended Kranzler and Jensen's [Intelligence 13 (1989) 329] meta-analysis of the relationship between inspection time (IT) and intelligence (IQ). Separate meta-analyses were conducted on obtained correlations (r's) between IT and general IQ for the total sample and for studies using samples of adults and children. Two new meta-analyses were also conducted. The first compared the IT–IQ r between visual and auditory IT; the second compared the r between strategy users and nonusers. For the total sample (N>4100), the r was −.51 after correction for artifactual effects (−.30 prior to correction). No statistically significant difference was observed between the mean corrected r of −.51 for adults and −.44 for children. The mean corrected r for visual and auditory IT measures were −.49 and −.58, respectively, suggesting that the relationship between IT and IQ is comparable across type of IT task. The mean corrected r of −.77 for strategy nonusers was statistically significantly higher than the r of −.60 for strategy users. Implications of these findings for future research are discussed.     

Defineability as an index of word meaning

A sample of 277 nouns was scaled for rated defineability and rated pronunciability. Intercorrelations were computed among the following variables: defineability (D), pronunciability (P), imagery (I), concreteness (C), associative meaningfulness (m), familiarity (F), and Thorndike-Lorge frequencies (TLF). RatedD was substantially related to all the other variables, notably to ratedI. Correlations were highest amongD, I, C, andm, on the one hand, and amongF, TLF, andP, on the other. The pattern of correlations was interpreted in terms of Paivio's (1970) distinction between higher-order and lower-order meaning. Two experiments were conducted to test the relationship between the word attributes and recall of word labels with dictionary definitions provided as retrieval cues. The positive effect ofD on label retrieval was independent of theI values of the defined words but the effectiveness ofI on retrieval was dependent on word defineability.This research was supported in part by grants to Dr. A. Paivio from the National Research Council of Canada (Grant 0087).     

DETECTING MODERATORS WITH META-ANALYSIS: AN EVALUATION AND COMPARISON OF TECHNIQUES

The present study evaluated accuracy levels of seven techniques for ascertaining, after a meta-analysis, whether moderators are present or not: (a) SH-75% rule for uncorrected r , (b) SH-75% rule for corrected r, (c) SH-95% rule for uncorrected r , (d) SH-95% rule for corrected r, (e) the Q statistic; (f) inclusion of 0 in the credibility interval around the corrected r, and (g) the size of the interval. Using Monte Carlo data which were defined by various parameters including sample based artifacts, comparisons of Type I and power determinations were generated. Findings showed that when differences between population correlations were small, power levels for all techniques were relatively low. Overall, SH rules and the Q statistic had greater power but higher Type I error rate than credibility intervals. Because of the high Type I error rate associated with both of the SH-95% techniques and the low power found with the credibility intervals, the SH-75% rules and Q statistic are recommended. Limitations and some practical implications for the findings are discussed.