What Causes the Mean Bias of the Likelihood Ratio Statistic with Many Variables?

Abstract

Survey data often contain many variables. Structural equation modeling (SEM) is commonly used in analyzing such data. However, conventional SEM methods are not crafted to handle data with a large number of variables (p). A large p can cause T_ml, the most widely used likelihood ratio statistic, to depart drastically from the assumed chi-square distribution even with normally distributed data and a relatively large sample size N. A key element affecting this behavior of T_ml is its mean bias. The focus of this article is to determine the cause of the bias. To this end, empirical means of T_ml via Monte Carlo simulation are used to obtain the empirical bias. The most effective predictors of the mean bias are subsequently identified and their predictive utility examined. The results are further used to predict type I errors of T_ml. The article also illustrates how to use the obtained results to determine the required sample size for T_ml to behave reasonably well. A real data example is presented to show the effect of the mean bias on model inference as well as how to correct the bias in practice.     

Mean Comparison: Manifest Variable Versus Latent Variable

Mean comparisons are of great importance in the application of statistics. Procedures for mean comparison with manifest variables have been well studied. However, few rigorous studies have been conducted on mean comparisons with latent variables, although the methodology has been widely used and documented. This paper studies the commonly used statistics in latent variable mean modeling and compares them with parallel manifest variable statistics. Our results indicate that, under certain conditions, the likelihood ratio and Wald statistics used for latent mean comparisons do not always have greater power than the Hotelling T² statistics used for manifest mean comparisons. The noncentrality parameter corresponding to the T² statistic can be much greater than those corresponding to the likelihood ratio and Wald statistics, which we find to be different from those provided in the literature. Under a fixed alternative hypothesis, our results also indicate that the likelihood ratio statistic can be stochastically much greater than the corresponding Wald statistic. The robustness property of each statistic is also explored when the model is misspecified or when data are nonnormally distributed. Recommendations and advice are provided for the use of each statistic. The research was supported by NSF grant DMS-0437167 and Grant DA01070 from the National Institute on Drug Abuse. We would like to thank three referees for suggestions that helped in improving the paper.     

Correcting Too Much or Too Little? The Performance of Three Chi-Square Corrections

This simulation study investigates the performance of three test statistics, T₁, T₂, and T₃, used to evaluate structural equation model fit under non normal data conditions. T₁ is the well-known mean-adjusted statistic of Satorra and Bentler. T₂ is the mean-and-variance adjusted statistic of Sattertwaithe type where the degrees of freedom is manipulated. T₃ is a recently proposed version of T₂ that does not manipulate degrees of freedom. Discrepancies between these statistics and their nominal chi-square distribution in terms of errors of Type I and Type II are investigated. All statistics are shown to be sensitive to increasing kurtosis in the data, with Type I error rates often far off the nominal level. Under excess kurtosis true models are generally over-rejected by T₁ and under-rejected by T₂ and T₃, which have similar performance in all conditions. Under misspecification there is a loss of power with increasing kurtosis, especially for T₂ and T₃. The coefficient of variation of the nonzero eigenvalues of a certain matrix is shown to be a reliable indicator for the adequacy of these statistics.     

Evaluating the robustness of repeated measures analyses: The case of small sample sizes and nonnormal data

Repeated measures analyses of variance are the method of choice in many studies from experimental psychology and the neurosciences. Data from these fields are often characterized by small sample sizes, high numbers of factor levels of the within-subjects factor(s), and nonnormally distributed response variables such as response times. For a design with a single within-subjects factor, we investigated Type I error control in univariate tests with corrected degrees of freedom, the multivariate approach, and a mixed-model (multilevel) approach (SAS PROC MIXED) with Kenward–Roger’s adjusted degrees of freedom. We simulated multivariate normal and nonnormal distributions with varied population variance–covariance structures (spherical and nonspherical), sample sizes (N), and numbers of factor levels (K). For normally distributed data, as expected, the univariate approach with Huynh–Feldt correction controlled the Type I error rate with only very few exceptions, even if samples sizes as low as three were combined with high numbers of factor levels. The multivariate approach also controlled the Type I error rate, but it requires N ≥ K. PROC MIXED often showed acceptable control of the Type I error rate for normal data, but it also produced several liberal or conservative results. For nonnormal data, all of the procedures showed clear deviations from the nominal Type I error rate in many conditions, even for sample sizes greater than 50. Thus, none of these approaches can be considered robust if the response variable is nonnormally distributed. The results indicate that both the variance heterogeneity and covariance heterogeneity of the population covariance matrices affect the error rates.     

A scaled difference chi-square test statistic for moment structure analysis

A family of scaling corrections aimed to improve the chi-square approximation of goodness-of-fit test statistics in small samples, large models, and nonnormal data was proposed in Satorra and Bentler (1994). For structural equations models, Satorra-Bentler's (SB) scaling corrections are available in standard computer software. Often, however, the interest is not on the overall fit of a model, but on a test of the restrictions that a null model sayM ₀ implies on a less restricted oneM ₁. IfT ₀ andT ₁ denote the goodness-of-fit test statistics associated toM ₀ andM ₁, respectively, then typically the differenceT _d =T ₀−T ₁ is used as a chi-square test statistic with degrees of freedom equal to the difference on the number of independent parameters estimated under the modelsM ₀ andM ₁. As in the case of the goodness-of-fit test, it is of interest to scale the statisticT _d in order to improve its chi-square approximation in realistic, that is, nonasymptotic and nonormal, applications. In a recent paper, Satorra (2000) shows that the difference between two SB scaled test statistics for overall model fit does not yield the correct SB scaled difference test statistic. Satorra developed an expression that permits scaling the difference test statistic, but his formula has some practical limitations, since it requires heavy computations that are not available in standard computer software. The purpose of the present paper is to provide an easy way to compute the scaled difference chi-square statistic from the scaled goodness-of-fit test statistics of modelsM ₀ andM ₁. A Monte Carlo study is provided to illustrate the performance of the competing statistics. This research was supported by the Spanish grants PB96-0300 and BEC2000-0983, and USPHS grants DA00017 and DA01070.     

Evaluating Structural Equation Models for Categorical Outcomes: A New Test Statistic and a Practical Challenge of Interpretation

This research is concerned with two topics in assessing model fit for categorical data analysis. The first topic involves the application of a limited-information overall test, introduced in the item response theory literature, to structural equation modeling (SEM) of categorical outcome variables. Most popular SEM test statistics assess how well the model reproduces estimated polychoric correlations. In contrast, limited-information test statistics assess how well the underlying categorical data are reproduced. Here, the recently introduced C₂ statistic of Cai and Monroe (2014) is applied. The second topic concerns how the root mean square error of approximation (RMSEA) fit index can be affected by the number of categories in the outcome variable. This relationship creates challenges for interpreting RMSEA. While the two topics initially appear unrelated, they may conveniently be studied in tandem since RMSEA is based on an overall test statistic, such as C₂. The results are illustrated with an empirical application to data from a large-scale educational survey.     

Four improved statistics for contrasting means by correcting skewness and kurtosis

This paper is concerned with removing the influence of non‐normality in the classical t‐statistic for contrasting means. Using higher‐order expansion to quantify the effect of non‐normality, four corrected statistics are provided. Two aim to correct the mean bias and two to correct the overall distribution. The classical t‐statistic is also robust against non‐normality when the observed variables satisfy certain structures. A special case is when the marginal distributions of the contrast are independent and identically distributed.     

A note on moran's measure of multiple rank correlation

A measure of multiple rank correlation,T _y.12 ², is proposed for the situation with no tied observations in the variables. The measure is a weighted average of two squared Kendall taus. It is shown thatT _y.12 ² is equivalent to a statistic previously proposed by Moran and thus a new interpretation is given to Moran's statistic.The author wishes to thank Nancy Anderson, Willard Larkin, and Kent Norman for their helpful comments.     

Two simple approximations to the distributions of quadratic forms

Many test statistics are asymptotically equivalent to quadratic forms of normal variables, which are further equivalent to with z_i being independent and following N(0,1). Two approximations to the distribution of T have been implemented in popular software and are widely used in evaluating various models. It is important to know how accurate these approximations are when compared to each other and to the exact distribution of T. The paper systematically studies the quality of the two approximations and examines the effect of the λ_i and the degrees of freedom d by analysis and Monte Carlo. The results imply that the adjusted distribution for T can be as good as knowing its exact distribution. When the coefficient of variation of the λ_i is small, the rescaled statistic is also adequate for practical model inference. But comparing T_R against will inflate type I errors when substantial differences exist among the λ_i, especially, when d is also large.     

Ensuring Positiveness of the Scaled Difference Chi-square Test Statistic

A scaled difference test statistic [(T)\tilde]_d\tilde{T}{}_{d} that can be computed from standard software of structural equation models (SEM) by hand calculations was proposed in Satorra and Bentler (Psychometrika 66:507–514, 2001). The statistic [(T)\tilde]_d\tilde{T}_{d} is asymptotically equivalent to the scaled difference test statistic [`(T)]<sub>d</sub>\bar{T}_{d} introduced in Satorra (Innovations in Multivariate Statistical Analysis: A Festschrift for Heinz Neudecker, pp. 233–247, 2000), which requires more involved computations beyond standard output of SEM software. The test statistic [(T)\tilde]<sub>d</sub>\tilde{T}_{d} has been widely used in practice, but in some applications it is negative due to negativity of its associated scaling correction. Using the implicit function theorem, this note develops an improved scaling correction leading to a new scaled difference statistic [`(T)]_d\bar{T}_{d} that avoids negative chi-square values.     

Exploratory Factor Analysis With Small Sample Sizes

Exploratory factor analysis (EFA) is generally regarded as a technique for large sample sizes (N), with N = 50 as a reasonable absolute minimum. This study offers a comprehensive overview of the conditions in which EFA can yield good quality results for N below 50. Simulations were carried out to estimate the minimum required N for different levels of loadings (λ), number of factors (f), and number of variables (p) and to examine the extent to which a small N solution can sustain the presence of small distortions such as interfactor correlations, model error, secondary loadings, unequal loadings, and unequal p/f. Factor recovery was assessed in terms of pattern congruence coefficients, factor score correlations, Heywood cases, and the gap size between eigenvalues. A subsampling study was also conducted on a psychological dataset of individuals who filled in a Big Five Inventory via the Internet. Results showed that when data are well conditioned (i.e., high λ, low f, high p), EFA can yield reliable results for N well below 50, even in the presence of small distortions. Such conditions may be uncommon but should certainly not be ruled out in behavioral research data.     

Assessing Approximate Fit in Categorical Data Analysis

A family of Root Mean Square Error of Approximation (RMSEA) statistics is proposed for assessing the goodness of approximation in discrete multivariate analysis with applications to item response theory (IRT) models. The family includes RMSEAs to assess the approximation up to any level of association of the discrete variables. Two members of this family are RMSEA₂, which uses up to bivariate moments, and the full information RMSEA_n. The RMSEA₂ is estimated using the M₂ statistic of Maydeu-Olivares and Joe (2005, 2006), whereas for maximum likelihood estimation, RMSEA_n is estimated using Pearson's X² statistic. Using IRT models, we provide cutoff criteria of adequate, good, and excellent fit using the RMSEA₂. When the data are ordinal, we find a strong linear relationship between the RMSEA₂ and the Standardized Root Mean Squared Residual goodness-of-fit index. We are unable to offer cutoff criteria for the RMSEA_n as its population values decrease as the number of variables and categories increase.     

Equivalence of the statistics for replicability and area under the ROC curve

Two statistics, one recent and one well known, are shown to be equivalent. The recent statistic, p_rep, gives the probability that the sign of an experimental effect is replicable by an experiment of equal power. That statistic is equivalent to the well‐known measure for the area under a receiver operating characteristic (ROC) curve for statistical power against significance level. Both statistics can be seen as exemplifying the area theorem of psychophysics.     

Abstract: Evaluation of Test Statistics for Robust Structural Equation Modeling With Nonnormal Missing Data

Traditional structural equation modeling (SEM) techniques have trouble dealing with incomplete and/or nonnormal data that are often encountered in practice. Yuan and Zhang (2011a) developed a two-stage procedure for SEM to handle nonnormal missing data and proposed four test statistics for overall model evaluation. Although these statistics have been shown to work well with complete data, their performance for incomplete data has not been investigated in the context of robust statistics.

Focusing on a linear growth curve model, a systematic simulation study is conducted to evaluate the accuracy of the parameter estimates and the performance of five test statistics including the naive statistic derived from normal distribution based maximum likelihood (ML), the Satorra-Bentler scaled chi-square statistic (RML), the mean- and variance-adjusted chi-square statistic (AML), Yuan-Bentler residual-based test statistic (CRADF), and Yuan-Bentler residual-based F statistic (RF). Data are generated and analyzed in R using the package rsem (Yuan & Zhang, 2011b).

Based on the simulation study, we can observe the following: (a) The traditional normal distribution-based method cannot yield accurate parameter estimates for nonnormal data, whereas the robust method obtains much more accurate model parameter estimates for nonnormal data and performs almost as well as the normal distribution based method for normal distributed data. (b) With the increase of sample size, or the decrease of missing rate or the number of outliers, the parameter estimates are less biased and the empirical distributions of test statistics are closer to their nominal distributions. (c) The ML test statistic does not work well for nonnormal or missing data. (d) For nonnormal complete data, CRADF and RF work relatively better than RML and AML. (e) For missing completely at random (MCAR) missing data, in almost all the cases, RML and AML work better than CRADF and RF. (f) For nonnormal missing at random (MAR) missing data, CRADF and RF work better than AML. (g) The performance of the robust method does not seem to be influenced by the symmetry of outliers.     

Explanatory,Multilevel Person-Fit Analysis of Response Consistency on the Spielberger State-Trait Anxiety Inventory

Self-report measures are vulnerable to concentration and motivation problems, leading to responses that may be inconsistent with the respondent's latent trait value. We investigated response consistency in a sample (N = 860) of cardiac patients with an implantable cardioverter defibrillator and their partners who completed the Spielberger State-Trait Anxiety Inventory on five measurement occasions. For each occasion and for both the state and trait subscales, we used the l ^p _z person-fit statistic to assess response consistency. We used multilevel analysis to model the between-person and within-person differences in the repeated observations of response consistency using time-dependent (e.g., mood states) and time-invariant explanatory variables (e.g., demographic characteristics). Respondents with lower education, undergoing psychological treatment, and with more post-traumatic stress disorder symptoms tended to respond less consistently. The percentages of explained variance in response consistency were small. Hence, we conclude that the results give insight into the causes of response inconsistency but that the identified explanatory variables are of limited practical value for identifying respondents at risk of producing invalid test results. We discuss explanations for the small percentage of explained variance and suggest alternative methods for studying causes of response inconsistency.     

Significance Testing with No Alternative Hypothesis: A Measure of Surprise

A pure significance test would check the agreement of a statistical model with the observed data even when no alternative model was available. The paper proposes the use of a modified p-value to make such a test. The model will be rejected if something surprising is observed (relative to what else might have been observed). It is shown that the relation between this measure of surprise (the s-value) and the surprise indices of Weaver and Good is similar to the relationship between a p-value, a corresponding odds-ratio, and a logit or log-odds statistic. The s-value is always larger than the corresponding p-value, and is not uniformly distributed. Difficulties with the whole approach are discussed.     

Correcting an analysis of variance for clustering

A great deal of educational and social data arises from cluster sampling designs where clusters involve schools, classrooms, or communities. A mistake that is sometimes encountered in the analysis of such data is to ignore the effect of clustering and analyse the data as if it were based on a simple random sample. This typically leads to an overstatement of the precision of results and too liberal conclusions about precision and statistical significance of mean differences. This paper gives simple corrections to the test statistics that would be computed in an analysis of variance if clustering were (incorrectly) ignored. The corrections are multiplicative factors depending on the total sample size, the cluster size, and the intraclass correlation structure. For example, the corrected F statistic has Fisher's F distribution with reduced degrees of freedom. The corrected statistic reduces to the F statistic computed by ignoring clustering when the intraclass correlations are zero. It reduces to the F statistic computed using cluster means when the intraclass correlations are unity, and it is in between otherwise. A similar adjustment to the usual statistic for testing a linear contrast among group means is described.     

Two new test statistics for the rasch model

It is shown that the presently available statistical tests for the Rasch model are insensitive to violation of the unidimensionality axiom. Two new test statistics are presented. The first one,Q ₁, is sensitive to the same effects as the presently available statistics, but has some desirable properties of a nonstatistical nature. The second statistic,Q ₂, is sensitive to violation of local stochastic independence and unidimensionality and thus fills an existing gap.     

Identifying the Source of Misfit in Item Response Theory Models

When an item response theory model fails to fit adequately, the items for which the model provides a good fit and those for which it does not must be determined. To this end, we compare the performance of several fit statistics for item pairs with known asymptotic distributions under maximum likelihood estimation of the item parameters: (a) a mean and variance adjustment to bivariate Pearson's X², (b) a bivariate subtable analog to Reiser's (1996) overall goodness-of-fit test, (c) a z statistic for the bivariate residual cross product, and (d) Maydeu-Olivares and Joe's (2006) M₂ statistic applied to bivariate subtables. The unadjusted Pearson's X² with heuristically determined degrees of freedom is also included in the comparison. For binary and ordinal data, our simulation results suggest that the z statistic has the best Type I error and power behavior among all the statistics under investigation when the observed information matrix is used in its computation. However, if one has to use the cross-product information, the mean and variance adjusted X² is recommended. We illustrate the use of pairwise fit statistics in 2 real-data examples and discuss possible extensions of the current research in various directions.     

Opportunities and Issues in Modeling Intensive Longitudinal Data: Learning from the COGITO Project

Abstract

Technological developments increasingly permit the collection of longitudinal data sets in which the data structure contains a large number of participants N and a large number of measurement occasions T. Promising new dynamical systems approaches to the analysis of large N, large T data sets have been proposed that utilize both between-subjects and within-subjects information. The COGITO project, begun over a decade ago, is an early large N?=?204, large T?=?100 study that collected high quality cognitive and psychosocial data. In this introduction, I describe the COGITO project and conceptual and statistical issues that arise in the analysis of large N, large T data sets. I provide a brief overview of the five papers in the special section which include conceptual pieces, a didactic presentation of a dynamic structural equation approach, and papers reporting new statistical analyses of the COGITO data set to answer substantive questions. Although many challenges remain, these new approaches offer the promise of improving scientific inquiry in the behavioral sciences.