Evaluating Small Sample Approaches for Model Test Statistics in Structural Equation Modeling

Through Monte Carlo simulation, small sample methods for evaluating overall data-model fit in structural equation modeling were explored. Type I error behavior and power were examined using maximum likelihood (ML), Satorra-Bentler scaled and adjusted (SB; Satorra & Bentler, 1988, 1994), residual-based (Browne, 1984), and asymptotically distribution free (ADF; Browne, 1982, 1984) test statistics. To accommodate small sample sizes the ML and SB statistics were adjusted using a k-factor correction (Bartlett, 1950); the residual-based and ADF statistics were corrected using modified x² and F statistics (Yuan & Bentler, 1998, 1999). Design characteristics include model type and complexity, ratio of sample size to number of estimated parameters, and distributional form. The k-factor-corrected SB scaled test statistic was especially stable at small sample sizes with both normal and nonnormal data. Methodologists are encouraged to investigate its behavior under a wider variety of models and distributional forms.     

Abstract: Evaluation of Test Statistics for Robust Structural Equation Modeling With Nonnormal Missing Data

Traditional structural equation modeling (SEM) techniques have trouble dealing with incomplete and/or nonnormal data that are often encountered in practice. Yuan and Zhang (2011a) developed a two-stage procedure for SEM to handle nonnormal missing data and proposed four test statistics for overall model evaluation. Although these statistics have been shown to work well with complete data, their performance for incomplete data has not been investigated in the context of robust statistics.

Focusing on a linear growth curve model, a systematic simulation study is conducted to evaluate the accuracy of the parameter estimates and the performance of five test statistics including the naive statistic derived from normal distribution based maximum likelihood (ML), the Satorra-Bentler scaled chi-square statistic (RML), the mean- and variance-adjusted chi-square statistic (AML), Yuan-Bentler residual-based test statistic (CRADF), and Yuan-Bentler residual-based F statistic (RF). Data are generated and analyzed in R using the package rsem (Yuan & Zhang, 2011b).

Based on the simulation study, we can observe the following: (a) The traditional normal distribution-based method cannot yield accurate parameter estimates for nonnormal data, whereas the robust method obtains much more accurate model parameter estimates for nonnormal data and performs almost as well as the normal distribution based method for normal distributed data. (b) With the increase of sample size, or the decrease of missing rate or the number of outliers, the parameter estimates are less biased and the empirical distributions of test statistics are closer to their nominal distributions. (c) The ML test statistic does not work well for nonnormal or missing data. (d) For nonnormal complete data, CRADF and RF work relatively better than RML and AML. (e) For missing completely at random (MCAR) missing data, in almost all the cases, RML and AML work better than CRADF and RF. (f) For nonnormal missing at random (MAR) missing data, CRADF and RF work better than AML. (g) The performance of the robust method does not seem to be influenced by the symmetry of outliers.     

Testing latent longitudinal models of social ties and depression among the elderly: a comparison of distribution-free and maximum likelihood estimates with nonnormal data.

Using a latent-variable modeling approach, relationships between social ties and depression were studied in a sample of 201 older adults. Both positive and negative ties were related to concurrent depression, whereas only negative ties predicted future depression. Nonnormally distributed scores were observed for several variables, and results based on maximum likelihood (ML), which assumes multivariate normality, were compared with those obtained using Browne's (1982, 1984) arbitrary distribution function (ADF) estimator for nonnormal variables. Inappropriate use of ML with nonnormal data yielded model chi-square values that were too large and standard errors that were too small. ML also failed to detect the over-time effect of negative ties on depression. The results suggest that the negative functions of social networks may causally influence depression and illustrate the need to test distributional assumptions when estimating latent-variable models.     

Identifying Variables Responsible for Data not Missing at Random

When data are not missing at random (NMAR), maximum likelihood (ML) procedure will not generate consistent parameter estimates unless the missing data mechanism is correctly modeled. Understanding NMAR mechanism in a data set would allow one to better use the ML methodology. A survey or questionnaire may contain many items; certain items may be responsible for NMAR values in other items. The paper develops statistical procedures to identify the responsible items. By comparing ML estimates (MLE), statistics are developed to test whether the MLEs are changed when excluding items. The items that cause a significant change of the MLEs are responsible for the NMAR mechanism. Normal distribution is used for obtaining the MLEs; a sandwich-type covariance matrix is used to account for distribution violations. The class of nonnormal distributions within which the procedure is valid is provided. Both saturated and structural models are considered. Effect sizes are also defined and studied. The results indicate that more missing data in a sample does not necessarily imply more significant test statistics due to smaller effect sizes. Knowing the true population means and covariances or the parameter values in structural equation models may not make things easier either. The research was supported by NSF grant DMS04-37167, the James McKeen Cattell Fund.     

Evaluation of the Bayesian and Maximum Likelihood Approaches in Analyzing Structural Equation Models with Small Sample Sizes

The main objective of this article is to investigate the empirical performances of the Bayesian approach in analyzing structural equation models with small sample sizes. The traditional maximum likelihood (ML) is also included for comparison. In the context of a confirmatory factor analysis model and a structural equation model, simulation studies are conducted with the different magnitudes of parameters and sample sizes n = da, where d = 2, 3, 4 and 5, and a is the number of unknown parameters. The performances are evaluated in terms of the goodness-of-fit statistics, and various measures on the accuracy of the estimates. The conclusion is: for data that are normally distributed, the Bayesian approach can be used with small sample sizes, whilst ML cannot.     

Higher-order approximations to the distributions of fit indexes under fixed alternatives in structural equation models

Higher-order approximations to the distributions of fit indexes for structural equation models under fixed alternative hypotheses are obtained in nonnormal samples as well as normal ones. The fit indexes include the normal-theory likelihood ratio chi-square statistic for a posited model, the corresponding statistic for the baseline model of uncorrelated observed variables, and various fit indexes as functions of these two statistics. The approximations are given by the Edgeworth expansions for the distributions of the fit indexes under arbitrary distributions. Numerical examples in normal and nonnormal samples with the asymptotic and simulated distributions of the fit indexes show the relative inappropriateness of the normal-theory approximation using noncentral chi-square distributions. A simulation for the confidence intervals of the fit indexes based on the normal-theory Studentized estimators under normality with a small sample size indicates an advantage for the approximation by the Cornish–Fisher expansion over those by the noncentral chi-square distribution and the asymptotic normality. The author is indebted to the reviewers for their comments and suggestions, which have led to the improvement of the previous versions of this paper. This work was partially supported by Grant-in-Aid for Scientific Research from the Japanese Ministry of Education, Culture, Sports, Science, and Technology.     

Generating Nonnormal Multivariate Data Using Copulas: Applications to SEM

This article develops a procedure based on copulas to simulate multivariate nonnormal data that satisfy a prespecified variance-covariance matrix. The covariance matrix used can comply with a specific moment structure form (e.g., a factor analysis or a general structural equation model). Thus, the method is particularly useful for Monte Carlo evaluation of structural equation models within the context of nonnormal data. The new procedure for nonnormal data simulation is theoretically described and also implemented in the widely used R environment. The quality of the method is assessed by Monte Carlo simulations. A 1-sample test on the observed covariance matrix based on the copula methodology is proposed. This new test for evaluating the quality of a simulation is defined through a particular structural model specification and is robust against normality violations.     

A New Procedure to Test Mediation With Missing Data Through Nonparametric Bootstrapping and Multiple Imputation

This article proposes a new procedure to test mediation with the presence of missing data by combining nonparametric bootstrapping with multiple imputation (MI). This procedure performs MI first and then bootstrapping for each imputed data set. The proposed procedure is more computationally efficient than the procedure that performs bootstrapping first and then MI for each bootstrap sample. The validity of the procedure is evaluated using a simulation study under different sample size, missing data mechanism, missing data proportion, and shape of distribution conditions. The result suggests that the proposed procedure performs comparably to the procedure that combines bootstrapping with full information maximum likelihood under most conditions. However, caution needs to be taken when using this procedure to handle missing not-at-random or nonnormal data.     

当结构假设和分布假设不满足时的验证性因子分析：稳健极大似然法估计和贝叶斯估计的比较研究

结构方程模型已被广泛应用于心理学、教育学、以及社会科学领域的统计分析中。结构方程模型分析中最常用的估计方法是基于正 态分布的估计量,比如极大似然估计法。这些方法需要满足两个假设。第一, 理论模型必须正确地反映变量与变量之间的关系,称为结构假 设。第二,数据必须符合多元正态分布,称为分布假设。如果这些假设不满足,基于正态分布的估计量就有可能导致不正确的卡方指数、不 正确的拟合度、以及有偏差的参数估计和参数估计的标准误。在实际应用中,几乎所有的理论模型都不能准确地解释变量与变量之间的关系, 数据也常常呈非多元正态分布。为此,一些新的估计方法得以发展。这些方法要么在理论上不要求数据呈多元正态分布,要么对因数据呈非 正态分布而导致的不正确结果进行纠正。当前较为流行的两种方法是稳健极大似然估计和贝叶斯估计。稳健极大似然估计是应用 Satorra and Bentler (1994) 的方法对不正确的卡方指数和参数估计的标准误进行调整,而参数估计和用极大似然方法得出的完全等同。贝叶斯估计方法则是 基于贝叶斯定理,其要点是：参数的后验分布是由参数的先验分布和数据似然值相乘而得来。后验分布常用马尔科夫蒙特卡洛算法来进行模拟。 对于稳健极大似然估计和贝叶斯估计这两种方法之间的优劣比较,先前的研究只局限于理论模型是正确的情境。而本研究则着重于理论模型 是错误的情境,同时也考虑到数据呈非正态分布的情境。本研究所采用的模型是验证性因子模型,数据全部由计算机模拟而来。数据的生成 取决于三个因素：8 类因子结构,3 种变量分布,和3 组样本量。这三个因素产生72 个模拟条件（72=8x3x3）。每个模拟条件下生成2000 个 数据组,每个数据组都拟合两个模型,一个是正确模型、一个是错误模型。每个模型都用两种估计方法来拟合：稳健极大似然估计法和贝叶 斯估计方法。贝叶斯估计方法中所使用的先验分布是无信息先验分布。结果分析主要着重于模型拒绝率、拟合度、参数估计、和参数估计的 标准误。研究的结果表明：在样本量充足的情况下,两种方法得出的参数估计非常相似。当数据呈非正态分布时,贝叶斯估计法比稳健极大 似然估计法更好地拒绝错误模型。但是,当样本量不足且数据呈正态分布时,贝叶斯估计在拒绝错误模型和参数估计上几乎没有优势,甚至 在一些条件下,比稳健极大似然法要差。     

Sample size determination for a matched-pairs study with incomplete data using exact approach

This research was motivated by a clinical trial design for a cognitive study. The pilot study was a matched-pairs design where some data are missing, specifically the missing data coming at the end of the study. Existing approaches to determine sample size are all based on asymptotic approaches (e.g., the generalized estimating equation (GEE) approach). When the sample size in a clinical trial is small to medium, these asymptotic approaches may not be appropriate for use due to the unsatisfactory Type I and II error rates. For this reason, we consider the exact unconditional approach to compute the sample size for a matched-pairs study with incomplete data. Recommendations are made for each possible missingness pattern by comparing the exact sample sizes based on three commonly used test statistics, with the existing sample size calculation based on the GEE approach. An example from a real surgeon-reviewers study is used to illustrate the application of the exact sample size calculation in study designs.     

Statistical inference based on latent ability estimates

The quality of approximations to first and second order moments (e.g., statistics like means, variances, regression coefficients) based on latent ability estimates is being discussed. The ability estimates are obtained using either the Rasch, or the two-parameter logistic model. Straightforward use of such statistics to make inferences with respect to true latent ability is not recommended, unless we account for the fact that the basic quantities are estimates. In this paper true score theory is used to account for the latter; the counterpart of observed/true score being estimated/true latent ability. It is shown that statistics based on the true score theory are virtually unbiased if the number of items presented to each examinee is larger than fifteen. Three types of estimators are compared: maximum likelihood, weighted maximum likelihood, and Bayes modal. Furthermore, the (dis)advantages of the true score method and direct modeling of latent ability is discussed.     

Heteroscedastic one‐factor models and marginal maximum likelihood estimation

In the present paper, a general class of heteroscedastic one‐factor models is considered. In these models, the residual variances of the observed scores are explicitly modelled as parametric functions of the one‐dimensional factor score. A marginal maximum likelihood procedure for parameter estimation is proposed under both the assumption of multivariate normality of the observed scores conditional on the single common factor score and the assumption of normality of the common factor score. A likelihood ratio test is derived, which can be used to test the usual homoscedastic one‐factor model against one of the proposed heteroscedastic models. Simulation studies are carried out to investigate the robustness and the power of this likelihood ratio test. Results show that the asymptotic properties of the test statistic hold under both small test length conditions and small sample size conditions. Results also show under what conditions the power to detect different heteroscedasticity parameter values is either small, medium, or large. Finally, for illustrative purposes, the marginal maximum likelihood estimation procedure and the likelihood ratio test are applied to real data.     

Adjusting Incremental Fit Indices for Nonnormality

A variety of indices are commonly used to assess model fit in structural equation modeling. However, fit indices obtained from the normal theory maximum likelihood fit function are affected by the presence of nonnormality in the data. We present a nonnormality correction for 2 commonly used incremental fit indices, the comparative fit index and the Tucker-Lewis index. This correction uses the Satorra-Bentler scaling constant to modify the sample estimate of these fit indices but does not affect the population value. We argue that this type of nonnormality correction is superior to the correction that changes the population value of the fit index implemented in some software programs. In a simulation study, we demonstrate that our correction performs well across a variety of sample sizes, model types, and misspecification types.     

测验相对拟合检验方法CVLL法在认知诊断中的拓展及应用

本文将IRT中表现较好的CVLL法引入到认知诊断领域,同时比较并分析CVLL及认知诊断领域已有的测验相对拟合检验统计量的表现,为实际工作者在认知诊断模型选用上提供方法学支持和借鉴。结果表明:CVLL的表现比其它传统测验相对拟合统计量要好;且当对Q矩阵进行误设时,该统计量也能选择较优的Q矩阵,说明CVLL在Q矩阵侦查上有较好的应用前景。     

Effects of skewness and kurtosis on normal-theory based maximum likelihood test statistic in multilevel structural equation modeling

A simulation study investigated the effects of skewness and kurtosis on level-specific maximum likelihood (ML) test statistics based on normal theory in multilevel structural equation models. The levels of skewness and kurtosis at each level were manipulated in multilevel data, and the effects of skewness and kurtosis on level-specific ML test statistics were examined. When the assumption of multivariate normality was violated, the level-specific ML test statistics were inflated, resulting in Type I error rates that were higher than the nominal level for the correctly specified model. Q-Q plots of the test statistics against a theoretical chi-square distribution showed that skewness led to a thicker upper tail and kurtosis led to a longer upper tail of the observed distribution of the level-specific ML test statistic for the correctly specified model.     

Pairwise Likelihood Ratio Tests and Model Selection Criteria for Structural Equation Models with Ordinal Variables

Correlated multivariate ordinal data can be analysed with structural equation models. Parameter estimation has been tackled in the literature using limited-information methods including three-stage least squares and pseudo-likelihood estimation methods such as pairwise maximum likelihood estimation. In this paper, two likelihood ratio test statistics and their asymptotic distributions are derived for testing overall goodness-of-fit and nested models, respectively, under the estimation framework of pairwise maximum likelihood estimation. Simulation results show a satisfactory performance of type I error and power for the proposed test statistics and also suggest that the performance of the proposed test statistics is similar to that of the test statistics derived under the three-stage diagonally weighted and unweighted least squares. Furthermore, the corresponding, under the pairwise framework, model selection criteria, AIC and BIC, show satisfactory results in selecting the right model in our simulation examples. The derivation of the likelihood ratio test statistics and model selection criteria under the pairwise framework together with pairwise estimation provide a flexible framework for fitting and testing structural equation models for ordinal as well as for other types of data. The test statistics derived and the model selection criteria are used on data on ‘trust in the police’ selected from the 2010 European Social Survey. The proposed test statistics and the model selection criteria have been implemented in the R package lavaan.     

When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions

A simulation study compared the performance of robust normal theory maximum likelihood (ML) and robust categorical least squares (cat-LS) methodology for estimating confirmatory factor analysis models with ordinal variables. Data were generated from 2 models with 2-7 categories, 4 sample sizes, 2 latent distributions, and 5 patterns of category thresholds. Results revealed that factor loadings and robust standard errors were generally most accurately estimated using cat-LS, especially with fewer than 5 categories; however, factor correlations and model fit were assessed equally well with ML. Cat-LS was found to be more sensitive to sample size and to violations of the assumption of normality of the underlying continuous variables. Normal theory ML was found to be more sensitive to asymmetric category thresholds and was especially biased when estimating large factor loadings. Accordingly, we recommend cat-LS for data sets containing variables with fewer than 5 categories and ML when there are 5 or more categories, sample size is small, and category thresholds are approximately symmetric. With 6-7 categories, results were similar across methods for many conditions; in these cases, either method is acceptable. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

Accommodating Small Sample Sizes in Three-Level Models When the Third Level is Incidental

Small samples sizes are a pervasive problem when modeling clustered data. In two-level models, this problem has been well studied, and several resources provide guidance for modeling such data. However, a recent review of small-sample clustered data methods has noted that no studies have investigated methods for modeling three-level data with small sample sizes. Furthermore, strategies for two-level models do not necessarily translate to the three-level context. Moreover, three-level models are prone to small samples because the “small sample” designation is primarily based on the sample size of the highest level, and large samples are increasingly difficult to amass as one progresses up a hierarchy. In this study, we focus on the case when the third level is incidental, meaning that the third level is important to consider but there are no explicit research questions at the third level. This study performs a simulation study to examine the performance of seven methods for modeling three-level data with a small sample at the third level. A motivating educational psychology example is also provided to demonstrate how the choice of method can greatly affect results.     

The use of item parcels in structural equation modelling: Non‐normal data and small sample sizes

Maximum likelihood estimation in confirmatory factor analysis requires large sample sizes, normally distributed item responses, and reliable indicators of each latent construct, but these ideals are rarely met. We examine alternative strategies for dealing with non‐normal data, particularly when the sample size is small. In two simulation studies, we systematically varied: the degree of non‐normality; the sample size from 50 to 1000; the way of indicator formation, comparing items versus parcels; the parcelling strategy, evaluating uniformly positively skews and kurtosis parcels versus those with counterbalancing skews and kurtosis; and the estimation procedure, contrasting maximum likelihood and asymptotically distribution‐free methods. We evaluated the convergence behaviour of solutions, as well as the systematic bias and variability of parameter estimates, and goodness of fit.     

Tests of homogeneity of means and covariance matrices for multivariate incomplete data

Existing test statistics for assessing whether incomplete data represent a missing completely at random sample from a single population are based on a normal likelihood rationale and effectively test for homogeneity of means and covariances across missing data patterns. The likelihood approach cannot be implemented adequately if a pattern of missing data contains very few subjects. A generalized least squares rationale is used to develop parallel tests that are expected to be more stable in small samples. Three factors were varied for a simulation: number of variables, percent missing completely at random, and sample size. One thousand data sets were simulated for each condition. The generalized least squares test of homogeneity of means performed close to an ideal Type I error rate for most of the conditions. The generalized least squares test of homogeneity of covariance matrices and a combined test performed quite well also.Preliminary results on this research were presented at the 1999 Western Psychological Association convention, Irvine, CA, and in the UCLA Statistics Preprint No. 265 (http://www.stat.ucla.edu). The assistance of Ke-Hai Yuan and several anonymous reviewers is gratefully acknowledged.