Evaluating Small Sample Approaches for Model Test Statistics in Structural Equation Modeling

Through Monte Carlo simulation, small sample methods for evaluating overall data-model fit in structural equation modeling were explored. Type I error behavior and power were examined using maximum likelihood (ML), Satorra-Bentler scaled and adjusted (SB; Satorra & Bentler, 1988, 1994), residual-based (Browne, 1984), and asymptotically distribution free (ADF; Browne, 1982, 1984) test statistics. To accommodate small sample sizes the ML and SB statistics were adjusted using a k-factor correction (Bartlett, 1950); the residual-based and ADF statistics were corrected using modified x² and F statistics (Yuan & Bentler, 1998, 1999). Design characteristics include model type and complexity, ratio of sample size to number of estimated parameters, and distributional form. The k-factor-corrected SB scaled test statistic was especially stable at small sample sizes with both normal and nonnormal data. Methodologists are encouraged to investigate its behavior under a wider variety of models and distributional forms.     

Structural Equation Modeling with Small Samples: Test Statistics

Structural equation modeling is a well-known technique for studying relationships among multivariate data. In practice, high dimensional nonnormal data with small to medium sample sizes are very common, and large sample theory, on which almost all modeling statistics are based, cannot be invoked for model evaluation with test statistics. The most natural method for nonnormal data, the asymptotically distribution free procedure, is not defined when the sample size is less than the number of nonduplicated elements in the sample covariance. Since normal theory maximum likelihood estimation remains defined for intermediate to small sample size, it may be invoked but with the probable consequence of distorted performance in model evaluation. This article studies the small sample behavior of several test statistics that are based on maximum likelihood estimator, but are designed to perform better with nonnormal data. We aim to identify statistics that work reasonably well for a range of small sample sizes and distribution conditions. Monte Carlo results indicate that Yuan and Bentler's recently proposed F-statistic performs satisfactorily.     

Abstract: Evaluation of Test Statistics for Robust Structural Equation Modeling With Nonnormal Missing Data

Traditional structural equation modeling (SEM) techniques have trouble dealing with incomplete and/or nonnormal data that are often encountered in practice. Yuan and Zhang (2011a) developed a two-stage procedure for SEM to handle nonnormal missing data and proposed four test statistics for overall model evaluation. Although these statistics have been shown to work well with complete data, their performance for incomplete data has not been investigated in the context of robust statistics.

Focusing on a linear growth curve model, a systematic simulation study is conducted to evaluate the accuracy of the parameter estimates and the performance of five test statistics including the naive statistic derived from normal distribution based maximum likelihood (ML), the Satorra-Bentler scaled chi-square statistic (RML), the mean- and variance-adjusted chi-square statistic (AML), Yuan-Bentler residual-based test statistic (CRADF), and Yuan-Bentler residual-based F statistic (RF). Data are generated and analyzed in R using the package rsem (Yuan & Zhang, 2011b).

Based on the simulation study, we can observe the following: (a) The traditional normal distribution-based method cannot yield accurate parameter estimates for nonnormal data, whereas the robust method obtains much more accurate model parameter estimates for nonnormal data and performs almost as well as the normal distribution based method for normal distributed data. (b) With the increase of sample size, or the decrease of missing rate or the number of outliers, the parameter estimates are less biased and the empirical distributions of test statistics are closer to their nominal distributions. (c) The ML test statistic does not work well for nonnormal or missing data. (d) For nonnormal complete data, CRADF and RF work relatively better than RML and AML. (e) For missing completely at random (MCAR) missing data, in almost all the cases, RML and AML work better than CRADF and RF. (f) For nonnormal missing at random (MAR) missing data, CRADF and RF work better than AML. (g) The performance of the robust method does not seem to be influenced by the symmetry of outliers.     

Robust Structural Equation Modeling with Missing Data and Auxiliary Variables

The paper develops a two-stage robust procedure for structural equation modeling (SEM) and an R package rsem to facilitate the use of the procedure by applied researchers. In the first stage, M-estimates of the saturated mean vector and covariance matrix of all variables are obtained. Those corresponding to the substantive variables are then fitted to the structural model in the second stage. A sandwich-type covariance matrix is used to obtain consistent standard errors (SE) of the structural parameter estimates. Rescaled, adjusted as well as corrected and F-statistics are proposed for overall model evaluation. Using R and EQS, the R package rsem combines the two stages and generates all the test statistics and consistent SEs. Following the robust analysis, multiple model fit indices and standardized solutions are provided in the corresponding output of EQS. An example with open/closed book examination data illustrates the proper use of the package. The method is further applied to the analysis of a data set from the National Longitudinal Survey of Youth 1997 cohort, and results show that the developed procedure not only gives a better endorsement of the substantive models but also yields estimates with uniformly smaller standard errors than the normal-distribution-based maximum likelihood.     

Empirically Corrected Rescaled Statistics for SEM with Small N and Large p

Survey data often contain many variables. Structural equation modeling (SEM) is commonly used in analyzing such data. With typical nonnormally distributed data in practice, a rescaled statistic T_rml proposed by Satorra and Bentler was recommended in the literature of SEM. However, T_rml has been shown to be problematic when the sample size N is small and/or the number of variables p is large. There does not exist a reliable test statistic for SEM with small N or large p, especially with nonnormally distributed data. Following the principle of Bartlett correction, this article develops empirical corrections to T_rml so that the mean of the empirically corrected statistics approximately equals the degrees of freedom of the nominal chi-square distribution. Results show that empirically corrected statistics control type I errors reasonably well even when N is smaller than 2p, where T_rml may reject the correct model 100% even for normally distributed data. The application of the empirically corrected statistics is illustrated via a real data example.     

Structural Equation Modeling Approaches for Analyzing Partially Nested Data

Study designs involving clustering in some study arms, but not all study arms, are common in clinical treatment-outcome and educational settings. For instance, in a treatment arm, persons may be nested in therapy groups, whereas in a control arm there are no groups. Methodological approaches for handling such partially nested designs have recently been developed in a multilevel modeling framework (MLM-PN) and have proved very useful. We introduce two alternative structural equation modeling (SEM) approaches for analyzing partially nested data: a multivariate single-level SEM (SSEM-PN) and a multiple-arm multilevel SEM (MSEM-PN). We show how SSEM-PN and MSEM-PN can produce results equivalent to existing MLM-PNs and can be extended to flexibly accommodate several modeling features that are difficult or impossible to handle in MLM-PNs. For instance, using an SSEM-PN or MSEM-PN, it is possible to specify complex structural models involving cluster-level outcomes, obtain absolute model fit, decompose person-level predictor effects in the treatment arm using latent cluster means, and include traditional factors as predictors/outcomes. Importantly, implementation of such features for partially nested designs differs from that for fully nested designs. An empirical example involving a partially nested depression intervention combines several of these features in an analysis of interest for treatment-outcome studies.     

Evaluation of the Bayesian and Maximum Likelihood Approaches in Analyzing Structural Equation Models with Small Sample Sizes

The main objective of this article is to investigate the empirical performances of the Bayesian approach in analyzing structural equation models with small sample sizes. The traditional maximum likelihood (ML) is also included for comparison. In the context of a confirmatory factor analysis model and a structural equation model, simulation studies are conducted with the different magnitudes of parameters and sample sizes n = da, where d = 2, 3, 4 and 5, and a is the number of unknown parameters. The performances are evaluated in terms of the goodness-of-fit statistics, and various measures on the accuracy of the estimates. The conclusion is: for data that are normally distributed, the Bayesian approach can be used with small sample sizes, whilst ML cannot.     

Multilevel Factor Analysis and Structural Equation Modeling of Daily Diary Coping Data: Modeling Trait and State Variation

This study used multilevel modeling of daily diary data to model within-person (state) and between-person (trait) components of coping variables. This application included the introduction of multilevel factor analysis (MFA) and a comparison of the predictive ability of these trait/state factors. Daily diary data were collected on a large (n = 366) multiethnic sample over the course of 5 days. Intraclass correlation coefficient for the derived factors suggested approximately equal amounts of variability in coping usage at the state and trait levels. MFAs showed that Problem-Focused Coping and Social Support emerged as stable factors at both the within-person and between-person levels. Other factors (Minimization, Emotional Rumination, Avoidance, Distraction) were specific to the within-person or between-person levels but not both. Multilevel structural equation modeling (MSEM) showed that the prediction of daily positive and negative affect differed as a function of outcome and level of coping factor. The Discussion section focuses primarily on a conceptual and methodological understanding of modeling state and trait coping using daily diary data with MFA and MSEM to examine covariation among coping variables and predicting outcomes of interest.     

Moment Testing for Interaction Terms in Structural Equation Modeling

Starting with Kenny and Judd (Psychol. Bull. 96:201–210, 1984) several methods have been introduced for analyzing models with interaction terms. In all these methods more information from the data than just means and covariances is required. In this paper we also use more than just first- and second-order moments; however, we are aiming to adding just a selection of the third-order moments. The key issue in this paper is to develop theoretical results that will allow practitioners to evaluate the strength of different third-order moments in assessing interaction terms of the model. To select the third-order moments, we propose to be guided by the power of the goodness-of-fit test of a model with no interactions, which varies with each selection of third-order moments. A theorem is presented that relates the power of the usual goodness-of-fit test of the model with the power of a moment test for the significance of third-order moments; the latter has the advantage that it can be computed without fitting a model. The main conclusion is that the selection of third-order moments can be based on the power of a moment test, thus assessing the relevance in the analysis of different sets of third-order moments can be computationally simple. The paper gives an illustration of the method and argues for the need of refraining from adding into the analysis an excess of higher-order moments.     

Alternative Multiple Imputation Inference for Categorical Structural Equation Modeling

The use of item responses from questionnaire data is ubiquitous in social science research. One side effect of using such data is that researchers must often account for item level missingness. Multiple imputation is one of the most widely used missing data handling techniques. The traditional multiple imputation approach in structural equation modeling has a number of limitations. Motivated by Lee and Cai’s approach, we propose an alternative method for conducting statistical inference from multiple imputation in categorical structural equation modeling. We examine the performance of our proposed method via a simulation study and illustrate it with one empirical data set.     

A Bayesian Modeling Approach for Generalized Semiparametric Structural Equation Models

In behavioral, biomedical, and psychological studies, structural equation models (SEMs) have been widely used for assessing relationships between latent variables. Regression-type structural models based on parametric functions are often used for such purposes. In many applications, however, parametric SEMs are not adequate to capture subtle patterns in the functions over the entire range of the predictor variable. A different but equally important limitation of traditional parametric SEMs is that they are not designed to handle mixed data types—continuous, count, ordered, and unordered categorical. This paper develops a generalized semiparametric SEM that is able to handle mixed data types and to simultaneously model different functional relationships among latent variables. A structural equation of the proposed SEM is formulated using a series of unspecified smooth functions. The Bayesian P-splines approach and Markov chain Monte Carlo methods are developed to estimate the smooth functions and the unknown parameters. Moreover, we examine the relative benefits of semiparametric modeling over parametric modeling using a Bayesian model-comparison statistic, called the complete deviance information criterion (DIC). The performance of the developed methodology is evaluated using a simulation study. To illustrate the method, we used a data set derived from the National Longitudinal Survey of Youth.     

A Propensity Score Adjustment for Multiple Group Structural Equation Modeling

In the behavioral and social sciences, quasi-experimental and observational studies are used due to the difficulty achieving a random assignment. However, the estimation of differences between groups in observational studies frequently suffers from bias due to differences in the distributions of covariates. To estimate average treatment effects when the treatment variable is binary, Rosenbaum and Rubin (1983a) proposed adjustment methods for pretreatment variables using the propensity score. However, these studies were interested only in estimating the average causal effect and/or marginal means. In the behavioral and social sciences, a general estimation method is required to estimate parameters in multiple group structural equation modeling where the differences of covariates are adjusted. We show that a Horvitz–Thompson-type estimator, propensity score weighted M estimator (PWME) is consistent, even when we use estimated propensity scores, and the asymptotic variance of the PWME is shown to be less than that with true propensity scores. Furthermore, we show that the asymptotic distribution of the propensity score weighted statistic under a null hypothesis is a weighted sum of independent χ² ₁ variables. We show the method can compare latent variable means with covariates adjusted using propensity scores, which was not feasible by previous methods. We also apply the proposed method for correlated longitudinal binary responses with informative dropout using data from the Longitudinal Study of Aging (LSOA). The results of a simulation study indicate that the proposed estimation method is more robust than the maximum likelihood (ML) estimation method, in that PWME does not require the knowledge of the relationships among dependent variables and covariates.     

OpenMx 2.0: Extended Structural Equation and Statistical Modeling

The new software package OpenMx 2.0 for structural equation and other statistical modeling is introduced and its features are described. OpenMx is evolving in a modular direction and now allows a mix-and-match computational approach that separates model expectations from fit functions and optimizers. Major backend architectural improvements include a move to swappable open-source optimizers such as the newly written CSOLNP. Entire new methodologies such as item factor analysis and state space modeling have been implemented. New model expectation functions including support for the expression of models in LISREL syntax and a simplified multigroup expectation function are available. Ease-of-use improvements include helper functions to standardize model parameters and compute their Jacobian-based standard errors, access to model components through standard R $ mechanisms, and improved tab completion from within the R Graphical User Interface.     

基于多层结构方程模型的情境效应分析—— 兼与多层线性模型比较

多层(嵌套)数据的变量关系研究, 必须借助多层模型来实现。两层模型中, 层一自变量Xij按组均值中心化, 并将组均值 置于层2截距方程式中, 可将Xij对因变量Yij的效应分解为组间和组内部分, 二者之差被称为情境效应, 称为情境变量。多层结构方程模型(MSEM)将多层线性模型(MLM)和结构方程模型(SEM)相结合, 通过设置潜变量和多指标的方法校正了MLM在情境效应分析中出现的抽样误差和测量误差, 同时解决了数据的多层(嵌套)结构和潜变量的估计问题。除了分析原理的说明, 还以班级平均竞争氛围对学生竞争表现的情境效应为例进行分析方法的示范, 并比较MSEM和MLM的异同, 随后展望了MSEM情境效应模型、情境效应无偏估计方法和情境变量研究的拓展方向。     

The Influence of Number of Categories and Threshold Values on Fit Indices in Structural Equation Modeling with Ordered Categorical Data

This study examines the unscaled and scaled root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker–Lewis index (TLI) of diagonally weighted least squares (DWLS) and unweighted least squares (ULS) estimators in structural equation modeling with ordered categorical data. We show that the number of categories and threshold values for categorization can unappealingly impact the DWLS unscaled and scaled fit indices, as well as the ULS scaled fit indices in the population, given that analysis models are misspecified and that the threshold structure is saturated. Consequently, a severely misspecified model may be considered acceptable, depending on how the underlying continuous variables are categorized. The corresponding CFI and TLI are less dependent on the categorization than RMSEA but are less sensitive to model misspecification in general. In contrast, the number of categories and threshold values do not impact the ULS unscaled fit indices in the population.     

Mixture Factor Analysis for Approximating a Nonnormally Distributed Continuous Latent Factor With Continuous and Dichotomous Observed Variables

Mixture factor analysis is examined as a means of flexibly estimating nonnormally distributed continuous latent factors in the presence of both continuous and dichotomous observed variables. A simulation study compares mixture factor analysis with normal maximum likelihood (ML) latent factor modeling. Different results emerge for continuous versus dichotomous outcomes. For dichotomous outcomes, normal ML path estimates have bias that worsens as latent factor skew/kurtosis increases and does not diminish as sample size increases, whereas the mixture factor analysis model produces nearly unbiased estimators as sample sizes increase (500 and greater) and offers near nominal coverage probability. For continuous outcome variables, both methods produce factor loading estimates with minimal bias regardless of latent factor skew, but the mixture factor analysis is more efficient. The method is demonstrated using data motivated by a study on youth with cystic fibrosis examining predictors of treatment adherence. In summary, mixture factor analysis provides improvements over normal ML estimation in the presence of skewed/kurtotic latent factors, but due to variability in the estimator relating the latent factor to dichotomous outcomes and computational issues, the improvements were only fully realized, in this study, at larger sample sizes (500 and greater).     

The Value of Exploratory Structural Equation Modeling in Identifying Factor Overlap in the Mental Health Continuum-Short Form (MHC-SF): A Study with a New Zealand Sample

This study is the first to investigate the factor structure of the Mental Health Continuum-Short Form (MHC-SF) in New Zealand. Towards this end, traditional Confirmatory Factor Analysis (CFA) and the new method of Exploratory Structural Equation Modeling (ESEM) were used. Both ESEM and CFA supported the tripartite model of mental well-being in comparison to the one- and two-factor models; however, ESEM provided better fit with the data. Moreover, interfactor correlations were considerably lower in ESEM than they were in CFA, indicating greater factor distinctiveness in ESEM. ESEM also revealed a number of important cross-loadings for items in the measurement model of the MHC-SF. The results supported full metric and full scalar invariance of the MHC-SF across gender. The attenuated correlations among well-being factors obtained by ESEM here provide an important insight about the ongoing controversy regarding the failure of empirical research to identify distinct eudaimonic and hedonic factors in well-being measures. An overreliance on CFA methods may have led the field to rely on inflated estimates of shared variance between eudaimonia and hedonia.     

Identification and Small Sample Estimation of Thurstone's Unrestricted Model for Paired Comparisons Data

The interpretation of a Thurstonian model for paired comparisons where the utilities' covariance matrix is unrestricted proved to be difficult due to the comparative nature of the data. We show that under a suitable constraint the utilities' correlation matrix can be estimated, yielding a readily interpretable solution. This set of identification constraints can recover any true utilities' covariance matrix, but it is not unique. Indeed, we show how to transform the estimated correlation matrix into alternative correlation matrices that are equally consistent with the data but may be more consistent with substantive theory. Also, we show how researchers can investigate the sample size needed to estimate a particular model by exploiting the simulation capabilities of a popular structural equation modeling statistical package.     

Bayesian Analysis of Nonlinear Structural Equation Models with Nonignorable Missing Data

A Bayesian approach is developed for analyzing nonlinear structural equation models with nonignorable missing data. The nonignorable missingness mechanism is specified by a logistic regression model. A hybrid algorithm that combines the Gibbs sampler and the Metropolis–Hastings algorithm is used to produce the joint Bayesian estimates of structural parameters, latent variables, parameters in the nonignorable missing model, as well as their standard errors estimates. A goodness-of-fit statistic for assessing the plausibility of the posited nonlinear structural equation model is introduced, and a procedure for computing the Bayes factor for model comparison is developed via path sampling. Results obtained with respect to different missing data models, and different prior inputs are compared via simulation studies. In particular, it is shown that in the presence of nonignorable missing data, results obtained by the proposed method with a nonignorable missing data model are significantly better than those that are obtained under the missing at random assumption. A real example is presented to illustrate the newly developed Bayesian methodologies. This research is fully supported by a grant (CUHK 4243/03H) from the Research Grant Council of the Hong Kong Special Administration Region. The authors are thankful to the editor and reviewers for valuable comments for improving the paper, and also to ICPSR and the relevant funding agency for allowing the use of the data. Requests for reprints should be sent to Professor S.Y. Lee, Department of Statistics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong.