Sensitivity of Fit Indices to Model Misspecification and Model Types

The search for cut-off criteria of fit indices for model fit evaluation (e.g., Hu &; Bentler, 1999 Hu, L. and Bentler, P. M. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling., 6: 1–55. [Taylor &; Francis Online], [Web of Science ®] , [Google Scholar]) assumes that these fit indices are sensitive to model misspecification, but not to different types of models. If fit indices were sensitive to different types of models that are misspecified to the same degree, it would be very difficult to establish cut-off criteria that would be generally useful. The issue about SEM fit indices being sensitive to different types of models has not received sufficient attention, although there is some research suggesting that this might be the case (e.g., Kenny &; McCoach, 2003 Kenny, D. A. and McCoach, D. B. 2003. Effect of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling., 10: 333–351. [Taylor &; Francis Online], [Web of Science ®] , [Google Scholar]). This study examines if fit indices are sensitive to different types of models while controlling for the severity of model misspecification. The findings show that most fit indices, including some very popular ones (e.g., RMSEA), may be sensitive to different types of models that have the same degree of specification error. The findings suggest that, for most fit indices, it would be difficult to establish cut-off criteria that would be generally useful in SEM applications.     

认知诊断模型下整体和项目拟合指标

认知诊断模型能否拟合测验数据,直接决定诊断结果的准确性。目前国内鲜有研究涉及认知诊断测验下的模型-资料拟合检验。文章将模型整体拟合指标及基于PPMC的项目拟合指标应用于认知诊断模型-资料拟合检验。模拟研究基于DINA,R-DINA和R-RUM三个诊断模型检验各拟合指标的表现。结果显示整体和项目拟合指标在识别数据产生模型时皆有较高准确率。采用整体和项目拟合指标比较了三个竞争模型与Tatsuoka带分数减法数据的拟合情况,显示R-RUM拟合最好。     

Sensitivity of Fit Indices to Fake Perturbation of Ordinal Data: A Sample by Replacement Approach

In many psychological questionnaires the need to analyze empirical data raises the fundamental problem of possible fake or fraudulent observations in the data. This aspect is particularly relevant for researchers working on sensitive topics such as, for example, risky sexual behaviors and drug addictions. Our contribution presents a new probabilistic approach, called Sample Generation by Replacement (SGR), to address the problem of evaluating the sensitivity of 8 commonly used SEM-based fit indices (Goodness of Fit Index, GFI; Adjusted Goodness of Fit Index, AGFI; Expected Cross Validation Index, ECVI; Standardized Root-Mean-Square Residual Index, SRMR; Root-Mean-Square Error of Approximation, RMSEA; Comparative Fit Index, CFI; Nonnormed Fit Index, NNFI; and Normed Fit Index, NFI) to fake-good ordinal data. We used SGR to perform a simulation study involving 3 different SEM models, 2 sample size conditions, and 2 estimation methods: maximum likelihood (ML) and weighted least squares (WLS). Our results show that the incremental fit indices (CFI, NNFI, and NFI) are clearly more sensitive to fake perturbation than the absolute fit indices (GFI, AGFI, and ECVI). Overall, NFI turned out to be the best and most reliable fit index. We also applied SGR to real behavioral data on (non)compliance in liver transplant patients.     

The Multigroup Multilevel Categorical Latent Growth Curve Models

Longitudinal data describe developmental patterns and enable predictions of individual changes beyond sampled time points. Major methodological issues in longitudinal data include modeling random effects, subject effects, growth curve parameters, and autoregressive residuals. This study embedded the longitudinal model within a multigroup multilevel framework and allowed for autoregressive residuals. The parameter in the new model can be estimated using the computer program WinBUGS, which adopts Markov Chain Monte Carlo algorithms. Two simulation studies were conducted. An empirical example was raised and established based on models generated by the results of empirical data, which have been fitted and compared.     

Nonlinear Growth Models as Measurement Models: A Second-Order Growth Curve Model for Measuring Potential

Recent methodological work has highlighted the promise of nonlinear growth models for addressing substantive questions in the behavioral sciences. In this article, we outline a second-order nonlinear growth model in order to measure a critical notion in development and education: potential. Here, potential is conceptualized as having three components—ability, capacity, and availability—where ability is the amount of skill a student is estimated to have at a given timepoint, capacity is the maximum amount of ability a student is predicted to be able to develop asymptotically, and availability is the difference between capacity and ability at any particular timepoint. We argue that single timepoint measures are typically insufficient for discerning information about potential, and we therefore describe a general framework that incorporates a growth model into the measurement model to capture these three components. Then, we provide an illustrative example using the public-use Early Childhood Longitudinal Study–Kindergarten data set using a Michaelis-Menten growth function (reparameterized from its common application in biochemistry) to demonstrate our proposed model as applied to measuring potential within an educational context. The advantage of this approach compared to currently utilized methods is discussed as are future directions and limitations.     

On Insensitivity of the Chi-Square Model Test to Nonlinear Misspecification in Structural Equation Models

In this paper, we show that for some structural equation models (SEM), the classical chi-square goodness-of-fit test is unable to detect the presence of nonlinear terms in the model. As an example, we consider a regression model with latent variables and interactions terms. Not only the model test has zero power against that type of misspecifications, but even the theoretical (chi-square) distribution of the test is not distorted when severe interaction term misspecification is present in the postulated model. We explain this phenomenon by exploiting results on asymptotic robustness in structural equation models. The importance of this paper is to warn against the conclusion that if a proposed linear model fits the data well according to the chi-quare goodness-of-fit test, then the underlying model is linear indeed; it will be shown that the underlying model may, in fact, be severely nonlinear. In addition, the present paper shows that such insensitivity to nonlinear terms is only a particular instance of a more general problem, namely, the incapacity of the classical chi-square goodness-of-fit test to detect deviations from zero correlation among exogenous regressors (either being them observable, or latent) when the structural part of the model is just saturated.     

Sample Sizes for Two-Group Second-Order Latent Growth Curve Models

Second-order latent growth curve models (S. C. Duncan &; Duncan, 1996 Duncan, S. C. and Duncan, T. E. 1996. A multivariate growth curve analysis of adolescent substance use.. Structural Equation Modeling, 3: 323–347. [Taylor &; Francis Online], [Web of Science ®] , [Google Scholar]; McArdle, 1988 McArdle, J. J. 1988. “Dynamic but structural equation modeling of repeated measures data.”. In Handbook of multivariate experimental psychology, , 2nd ed. Edited by: Cattell, R. B. and Nesselroade, J. 564–614. New York: Plenum.. [Crossref] , [Google Scholar]) can be used to study group differences in change in latent constructs. We give exact formulas for the covariance matrix of the parameter estimates and an algebraic expression for the estimation of slope differences. Formulas for calculations of the required sample size are presented, illustrated, and discussed. They are checked by Monte Carlo simulations in Mplus and also by Satorra and Saris's (1985) Satorra, A. and Saris, W. E. 1985. The power of the likelihood ratio test in covariance structure analysis.. Psychometrika, 50: 83–90. [Crossref], [Web of Science ®] , [Google Scholar] power approximation techniques for small and medium effect sizes (Cohen, 1988 Cohen, J. 1988. Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum..  [Google Scholar]). Results are similar across methods. Not surprisingly, sample sizes decrease with effect sizes, indicator reliabilities, number of indicators, frequency of observation, and duration of study. The relative importance of these factors is also discussed, alone and in combination. The use of the sample size formula is illustrated using a modification of empirical results from Stoel, Peetsma, and Roeleveld (2003) Stoel, R. D., Peetsma, T. T. and Roeleveld, J. 2003. Relations between the development of school investment, self-confidence, and language achievement in elementary education: A multivariate latent growth curve approach.. Learning and Individual Differences, 13: 313–333. [Crossref], [Web of Science ®] , [Google Scholar].     

Evaluation of Model Fit in Cognitive Diagnosis Models

Cognitive diagnosis models (CDMs) estimate student ability profiles using latent attributes. Model fit to the data needs to be ascertained in order to determine whether inferences from CDMs are valid. This study investigated the usefulness of some popular model fit statistics to detect CDM fit including relative fit indices (AIC, BIC, and CAIC), and absolute fit indices (RMSEA2, ABS(fcor) and MAX(χ²_jj′)). These fit indices were assessed under different CDM settings with respect to Q-matrix misspecification and CDM misspecification. Results showed that relative fit indices selected the correct DINA model most of the times and selected the correct G-DINA model well across most conditions. Absolute fit indices rejected the true DINA model if the Q-matrix was misspecified in any way. Absolute fit indices rejected the true G-DINA model whenever the Q-matrix was under-specified. RMSEA2 could be artificially low when the Q-matrix was over-specified.     

Fuzzy Clusterwise Growth Curve Models via Generalized Estimating Equations: An Application to the Antisocial Behavior of Children

The growth curve model has been a useful tool for the analysis of repeated measures data. However, it is designed for an aggregate-sample analysis based on the assumption that the entire sample of respondents are from a single homogenous population. Thus, this method may not be suitable when heterogeneous subgroups exist in the population with qualitatively distinct patterns of trajectories. In this paper, the growth curve model is generalized to a fuzzy clustering framework, which explicitly accounts for such group-level heterogeneity in trajectories of change over time. Moreover, the proposed method estimates parameters based on generalized estimating equations thereby relaxing the assumption of correct specification of the population covariance structure among repeated responses. The performance of the proposed method in recovering parameters and the number of clusters is investigated based on two Monte Carlo analyses involving synthetic data. In addition, the empirical usefulness of the proposed method is illustrated by an application concerning the antisocial behavior of a sample of children.     

Latent Growth Curve and Repeated Measures ANOVA Contrasts: What the Models are Telling You

The relationship between the latent growth curve and repeated measures ANOVA models is often misunderstood. Although a number of investigators have looked into the similarities and differences among these models, a cursory reading of the literature can give the impression that they are very different models. Here we show that each model represents a set of contrasts on the occasion means. We demonstrate that the fixed effects parameters of the estimated basis vector latent growth curve model are merely a transformation of the repeated measures ANOVA fixed effects parameters. We further show that differences in fit in models that estimate the same means structure can be due to the different error covariance structures implied by the model. We show these relationships both algebraically and through using data from a simulation.     

不同条件下拟合指数的表现及临界值的选择

在本模拟研究中设计了6种样本容量,6种因子载荷,和4种评分等级,并考察了正态和非正态分布两种情况。采用的错误模型为参数误置（真模型中每个因子各由5个题目来测量,错误模型中则是第一个因子由6个题测量,另两个因子各由4个和5个题来测量,即有一个因子载荷被误置）模型。结果发现（1）样本量、载荷量、评分等级数和分布形态都对GOF的取值确有影响。其中分布形态的影响最大。NNFI、IFI在不同条件下的平均值是最稳定的,其次是CFI、RMSEA和SRMR。它们都算是值得推荐的GOF,尤其是NNFI和IFI。（2）在正态分布中,当样本量≥1000时,根据NNFI、IFI、CFI、RMSEA、SRMR对模型是否拟合做出判断时有很低的两类错误率,在样本量<1000时则不理想。在偏态条件下无论选择哪个GOF两类错误率都很高。（3）采用2指数策略在很多情况下也不能显著降低两类错误率。（4）由于在数据分布非正态,或正态但样本量<1000时是难判断模型是否拟合的。因此我们提出了2界值策略。即为每个GOF确定上下两个界值。低于下界值时可判断模型是不正确的,而高于上界值时则可判断模型是正确的。GOF取值处于上下界值之间时难以判断模型是否拟合,只能说越高拟合的可能性越大。这时就要通过跨样本验证和增加样本量来确定模型是否正确     

Time-specific Errors in Growth Curve Modeling: Type-1 Error Inflation and a Possible Solution with Mixed-Effects Models

Growth curve modeling (GCM) has been one of the most popular statistical methods to examine participants’ growth trajectories using longitudinal data. In spite of the popularity of GCM, little attention has been paid to the possible influence of time-specific errors, which influence all participants at each timepoint. In this article, we demonstrate that the failure to take into account such time-specific errors in GCM produces considerable inflation of type-1 error rates in statistical tests of fixed effects (e.g., coefficients for the linear and quadratic terms). We propose a GCM that appropriately incorporates time-specific errors using mixed-effects models to address the problem. We also provide an applied example to illustrate that GCM with and without time-specific errors would lead to different substantive conclusions about the true growth trajectories. Comparisons with other models in longitudinal data analysis and potential issues of model misspecification are discussed.     

Outlying Observation Diagnostics in Growth Curve Modeling

Growth curve models are widely used for investigating growth and change phenomena. Many studies in social and behavioral sciences have demonstrated that data without any outlying observation are rather an exception, especially for data collected longitudinally. Ignoring the existence of outlying observations may lead to inaccurate or even incorrect statistical inferences. Therefore, it is crucial to identify outlying observations in growth curve modeling. This study comparatively evaluates six methods in outlying observation diagnostics through a Monte Carlo simulation study on a linear growth curve model, by varying factors of sample size, number of measurement occasions, as well as proportion, geometry, and type of outlying observations. It is suggested that the greatest chance of success in detecting outlying observations comes from use of multiple methods, comparing their results and making a decision based on research purposes. A real data analysis example is also provided to illustrate the application of the six outlying observation diagnostic methods.     

结构方程模型常用拟合指数检验的实质

拟合指数检验是评价结构方程模型(SEM)的重要环节。从协方差结构分析的角度将SEM与传统的回归模型比较,容易理解为什么SEM需要拟合指数。揭示了目前几种流行的拟合指数检验的实质:基于卡方的绝对拟合指数(如RMSEA)检验的实质是重新设定卡方检验的显著性水平(不同于通常的.05),相对拟合指数(如NNFI和CFI)检验的实质是基于虚模型设定均方(卡方与自由度之比)降低到的比例;在NNFI大于临界值后,报告和检验CFI是不必要的。根据研究结果提出了一些方便实用的拟合检验建议。     

Misspecification in Latent Change Score Models: Consequences for Parameter Estimation,Model Evaluation,and Predicting Change

Latent change score models (LCS) are conceptually powerful tools for analyzing longitudinal data (McArdle & Hamagami, 2001). However, applications of these models typically include constraints on key parameters over time. Although practically useful, strict invariance over time in these parameters is unlikely in real data. This study investigates the robustness of LCS when invariance over time is incorrectly imposed on key change-related parameters. Monte Carlo simulation methods were used to explore the impact of misspecification on parameter estimation, predicted trajectories of change, and model fit in the dual change score model, the foundational LCS. When constraints were incorrectly applied, several parameters, most notably the slope (i.e., constant change) factor mean and autoproportion coefficient, were severely and consistently biased, as were regression paths to the slope factor when external predictors of change were included. Standard fit indices indicated that the misspecified models fit well, partly because mean level trajectories over time were accurately captured. Loosening constraint improved the accuracy of parameter estimates, but estimates were more unstable, and models frequently failed to converge. Results suggest that potentially common sources of misspecification in LCS can produce distorted impressions of developmental processes, and that identifying and rectifying the situation is a challenge.     

等级反应模型下项目特征曲线等值法在大型考试中的应用

在中国最大的资格考试之一的经济专业资格考试中,为保证不同年度间考试的可比性、进行题库建设和为计算机自适应考试做准备,应用项目反应理论中等级反应模型下的项目特征曲线等值法,采用铆测验等值设计,实现了4个年度考试资料的项目参数和能力参数的等值,并成功地组建了经济专业题库。在此基础上,利用等值技术对不同年份试卷的划界分数进行了比较,为经济考试的合格标准制定、确保考试的公平性提供了实证依据。