Masking misfit in confirmatory factor analysis by increasing unique variances: a cautionary note on the usefulness of cutoff values of fit indices

Fit indices are widely used in order to test the model fit for structural equation models. In a highly influential study, Hu and Bentler (1999) showed that certain cutoff values for these indices could be derived, which, over time, has led to the reification of these suggested thresholds as "golden rules" for establishing the fit or other aspects of structural equation models. The current study shows how differences in unique variances influence the value of the global chi-square model test and the most commonly used fit indices: Root-mean-square error of approximation, standardized root-mean-square residual, and the comparative fit index. Using data simulation, the authors illustrate how the value of the chi-square test, the root-mean-square error of approximation, and the standardized root-mean-square residual are decreased when unique variances are increased although model misspecification is present. For a broader understanding of the phenomenon, the authors used different sample sizes, number of observed variables per factor, and types of misspecification. A theoretical explanation is provided, and implications for the application of structural equation modeling are discussed.     

Analysis of residuals for the multionmial item response model

Using the item response model as developed on the multinomial distribution, asymptotic variances are obtained for residuals associated with response patterns and first-, and second-order marginal frequencies of manifest variables. When the model does not fit well, an examination of these residuals may reveal the source of the poor fit. Finally, a limited-information test of fit for the model is developed by using residuals defined for the first-, and second-order marginals. Model evaluation based on residuals for these marginals is particularly useful when the response pattern frequencies are sparse.The author would like to thank Yasuo Amemiya and Joseph Lucke for helpful suggestions. This research was supported by a Research Incentive Grant from Arizona State University.     

The Factor Structure and Measurement Invariance of the Daily Spiritual Experiences Scale

The current study has two main goals: (a) to identify a factor structure of the Daily Spiritual Experiences Scale (DSES) on a large archival data, collected from 1,325 adults in the United States (709 women, 616 men) by the U.S. General Social Survey in 2004 and (b) to examine the measurement invariance of the 16 DSES items between women and men in the same data to see whether any of the items are favoring or biased toward either women or men. The one-factor confirmatory factor analysis (CFA) model fit our data better than the two-factor CFA models because of high correlations between the two factors (r > .90). The fit of the one-factor CFA to our sample data was improved when we specified seven correlated residuals suggested by overlapping item content and large modification indices. The ensuing measurement invariance testing of the one-factor CFA model with seven correlated residuals supported full measurement invariance of factor loadings, thresholds, and residual variances, as well as factor variances between the women and the men. Yet the factor mean for the women was .841 units (Cohen’s d = .496) higher than it was for the men, indicating that higher levels of daily spiritual experiences for women reported in gender comparison studies in the United States are not likely to be an artifact of bias in the questionnaire.     

Fitting correlated residual error structures in nonlinear mixed-effects models using SAS PROC NLMIXED

Nonlinear mixed-effects (NLME) models remain popular among practitioners for analyzing continuous repeated measures data taken on each of a number of individuals when interest centers on characterizing individual-specific change. Within this framework, variation and correlation among the repeated measurements may be partitioned into interindividual variation and intraindividual variation components. The covariance structure of the residuals are, in many applications, consigned to be independent with homogeneous variances, $ {\sigma}^2{\mathbf{I}}_{n_i} $ , not because it is believed that intraindividual variation adheres to this structure, but because many software programs that estimate parameters of such models are not well-equipped to handle other, possibly more realistic, patterns. In this article, we describe how the programmatic environment within SAS may be utilized to model residual structures for serial correlation and variance heterogeneity. An empirical example is used to illustrate the capabilities of the module.     

The relationship between multilevel models and non‐parametric multilevel mixture models: Discrete approximation of intraclass correlation,random coefficient distributions,and residual heteroscedasticity

Multilevel data structures are common in the social sciences. Often, such nested data are analysed with multilevel models (MLMs) in which heterogeneity between clusters is modelled by continuously distributed random intercepts and/or slopes. Alternatively, the non‐parametric multilevel regression mixture model (NPMM) can accommodate the same nested data structures through discrete latent class variation. The purpose of this article is to delineate analytic relationships between NPMM and MLM parameters that are useful for understanding the indirect interpretation of the NPMM as a non‐parametric approximation of the MLM, with relaxed distributional assumptions. We define how seven standard and non‐standard MLM specifications can be indirectly approximated by particular NPMM specifications. We provide formulas showing how the NPMM can serve as an approximation of the MLM in terms of intraclass correlation, random coefficient means and (co)variances, heteroscedasticity of residuals at level 1, and heteroscedasticity of residuals at level 2. Further, we discuss how these relationships can be useful in practice. The specific relationships are illustrated with simulated graphical demonstrations, and direct and indirect interpretations of NPMM classes are contrasted. We provide an R function to aid in implementing and visualizing an indirect interpretation of NPMM classes. An empirical example is presented and future directions are discussed.     

Some Mathematical Properties of the Matrix Decomposition Solution in Factor Analysis

A new factor analysis (FA) procedure has recently been proposed which can be called matrix decomposition FA (MDFA). All FA model parameters (common and unique factors, loadings, and unique variances) are treated as fixed unknown matrices. Then, the MDFA model simply becomes a specific data matrix decomposition. The MDFA parameters are found by minimizing the discrepancy between the data and the MDFA model. Several algorithms have been developed and some properties have been discussed in the literature (notably by Stegeman in Comput Stat Data Anal 99:189–203, 2016), but, as a whole, MDFA has not been studied fully yet. A number of new properties are discovered in this paper, and some existing ones are derived more explicitly. The properties provided concern the uniqueness of results, covariances among common factors, unique factors, and residuals, and assessment of the degree of indeterminacy of common and unique factor scores. The properties are illustrated using a real data example.     

等效性检验——结构方程模型评价和测量不变性分析的新视角

常用的结构方程模型拟合指数存在一定局限, 如χ ²以传统零假设为目标假设, 无法验证模型, 而RMSEA和CFI等描述性的拟合指数不具备推断统计性质, 等效性检验有效弥补了这些问题。首先说明等效性检验如何评价单个模型的拟合, 并解释其与零假设检验的不同, 然后介绍等效性检验如何分析测量不变性, 接着用实证数据展示了等效性检验在单个模型评价和测量不变性检验中的效果, 并与传统模型评价方法比较。     

Exact F Tests in an ANOVA Procedure for Dependent Observations

Several general correlation patterns are shown in this paper which give exact F tests in an ANOVA procedure with dependent observations. This paper presents the most general correlation patterns one can assume in a one-way and two-way layout and still have the F tests be valid. Exact F tests are given for various designs. These include the unbalanced ANOVA design, analysis of covariance, random effects models, and mixed models. Bartlett's test for homogeneity of variances is shown to be exact when the independence assumption is relaxed. An example is provided to illustrate how the general correlation can occur in an experimental design.     

Generating Multivariate Ordinal Data via Entropy Principles

When conducting robustness research where the focus of attention is on the impact of non-normality, the marginal skewness and kurtosis are often used to set the degree of non-normality. Monte Carlo methods are commonly applied to conduct this type of research by simulating data from distributions with skewness and kurtosis constrained to pre-specified values. Although several procedures have been proposed to simulate data from distributions with these constraints, no corresponding procedures have been applied for discrete distributions. In this paper, we present two procedures based on the principles of maximum entropy and minimum cross-entropy to estimate the multivariate observed ordinal distributions with constraints on skewness and kurtosis. For these procedures, the correlation matrix of the observed variables is not specified but depends on the relationships between the latent response variables. With the estimated distributions, researchers can study robustness not only focusing on the levels of non-normality but also on the variations in the distribution shapes. A simulation study demonstrates that these procedures yield excellent agreement between specified parameters and those of estimated distributions. A robustness study concerning the effect of distribution shape in the context of confirmatory factor analysis shows that shape can affect the robust \(\chi ^2\) and robust fit indices, especially when the sample size is small, the data are severely non-normal, and the fitted model is complex.     

Direction of Effects in Multiple Linear Regression Models

Previous studies analyzed asymmetric properties of the Pearson correlation coefficient using higher than second order moments. These asymmetric properties can be used to determine the direction of dependence in a linear regression setting (i.e., establish which of two variables is more likely to be on the outcome side) within the framework of cross-sectional observational data. Extant approaches are restricted to the bivariate regression case. The present contribution extends the direction of dependence methodology to a multiple linear regression setting by analyzing distributional properties of residuals of competing multiple regression models. It is shown that, under certain conditions, the third central moments of estimated regression residuals can be used to decide upon direction of effects. In addition, three different approaches for statistical inference are discussed: a combined D’Agostino normality test, a skewness difference test, and a bootstrap difference test. Type I error and power of the procedures are assessed using Monte Carlo simulations, and an empirical example is provided for illustrative purposes. In the discussion, issues concerning the quality of psychological data, possible extensions of the proposed methods to the fourth central moment of regression residuals, and potential applications are addressed.     

Some contributions to maximum likelihood factor analysis

A new computational method for the maximum likelihood solution in factor analysis is presented. This method takes into account the fact that the likelihood function may not have a maximum in a point of the parameter space where all unique variances are positive. Instead, the maximum may be attained on the boundary of the parameter space where one or more of the unique variances are zero. It is demonstrated that suchimproper (Heywood) solutions occur more often than is usually expected. A general procedure to deal with such improper solutions is proposed. The proposed methods are illustrated using two small sets of empirical data, and results obtained from the analyses of many other sets of data are reported. These analyses verify that the new computational method converges rapidly and that the maximum likelihood solution can be determined very accurately. A by-product obtained by the method is a large sample estimate of the variance-covariance matrix of the estimated unique variances. This can be used to set up approximate confidence intervals for communalities and unique variances.The first part of this work was done at the University of Uppsala, Sweden and supported by the Swedish Council for Social Science Research. The second part was done at Educational Testing Service and supported by a grant (NSF-GB-1985) from National Science Foundation to Educational Testing Service.The author is deeply indebted to Dr. D. N. Lawley whose contributions amounted nearly to coauthorship. The author also wishes to thank Mr. G. Gruvaeus for much valuable assistance in constructing the computer program.     

Generating correlation matrices with model error for simulation studies in factor analysis: A combination of the Tucker-Koopman-Linn model and Wijsman’s algorithm

Most simulation studies in factor analysis follow a process of constructing population correlation matrices from the common-factor model and generating sample correlation matrices from the population matrices. In the common-factor model, the population correlation matrix is perfectly fit by the model’s containing common and unique factors. However, since no mathematical model accounts exactly for the real-world phenomena that it is intended to represent, the Tucker-Koopman-Linn model (1969) is more realistic for generating correlation matrices than the conventional common-factor model because the former incorporates model error. In this paper, a procedure for generating population and sample correlation matrices with model error by combining the Tucker-Koopman-Linn model and Wijsman’s algorithm (1959) is presented. The SAS/ IML program for generating correlation matrices is described, and an example is also provided.     

预测视角下双因子模型与高阶模型的模拟比较

双因子模型和高阶因子模型,作为既有全局因子又有局部因子的两个竞争模型,在研究中得到了广泛应用。本文采用Monte Carlo模拟方法,在模型拟合比较的基础上,比较了效标分别为外显变量和内潜变量时,两个模型在各种负荷水平下预测准确度的差异。结果发现,两种模型在拟合效果方面无显著差异;但在预测效度方面,当效标为显变量时,两个模型的结构系数估计值皆为无偏估计;而效标为潜变量时,高阶因子模型表现优于双因子模型:高阶因子模型的结构系数为无偏估计,双因子模型的结构系数估计值则在50%左右的情况下存在偏差。     

A formal version of the Guided Search (GS2) model

Guided Search 2 (GS2) is currently one of the most detailed models of visual search and has been used to predict search times for different stimulus conditions by means of detailed computer simulations. The present article goes a step further and presents formulas that allow for the calculation of the search times and their variances. Moreover, these formulas can be applied to fit GS2 to data. An example is provided in which GS2 is fitted to search functions representing search asymmetries.     

A formal version of the Guided Search (GS2) model

Guided Search 2 (GS2) is currently one of the most detailed models of visual search and has been used to predict search times for different stimulus conditions by means of detailed computer simulations. The present article goes a step further and presents formulas that allow for the calculation of the search times and their variances. Moreover, these formulas can be applied to fit GS2 to data. An example is provided in which GS2 is fitted to search functions representing search asymmetries.     

Introducing the New Indices of Religious Orientation (NIRO): Conceptualization and measurement

The notion of religious orientation as proposed by Allport and refined by Batson has provided a useful tool for identifying and discussing individual differences in religiosity within Christian and post-Christian contexts. The present paper accepts the conceptual and empirical usefulness of distinctions between the three constructs of intrinsic, extrinsic, and quest religious orientations; reviews the conceptual and empirical strengths, weaknesses, and limitations of existing measures in this field; and proposes three new indices to assess these orientations. The New Indices of Religious Orientation (NIRO) are designed to be of equal length, to give equal empirical weight to three conceptual components within each construct, and to employ direct and accessible language. Data provided from a sample of 517 undergraduate students in Wales demonstrate the satisfactory psychometric properties of these new indices. A clear distinction is made between the appropriate use and the inappropriate misuse of these indices in future research.     

Factor-analytic Procedures for Assessing Response Pattern Scalability

This paper proposes procedures for assessing the fit of a psychometric model at the level of the individual respondent. The procedures are intended for personality measures made up of Likert-type items, which, in applied research, are usually analyzed by means of factor analysis. Two scalability indices are proposed, which can be considered as factor-analytic counterparts of the lo and lz IRT-based person-fit indices. The present indices can be derived both as likelihood-based measures and as residual measures, and this makes it possible to relate them with previous methodologies such as Bollen's residual analysis and Lanning's scalability index. The indices are intended to be used mainly as exploratory devices, and a factor-analytic version of the person response curve is also proposed as a tool for obtaining further information about the possible causes of misfit. The behavior of the procedures is assessed by means of simulation studies. Finally, an empirical example in personality measurement shows how the procedures can be used in applied research.     

A Comparison of Inverse-Wishart Prior Specifications for Covariance Matrices in Multilevel Autoregressive Models

Multilevel autoregressive models are especially suited for modeling between-person differences in within-person processes. Fitting these models with Bayesian techniques requires the specification of prior distributions for all parameters. Often it is desirable to specify prior distributions that have negligible effects on the resulting parameter estimates. However, the conjugate prior distribution for covariance matrices—the Inverse-Wishart distribution—tends to be informative when variances are close to zero. This is problematic for multilevel autoregressive models, because autoregressive parameters are usually small for each individual, so that the variance of these parameters will be small. We performed a simulation study to compare the performance of three Inverse-Wishart prior specifications suggested in the literature, when one or more variances for the random effects in the multilevel autoregressive model are small. Our results show that the prior specification that uses plug-in ML estimates of the variances performs best. We advise to always include a sensitivity analysis for the prior specification for covariance matrices of random parameters, especially in autoregressive models, and to include a data-based prior specification in this analysis. We illustrate such an analysis by means of an empirical application on repeated measures data on worrying and positive affect.     

An investigation of the sampling distributions of item discrimination indices

The sampling properties of four item discrimination indices (biserialr, Cook's indexB, theU–L 27 per cent index, and DeltaP) were investigated in order to ascertain their sampling properties when small samples drawn from actual test data rather than constructed data were employed. The empirical results indicated that the mean index values approximated the population values and that values of the standard deviations computed from large sample formulas were good approximations to the standard deviations of the observed distributions based on samples of size 120 or less. Goodness of fit tests comparing the observed distributions with the corresponding distribution of the product-moment correlation coefficient based upon a bivariate normal population indicated that this correlational model was inappropriate for the data. The lack of adequate mathematical models for the sampling distributions of item discrimination indices suggests that one should avoid indices whose only real reason for existence was the simplification of computational procedures.This research reported herein was performed pursuant to a contract (OE-2-10-071) with the United States Office of Education, Department of Health, Education and Welfare.     

Inferential theory for partially disattenuated correlation coefficients

Four measurement designs are presented for use with correlation coefficients corrected, in one variable, for attenuation due to unreliability—coefficients that we term partially disattenuated correlation coefficients. Asymptotic expressions are derived for the variances and covariances of the estimates accompanying each design. Empirical simulation results that bear on the preceding mathematical developments are then presented. In addition to providing insights into the distributions of the estimates, the empirical results demonstrate satisfactory Type I error control for typical inferential applications. Power is shown to be equal to or greater than that of corresponding product-moment correlations in three of the four designs. Implications for practice are discussed.Support for the research reported in this article was provided by the Natural Sciences and Engineering Research Council of Canada. The authors acknowledge with thanks the contributions of Nancy E. Heckman to some of the theoretical aspects of the study.