Model selection in covariance structures analysis and the "problem" of sample size: a clarification

Complex models for covariance matrices are structures that specify many parameters, whereas simple models require only a few. When a set of models of differing complexity is evaluated by means of some goodness of fit indices, structures with many parameters are more likely to be selected when the number of observations is large, regardless of other utility considerations. This is known as the sample size problem in model selection decisions. This article argues that this influence of sample size is not necessarily undesirable. The rationale behind this point of view is described in terms of the relationships among the population covariance matrix and 2 model-based estimates of it. The implications of these relationships for practical use are discussed.     

Information Matrices and Standard Errors for MLEs of Item Parameters in IRT

The paper clarifies the relationship among several information matrices for the maximum likelihood estimates (MLEs) of item parameters. It shows that the process of calculating the observed information matrix also generates a related matrix that is the middle piece of a sandwich-type covariance matrix. Monte Carlo results indicate that standard errors (SEs) based on the observed information matrix are robust to many, but not all, conditions of model/distribution misspecifications. SEs based on the sandwich-type covariance matrix perform most consistently across conditions. Results also suggest that SEs based on other matrices are either not consistent or perform not as robust as those based on the sandwich-type covariance matrix or the observed information matrix.     

How many parameters can a model have and still be testable?

A standard rule of thumb states that a model has too many parameters to be testable if and only if it has at least as many parameters as empirically observable quantities. We argue that when one asks whether a model has too many parameters to be testable, one implicitly refers to a particular type of testability, which we call quantitative testability. A model is defined to be quantitatively testable if the model's predictions have zero probability of being correct by chance. Next, we propose a new rule of thumb, based on the rank of the Jacobian matrix of a model (i.e., the matrix of partial derivatives of the function that maps the model's parameter values onto predicted experimental outcomes). According to this rule, a model is quantitatively testable if and only if the rank of the Jacobian matrix is less than the number of observables. (The rank of his matrix can be found with standard computer algorithms.) Using Sard's theorem, we prove that the proposed new rule of thumb is correct provided that certain “smoothness” conditions are satisfied. We also discuss the relation between quantitative testability and reparameterization, identifiability, and goodness-of-fit testing.     

A new algorithm for the least-squares solution in factor analysis

A new algorithm to obtain the least-squares or MINRES solution in common factor analysis is presented. It is based on the up-and-down Marquardt algorithm developed by the present authors for a general nonlinear least-squares problem. Experiments with some numerical models and some empirical data sets showed that the algorithm worked nicely and that SMC (Squared Multiple Correlation) performed best among four sets of initial values for common variances but that the solution might sometimes be very sensitive to fluctuations in the sample covariance matrix.Numerical computation was made on a NEAC S-1000 computer in the Computer Center, Osaka University.     

Information matrix estimation procedures for cognitive diagnostic models

Two new methods to estimate the asymptotic covariance matrix for marginal maximum likelihood estimation of cognitive diagnosis models (CDMs), the inverse of the observed information matrix and the sandwich-type estimator, are introduced. Unlike several previous covariance matrix estimators, the new methods take into account both the item and structural parameters. The relationships between the observed information matrix, the empirical cross-product information matrix, the sandwich-type covariance matrix and the two approaches proposed by de la Torre (2009, J. Educ. Behav. Stat., 34, 115) are discussed. Simulation results show that, for a correctly specified CDM and Q-matrix or with a slightly misspecified probability model, the observed information matrix and the sandwich-type covariance matrix exhibit good performance with respect to providing consistent standard errors of item parameter estimates. However, with substantial model misspecification only the sandwich-type covariance matrix exhibits robust performance.     

Multilevel models for multiple-baseline data: modeling across-participant variation in autocorrelation and residual variance

Multilevel models (MLM) have been used as a method for analyzing multiple-baseline single-case data. However, some concerns can be raised because the models that have been used assume that the Level-1 error covariance matrix is the same for all participants. The purpose of this study was to extend the application of MLM of single-case data in order to accommodate across-participant variation in the Level-1 residual variance and autocorrelation. This more general model was then used in the analysis of single-case data sets to illustrate the method, to estimate the degree to which the autocorrelation and residual variances differed across participants, and to examine whether inferences about treatment effects were sensitive to whether or not the Level-1 error covariance matrix was allowed to vary across participants. The results from the analyses of five published studies showed that when the Level-1 error covariance matrix was allowed to vary across participants, some relatively large differences in autocorrelation estimates and error variance estimates emerged. The changes in modeling the variance structure did not change the conclusions about which fixed effects were statistically significant in most of the studies, but there was one exception. The fit indices did not consistently support selecting either the more complex covariance structure, which allowed the covariance parameters to vary across participants, or the simpler covariance structure. Given the uncertainty in model specification that may arise when modeling single-case data, researchers should consider conducting sensitivity analyses to examine the degree to which their conclusions are sensitive to modeling choices.     

Estimation of covariance structure models with parameters subject to functional restraints

This paper demonstrates the feasibility of using the penalty function method to estimate parameters that are subject to a set of functional constraints in covariance structure analysis. Both types of inequality and equality constraints are studied. The approaches of maximum likelihood and generalized least squares estimation are considered. A modified Scoring algorithm and a modified Gauss-Newton algorithm are implemented to produce the appropriate constrained estimates. The methodology is illustrated by its applications to Heywood cases in confirmatory factor analysis, quasi-Weiner simplex model, and multitrait-multimethod matrix analysis.The author is indebted to several anonymous reviewers for creative suggestions for improvement of this paper. Computer funding is provided by the Computer Services Centre, The Chinese University of Hong Kong.     

A Class of Population Covariance Matrices in the Bootstrap Approach to Covariance Structure Analysis

Model evaluation in covariance structure analysis is critical before the results can be trusted. Due to finite sample sizes and unknown distributions of real data, existing conclusions regarding a particular statistic may not be applicable in practice. The bootstrap procedure automatically takes care of the unknown distribution and, for a given sample size, also provides more accurate results than those based on standard asymptotics. But the procedure needs a matrix to play the role of the population covariance matrix. The closer the matrix is to the true population covariance matrix, the more valid the bootstrap inference is. The current paper proposes a class of covariance matrices by combining theory and data. Thus, a proper matrix from this class is closer to the true population covariance matrix than those constructed by any existing methods. Each of the covariance matrices is easy to generate and also satisfies several desired properties. An example with nine cognitive variables and a confirmatory factor model illustrates the details for creating population covariance matrices with different misspecifications. When evaluating the substantive model, bootstrap or simulation procedures based on these matrices will lead to more accurate conclusion than that based on artificial covariance matrices.     

融合反应时的多级评分IRT模型开发及其应用研究

当前大多数融合反应时的IRT模型仅适用于0-1评分数据资料,极大的限制了IRT反应时模型在实际中的应用。本文在传统的二级计分反应时IRT模型基础上,拟开发一种多级评分反应时模型。在层次建模框架下,分别采用拓广分部评分模型(GPCM)和对数正态模型构建融合反应时的多级评分IRT模型(本文记为JRT-GPCM),并采用全息贝叶斯MCMC算法实现新模型的参数估计。为验证新开发的JRT-GPCM模型的可行性及其在实践中的应用,本文开展了两项研究:研究1为模拟实验研究,研究2为新模型在大五人格-神经质分量表中的应用。研究1结果表明,JRT-GPCM模型的估计精度较高,且具有较好的稳健性。研究2表明,被试的潜在特质与作答速度具有一定的正相关,且本研究结果支持Ferrando和Lorenzo-Seva(2007)提出的“距离-困难度假设”,即当被试的潜在特质与项目的难度阈限距离越远,那么被试会花费更多的时间对项目进行作答。总之,本研究为拓展反应时信息在心理测量及教育中的应用提供新的方法支持。     

An estimate of the covariance between variables which are not jointly observed

Situations sometimes arise in which variables collected in a study are not jointly observed. This typically occurs because of study design. An example is an equating study where distinct groups of subjects are administered different sections of a test. In the normal maximum likelihood function to estimate the covariance matrix among all variables, elements corresponding to those that are not jointly observed are unidentified. If a factor analysis model holds for the variables, however, then all sections of the matrix can be accurately estimated, using the fact that the covariances are a function of the factor loadings. Standard errors of the estimated covariances can be obtained by the delta method. In addition to estimating the covariance matrix in this design, the method can be applied to other problems such as regression factor analysis. Two examples are presented to illustrate the method. This research was partially supported by NIMH grant MH5-4576     

Hierarchical Multinomial Processing Tree Models: A Latent-Trait Approach

Multinomial processing tree models are widely used in many areas of psychology. A hierarchical extension of the model class is proposed, using a multivariate normal distribution of person-level parameters with the mean and covariance matrix to be estimated from the data. The hierarchical model allows one to take variability between persons into account and to assess parameter correlations. The model is estimated using Bayesian methods with weakly informative hyperprior distribution and a Gibbs sampler based on two steps of data augmentation. Estimation, model checks, and hypotheses tests are discussed. The new method is illustrated using a real data set, and its performance is evaluated in a simulation study.     

Structural equation modeling with heavy tailed distributions

Data in social and behavioral sciences typically possess heavy tails. Structural equation modeling is commonly used in analyzing interrelations among variables of such data. Classical methods for structural equation modeling fit a proposed model to the sample covariance matrix, which can lead to very inefficient parameter estimates. By fitting a structural model to a robust covariance matrix for data with heavy tails, one generally gets more efficient parameter estimates. Because many robust procedures are available, we propose using the empirical efficiency of a set of invariant parameter estimates in identifying an optimal robust procedure. Within the class of elliptical distributions, analytical results show that the robust procedure leading to the most efficient parameter estimates also yields a most powerful test statistic. Examples illustrate the merit of the proposed procedure. The relevance of this procedure to data analysis in a broader context is noted. The authors thank the editor, an associate editor and four referees for their constructive comments, which led to an improved version of the paper.     

Constructing a covariance matrix that yields a specified minimizer and a specified minimum discrepancy function value

A method is presented for constructing a covariance matrix Σ*₀ that is the sum of a matrix Σ(γ₀) that satisfies a specified model and a perturbation matrix,E, such that Σ*₀=Σ(γ0) +E. The perturbation matrix is chosen in such a manner that a class of discrepancy functionsF(Σ*₀, Σ(γ₀)), which includes normal theory maximum likelihood as a special case, has the prespecified parameter value γ₀ as minimizer and a prespecified minimum δ A matrix constructed in this way seems particularly valuable for Monte Carlo experiments as the covariance matrix for a population in which the model does not hold exactly. This may be a more realistic conceptualization in many instances. An example is presented in which this procedure is employed to generate a covariance matrix among nonnormal, ordered categorical variables which is then used to study the performance of a factor analysis estimator. We are grateful to Alexander Shapiro for suggesting the proof of the solution in section 2.     

Identification and Small Sample Estimation of Thurstone's Unrestricted Model for Paired Comparisons Data

The interpretation of a Thurstonian model for paired comparisons where the utilities' covariance matrix is unrestricted proved to be difficult due to the comparative nature of the data. We show that under a suitable constraint the utilities' correlation matrix can be estimated, yielding a readily interpretable solution. This set of identification constraints can recover any true utilities' covariance matrix, but it is not unique. Indeed, we show how to transform the estimated correlation matrix into alternative correlation matrices that are equally consistent with the data but may be more consistent with substantive theory. Also, we show how researchers can investigate the sample size needed to estimate a particular model by exploiting the simulation capabilities of a popular structural equation modeling statistical package.     

Three-Mode Factor Analysis by Means of Candecomp/Parafac

A three-mode covariance matrix contains covariances of N observations (e.g., subject scores) on J variables for K different occasions or conditions. We model such an JK×JK covariance matrix as the sum of a (common) covariance matrix having Candecomp/Parafac form, and a diagonal matrix of unique variances. The Candecomp/Parafac form is a generalization of the two-mode case under the assumption of parallel factors. We estimate the unique variances by Minimum Rank Factor Analysis. The factors can be chosen oblique or orthogonal. Our approach yields a model that is easy to estimate and easy to interpret. Moreover, the unique variances, the factor covariance matrix, and the communalities are guaranteed to be proper, a percentage of explained common variance can be obtained for each variable-condition combination, and the estimated model is rotationally unique under mild conditions. We apply our model to several datasets in the literature, and demonstrate our estimation procedure in a simulation study.     

Some properties of estimated scale invariant covariance structures

Scale invariance is a property shared by many covariance structure models employed in practice. An example is provided by the well-known LISREL model subject only to classical normalizations and zero constraints on the parameters. It is shown that scale invariance implies that the estimated covariannce matrix must satisfy certain equations, and the nature of these equations depends on the fitting function used. In this context, the paper considers two classes of fitting functions: weighted least squares and the class of functions proposed by Swain.Constructive comments by the referees are greatly appreciated. The author gratefully acknowledges Michael Browne's interest in his work.     

A note on structural models for the circumplex

Jöreskog (1974) developed a latent variable model for the covariance structure of the circumplex which, under certain conditions, includes a model for a patterned correlation matrix (Browne, 1977). This model is of limited usefulness, however, in that it employs a known matrix that is rank deficient for many problems. Furthermore, the model is inappropriate for the circumplex which contains negative covariances. This paper presents alternative models for the perfect circumplex and quasi-circumplex that avoids these difficulties, and that includes the important model for a patterned correlation circumplex matrix. Two numerical examples are provided.This research was supported in part by a grant from the Graduate School of the University of Minnesota. I wish to thank M. W. Browne for suggesting the final model presented in this paper. James Steiger and the Editor also made several valuable suggestions.     

SEM of another flavour: Two new applications of the supplemented EM algorithm

The supplemented EM (SEM) algorithm is applied to address two goodness‐of‐fit testing problems in psychometrics. The first problem involves computing the information matrix for item parameters in item response theory models. This matrix is important for limited‐information goodness‐of‐fit testing and it is also used to compute standard errors for the item parameter estimates. For the second problem, it is shown that the SEM algorithm provides a convenient computational procedure that leads to an asymptotically chi‐squared goodness‐of‐fit statistic for the ‘two‐stage EM’ procedure of fitting covariance structure models in the presence of missing data. Both simulated and real data are used to illustrate the proposed procedures.     

The kinetic limit of superheating induced by semicoherent interfaces

The superheating behaviour of embedded particles induced by semicoherent interfaces has been observed in many circumstances. In this paper, a phenomeno‐ logical model for melt nucleation on misfit dislocations at a semicoherent interface is proposed. A kinetic limit for semicoherent-interface-induced superheating, which is in good agreement with the results of experiments and computer simulations, is derived from this model. Calculations and analyses based on the model reveal that melting prefers to initiate at the semicoherent interface and that superheating of embedded particles is possible for a melt nucleation contact angle less than 90°. Among the matrix-dependent parameters, the contact angle and the shear modulus of the matrix are found to be dominant in determining the superheating of embedded particles.     

Linear structural equations with latent variables

An interdependent multivariate linear relations model based on manifest, measured variables as well as unmeasured and unmeasurable latent variables is developed. The latent variables include primary or residual common factors of any order as well as unique factors. The model has a simpler parametric structure than previous models, but it is designed to accommodate a wider range of applications via its structural equations, mean structure, covariance structure, and constraints on parameters. The parameters of the model may be estimated by gradient and quasi-Newton methods, or a Gauss-Newton algorithm that obtains least-squares, generalized least-squares, or maximum likelihood estimates. Large sample standard errors and goodness of fit tests are provided. The approach is illustrated by a test theory model and a longitudinal study of intelligence.This investigation was supported in part by a Research Scientist Development Award (KO2-DA00017) and a research grant (DA01070) from the U. S. Public Health Service.