Structural analysis of covariance and correlation matrices

A general approach to the analysis of covariance structures is considered, in which the variances and covariances or correlations of the observed variables are directly expressed in terms of the parameters of interest. The statistical problems of identification, estimation and testing of such covariance or correlation structures are discussed.Several different types of covariance structures are considered as special cases of the general model. These include models for sets of congeneric tests, models for confirmatory and exploratory factor analysis, models for estimation of variance and covariance components, regression models with measurement errors, path analysis models, simplex and circumplex models. Many of the different types of covariance structures are illustrated by means of real data.1978 Psychometric Society Presidential Address.This research has been supported by the Bank of Sweden Tercentenary Foundation under the project entitledStructural Equation Models in the Social Sciences, Karl G. Jöreskog, project director.     

An estimate of the covariance between variables which are not jointly observed

Situations sometimes arise in which variables collected in a study are not jointly observed. This typically occurs because of study design. An example is an equating study where distinct groups of subjects are administered different sections of a test. In the normal maximum likelihood function to estimate the covariance matrix among all variables, elements corresponding to those that are not jointly observed are unidentified. If a factor analysis model holds for the variables, however, then all sections of the matrix can be accurately estimated, using the fact that the covariances are a function of the factor loadings. Standard errors of the estimated covariances can be obtained by the delta method. In addition to estimating the covariance matrix in this design, the method can be applied to other problems such as regression factor analysis. Two examples are presented to illustrate the method. This research was partially supported by NIMH grant MH5-4576     

Modified Distribution-Free Goodness-of-Fit Test Statistic

Covariance structure analysis and its structural equation modeling extensions have become one of the most widely used methodologies in social sciences such as psychology, education, and economics. An important issue in such analysis is to assess the goodness of fit of a model under analysis. One of the most popular test statistics used in covariance structure analysis is the asymptotically distribution-free (ADF) test statistic introduced by Browne (Br J Math Stat Psychol 37:62–83, 1984). The ADF statistic can be used to test models without any specific distribution assumption (e.g., multivariate normal distribution) of the observed data. Despite its advantage, it has been shown in various empirical studies that unless sample sizes are extremely large, this ADF statistic could perform very poorly in practice. In this paper, we provide a theoretical explanation for this phenomenon and further propose a modified test statistic that improves the performance in samples of realistic size. The proposed statistic deals with the possible ill-conditioning of the involved large-scale covariance matrices.     

Conditional Covariance Theory and Detect for Polytomous Items

This paper extends the theory of conditional covariances to polytomous items. It has been proven that under some mild conditions, commonly assumed in the analysis of response data, the conditional covariance of two items, dichotomously or polytomously scored, given an appropriately chosen composite is positive if, and only if, the two items measure similar constructs besides the composite. The theory provides a theoretical foundation for dimensionality assessment procedures based on conditional covariances or correlations, such as DETECT and DIMTEST, so that the performance of these procedures is theoretically justified when applied to response data with polytomous items. Various estimators of conditional covariances are constructed, and special attention is paid to the case of complex sampling data, such as those from the National Assessment of Educational Progress (NAEP). As such, the new version of DETECT can be applied to response data sets not only with polytomous items but also with missing values, either by design or at random. DETECT is then applied to analyze the dimensional structure of the 2002 NAEP reading samples of grades 4 and 8. The DETECT results show that the substantive test structure based on the purposes for reading is consistent with the statistical dimensional structure for either grade. This research was supported by the Educational Testing Service and the National Assessment of Educational Progress (Grant R902F980001), US Department of Education. The opinions expressed herein are solely those of the author and do not necessarily represent those of the Educational Testing Service. The author would like to thank Ting Lu, Paul Holland, Shelby Haberman, and Feng Yu for their comments and suggestions. Requests for reprints should be sent to Jinming Zhang, Educational Testing Service, MS 02-T, Rosedale Road, Princeton, NJ 08541, USA. E-mail: jzhang@ets.org     

The Nonsingularity of Γ in Covariance Structure Analysis of Nonnormal Data

Covariance structure analysis of nonnormal data is important because in practice all data are nonnormal. When applying covariance structure analysis to nonnormal data, it is generally assumed that the asymptotic covariance matrix Γ for the nonredundant terms in the sample covariance matrix S is nonsingular. It is shown this need not be the case, which raises a question of how restrictive this assumption may be and how difficult it may be to verify it. It is shown that Γ is nonsingular whenever sampling is from a nonsingular distribution, including any distribution defined by a density function. In the discrete case necessary and sufficient conditions are given for the nonsingularity of Γ, and it is shown how to demonstrate Γ is nonsingular with high probability. Thus, the nonsingularity of Γ assumption is mild and one should feel comfortable about making it. These observations also apply to the asymptotic covariance matrix Γ that arises in structural equation modeling.     

On the Relation Between the Linear Factor Model and the Latent Profile Model

The relationship between linear factor models and latent profile models is addressed within the context of maximum likelihood estimation based on the joint distribution of the manifest variables. Although the two models are well known to imply equivalent covariance decompositions, in general they do not yield equivalent estimates of the unconditional covariances. In particular, a 2-class latent profile model with Gaussian components underestimates the observed covariances but not the variances, when the data are consistent with a unidimensional Gaussian factor model. In explanation of this phenomenon we provide some results relating the unconditional covariances to the goodness of fit of the latent profile model, and to its excess multivariate kurtosis. The analysis also leads to some useful parameter restrictions related to symmetry.     

Conditional covariance structure of generalized compensatory multidimensional items

Some nonparametric dimensionality assessment procedures, such as DIMTEST and DETECT, use nonparametric estimates of item pair conditional covariances given an appropriately chosen subtest score as their basic building blocks. Such conditional covariances given some subtest score can be regarded as an approximation to the conditional covariances given an appropriately chosen unidimensional latent composite, where the composite is oriented in the multidimensional test space direction in which the subtest score measures best. In this paper, the structure and properties of such item pair conditional covariances given a unidimensional latent composite are thoroughly investigated, assuming a semiparametric IRT modeling framework called a generalized compensatory model. It is shown that such conditional covariances are highly informative about the multidimensionality structure of a test. The theory developed here is very useful in establishing properties of dimensionality assessment procedures, current and yet to be developed, that are based upon estimating such conditional covariances.In particular, the new theory is used to justify the DIMTEST procedure. Because of the importance of conditional covariance estimation, a new bias reducing approach is presented. A byproduct of likely independent importance beyond the study of conditional covariances is a rigorous score information based definition of an item's and a score's direction of best measurement in the multidimensional test space.This paper is based on a chapter of the first author's doctoral dissertation, written at the University of Illinois and supervised by the second author. Part of this research has been presented at the annual meeting of the National Council on Measurement in Education, San Francisco, April 1995.The authors would like to thank Jeff Douglas, Xuming He and Ming-mei Wang for their comments and suggestions. The research of the first author was partially supported by an ETS/GREB Psychometric Fellowship, and by Educational Testing Service Research Allocation Project 884-01. The research of the second author was partially supported by NSF grant DMS 97-04474.     

A note on structural models for the circumplex

Jöreskog (1974) developed a latent variable model for the covariance structure of the circumplex which, under certain conditions, includes a model for a patterned correlation matrix (Browne, 1977). This model is of limited usefulness, however, in that it employs a known matrix that is rank deficient for many problems. Furthermore, the model is inappropriate for the circumplex which contains negative covariances. This paper presents alternative models for the perfect circumplex and quasi-circumplex that avoids these difficulties, and that includes the important model for a patterned correlation circumplex matrix. Two numerical examples are provided.This research was supported in part by a grant from the Graduate School of the University of Minnesota. I wish to thank M. W. Browne for suggesting the final model presented in this paper. James Steiger and the Editor also made several valuable suggestions.     

Heterogeneous factor analysis models: A bayesian approach

Multilevel factor analysis models are widely used in the social sciences to account for heterogeneity in mean structures. In this paper we extend previous work on multilevel models to account for general forms of heterogeneity in confirmatory factor analysis models. We specify various models of mean and covariance heterogeneity in confirmatory factor analysis and develop Markov Chain Monte Carlo (MCMC) procedures to perform Bayesian inference, model checking, and model comparison.We test our methodology using synthetic data and data from a consumption emotion study. The results from synthetic data show that our Bayesian model perform well in recovering the true parameters and selecting the appropriate model. More importantly, the results clearly illustrate the consequences of ignoring heterogeneity. Specifically, we find that ignoring heterogeneity can lead to sign reversals of the factor covariances, inflation of factor variances and underappreciation of uncertainty in parameter estimates. The results from the emotion study show that subjects vary both in means and covariances. Thus traditional psychometric methods cannot fully capture the heterogeneity in our data.     

The dynamical structure of handball penalty shots as a function of target location

We used principal components analysis (PCA) to investigate variations in the dynamical structure of handball penalty shots as a factor of target location and phase of shot. Participants completed a total of 10 successful shots to each of four target locations in the handball goal. Three dimensional movement time series data were analyzed. Also, data were analyzed across three temporally distinct time windows in line with the evolving kinematic chain. Statistical analyses were undertaken to determine differences across target locations. There were no significant differences between dynamical structures as a factor of target or phase. Covariance between time evolutions as a factor of target reduced in line with the ranking of the component. When shots were analyzed as three distinct time windows, only the low time evolution covariances suggested differences between targets in any time window. Our findings show that the dynamical structure underpinning the handball penalty shot does not differ greatly across locations. However, the time evolution of principal components suggests there are some variations in dynamics which may differentiate shot direction.     

Using the linear mixed model to analyze nonnormal data distributions in longitudinal designs

Using a Monte Carlo simulation and the Kenward–Roger (KR) correction for degrees of freedom, in this article we analyzed the application of the linear mixed model (LMM) to a mixed repeated measures design. The LMM was first used to select the covariance structure with three types of data distribution: normal, exponential, and log-normal. This showed that, with homogeneous between-groups covariance and when the distribution was normal, the covariance structure with the best fit was the unstructured population matrix. However, with heterogeneous between-groups covariance and when the pairing between covariance matrices and group sizes was null, the best fit was shown by the between-subjects heterogeneous unstructured population matrix, which was the case for all of the distributions analyzed. By contrast, with positive or negative pairings, the within-subjects and between-subjects heterogeneous first-order autoregressive structure produced the best fit. In the second stage of the study, the robustness of the LMM was tested. This showed that the KR method provided adequate control of Type I error rates for the time effect with normally distributed data. However, as skewness increased—as occurs, for example, in the log-normal distribution—the robustness of KR was null, especially when the assumption of sphericity was violated. As regards the influence of kurtosis, the analysis showed that the degree of robustness increased in line with the amount of kurtosis.     

Robustness of normal theory methods in the analysis of linear latent variate models

The structure of the covariance matrix of sample covariances under the class of linear latent variate models is derived using properties of cumulants. This is employed to provide a general framework for robustness of statistical inference in the analysis of covariance structures arising from linear latent variate models. Conditions for normal theory estimators and test statistics to retain each of their usual asymptotic properties under non-normality of latent variates are given. Factor analysis, LISREL and other models are discussed as examples.     

Extending the Debate Between Spearman and Wilson 1929: When do Single Variables Optimally Reproduce the Common Part of the Observed Covariances?

The covariances of observed variables reproduced from conventional factor score predictors are generally not the same as the covariances reproduced from the common factors. We sought to find a factor score predictor that optimally reproduces the common part of the observed covariances. It was found algebraically that—under some conditions—the single observed variable with highest loading on a factor reproduces the non-diagonal elements of the observed covariance matrix more exactly than the conventional factor score predictors. This finding is linked to Spearman's and Wilson's 1929 debate on the use of single variables as factor score predictors. A population-based and a sample-based simulation study confirmed the algebraic result that taking a single variable can outperform conventional factor score predictors in reproducing the non-diagonal covariances when the nonzero loading size and the number of nonzero loadings per factor are small. The results indicated that a weighted aggregation of variables does not necessarily lead to an improvement of the score over the variable with the highest loading.     

Structural equation models with continuous and polytomous variables

A two-stage procedure is developed for analyzing structural equation models with continuous and polytomous variables. At the first stage, the maximum likelihood estimates of the thresholds, polychoric covariances and variances, and polyserial covariances are simultaneously obtained with the help of an appropriate transformation that significantly simplifies the computation. An asymptotic covariance matrix of the estiates is also computed. At the second stage, the parameters in the structural covariance model are obtained via the generalized least squares approach. Basic statistical properties of the estimates are derived and some illustrative examples and a small simulation study are reported.This research was supported in part by a research grant DA01070 from the U. S. Public Health Service. We are indebted to several referees and the editor for very valuable comments and suggestions for improvement of this paper. The computing assistance of King-Hong Leung and Man-Lai Tang is also gratefully acknowledged.     

Sources of structure: genetic, environmental, and artifactual influences on the covariation of personality traits

The phenotypic structure of personality traits has been well described, but it has not yet been explained causally. Behavior genetic covariance analyses can identify the underlying causes of phenotypic structure; previous behavior genetic research has suggested that the effects from both genetic and nonshared environmental influences mirror the phenotype. However, nonshared environmental effects are usually estimated as a residualterm that may also include systematic bias, such as that introduced by implicit personality theory. To reduce that bias, we supplemented data from Canadian and German twin studies with cross-observer correlations on the Revised NEO Personality Inventory. The hypothesized five-factor structure was found in both the phenotypic and genetic/familial covariances. When the residual covariance was decomposed into true nonshared environmental influences and method bias, only the latter showed the five-factor structure. True nonshared environmental influences are not structured as genetic influences are, although there was some suggestion that they do affect two personality dimensions, Conscientiousness and Love. These data reaffirm the value of behavior genetic analyses for research on the underlying causes of personality traits.     

Interrelations Among Models For The Analysis Of Moment Structures

Factor analysis in several populations, covariance structure models, three-mode factor analysis, structural equation systems with measurement model, and analysis of covariance with measurement model are all shown to be specializations of a general moment structure model published previously in this journal. Some new structured linear models are also described; they may be considered either generalizations or special cases of existing models. Simple representations are developed for complex linear models, and some applications to behavioral data are cited.     

Picture of all Solutions of Successive 2-Block Maxbet Problems

The Maxbet method is a generalized principal components analysis of a data set, where the group structure of the variables is taken into account. Similarly, 3-block[12,13] partial Maxdiff method is a generalization of covariance analysis, where only the covariances between blocks (1, 2) and (1, 3) are taken into account. The aim of this paper is to give the global maximum for the 2-block Maxbet and 3-block[12,13] partial Maxdiff problems by picking the best solution from the complete solution set for the multivariate eigenvalue problem involved. To do this, we generalize the characteristic polynomial of a matrix to a system of two characteristic polynomials, and provide the complete solution set of the latter via Sylvester resultants. Examples are provided.     

Three‐mode analysis of multimode covariance matrices

Multimode covariance matrices, such as multitrait‐multimethod matrices, contain the covariances of subject scores on variables for different occasions or conditions. This paper presents a comparison of three‐mode component analysis and three‐mode factor analysis applied to such covariance matrices. The differences and similarities between the non‐stochastic and stochastic approaches are demonstrated by two examples, one of which has a longitudinal design. The empirical comparison is facilitated by deriving, as a heuristic device, a statistic based on the maximum likelihood function for three‐mode factor analysis and its associated degrees of freedom for the three‐mode component models. Furthermore, within the present context a case is made for interpreting the core array as second‐order components.     

Multilevel models for multiple-baseline data: modeling across-participant variation in autocorrelation and residual variance

Multilevel models (MLM) have been used as a method for analyzing multiple-baseline single-case data. However, some concerns can be raised because the models that have been used assume that the Level-1 error covariance matrix is the same for all participants. The purpose of this study was to extend the application of MLM of single-case data in order to accommodate across-participant variation in the Level-1 residual variance and autocorrelation. This more general model was then used in the analysis of single-case data sets to illustrate the method, to estimate the degree to which the autocorrelation and residual variances differed across participants, and to examine whether inferences about treatment effects were sensitive to whether or not the Level-1 error covariance matrix was allowed to vary across participants. The results from the analyses of five published studies showed that when the Level-1 error covariance matrix was allowed to vary across participants, some relatively large differences in autocorrelation estimates and error variance estimates emerged. The changes in modeling the variance structure did not change the conclusions about which fixed effects were statistically significant in most of the studies, but there was one exception. The fit indices did not consistently support selecting either the more complex covariance structure, which allowed the covariance parameters to vary across participants, or the simpler covariance structure. Given the uncertainty in model specification that may arise when modeling single-case data, researchers should consider conducting sensitivity analyses to examine the degree to which their conclusions are sensitive to modeling choices.     

Analysis of covariance structures

A general method is presented for estimating variance components when the experimental design has one random way of classification and a possibly unbalanced fixed classification. The procedure operates on a sample covariance matrix in which the fixed classes play the role of variables and the random classes correspond to observations. Cases are considered which assume (i) homogeneous and (ii) nonhomogeneous error variance, and (iii) arbitrary scale factors in the measurements and homogeneous error variance. The results include maximum-likelihood estimations of the variance components and scale factors, likelihood-ratio tests of the goodness-of-fit of the model assumed for the design, and large-sample variances and covariances of the estimates. Applications to mental test data are presented. In these applications the subjects constitute the random dimension of the design, and a classification of the mental tests according to objective features of format or content constitute the fixed dimensions.Preparation of this paper has been supported in part by NSF Grant GB-939 and U. S. P. H. Grant GM-1286-01. Computer time was donated by the Computation Center, University of Chicago.Now at the University of Chicago.Now at the University of Georgia.