Identification with deficient rank loading matrices in confirmatory factor analysis: Multitrait-multimethod models

This paper presents some results on identification in multitrait-multimethod (MTMM) confirmatory factor analysis (CFA) models. Some MTMM models are not identified when the (factorial-patterned) loadings matrix is of deficient column rank. For at least one other MTMM model, identification does exist despite such deficiency. It is also shown that for some MTMM CFA models, Howe's (1955) conditions sufficient for rotational uniqueness can fail, yet the model may well be identified and rotationally unique. Implications of these results for CFA models in general are discussed.     

Modeling Common Traits and Method Effects in Multitrait-Multimethod Analysis

Method effects often occur when constructs are measured by different methods. In traditional multitrait-multimethod (MTMM) models method effects are regarded as residuals, which implies a mean method effect of zero and no correlation between trait and method effects. Furthermore, in some recent MTMM models, traits are modeled to be specific to a certain method. However, often we are not interested in a method-specific trait but in a trait that is common to all methods. Here we present the Method Effect model with common trait factors, which allows modeling “common” trait factors and method factors that represent method “effects” rather than residuals. The common trait factors are defined as the mean of the true-score variables of all variables measuring the same trait and the method factors are defined as differences between true-score variables and means of true-score variables. Because the model allows estimating mean method effects, correlations between method factors, and correlations between trait and method factors, new research questions may be investigated. The application of the model is demonstrated by 2 examples studying the effect of negative, as compared with positive, item wording for the measurement of mood states.     

Controlling Correlational Bias via Confirmatory Factor Analysis of MTMM Data

Confirmatory factor analysis of multitrait-multimethod (MTMM) data has proven to be a useful tool for assessing convergent and discriminant validity. However, researchers have not made full use of the results of MTMM analyses in examining the relationship between MTMM factors and variables outside the MTMM. Often, researchers simply average the various measures of each trait. Alternatively, they estimate LISREL MTMM models, but estimate only relationships between MTMM traits and the outside variables. In the present article, we show that these two approaches to analyzing data outside the MTMM produce equally highly biased parameter estimates when the actual correlations between MTMM method factors and the outside variables are substantial. An algebraic explanation and a simulated data illustration are given for the bias due to misspecification. Also, the problem is illustrated with a brief empirical example. Implications for applied research are discussed.     

Separating trait effects from trait-specific method effects in multitrait-multimethod models: a multiple-indicator CT-C(M-1) model

An overview of several models of confirmatory factor analysis for analyzing multitrait-multimethod (MTMM) data and a discussion of their advantages and limitations are provided. A new class of multi-indicator MTMM models combines several strengths and avoids a number of serious shortcomings inherent in previously developed MTMM models. The new models enable researchers to specify and to test trait-specific-method effects. The trait and method concepts composing these models are explained in detail and are contrasted with those of previously developed MTMM models for multiple indicators. The definitions of the models are explained step by step, and a practical empirical application of the models to the measurement of 3 traits x 3 methods is used to demonstrate their advantages and limitations.     

The analysis of multitrait-multimethod matrices via constrained components analysis

Multitrait-Multimethod (MTMM) matrices are often analyzed by means of confirmatory factor analysis (CFA). However, fitting MTMM models often leads to improper solutions, or non-convergence. In an attempt to overcome these problems, various alternative CFA models have been proposed, but with none of these the problem of finding improper solutions was solved completely. In the present paper, an approach is proposed where improper solutions are ruled out altogether and convergence is guaranteed. The approach is based on constrained variants of components analysis (CA). Besides the fact that these methods do not give improper solutions, they have the advantage that they provide component scores which can later on be used to relate the components to external variables. The new methods are illustrated by means of simulated data, as well as empirical data sets.This research has been made possible by a fellowship from the Royal Netherlands Academy of Arts and Sciences to the first author. The authors are obliged to three anonymous reviewers and an associate editor for constructive suggestions on the first version of this paper.     

Confirmatory Factor Analysis of Multitrait-Multimethod Data: The Construct Validation of Multidimensional Self-Concept Responses

ABSTRACT The multitrait-multimethod (MTMM) design is frequently used to test construct validity and is particularly appropriate for evaluating multidimensional instruments in personality research Despite its popularity there are important problems with both the traditional and confirmatory factor analysis (CFA) approaches to MTMM data In previous research (1989a, 1989b) I described a new CFA approach that I claimed to be relatively immune to many of these problems This approach is applied here to test the construct validity of preadolescent responses to three multidimensional self-concept instruments The substantive contribution of the present investigation is to demonstrate good support for the construct validity of two of the three instruments The methodological contributions are to further substantiate my claims about the new CFA approaches and to demonstrate this approach in a substantively meaningful context that has broad applicability to other personality research     

Structural equation modeling of multitrait-multimethod data: different models for different types of methods

The question as to which structural equation model should be selected when multitrait-multimethod (MTMM) data are analyzed is of interest to many researchers. In the past, attempts to find a well-fitting model have often been data-driven and highly arbitrary. In the present article, the authors argue that the measurement design (type of methods used) should guide the choice of the statistical model to analyze the data. In this respect, the authors distinguish between (a) interchangeable methods, (b) structurally different methods, and (c) the combination of both kinds of methods. The authors present an appropriate model for each type of method. All models allow separating measurement error from trait influences and trait-specific method effects. With respect to interchangeable methods, a multilevel confirmatory factor model is presented. For structurally different methods, the correlated trait-correlated (method-1) model is recommended. Finally, the authors demonstrate how to appropriately analyze data from MTMM designs that simultaneously use interchangeable and structurally different methods. All models are applied to empirical data to illustrate their proper use. Some implications and guidelines for modeling MTMM data are discussed.     

Combining Methodologies in the Construct Validation Process: An Empirical Illustration

Confirmatory Factor Analysis (CFA) has received considerable support as a methodology for assessing construct validity. As with other methodologies, however, numerous problems can be encountered when using CFA to assess construct validity. Given the limitations of the CFA, Multitrait-Multimethod (MTMM), and analysis of variance (ANOVA) methodologies, a set of guidelines was developed. The guidelines recommend that the characteristics of the data set be used to determine how the three alternative construct validation methodologies can be used in combination. Examples from the research literature are discussed in order to demonstrate the application of these guidelines.     

Analysing multitrait–multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: What sample size is needed for valid results?

Convergent and discriminant validity of psychological constructs can best be examined in the framework of multitrait–multimethod (MTMM) analysis. To gain information at the level of single items, MTMM models for categorical variables have to be applied. The CTC(M?1) model is presented as an example of an MTMM model for ordinal variables. Based on an empirical application of the CTC(M?1) model, a complex simulation study was conducted to examine the sample size requirements of the robust weighted least squares mean‐ and variance‐adjusted χ² test of model fit (WLSMV estimator) implemented in Mplus. In particular, the simulation study analysed the χ² approximation, the parameter estimation bias, the standard error bias, and the reliability of the WLSMV estimator depending on the varying number of items per trait–method unit (ranging from 2 to 8) and varying sample sizes (250, 500, 750, and 1000 observations). The results showed that the WLSMV estimator provided a good – albeit slightly liberal – χ² approximation and stable and reliable parameter estimates for models of reasonable complexity (2–4 items) and small sample sizes (at least 250 observations). When more complex models with 5 or more items were analysed, larger sample sizes of at least 500 observations were needed. The most complex model with 9 trait–method units and 8 items (72 observed variables) requires sample sizes of at least 1000 observations.     

Testing the convergent and discriminant validity of the Decisional Balance Scale of the Transtheoretical Model using the Multi-Trait Multi-Method approach.

The authors extended research on the construct validity of the Decisional Balance Scale for smoking in adolescence by testing its convergent and discriminant validity. Hierarchical confirmatory factor analysis multi-trait multi-method approach (HCFA MTMM) was used with data from 2,334 UK adolescents, both smokers and non-smokers. They completed computerized and paper versions of the questionnaire on 3 occasions over 2 years. The results indicated a 3-factor solution; Social Pros, Coping Pros, and Cons fit the data best. The HCFA MTMM model fit the data well, with correlated methods and correlated trait factors. Subsequent testing confirmed discriminant validity between the factors and convergent validity of both methods of administering the questionnaire. There was, however, clear evidence of a method effect, which may have arisen due to different response formats or may be a function of the method of presentation. Taken with other data, there is strong evidence for construct validity of Decisional Balance for smoking in adolescence, but evidence of predictive validity is required.     

Bayesian analysis of longitudinal multitrait–multimethod data with ordinal response variables

A new multilevel latent state graded response model for longitudinal multitrait–multimethod (MTMM) measurement designs combining structurally different and interchangeable methods is proposed. The model allows researchers to examine construct validity over time and to study the change and stability of constructs and method effects based on ordinal response variables. We show how Bayesian estimation techniques can address a number of important issues that typically arise in longitudinal multilevel MTMM studies and facilitates the estimation of the model presented. Estimation accuracy and the impact of between‐ and within‐level sample sizes as well as different prior specifications on parameter recovery were investigated in a Monte Carlo simulation study. Findings indicate that the parameters of the model presented can be accurately estimated with Bayesian estimation methods in the case of low convergent validity with as few as 250 clusters and more than two observations within each cluster. The model was applied to well‐being data from a longitudinal MTMM study, assessing the change and stability of life satisfaction and subjective happiness in young adults after high‐school graduation. Guidelines for empirical applications are provided and advantages and limitations of a Bayesian approach to estimating longitudinal multilevel MTMM models are discussed.     

The use of configural frequency analysis for explorative data analysis

Configural frequency analysis (CFA) is a widely used method of explorative data analysis. It tries to detect patterns in the data that occur significantly more or significantly less often than expected by chance. Patterns which occur more often than expected by chance are called CFA types, while those which occur less often than expected by chance are called CFA antitypes. The patterns detected are used to generate knowledge about the mechanisms underlying the data. We investigate the ability of CFA to detect adequate types and antitypes in a number of simulation studies. The basic idea of these studies is to predefine sets of types and antitypes and a mechanism which uses them to create a simulated data set. This simulated data set is then analysed with CFA and the detected types and antitypes are compared to the predefined ones. The predefined types and antitypes together with the method to generate the data are called a data generation model. The results of the simulation studies show that CFA can be used in quite different research contexts to detect structural dependencies in observed data. In addition, we can learn from these simulation studies how much data is necessary to enable CFA to reconstruct the predefined types and antitypes with sufficient accuracy. For one of the data generation models investigated, implicitly underlying knowledge space theory, it was shown that zero‐order CFA can be used to reconstruct the predefined types (which can be interpreted in this context as knowledge states) with sufficient accuracy. Theoretical considerations show that first‐order CFA cannot be used for this data generation model. Thus, it is wrong to consider first‐order CFA, as is done in many publications, as the standard or even only method of CFA.     

Longitudinal Multi-Trait-State-Method Model Using Ordinal Data

Multi-trait multi-method (MTMM) models provide a way to assess convergent and discriminant validity when multiple traits are measured by multiple methods. In recent years, longitudinal extensions of MTMM models have been proposed in the structural equation modeling framework to evaluate whether and how the trait as well as method factors change over time. We propose a novel longitudinal ordinal MTMM model that can be used to effectively distinguish volatile “state” processes from “trait” processes that tend to remain stable and invariant over time. The proposed model, termed a longitudinal multi-trait-state-method (LM-TSM) model, combines 3 key modeling components: (a) a measurement model for ordinal data, (b) a vector autoregressive moving average model at the latent level to examine changes in the state as well as the method factors over time, and (c) a second-order factor-analytic model to capture time-invariant traits as shared variances among the state factors across all measurement occasions. Data from the Affective Dynamics and Individual Differences (ADID; Emotions and Dynamic Systems Laboratory, 2010 Emotions and Dynamic Systems Laboratory. (2010). The Affective Dynamics and Individual Differences (ADID) study: Developing non-stationary and network-based methods for modeling the perception and physiology of emotions. . Unpublished manual, University of North Carolina at Chapel Hill. [Google Scholar]) study was used to illustrate the proposed longitudinal LM-TSM model. Methodological issues associated with fitting the LM-TSM model are discussed.     

Constant latent odds-ratios models and the mantel-haenszel null hypothesis

In the present paper, a new family of item response theory (IRT) models for dichotomous item scores is proposed. Two basic assumptions define the most general model of this family. The first assumption is local independence of the item scores given a unidimensional latent trait. The second assumption is that the odds-ratios for all item-pairs are constant functions of the latent trait. Since the latter assumption is characteristic of the whole family, the models are called constant latent odds-ratios (CLORs) models. One nonparametric special case and three parametric special cases of the general CLORs model are shown to be generalizations of the one-parameter logistic Rasch model. For all CLORs models, the total score (the unweighted sum of the item scores) is shown to be a sufficient statistic for the latent trait. In addition, conditions under the general CLORs model are studied for the investigation of differential item functioning (DIF) by means of the Mantel-Haenszel procedure. This research was supported by the Dutch Organization for Scientific Research (NWO), grant number 400-20-026.     

Personality Correlates of Midlife Cardiometabolic Risk: The Explanatory Role of Higher‐Order Factors of the Five‐Factor Model

Varying associations are reported between Five‐Factor Model (FFM) personality traits and cardiovascular disease risk. Here, we further examine dispositional correlates of cardiometabolic risk within a hierarchical model of personality that proposes higher‐order traits of Stability (shared variance of Agreeableness, Conscientiousness, inverse Neuroticism) and Plasticity (Extraversion, Openness), and we test hypothesized mediation via biological and behavioral factors. In an observational study of 856 community volunteers aged 30–54 years (46% male, 86% Caucasian), latent variable FFM traits (using multiple‐informant reports) and aggregated cardiometabolic risk (indicators: insulin resistance, dyslipidemia, blood pressure, adiposity) were estimated using confirmatory factor analysis (CFA). The cardiometabolic factor was regressed on each personality factor or higher‐order trait. Cross‐sectional indirect effects via systemic inflammation, cardiac autonomic control, and physical activity were tested. CFA models confirmed the Stability “meta‐trait,” but not Plasticity. Lower Stability was associated with heightened cardiometabolic risk. This association was accounted for by inflammation, autonomic function, and physical activity. Among FFM traits, only Openness was associated with risk over and above Stability, and, unlike Stability, this relationship was unexplained by the intervening variables. A Stability meta‐trait covaries with midlife cardiometabolic risk, and this association is accounted for by three candidate biological and behavioral factors.     

The Value of Exploratory Structural Equation Modeling in Identifying Factor Overlap in the Mental Health Continuum-Short Form (MHC-SF): A Study with a New Zealand Sample

This study is the first to investigate the factor structure of the Mental Health Continuum-Short Form (MHC-SF) in New Zealand. Towards this end, traditional Confirmatory Factor Analysis (CFA) and the new method of Exploratory Structural Equation Modeling (ESEM) were used. Both ESEM and CFA supported the tripartite model of mental well-being in comparison to the one- and two-factor models; however, ESEM provided better fit with the data. Moreover, interfactor correlations were considerably lower in ESEM than they were in CFA, indicating greater factor distinctiveness in ESEM. ESEM also revealed a number of important cross-loadings for items in the measurement model of the MHC-SF. The results supported full metric and full scalar invariance of the MHC-SF across gender. The attenuated correlations among well-being factors obtained by ESEM here provide an important insight about the ongoing controversy regarding the failure of empirical research to identify distinct eudaimonic and hedonic factors in well-being measures. An overreliance on CFA methods may have led the field to rely on inflated estimates of shared variance between eudaimonia and hedonia.     

Robustness of Parameter Estimation to Assumptions of Normality in the Multidimensional Graded Response Model

A central assumption that is implicit in estimating item parameters in item response theory (IRT) models is the normality of the latent trait distribution, whereas a similar assumption made in categorical confirmatory factor analysis (CCFA) models is the multivariate normality of the latent response variables. Violation of the normality assumption can lead to biased parameter estimates. Although previous studies have focused primarily on unidimensional IRT models, this study extended the literature by considering a multidimensional IRT model for polytomous responses, namely the multidimensional graded response model. Moreover, this study is one of few studies that specifically compared the performance of full-information maximum likelihood (FIML) estimation versus robust weighted least squares (WLS) estimation when the normality assumption is violated. The research also manipulated the number of nonnormal latent trait dimensions. Results showed that FIML consistently outperformed WLS when there were one or multiple skewed latent trait distributions. More interestingly, the bias of the discrimination parameters was non-ignorable only when the corresponding factor was skewed. Having other skewed factors did not further exacerbate the bias, whereas biases of boundary parameters increased as more nonnormal factors were added. The item parameter standard errors recovered well with both estimation algorithms regardless of the number of nonnormal dimensions.     

Temporal patterns of variable relationships in person-oriented research: longitudinal models of configural frequency analysis

This article reviews the premises of configural frequency analysis (CFA), including methods of choosing significance tests and base models, as well as protecting alpha, and discusses why CFA is a useful approach when conducting longitudinal person-oriented research. CFA operates at the manifest variable level. Longitudinal CFA seeks to identify those temporal patterns that stand out as more frequent (CFA types) or less frequent (CFA antitypes) than expected with reference to a base model. A base model that has been used frequently in CFA applications, prediction CFA, and a new base model, auto-association CFA, are discussed for analysis of cross-classifications of longitudinal data. The former base model takes the associations among predictors and among criteria into account. The latter takes the auto-associations among repeatedly observed variables into account. Application examples of each are given using data from a longitudinal study of domestic violence. It is demonstrated that CFA results are not redundant with results from log-linear modeling or multinomial regression and that, of these approaches, CFA shows particular utility when conducting person-oriented research.