Bayesian analysis of longitudinal multitrait–multimethod data with ordinal response variables

A new multilevel latent state graded response model for longitudinal multitrait–multimethod (MTMM) measurement designs combining structurally different and interchangeable methods is proposed. The model allows researchers to examine construct validity over time and to study the change and stability of constructs and method effects based on ordinal response variables. We show how Bayesian estimation techniques can address a number of important issues that typically arise in longitudinal multilevel MTMM studies and facilitates the estimation of the model presented. Estimation accuracy and the impact of between‐ and within‐level sample sizes as well as different prior specifications on parameter recovery were investigated in a Monte Carlo simulation study. Findings indicate that the parameters of the model presented can be accurately estimated with Bayesian estimation methods in the case of low convergent validity with as few as 250 clusters and more than two observations within each cluster. The model was applied to well‐being data from a longitudinal MTMM study, assessing the change and stability of life satisfaction and subjective happiness in young adults after high‐school graduation. Guidelines for empirical applications are provided and advantages and limitations of a Bayesian approach to estimating longitudinal multilevel MTMM models are discussed.     

A Cross-Classified CFA-MTMM Model for Structurally Different and Nonindependent Interchangeable Methods

Multirater (multimethod, multisource) studies are increasingly applied in psychology. Eid and colleagues (2008) proposed a multilevel confirmatory factor model for multitrait-multimethod (MTMM) data combining structurally different and multiple independent interchangeable methods (raters). In many studies, however, different interchangeable raters (e.g., peers, subordinates) are asked to rate different targets (students, supervisors), leading to violations of the independence assumption and to cross-classified data structures. In the present work, we extend the ML-CFA-MTMM model by Eid and colleagues (2008) to cross-classified multirater designs. The new C4 model (Cross-Classified CTC[M-1] Combination of Methods) accounts for nonindependent interchangeable raters and enables researchers to explicitly model the interaction between targets and raters as a latent variable. Using a real data application, it is shown how credibility intervals of model parameters and different variance components can be obtained using Bayesian estimation techniques.     

A multitrait-multimethod model with minimal assumptions

A new model of confirmatory factor analysis (CFA) for multitrait-multimethod (MTMM) data sets is presented. It is shown that this model can be defined by only three assumptions in the framework of classical psychometric test theory (CTT). All other properties of the model, particularly the uncorrelated-ness of the trait with the method factors are logical consequences of the definition of the model. In the model proposed there are as many trait factors as different traits considered, but the number of method factors is one fewer than the number of methods included in an MTMM study. The covariance structure implied by this model is derived, and it is shown that this model is identified even under conditions under which other CFA-MTMM models are not. The model is illustrated by two empirical applications. Furthermore, its advantages and limitations are discussed with respect to previously developed CFA models for MTMM data.     

Separating trait effects from trait-specific method effects in multitrait-multimethod models: a multiple-indicator CT-C(M-1) model

An overview of several models of confirmatory factor analysis for analyzing multitrait-multimethod (MTMM) data and a discussion of their advantages and limitations are provided. A new class of multi-indicator MTMM models combines several strengths and avoids a number of serious shortcomings inherent in previously developed MTMM models. The new models enable researchers to specify and to test trait-specific-method effects. The trait and method concepts composing these models are explained in detail and are contrasted with those of previously developed MTMM models for multiple indicators. The definitions of the models are explained step by step, and a practical empirical application of the models to the measurement of 3 traits x 3 methods is used to demonstrate their advantages and limitations.     

Structural equation models for interchangeable dyads: being the same makes a difference

Structural equation modeling (SEM) offers a flexible method for studying the patterns of interdependence in partners' behavior, which lie at the heart of interactions and relationships. Although SEM has been applied to the study of distinguishable dyads, in which partners are distinguishable by type, such as male and female, it has rarely been applied to the study of interchangeable dyads, such as male-male or female-female pairs. The authors integrate a wide range of dyadic interdependence models--including actor-partner interdependence models, mutual-influence models, and common-fate or dyadic personality models--into an SEM framework for use with interchangeable dyads. The authors also address the use of latent variables at both the dyadic and individual levels, whereby substantive relationships in these models can be corrected for errors of measurement. Furthermore, the authors discuss the conceptual underpinnings of dyadic models and give examples of their application.     

Analyzing multitrait-mulitmethod data with multilevel confirmatory factor analysis: An application to the validation of the State-Trait Cheerfulness Inventory

This article gives a didactic introduction to the analysis of multitrait-multimethod data with models of multilevel confirmatory factor analysis. In particular, the principles of the multilevel CT-C(M-1) model for interchangeable and structural different methods are explained in detail, and the first application of this model to more than two structurally different methods is presented. The model is illustrated by an application to the analysis of the convergent and discriminant validity of the trait subscales of the State-Trait Cheerfulness Inventory (STCI-T, Ruch, Köhler, & van Thriel, 1996). The results show that the STCI is a reliable and valid questionnaire for assessing the temperamental basis of sense of humor.     

Controlling Correlational Bias via Confirmatory Factor Analysis of MTMM Data

Confirmatory factor analysis of multitrait-multimethod (MTMM) data has proven to be a useful tool for assessing convergent and discriminant validity. However, researchers have not made full use of the results of MTMM analyses in examining the relationship between MTMM factors and variables outside the MTMM. Often, researchers simply average the various measures of each trait. Alternatively, they estimate LISREL MTMM models, but estimate only relationships between MTMM traits and the outside variables. In the present article, we show that these two approaches to analyzing data outside the MTMM produce equally highly biased parameter estimates when the actual correlations between MTMM method factors and the outside variables are substantial. An algebraic explanation and a simulated data illustration are given for the bias due to misspecification. Also, the problem is illustrated with a brief empirical example. Implications for applied research are discussed.     

The Use of Multitrait–Multimethod Data for Detecting Nonlinear Relationships: The Case of Psychological Climate and Job Satisfaction

Previous monomethod research has found mixed support for nonlinear effects between certain job characteristics (e.g., job autonomy, job complexity) and outcome variables (e.g., job performance). We hypothesized that these weak nonlinear findings may be due to the prevalence of monomethod research that can lead to a lack of complete measurement and/or introduce common methods variance, either of which may mask the true shape of relationships. Using hierarchical regression analyses and a multitrait–multimethod (MTMM) research design, we found strikingly different results between monomethod and MTMM data when considering the relationships between three psychological climate variables and job satisfaction. While the monomethod results mirrored earlier inconclusive findings, the MTMM data indicated that nonlinear equations explained significantly more of the relationship between all three climate dimensions and job satisfaction. These results suggest that the use of MTMM data allows one to more effectively test for nonlinear effects. Furthermore, these nonlinear results suggest that the format of employee questionnaires should probably change from asking how much an employee has of certain constructs to asking does the employee want more or less of these constructs.     

The analysis of multitrait-multimethod matrices via constrained components analysis

Multitrait-Multimethod (MTMM) matrices are often analyzed by means of confirmatory factor analysis (CFA). However, fitting MTMM models often leads to improper solutions, or non-convergence. In an attempt to overcome these problems, various alternative CFA models have been proposed, but with none of these the problem of finding improper solutions was solved completely. In the present paper, an approach is proposed where improper solutions are ruled out altogether and convergence is guaranteed. The approach is based on constrained variants of components analysis (CA). Besides the fact that these methods do not give improper solutions, they have the advantage that they provide component scores which can later on be used to relate the components to external variables. The new methods are illustrated by means of simulated data, as well as empirical data sets.This research has been made possible by a fellowship from the Royal Netherlands Academy of Arts and Sciences to the first author. The authors are obliged to three anonymous reviewers and an associate editor for constructive suggestions on the first version of this paper.     

Modeling Common Traits and Method Effects in Multitrait-Multimethod Analysis

Method effects often occur when constructs are measured by different methods. In traditional multitrait-multimethod (MTMM) models method effects are regarded as residuals, which implies a mean method effect of zero and no correlation between trait and method effects. Furthermore, in some recent MTMM models, traits are modeled to be specific to a certain method. However, often we are not interested in a method-specific trait but in a trait that is common to all methods. Here we present the Method Effect model with common trait factors, which allows modeling “common” trait factors and method factors that represent method “effects” rather than residuals. The common trait factors are defined as the mean of the true-score variables of all variables measuring the same trait and the method factors are defined as differences between true-score variables and means of true-score variables. Because the model allows estimating mean method effects, correlations between method factors, and correlations between trait and method factors, new research questions may be investigated. The application of the model is demonstrated by 2 examples studying the effect of negative, as compared with positive, item wording for the measurement of mood states.     

Testing the convergent and discriminant validity of the Decisional Balance Scale of the Transtheoretical Model using the Multi-Trait Multi-Method approach.

The authors extended research on the construct validity of the Decisional Balance Scale for smoking in adolescence by testing its convergent and discriminant validity. Hierarchical confirmatory factor analysis multi-trait multi-method approach (HCFA MTMM) was used with data from 2,334 UK adolescents, both smokers and non-smokers. They completed computerized and paper versions of the questionnaire on 3 occasions over 2 years. The results indicated a 3-factor solution; Social Pros, Coping Pros, and Cons fit the data best. The HCFA MTMM model fit the data well, with correlated methods and correlated trait factors. Subsequent testing confirmed discriminant validity between the factors and convergent validity of both methods of administering the questionnaire. There was, however, clear evidence of a method effect, which may have arisen due to different response formats or may be a function of the method of presentation. Taken with other data, there is strong evidence for construct validity of Decisional Balance for smoking in adolescence, but evidence of predictive validity is required.     

General mixture item response models with different item response structures: Exposition with an application to Likert scales

This article proposes a general mixture item response theory (IRT) framework that allows for classes of persons to differ with respect to the type of processes underlying the item responses. Through the use of mixture models, nonnested IRT models with different structures can be estimated for different classes, and class membership can be estimated for each person in the sample. If researchers are able to provide competing measurement models, this mixture IRT framework may help them deal with some violations of measurement invariance. To illustrate this approach, we consider a two-class mixture model, where a person’s responses to Likert-scale items containing a neutral middle category are either modeled using a generalized partial credit model, or through an IRTree model. In the first model, the middle category (“neither agree nor disagree”) is taken to be qualitatively similar to the other categories, and is taken to provide information about the person’s endorsement. In the second model, the middle category is taken to be qualitatively different and to reflect a nonresponse choice, which is modeled using an additional latent variable that captures a person’s willingness to respond. The mixture model is studied using simulation studies and is applied to an empirical example.     

Identification with deficient rank loading matrices in confirmatory factor analysis: Multitrait-multimethod models

This paper presents some results on identification in multitrait-multimethod (MTMM) confirmatory factor analysis (CFA) models. Some MTMM models are not identified when the (factorial-patterned) loadings matrix is of deficient column rank. For at least one other MTMM model, identification does exist despite such deficiency. It is also shown that for some MTMM CFA models, Howe's (1955) conditions sufficient for rotational uniqueness can fail, yet the model may well be identified and rotationally unique. Implications of these results for CFA models in general are discussed.     

Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings

The authors reanalyzed assessment center (AC) multitrait-multimethod (MTMM) matrices containing correlations among postexercise dimension ratings (PEDRs) reported by F. Lievens and J. M. Conway (2001). Unlike F. Lievens and J. M. Conway, who used a correlated dimension-correlated uniqueness model, we used a different set of confirmatory-factor-analysis-based models (1-dimension-correlated Exercise and 1-dimension-correlated uniqueness models) to estimate dimension and exercise variance components in AC PEDRs. Results of reanalyses suggest that, consistent with previous narrative reviews, exercise variance components dominate over dimension variance components after all. Implications for AC construct validity and possible redirections of research on the validity of ACs are discussed.     

Longitudinal Multi-Trait-State-Method Model Using Ordinal Data

Multi-trait multi-method (MTMM) models provide a way to assess convergent and discriminant validity when multiple traits are measured by multiple methods. In recent years, longitudinal extensions of MTMM models have been proposed in the structural equation modeling framework to evaluate whether and how the trait as well as method factors change over time. We propose a novel longitudinal ordinal MTMM model that can be used to effectively distinguish volatile “state” processes from “trait” processes that tend to remain stable and invariant over time. The proposed model, termed a longitudinal multi-trait-state-method (LM-TSM) model, combines 3 key modeling components: (a) a measurement model for ordinal data, (b) a vector autoregressive moving average model at the latent level to examine changes in the state as well as the method factors over time, and (c) a second-order factor-analytic model to capture time-invariant traits as shared variances among the state factors across all measurement occasions. Data from the Affective Dynamics and Individual Differences (ADID; Emotions and Dynamic Systems Laboratory, 2010 Emotions and Dynamic Systems Laboratory. (2010). The Affective Dynamics and Individual Differences (ADID) study: Developing non-stationary and network-based methods for modeling the perception and physiology of emotions. . Unpublished manual, University of North Carolina at Chapel Hill. [Google Scholar]) study was used to illustrate the proposed longitudinal LM-TSM model. Methodological issues associated with fitting the LM-TSM model are discussed.     

Social support,coping, and depressive symptoms in cardiac illness among Hispanic and non-Hispanic White cardiac patients

In Multi-Trait Multi-Method (MTMM) studies of causal attributions for laboratory events, there is little evidence of convergent and discriminant validity for attribution measures. We report the first MTMM study to investigate the validity of two methods of eliciting causal beliefs for an illness, specifically, myocardial infarction. Adult respondents (N?=?107) listed causes of MI, then completed questionnaire rating scales for causal beliefs for MI. Measures were compared using both Campbell and Fiske's approach to MTMM analyses, and a Confirmatory Factor Analysis approach. Neither single item measures causal beliefs, nor scales of causal beliefs derived using exploratory factor analysis provided much evidence of convergent and discriminant validity. Confirmatory factor analysis showed that a model containing only causal belief factors provided a moderately good fit to the data. Adding a questionnaire method factor significantly improved the fit of the model, as well as substantially changing the pattern of factor loadings: loadings of questionnaire items on causal belief factors were markedly reduced. These results highlight major problems with the measurement of causal beliefs, and in particular question the validity of factor analysis of questionnaire measures of causal beliefs. They also suggest that at least some of the MI causal belief factors reported in the literature are artefacts of common questionnaire method variance.     

Structural equation modeling with interchangeable dyads

Structural equation modeling (SEM) can be adapted in a relatively straightforward fashion to analyze data from interchangeable dyads (i.e., dyads in which the 2 members cannot be differentiated). The authors describe a general strategy for SEM model estimation, comparison, and fit assessment that can be used with either dyad-level or pairwise (double-entered) dyadic data. They present applications illustrating this approach with the actor-partner interdependence model, confirmatory factor analysis, and latent growth curve analysis.     

基于遗传算法的模糊综合评价在心理测量中的应用

提出了运用模糊数学对利克特量表数据进行分析的方法, 探讨了人们在进行模糊综合评价时, 所采用的算子和对各个自变量的权重分配, 并且运用遗传算法(GA)来确定相关的权重。以大学生对康师傅红烧牛肉面的评价数据为例, 运用基于遗传算法的模糊综合评价方法, 发现男生采用了“最大最小”合成算子, 女生采用了“有界和、取小”合成算子。研究结果表明, 基于遗传算法的模糊综合评价方法可以对利克特量表的心理测量数据进行有效的分析。     

多维测验项目参数的估计：基于SEM与MIRT方法的比较

作者简要回顾了SEM框架下分类数据因素分析(CCFA)模型和MIRT框架下测验题目和潜在能力的关系模型, 对两种框架下的主要参数估计方法进行了总结。通过模拟研究, 比较了SEM框架下WLSc和WLSMV估计方法与MIRT框架下MLR和MCMC估计方法的差异。研究结果表明：(1) WLSc得到参数估计的偏差最大, 且存在参数收敛的问题; (2)随着样本量增大, 各种项目参数估计的精度均提高, WLSMV方法与MLR方法得到的参数估计精度差异很小, 大多数情况下不比MCMC方法差; (3)除WLSc方法外, 随着每个维度测验题目的增多参数估计的精度逐渐增高; (4)测验维度对区分度参数和难度参数的影响较大, 而测验维度对项目因素载荷和阈值的影响相对较小; (5)项目参数的估计精度受项目测量维度数的影响, 只测量一个维度的项目参数估计精度较高。另外文章还对两种方法在实际应用中应该注意的问题提供了一些建议。