Controlling Correlational Bias via Confirmatory Factor Analysis of MTMM Data

Confirmatory factor analysis of multitrait-multimethod (MTMM) data has proven to be a useful tool for assessing convergent and discriminant validity. However, researchers have not made full use of the results of MTMM analyses in examining the relationship between MTMM factors and variables outside the MTMM. Often, researchers simply average the various measures of each trait. Alternatively, they estimate LISREL MTMM models, but estimate only relationships between MTMM traits and the outside variables. In the present article, we show that these two approaches to analyzing data outside the MTMM produce equally highly biased parameter estimates when the actual correlations between MTMM method factors and the outside variables are substantial. An algebraic explanation and a simulated data illustration are given for the bias due to misspecification. Also, the problem is illustrated with a brief empirical example. Implications for applied research are discussed.     

A multitrait-multimethod model with minimal assumptions

A new model of confirmatory factor analysis (CFA) for multitrait-multimethod (MTMM) data sets is presented. It is shown that this model can be defined by only three assumptions in the framework of classical psychometric test theory (CTT). All other properties of the model, particularly the uncorrelated-ness of the trait with the method factors are logical consequences of the definition of the model. In the model proposed there are as many trait factors as different traits considered, but the number of method factors is one fewer than the number of methods included in an MTMM study. The covariance structure implied by this model is derived, and it is shown that this model is identified even under conditions under which other CFA-MTMM models are not. The model is illustrated by two empirical applications. Furthermore, its advantages and limitations are discussed with respect to previously developed CFA models for MTMM data.     

Bayesian analysis of longitudinal multitrait–multimethod data with ordinal response variables

A new multilevel latent state graded response model for longitudinal multitrait–multimethod (MTMM) measurement designs combining structurally different and interchangeable methods is proposed. The model allows researchers to examine construct validity over time and to study the change and stability of constructs and method effects based on ordinal response variables. We show how Bayesian estimation techniques can address a number of important issues that typically arise in longitudinal multilevel MTMM studies and facilitates the estimation of the model presented. Estimation accuracy and the impact of between‐ and within‐level sample sizes as well as different prior specifications on parameter recovery were investigated in a Monte Carlo simulation study. Findings indicate that the parameters of the model presented can be accurately estimated with Bayesian estimation methods in the case of low convergent validity with as few as 250 clusters and more than two observations within each cluster. The model was applied to well‐being data from a longitudinal MTMM study, assessing the change and stability of life satisfaction and subjective happiness in young adults after high‐school graduation. Guidelines for empirical applications are provided and advantages and limitations of a Bayesian approach to estimating longitudinal multilevel MTMM models are discussed.     

Analysing multitrait–multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: What sample size is needed for valid results?

Convergent and discriminant validity of psychological constructs can best be examined in the framework of multitrait–multimethod (MTMM) analysis. To gain information at the level of single items, MTMM models for categorical variables have to be applied. The CTC(M?1) model is presented as an example of an MTMM model for ordinal variables. Based on an empirical application of the CTC(M?1) model, a complex simulation study was conducted to examine the sample size requirements of the robust weighted least squares mean‐ and variance‐adjusted χ² test of model fit (WLSMV estimator) implemented in Mplus. In particular, the simulation study analysed the χ² approximation, the parameter estimation bias, the standard error bias, and the reliability of the WLSMV estimator depending on the varying number of items per trait–method unit (ranging from 2 to 8) and varying sample sizes (250, 500, 750, and 1000 observations). The results showed that the WLSMV estimator provided a good – albeit slightly liberal – χ² approximation and stable and reliable parameter estimates for models of reasonable complexity (2–4 items) and small sample sizes (at least 250 observations). When more complex models with 5 or more items were analysed, larger sample sizes of at least 500 observations were needed. The most complex model with 9 trait–method units and 8 items (72 observed variables) requires sample sizes of at least 1000 observations.     

The covariation of independent and dependant variables in neurofeedback: A proposal framework to identify cognitive processes and brain activity variables

This methodological article proposes a framework for analysing the relationship between cognitive processes and brain activity using variables measured by neurofeedback (NF) carried out by functional Magnetic Resonance Imagery (fMRI NF). Cognitive processes and brain activity variables can be analysed as either the dependant variable or the independent variable. Firstly, we propose two traditional approaches, defined in the article as the “neuropsychological” approach (NP) and the “psychophysiology” approach (PP), to extract dependent and independent variables in NF protocols. Secondly, we suggest that NF can be inspired by the style of inquiry used in neurophenomenology. fMRI NF allows participants to experiment with his or her own cognitive processes and their effects on brain region of interest (ROI) activations simultaneously. Thus, we suggest that fMRI NF could be improved by implementing “the elicitation interview method”, which allows the investigator to gather relevant verbatim from participants’ introspection on subjective experiences.     

Longitudinal Multi-Trait-State-Method Model Using Ordinal Data

Multi-trait multi-method (MTMM) models provide a way to assess convergent and discriminant validity when multiple traits are measured by multiple methods. In recent years, longitudinal extensions of MTMM models have been proposed in the structural equation modeling framework to evaluate whether and how the trait as well as method factors change over time. We propose a novel longitudinal ordinal MTMM model that can be used to effectively distinguish volatile “state” processes from “trait” processes that tend to remain stable and invariant over time. The proposed model, termed a longitudinal multi-trait-state-method (LM-TSM) model, combines 3 key modeling components: (a) a measurement model for ordinal data, (b) a vector autoregressive moving average model at the latent level to examine changes in the state as well as the method factors over time, and (c) a second-order factor-analytic model to capture time-invariant traits as shared variances among the state factors across all measurement occasions. Data from the Affective Dynamics and Individual Differences (ADID; Emotions and Dynamic Systems Laboratory, 2010 Emotions and Dynamic Systems Laboratory. (2010). The Affective Dynamics and Individual Differences (ADID) study: Developing non-stationary and network-based methods for modeling the perception and physiology of emotions. . Unpublished manual, University of North Carolina at Chapel Hill. [Google Scholar]) study was used to illustrate the proposed longitudinal LM-TSM model. Methodological issues associated with fitting the LM-TSM model are discussed.     

Some Problems Concerning the Assessment of Job Characteristics in Nursing Work

Abstract

The question of whether one should use subjective rather than “objective” assessments of job characteristics is still the subject of a vigorous discussion in analytical studies of jobs and in job design. In nursing work situations, too, the attention of managers and researchers for such studies is growing. In this article the results of a study are presented in which a comparison is made between two kinds of assessments: job characteristics as judged by task performers (nurses) and job characteristics as judged by non-task performers (nursing management). Differences and agreements between mean scores as well as between correlations with dependent variables like job satisfaction were analysed. Most agreement seems to be on the ratings of “complexity and difficulty”, “work pressure”, and “promotion and growth opportunities”. As to “feedback and clarity”, “autonomy”, and “patient caring opportunities”, differences between the ratings have been found. It appears that a larger number of correlations between the nurse-rated job characteristics and the reaction variables is found to be statistically significant than between the management-rated job characteristics and the reaction variables. Social, organizational, and methodological reasons for the observed differences are put forward.     

The Use of Multitrait–Multimethod Data for Detecting Nonlinear Relationships: The Case of Psychological Climate and Job Satisfaction

Previous monomethod research has found mixed support for nonlinear effects between certain job characteristics (e.g., job autonomy, job complexity) and outcome variables (e.g., job performance). We hypothesized that these weak nonlinear findings may be due to the prevalence of monomethod research that can lead to a lack of complete measurement and/or introduce common methods variance, either of which may mask the true shape of relationships. Using hierarchical regression analyses and a multitrait–multimethod (MTMM) research design, we found strikingly different results between monomethod and MTMM data when considering the relationships between three psychological climate variables and job satisfaction. While the monomethod results mirrored earlier inconclusive findings, the MTMM data indicated that nonlinear equations explained significantly more of the relationship between all three climate dimensions and job satisfaction. These results suggest that the use of MTMM data allows one to more effectively test for nonlinear effects. Furthermore, these nonlinear results suggest that the format of employee questionnaires should probably change from asking how much an employee has of certain constructs to asking does the employee want more or less of these constructs.     

项目表述效应对人格测验信度和效标效度的影响——以核心自我评价量表(CSES)为例

项目表述效应是指由项目表述方式的差异引起的与测量内容无关的系统变异,项目表述效应模型的统计本质是一种双因子模型。本研究以核心自我评价量表(CSES)为例,探讨项目表述效应对人格测验信效度的影响。采用核心自我评价量表、生活满意度量表和积极情感消极情感量表对340名“蚁族”进行测查。结果表明,CSES在核心自我评价特质以外,还存在一个反向题项目表述效应因子;忽视项目表述效应对CSES的同质性信度和效标关联效度有重要影响：高估CSES的同质性信度,低估核心自我评价与生活满意度、积极情感的正相关,高估核心自我评价与消极情感的负相关。     

Trait-Specific Versus Person-Specific Moderators of Cross-Situational Consistency

ABSTRACT Moderator variables for cross-situational consistency can be tested m two different ways (a) a trait-specific manner which examines differences in cross-situational consistency among traits, or (b) a person-specific manner which examines differences in cross-situational consistency among people The present study examined moderator effects–both trait- and person-specific–of the discrepancy between private self-ratings on trait dimensions (“How do you see yourself?”) and the corresponding public self-ratings (“How do others see you?”) Agreement between self- and peer ratings served as the dependent variable The results showed that public-private discrepancy moderated self-peer agreement when operationalized in a trait-specific manner, i e, for each trait, higher discrepancy was associated with lower self-peer agreement On the other hand, the results showed only minimal moderator effects when public-private discrepancy was operationalized in a person-specific manner, i e, when mean discrepancy across all traits served as the moderator Implications of the distinction between trait- and person-specific approaches to moderator effects are discussed     

Structural equation modeling of multitrait-multimethod data: different models for different types of methods

The question as to which structural equation model should be selected when multitrait-multimethod (MTMM) data are analyzed is of interest to many researchers. In the past, attempts to find a well-fitting model have often been data-driven and highly arbitrary. In the present article, the authors argue that the measurement design (type of methods used) should guide the choice of the statistical model to analyze the data. In this respect, the authors distinguish between (a) interchangeable methods, (b) structurally different methods, and (c) the combination of both kinds of methods. The authors present an appropriate model for each type of method. All models allow separating measurement error from trait influences and trait-specific method effects. With respect to interchangeable methods, a multilevel confirmatory factor model is presented. For structurally different methods, the correlated trait-correlated (method-1) model is recommended. Finally, the authors demonstrate how to appropriately analyze data from MTMM designs that simultaneously use interchangeable and structurally different methods. All models are applied to empirical data to illustrate their proper use. Some implications and guidelines for modeling MTMM data are discussed.     

Who is susceptible in three false memory tasks?

ABSTRACT

Decades of research show that people are susceptible to developing false memories. But if they do so in one task, are they likely to do so in a different one? The answer: “No”. In the current research, a large number of participants took part in three well-established false memory paradigms (a misinformation task, the Deese-Roediger-McDermott [DRM] list learning paradigm, and an imagination inflation exercise) as well as completed several individual difference measures. Results indicate that many correlations between false memory variables in all three inter-paradigm comparisons are null, though some small, positive, significant correlations emerged. Moreover, very few individual difference variables significantly correlated with false memories, and any significant correlations were rather small. It seems likely, therefore, that there is no false memory “trait”. In other words, no one type of person seems especially prone, or especially resilient, to the ubiquity of memory distortion.     

Separating trait effects from trait-specific method effects in multitrait-multimethod models: a multiple-indicator CT-C(M-1) model

An overview of several models of confirmatory factor analysis for analyzing multitrait-multimethod (MTMM) data and a discussion of their advantages and limitations are provided. A new class of multi-indicator MTMM models combines several strengths and avoids a number of serious shortcomings inherent in previously developed MTMM models. The new models enable researchers to specify and to test trait-specific-method effects. The trait and method concepts composing these models are explained in detail and are contrasted with those of previously developed MTMM models for multiple indicators. The definitions of the models are explained step by step, and a practical empirical application of the models to the measurement of 3 traits x 3 methods is used to demonstrate their advantages and limitations.     

The Role of Trait Emotional Intelligence in a Gender‐Specific Model of Organizational Variables1

This article investigated the relationships between trait emotional intelligence (“trait EI” or “emotional self‐efficacy”) and 4 job‐related variables (perceived job control, job stress, job satisfaction, and organizational commitment). Gender‐specific data (N=167, 87 females) were analyzed via multigroup structural equation modeling. Perceived job control had a negative effect on stress and a positive effect on satisfaction. Stress had a negative effect on satisfaction, which, in turn, had the strongest positive effect on commitment. There were many gender differences in the model, mainly concerning age, which was negatively related to control and commitment in the female sample only. Trait EI had specific, rather than widespread, effects in the model. Discussion focuses on trait EI's implications in the workplace.     

Burnout‐depression overlap: Nomological network examination and factor‐analytic approach

Burnout has been defined as a condition in which individuals are left exhausted by a long‐term confrontation with unmanageable job stressors. The question of whether burnout reflects anything other than depressive responses to unresolvable stress remains an object of debate. In this 911‐participant study (83% female; mean age: 42.36), we further addressed the issue of burnout‐depression overlap. Burnout was assessed with the exhaustion subscale of the Maslach Burnout Inventory‐General Survey (MBI ‐GS ) and depression with the PHQ ‐8. The relationships of burnout and depression with three job‐related variables – illegitimate work tasks, work‐nonwork interference, and job satisfaction – and three “context‐free” variables – social support, general health status, and trait anxiety – were examined. Burnout and depression were found to be strongly correlated, to cluster together, and to exhibit overlapping nomological networks. Remarkably, the average correlations of burnout and depression with job‐related variables were almost identical. A principal component analysis and a principal axis factor analysis both showed that the items of the MBI ‐GS and of the PHQ ‐8 loaded on a single dimension. All in all, our findings are consistent with the view that burnout is a depressive condition. The distinction between burnout and depression may be an instance of the jangle fallacy.     

Comparing Latent Means Without Mean Structure Models: A Projection-Based Approach

The conventional setup for multi-group structural equation modeling requires a stringent condition of cross-group equality of intercepts before mean comparison with latent variables can be conducted. This article proposes a new setup that allows mean comparison without the need to estimate any mean structural model. By projecting the observed sample means onto the space of the common scores and the space orthogonal to that of the common scores, the new setup allows identifying and estimating the means of the common and specific factors, although, without replicate measures, variances of specific factors cannot be distinguished from those of measurement errors. Under the new setup, testing cross-group mean differences of the common scores is done independently from that of the specific factors. Such independent testing eliminates the requirement for cross-group equality of intercepts by the conventional setup in order to test cross-group equality of means of latent variables using chi-square-difference statistics. The most appealing piece of the new setup is a validity index for mean differences, defined as the percentage of the sum of the squared observed mean differences that is due to that of the mean differences of the common scores. By analyzing real data with two groups, the new setup is shown to offer more information than what is obtained under the conventional setup.     

Self-presentational determinants of health risk behavior among college freshmen

Abstract

This study examined adolescents' use of unhealthy and potentially dangerous behaviors for self-presentational reasons. At the start of their first semester at college, 110 freshmen (M age = 18.2) completed trait measures of self-presentational concern. At the end of the semester they were asked about their use of health risk behaviors as impression management tactics. Seventy-five percent of respondents reported performing at least 1 risky behavior for self-presentational reasons during their first college semester. The most common behaviors were smoking, drinking, driving recklessly and performing dangerous stunts. The desire to be perceived as “cool” or a “risk-taker” often prompted healdi risks. Modest correlations between the trait measures and health risk behaviors provided additional evidence that self-presentational motives sometimes play a role in adolescent health risk behavior.     

Testing the convergent and discriminant validity of the Decisional Balance Scale of the Transtheoretical Model using the Multi-Trait Multi-Method approach.

The authors extended research on the construct validity of the Decisional Balance Scale for smoking in adolescence by testing its convergent and discriminant validity. Hierarchical confirmatory factor analysis multi-trait multi-method approach (HCFA MTMM) was used with data from 2,334 UK adolescents, both smokers and non-smokers. They completed computerized and paper versions of the questionnaire on 3 occasions over 2 years. The results indicated a 3-factor solution; Social Pros, Coping Pros, and Cons fit the data best. The HCFA MTMM model fit the data well, with correlated methods and correlated trait factors. Subsequent testing confirmed discriminant validity between the factors and convergent validity of both methods of administering the questionnaire. There was, however, clear evidence of a method effect, which may have arisen due to different response formats or may be a function of the method of presentation. Taken with other data, there is strong evidence for construct validity of Decisional Balance for smoking in adolescence, but evidence of predictive validity is required.     

Neurotic Nucleus and Test Anxiety

Undergraduate university students (N = 103) participated in a study of the relationship between test anxiety and the variables of trait anxiety, self-esteem, locus of control, mental ability, and gender. Results indicated bivariate associations between total test anxiety and the other measures except for mental ability. Further analyses revealed independent relationships between the “worry” component of test anxiety and the variables of trait anxiety, intemality, chance, and mental ability. We also found independent associations between the “emotionality” aspect of test anxiety and the measures of trait anxiety and chance.     

A criterion for significant common factor variance

Up to the present only empirical methods have been available for determining the number of factors to be extracted from a matrix of correlations. The problem has been confused by the implicit attitude that a matrix of intercorrelations between psychological variables has a rank which is determinable. A table of residuals always contains error variance and common factor variance. The extraction of successive factors increases the proportion of error variance remaining to common factor variance remaining, and a point is reached where the extraction of more dimensions would contain so much error variance that the common factor variance would be overshadowed. The critical value for this point is determined by probability theory and does not take into account the size of the residuals. Interpretation of the criterion is discussed.