Combining Methodologies in the Construct Validation Process: An Empirical Illustration

Confirmatory Factor Analysis (CFA) has received considerable support as a methodology for assessing construct validity. As with other methodologies, however, numerous problems can be encountered when using CFA to assess construct validity. Given the limitations of the CFA, Multitrait-Multimethod (MTMM), and analysis of variance (ANOVA) methodologies, a set of guidelines was developed. The guidelines recommend that the characteristics of the data set be used to determine how the three alternative construct validation methodologies can be used in combination. Examples from the research literature are discussed in order to demonstrate the application of these guidelines.     

Measuring motivation multidimensionally: development of the Assessment of Individual Motives-Questionnaire (AIM-Q)

We report the development of the Assessment of Individual Motives-Questionnaire (AIM-Q), a new instrument based on an evolutionary psychology theory of human motivation. It provides multitrait-multimethod (MTMM) assessment of individual differences on 15 motive scales. A total heterogeneous sample of N = 1,251 participated in eight studies that evaluated the homogeneity, internal consistency, test-retest reliability, and MTMM convergent and discriminant validities of the AIM-Q's three methods. These studies generally support the overall strategy of assessing individual differences in multiple evolutionary-based motives with multiple methods. Additional validity studies are underway and, when validated further, the AIM-Q may offer a promising option for evolutionary psychologists and behavioral geneticists wanting to incorporate individual differences into their research but have had to use existing self-report measures of personality, which were not designed for such a purpose. It may also offer clinical and counseling psychologists an additional approach to personality measures for the prediction of behavior.     

The analysis of multitrait-multimethod matrices via constrained components analysis

Multitrait-Multimethod (MTMM) matrices are often analyzed by means of confirmatory factor analysis (CFA). However, fitting MTMM models often leads to improper solutions, or non-convergence. In an attempt to overcome these problems, various alternative CFA models have been proposed, but with none of these the problem of finding improper solutions was solved completely. In the present paper, an approach is proposed where improper solutions are ruled out altogether and convergence is guaranteed. The approach is based on constrained variants of components analysis (CA). Besides the fact that these methods do not give improper solutions, they have the advantage that they provide component scores which can later on be used to relate the components to external variables. The new methods are illustrated by means of simulated data, as well as empirical data sets.This research has been made possible by a fellowship from the Royal Netherlands Academy of Arts and Sciences to the first author. The authors are obliged to three anonymous reviewers and an associate editor for constructive suggestions on the first version of this paper.     

Multitrait Multimethod validation of Implicit Association Test as a measure of individual differences in personality: Is there personality at all?

The objective of the study was to investigate the validity of IAT in the assessment of the Big five personality traits. We employed the Multi-Trait Multi-Method (MTMM) approach on a sample of 146 respondents. We collected measures on IATs assessing Big five personality traits, self-reports, and two ratings of close others of different gender on NEO PI-R. Additionally, we assessed behavior related to the Big five personality traits in a structured interview where three experts independently provided rating measures. MTMM analyses suggest that personality IAT measures have little in common with personality traits measured traditionally using explicit methods. The evidence does not support the claim that personality IAT can serve as a valid measure of individual differences in personality traits.     

A multitrait-multimethod model with minimal assumptions

A new model of confirmatory factor analysis (CFA) for multitrait-multimethod (MTMM) data sets is presented. It is shown that this model can be defined by only three assumptions in the framework of classical psychometric test theory (CTT). All other properties of the model, particularly the uncorrelated-ness of the trait with the method factors are logical consequences of the definition of the model. In the model proposed there are as many trait factors as different traits considered, but the number of method factors is one fewer than the number of methods included in an MTMM study. The covariance structure implied by this model is derived, and it is shown that this model is identified even under conditions under which other CFA-MTMM models are not. The model is illustrated by two empirical applications. Furthermore, its advantages and limitations are discussed with respect to previously developed CFA models for MTMM data.     

Validity of test score interpretations for three self-concept measures based on differing theoretical models

Three theoretical models have been proposed to represent self-concept: (a) unidimensional; (b) multidimensional; (c) multidimensional hierarchical. Inventories have been developed under each of the three competing theoretical models; which model best represents self-concept is unclear. Typically, self-concept construct validation has utilized various approaches including correlational, multitrait-multimethod, and factor analytic methods. Another method, however, for assessing validity would be to determine the consequences of score interpretations using different measures specific to each of the theoretical models. This paper examined Messick's notion (1989) of the validity of test-score interpretations as applied to three of the most widely used measures derived under each of the three different theoretical models of self-concept. Results suggest that overall multidimensional measures are more consistent in classifying individual's self-concept than unidimensional measures.     

New Wine in a New Bottle: Refining The Assessment of Authentic Leadership Using Exploratory Structural Equation Modeling (ESEM)

The aim of this study was to investigate the multidimensional nature of authentic leadership (AL) through the re-evaluation of the factor structure of the most commonly used scales in AL research, the Authentic Leadership Questionnaire (ALQ) and the Authentic Leadership Inventory (ALI). Results from Exploratory Structural Equation Modeling (ESEM) conducted among 538 workers of private (57,81%) and public (42,19%)? organizations provide support for the multidimensional nature of AL but also reveal excessive content overlap across subscales and an inability to properly reflect the a priori factor structure of both instruments. Further analyses enabled the identification of key items from both instruments providing a cleaner depiction of the a priori dimensionality of AL, leading to the development of an integrated optimized measure, the Authentic Leadership Integrated Questionnaire (AL-IQ). The main contribution of this study is the proposition of an alternative and optimized measure of authentic leadership that sheds light on the distinct theoretical facets of this positive leadership style. Our data suggest that the AL-IQ is a promising tool for characterizing the strengths and weaknesses of leaders and managers’ AL practices. To our knowledge, this is the first study to use a combination of classical Confirmatory Factor Analyses (CFA) and newly developed ESEM framework to assess the construct validity (factor structure, reliability, and criterion-related validity) of the two main AL measures: the ALQ and the ALI.     

Masculinity and Femininity: A Bipolar Construct and Independent Constructs

ABSTRACT The present investigation is a reanalysis of data from Antill and Cunningham (1979, 1980, Marsh, Antill, & Cunningham, 1987) consisting of responses to five masculinity-femininity (MF) instruments, two self-esteem instruments, and two social desirability scales Correlations between M and F for the five instruments varied from 23 to approximately - 10, support for distinguishable (nonbipolar) M and F factors was found for four of the instruments Applying confirmatory factor analysis (CFA) and hierarchical CFA (HCFA), the present study examined the dimensionality of MF and the influence of method/ halo effects m response to specific instruments The best fitting model identified three higher order factors, in support of traditional personality theories one factor was a bipolar MF construct, but m support of androgyny theory the other two factors were distinguishable M and F factors The factor structures were reasonably invanant for men and women, and methodological implications of this important finding were examined In subsequent analyses, the higher order MF factors were related to self-esteem, social desirability, and gender in order to further test interpretations of the MF factors.     

Testing the convergent and discriminant validity of the Decisional Balance Scale of the Transtheoretical Model using the Multi-Trait Multi-Method approach.

The authors extended research on the construct validity of the Decisional Balance Scale for smoking in adolescence by testing its convergent and discriminant validity. Hierarchical confirmatory factor analysis multi-trait multi-method approach (HCFA MTMM) was used with data from 2,334 UK adolescents, both smokers and non-smokers. They completed computerized and paper versions of the questionnaire on 3 occasions over 2 years. The results indicated a 3-factor solution; Social Pros, Coping Pros, and Cons fit the data best. The HCFA MTMM model fit the data well, with correlated methods and correlated trait factors. Subsequent testing confirmed discriminant validity between the factors and convergent validity of both methods of administering the questionnaire. There was, however, clear evidence of a method effect, which may have arisen due to different response formats or may be a function of the method of presentation. Taken with other data, there is strong evidence for construct validity of Decisional Balance for smoking in adolescence, but evidence of predictive validity is required.     

Controlling Correlational Bias via Confirmatory Factor Analysis of MTMM Data

Confirmatory factor analysis of multitrait-multimethod (MTMM) data has proven to be a useful tool for assessing convergent and discriminant validity. However, researchers have not made full use of the results of MTMM analyses in examining the relationship between MTMM factors and variables outside the MTMM. Often, researchers simply average the various measures of each trait. Alternatively, they estimate LISREL MTMM models, but estimate only relationships between MTMM traits and the outside variables. In the present article, we show that these two approaches to analyzing data outside the MTMM produce equally highly biased parameter estimates when the actual correlations between MTMM method factors and the outside variables are substantial. An algebraic explanation and a simulated data illustration are given for the bias due to misspecification. Also, the problem is illustrated with a brief empirical example. Implications for applied research are discussed.     

贝叶斯多组比较——渐近测量不变性

在行为科学研究领域中,检验测量工具的测量不变性是进行群体差异比较的前提。目前,多组验证性因子分析(多组CFA)方法被广泛用于检验测量不变性,但是它对跨组等值的限制过于严格,在实际应用中常常存在大量局限。贝叶斯渐近测量不变性方法基于贝叶斯思想的优良特性,放宽了传统多组CFA方法对跨组差异的严格限制,避免了传统方法的问题,具有较高的应用价值。文章详细介绍了贝叶斯渐近测量不变性方法的原理及优势,同时通过实例展示了渐近测量不变性方法在Mplus软件中的具体分析过程。     

Action-state orientation: construct validity of a revised measure and its relationship to work-related variables

Data from 6 studies (N = 945) were combined to examine the factor structure and construct validity of J. Kuhl's (1994a) measure of action-state orientation, the Action Control Scale (ACS). Initial confirmatory factor analysis (CFA) of 4 models showed poor fit with the data. Items that did not perform well were dropped, and the 4 CFA models were retested, resulting in support for a 3-factor solution corresponding to the Preoccupation, Hesitation, and Volatility subscales. Regression analyses demonstrated that the revised ACS subscales measure a construct distinct from cognitive ability and personality. The revised subscales showed differential statistically significant relationships with (a) self-regulatory and self-focus variables and (b) job attitudes and ratings of job performance. The authors conclude that action-state orientation is a construct relevant to applied settings and that future research should further examine this construct.     

Identification with deficient rank loading matrices in confirmatory factor analysis: Multitrait-multimethod models

This paper presents some results on identification in multitrait-multimethod (MTMM) confirmatory factor analysis (CFA) models. Some MTMM models are not identified when the (factorial-patterned) loadings matrix is of deficient column rank. For at least one other MTMM model, identification does exist despite such deficiency. It is also shown that for some MTMM CFA models, Howe's (1955) conditions sufficient for rotational uniqueness can fail, yet the model may well be identified and rotationally unique. Implications of these results for CFA models in general are discussed.     

情景判断测验的开发程序、构思效度及研究趋势

章详细介绍情景判断测验开发的一般程序,对情景判断测验的多种形式和记分方法进行了总结和比较。同时,从情景判断测验的结果与认知能力、个性和工作经验等关系的角度出发,分析了情景判断测验的构思效度,认为情景判断测验测量的是多维构思。文章最后认为,需要从与其他构思的关系、测评指定构思、影响效度的因素以及跨文化比较等四个方面对情景判断测验开展进一步研究。     

Bayesian analysis of longitudinal multitrait–multimethod data with ordinal response variables

A new multilevel latent state graded response model for longitudinal multitrait–multimethod (MTMM) measurement designs combining structurally different and interchangeable methods is proposed. The model allows researchers to examine construct validity over time and to study the change and stability of constructs and method effects based on ordinal response variables. We show how Bayesian estimation techniques can address a number of important issues that typically arise in longitudinal multilevel MTMM studies and facilitates the estimation of the model presented. Estimation accuracy and the impact of between‐ and within‐level sample sizes as well as different prior specifications on parameter recovery were investigated in a Monte Carlo simulation study. Findings indicate that the parameters of the model presented can be accurately estimated with Bayesian estimation methods in the case of low convergent validity with as few as 250 clusters and more than two observations within each cluster. The model was applied to well‐being data from a longitudinal MTMM study, assessing the change and stability of life satisfaction and subjective happiness in young adults after high‐school graduation. Guidelines for empirical applications are provided and advantages and limitations of a Bayesian approach to estimating longitudinal multilevel MTMM models are discussed.     

Development and validation of the Causes of Re-Injury Worry Questionnaire

Re-injury worry is an important construct in competitive sport that may influence performance and increase the risk of re-injury. However, there are currently no available instruments to measure the causes of re-injury worry. The purpose of this study was to develop the Causes of Re-Injury Worry Questionnaire (CR-IWQ). The study was conducted in three independent research phases to investigate the following: (a) the content relevance, (b) the factor structure and the factorial validity, (c) the concurrent validity, (d) the discriminant validity, and (e) the test-retest reliability (intraclass correlation coefficients; ICC), and the internal consistency of the instrument. Exploratory factor analysis (EFA) was chosen to examine the factor structure of the CR-IWQ. Confirmatory factor analysis (CFA) was used to examine further the factorial validity of the instrument. A number of valid constructs were used to assess the concurrent and discriminant validity of the CR-IWQ. The reliability of the new instrument was examined using Pearson r (ICC) and Cronbach α. Three hundred and seventy athletes with an acute musculoskeletal sport injury in the last year participated in the study. EFA revealed a 12-item model, representing two factors ("Re-injury worry due to rehabilitation" and "Re-injury worry due to opponent's ability"). CFA supported the two-factor model of the CR-IWQ. The concurrent and discriminant validity of the CR-IWQ was confirmed by examining correlations between the CR-IWQ with other constructs. The ICCs and the Cronbach α indices of the CR-IWQ were acceptable. We have demonstrated that the CR-IWQ is a good psychometric instrument that can be used for clinical and research purposes.     

Toward a brief multidimensional assessment of emotional intelligence: psychometric properties of the Emotional Quotient Inventory-Short Form

Although several brief instruments are available for the emotional intelligence (EI) construct, their conceptual coverage tends to be quite limited. One notable exception is the short form of the Emotional Quotient Inventory (EQ-i:S), which measures multiple EI dimensions in addition to a global EI index. Despite the unique advantage offered by the inventory, psychometric properties of the EQ-i:S scores have not yet been systematically evaluated. Such an evaluation was the main goal of the present investigation. Using data from 2,508 undergraduates, the authors conducted 2 studies involving factor structure, internal reliability, 6-month temporal stability, and construct validity of the EQ-i:S responses, both for the total EQ scale and for each constituent dimension. The results supported the multidimensional measurement structure of the EQ-i:S, with each dimension producing internally consistent, temporally stable, and theoretically meaningful responses. Scores on the EQ-i:S were associated more strongly with performance on an ability test of EI and with a conceptually similar construct of alexithymia than with the broader dimensions of basic personality and explained nontrivial amounts of incremental variance in the criterion symptoms of attention deficit/hyperactivity disorder. Moreover, scores on each EQ-i:S dimension exhibited unique patterns of associations with the validation variables. The discussion highlights the advantages of the multidimensional approach in the assessment and study of EI.     

Capturing the four-factor structure of psychopathy in college students via self-report

A number of self-report psychopathy scales have been used successfully in both clinical and nonclinical settings. However, their factor structure does not adequately capture the four factors (Interpersonal, Affective, Lifestyle, and Antisocial) recently identified in the Psychopathy Checklist-Revised (PCL-R; Hare, 2003) and related measures. This deficit was addressed by upgrading the Self Report Psychopathy Scale (SRP-II; Hare, Hemphill, & Harpur, 1989). In Study 1 (N = 249), an exploratory factor analysis of this experimental version revealed oblique factors similar to those outlined by Hare (2003). In Study 2 (N = 274), confirmatory factor analysis (CFA) confirmed this structure, that is, four distinct but intercorrelated factors. The factors exhibited appropriate construct validity in a nomological network of related personality measures. Links with self-reports of offensive activities (including entertainment preferences and behavior) also supported the construct validity of the oblique four-factor model.     

Psychometric properties of beliefs about relationship violence against women and gender stereotypes scale

This article describes psychometric properties of the Gender-Based Relationship Violence Beliefs Scale (BEREVIWOS) and the Gender Stereotypes and Beliefs (GESTABE) scale in a Nigerian setting. Analysis was based on a convenience sample of 202 respondents. Confirmatory factor analysis (CFA) was used to determine construct validity of the three-factor structure of each scale and measurement invariance procedures were utilised to determine whether the factor structure was equivalent across gender. Convergent and discriminant validity of the scales, and predictive and incremental validity were examined for consistency with theory on gender beliefs/stereotypes and violence in patriarchal society. A single-factor CFA model approach was used to examine common method bias. Results suggest the BEREVIWOS and GESTABE are multidimensional scales with an intercorrelated three-factor structure. The fit of the models and composite reliability were adequate. Scores from the BEREVIWOS and GESTABE appear reliable and valid measures of underlying beliefs and stereotypes associated with violence against women in Nigeria.