On Conducting Construct Validity Meta-Analyses for the Rorschach: A Reply to Tibon Czopp and Zeligman (2016)

We respond to Tibon Czopp and Zeligman's (2016) critique of our systematic reviews and meta-analyses of 65 Rorschach Comprehensive System (CS) variables published in Psychological Bulletin (2013). The authors endorsed our supportive findings but critiqued the same methodology when used for the 13 unsupported variables. Unfortunately, their commentary was based on significant misunderstandings of our meta-analytic method and results, such as thinking we used introspectively assessed criteria in classifying levels of support and reporting only a subset of our externally assessed criteria. We systematically address their arguments that our construct label and criterion variable choices were inaccurate and, therefore, meta-analytic validity for these 13 CS variables was artificially low. For example, the authors created new construct labels for these variables that they called “the customary CS interpretation,” but did not describe their methodology nor provide evidence that their labels would result in better validity than ours. They cite studies they believe we should have included; we explain how these studies did not fit our inclusion criteria and that including them would have actually reduced the relevant CS variables’ meta-analytic validity. Ultimately, criticisms alone cannot change meta-analytic support from negative to positive; Tibon Czopp and Zeligman would need to conduct their own construct validity meta-analyses.     

General Mental Ability and Selection in Private Sector Organizations: A Commentary

This article reviews a number of construct validation issues within the frarne- work of B i g and Barrett's (1989) Inference 8, which states that "the crite- rion measure is an adequate sample from the performance domain" (p. 480). Emphasis is placed on managerial selection, behavior, and criteria, but illus- trative data from Project A are included. The argument is made that more re- search on the construct validity of our criteria must be done if we are to understand what we are predicting in our selection research. Instead of con- tinuing to rely on ratings, we must understand what behaviors lead to what results, and how those behaviors and results relate to our predictors.     

Do overall dimension ratings from assessment centres show external construct-related validity?

ABSTRACT

There have been repeated calls for an external construct validation approach to advance our understanding of the construct-related validity of assessment centre dimension ratings beyond existing internal construct-related validity findings. Following an external construct validation approach, we examined whether linking assessment centre overall dimension ratings to ratings of the same dimensions that stem from sources external to the assessment centre provides evidence for construct-related validity of assessment centre ratings. We used data from one laboratory assessment centre sample and two field samples. External ratings of the same dimensions stemmed from assessees, assessees’ supervisors, and customers. Results converged across all three samples and showed that different dimension-same source correlations within the assessment centres were larger than same dimension-different source correlations. Moreover, confirmatory factor analyses revealed source factors but no dimension factors in the latent factor structure of overall dimension ratings from the assessment centre and from external sources. Hence, consistent results across the three samples provide no support that assessment centre overall dimension ratings and ratings of the same dimensions from other sources can be attributed to dimension factors. This questions arguments that assessment centre overall dimension ratings should have construct-related validity.     

Sex differences in scholastic aptitude: The Israeli scene

The major goal of the present study is two-fold: (a) To assess the direction, magnitude and profile of sex differences in scholastic aptitude test performance among Israeli college student candidates and (b) to test for possible sex bias in the construct validity of aptitude test scores. The analyses are based on the college admissions aptitude test scores of 1088 female and 690 male college candidates, applying for admissions to a major Israeli campus. On the whole, the data are consistent with much previous research, generated mainly from the American scene, showing small but consistent sex differences in favor of male examinees for the test as a whole and for the quantitative subscale, in particular. Also, the sex difference profiles for varying ethnic groups in the sample are surprisingly similar. Furthermore, our data show little evidence for differential construct validity of ability test scores by gender. On the whole, our data lend a greater degree of generality and cross-cultural validity to the bulk of previous research negating the sex bias contention and pointing to negligible gender differences in scholastic aptitude.     

Exploring trust of mobile applications based on user behaviors: an empirical study

This paper explores trust of mobile applications based on users' behaviors. It proposes a trust behavior construct through principal component analysis, reliability analysis, and confirmatory factor analysis based on the data collected from a questionnaire survey with more than 1,500 participants. It is indicated that a user's trust behavior is composed of three principal constructs: using behavior, reflection behavior, and correlation behavior. They are further delineated into 12 measurable sub‐constructs and relate to a number of external factors. The data analysis showed that the questionnaire has positive psychometric properties with respect to construct validity and reliability. We also discuss the practical significance and limitations of our work toward usable trust management.     

Evidence of construct validity for work values

Despite the importance of work values in the process of career adjustment (Dawis, 2002), little empirical research has focused on articulating the domains represented within the construct of work values and the examination of evidence of validity for the construct has been limited. Furthermore, the larger number of work values measures has made it difficult to determine the key domains that constitute the construct. The current study sought to examine multiple measures of work values to understand domains represented within the construct of work values and to establish evidence of validity for these domains. Principal Components Analysis utilizing scores on the Minnesota Importance Questionnaire (Rounds, Henley, Dawis, Lofquist, & Weiss, 1981), the revised edition of Super's Work Values Inventory (SWVI-R; Zytowski, 2006), and Manhardt's Work Values Inventory (Manhardt, 1972) found that six components best explained the data. These components reflected the importance of: the working environment, having challenging work, opportunities for status and income, autonomy, organizational support, and relationships. Implications for research and practice are discussed.     

An extension of construct validity for personality scales using twin-based criteria

A twin-based analysis of personality inventory scales yields information relevant to construct validity. If scales validly measure dimensions influenced developmentally either by environmental factors common to cotwins or by genetic similarities, these influences should be apparent in patterns of intraclass correlations and within-pair variance ratios for monozygotic and dizygotic pairs. Criteria for construct validity derived from adolescent twin statistics are illustrated with Wiggins' MMPI Content Scales. A type of discriminant validity indicates that some scales are not distinct in terms of the influences of familial similarity. Sex differences are examined.     

The Chinese Version of the Self-Report Family Inventory: Reliability and Validity

The reliability and validity of the Chinese Self-Report Family Inventory (C-SFI) were examined in three studies. In Study 1, data based on 361 adolescents showed that the C-SFI was temporally stable and internally consistent and there was support for its concurrent and construct validities. In Study 2, data analyzed from a clinical group and a nonclinical group (N = 281 and 451, respectively) showed that the C-SFI scores were able to discriminate these two groups. With data pertaining to 3649 secondary school students, Study 3 gives support for the internal consistency, concurrent validity, and construct validity of the C-SFI in different adolescent samples. These studies strongly suggest that the C-SFI possesses good psychometric properties in different Chinese adolescent samples.     

The internal consistency reliability and construct validity of the Francis Scale of Attitude toward Christianity among 8- to 11-year-old students in Wales

ABSTRACT

The Francis Scale of Attitude toward Christianity was designed in the mid-1970s to assess individual differences in attitude through childhood, adolescence, and adulthood from the age of 8?years upwards. This study examines the internal consistency reliability, factor structure, and construct validity of the instrument 40?years later among 1511 year 4 students (8–9 years), 1544 year 5 students (9–10 years), and 1526 year 6 students (10–11 years) in Wales attending Church in Wales voluntary aided and voluntary controlled primary schools. The data support the internal consistency reliability and construct validity of the instrument among the age range and commend the instrument for continued use in research.     

Servant Leadership Behaviour Scale: A hierarchical model and test of construct validity

Servant leadership is widely believed to be a multidimensional construct. However, existing measures of servant leadership typically suffer from highly correlated dimensions, raising concerns over discriminant validity. We set out in this study to examine the dimensionality of the hypothesized six-factor Servant Leadership Behaviour Scale (SLBS) and validate a hierarchical model of servant leadership. Using structural equation modelling, convergent and discriminant validity of the higher-order servant leadership construct was examined in relation to measures of character-related behaviour, Machiavellian leadership orientation, and social desirability. Our findings suggest that the SLBS assesses a single, higher-order construct of servant leadership with evidence of construct validity and absence of social desirability bias. The high correlation between servant leadership and character-related behaviour supports our expectation that servant leadership reflects in large part the character strength of the leader.     

Optimizing the validity of personality assessments: The importance of aggregation and item content

The predictive validity of a psychological measure can be improved by minimizing measurement errors through increases in the length of the assessment (aggregation) and, for an assessment of finite length, by making use of objective strategies for choosing from all available component measures. Two prominent considerations in selecting individual measures to be aggregated involve standards of (a) item content (construct approach) and (b) item/criterion association (empirical approach). Personality trait scales of different lengths were assembled for this study in order to represent features of the construct and empirical methods of selection. It was observed that (a) although reliability and validity generally increased with test length, aggregation beyond a certain point can fail to be expedient; and (b) although the prediction performance of empirically derived measures initially surpassed that of construct based assessments, the superiority of the empirical scales did not generalize to trait criteria that were not used as a basis for item selection. The data are interpreted as providing support for a theory-based program of test development where substantive considerations involving item content play a major role. The findings are also viewed as encouragement for conventional conceptualizations about organized dimensions of behavior.     

Structural equation models and personality research

This article is a nontechnical introduction to the use of structural equation models in personality research Although such models can be fruitfully used to address a variety of important theoretical issues, the substantive focus in this article is on the use of such models for elucidating the construct validity of personality measures We include numerous more specific topics under our treatment of construct validity First of all, we show how structural equation models can be applied to the issues of convergent and discriminant validity Do our variables measure the constructs we want them to measure and not other constructs that we would prefer not to measure? Second, we show the utility of structural equation models for predictive validity Do our variables reliably predict other constructs with which they are theoretically linked? Finally, we examine the stability of personality constructs through structural equation models Through-out, our emphasis is on the particular advantages that structural equation models bring to these analytic tasks Ultimately, such models must be used in the service of theory, and when used appropriately, they can help us to refine both our measures and our theories of individual differences     

The Association Between the Attitude of Food‐Waste‐Aversion and BMI: An Exploration in India and the United States

This research proposes the existence of a hitherto undocumented attitude related to food wastage: the attitude of food‐waste‐aversion. We develop a 6‐item scale including affective, cognitive, and conative components to measure this attitude and empirically investigate its properties in two countries using novel datasets. We test for food‐waste‐aversion scale's convergent validity by demonstrating that it is correlated in the expected direction with five theoretically related constructs—frugality, social responsibility, spendthriftness, self‐control, and materialism (Studies 1a and 1b)—and with BMI (Studies 2 and 3). We provide more indirect evidence of the scale's convergent validity by documenting that the link between food‐waste‐aversion and BMI is attenuated among those who practice refrigerating leftovers (Study 3). We also document that the food‐waste‐aversion scale is distinct from general waste aversion and external meal‐cessation rules, thus providing evidence of discriminant validity (Studies 1a, 1b, and 1c). Taken together, these results provide construct validity for the novel construct of food‐waste‐aversion. We discuss the theoretical and substantive contributions of our findings.     

The Construct Validity of the Strengths and Difficulties Questionnaire for Aboriginal Children Living in Urban New South Wales,Australia

The current study explores the construct validity of the standard Strengths and Difficulties Questionnaire (SDQ) for Aboriginal children living in urban communities in New South Wales, Australia. Parent report SDQ data from the first 717 Aboriginal children aged 4–17 years who participated in the baseline survey of the Study of Environment on Aboriginal Resilience and Child Health were analysed. The overall construct validity of the SDQ in our sample was acceptable but not “good.” The internal consistency reliability was excellent overall and good for all subscales with the exception of peer problems, a concept that may have a different significance for urban Aboriginal parents. Removing the peer relationships subscale, however, did not improve the fit of the model. The convergent validity of the SDQ was good. The prosocial behaviours scale and both the peer and conduct problems scales were highly correlated suggesting Aboriginal parents conceptualise these differently and that prosocial behaviours may be considered a key indicator of well‐being for Aboriginal children. Overall, the SDQ is a promising tool for urban Aboriginal children in New South Wales. Those working with Aboriginal young people should focus on the SDQ total difficulties score and limit their reliance on the peer relationships subscale.     

多重情绪智力量表(MEIS)的信度、结构效度及应用评价研究

对多重情绪智力量表（MEIS）的信度和结构效度进行了实证检验,结果发现,MEIS的内部一致性信度（a）系数偏低,各分量表的分半信度系数很低;MEIS总量表的结构效度明显不足,对七个分量表的因素分析结果显示,每个维度对总量表的贡献率很低;无法满足Mayer和Salovey情绪智力4维度理论框架。对4个维度逐一进行二阶因素分析结果均揭示,MEIS明显缺乏每个维度所设定的4因素的结构效度,项目效率明显不足,因此,MEIS尚不具备有效测量情绪智力的功能。     

WORK ENGAGEMENT: A QUANTITATIVE REVIEW AND TEST OF ITS RELATIONS WITH TASK AND CONTEXTUAL PERFORMANCE

Many researchers have concerns about work engagement's distinction from other constructs and its theoretical merit. The goals of this study were to identify an agreed‐upon definition of engagement, to investigate its uniqueness, and to clarify its nomological network of constructs. Using a conceptual framework based on Macey and Schneider (2008; Industrial and Organizational Psychology, 1, 3–30), we found that engagement exhibits discriminant validity from, and criterion related validity over, job attitudes. We also found that engagement is related to several key antecedents and consequences. Finally, we used meta‐analytic path modeling to test the role of engagement as a mediator of the relation between distal antecedents and job performance, finding support for our conceptual framework. In sum, our results suggest that work engagement is a useful construct that deserves further attention.     

The concurrent validity of three self-report measures of borderline personality

The recent accumulation of self-report measures of borderline personality disorder (BPD) affords the opportunity to evaluate both the construct validity of the concept and the quality of these measures. This study examines the relationship among three recently developed self-report instruments for assessing BPD from the Personality Assessment Inventory (PAI; Morey, 1991), the MMPI Personality Disorders Scales (MPD; Morey, Waugh, & Blashfield, 1985), and the Bell Object Relations Inventory (BORI: Bell, Billington, & Becker, 1986). Data on the three measures were provided by 119 undergraduate subjects from a southeastern university. A correlational analysis addresses the convergence of these measures of BPD, their divergence from measures of different but related traits, and their independence from variance due to method. Application of the Campbell-Fiske (1959) criteria indicates adequate convergence for all the BPD measures but a lack of discriminant validity for the BORI scales. The fit of the data to a structural model of construct validity is tested using confirmatory factor analysis, and these results are consistent with the hypothesis of a latent borderline trait factor independent of measurement method factors. In sum, the construct validity of the borderline personality concept using self-report methodologies receives support, and a strong association between borderline personality and paranoid phenomena is also suggested.     

Workaholism in Organizations: Measurement Validation and Replication

Although workaholism in organizations has received considerable attention in the popular press, our understanding of it based on research is quite limited. This deficit is the result of absence of both suitable definitions and measures of the concept. This study reports the replication of earlier work undertaken by Spence and Robbins (1992) applying their measures to a managerial population (N = 530). Considerable support was found for the usefulness of their measures based on reliability and construct validity data.