Do overall dimension ratings from assessment centres show external construct-related validity?

ABSTRACT

There have been repeated calls for an external construct validation approach to advance our understanding of the construct-related validity of assessment centre dimension ratings beyond existing internal construct-related validity findings. Following an external construct validation approach, we examined whether linking assessment centre overall dimension ratings to ratings of the same dimensions that stem from sources external to the assessment centre provides evidence for construct-related validity of assessment centre ratings. We used data from one laboratory assessment centre sample and two field samples. External ratings of the same dimensions stemmed from assessees, assessees’ supervisors, and customers. Results converged across all three samples and showed that different dimension-same source correlations within the assessment centres were larger than same dimension-different source correlations. Moreover, confirmatory factor analyses revealed source factors but no dimension factors in the latent factor structure of overall dimension ratings from the assessment centre and from external sources. Hence, consistent results across the three samples provide no support that assessment centre overall dimension ratings and ratings of the same dimensions from other sources can be attributed to dimension factors. This questions arguments that assessment centre overall dimension ratings should have construct-related validity.     

The Impact of Contextual Self‐Ratings and Observer Ratings of Personality on the Personality–Performance Relationship1

This study examined 2 possible ways of increasing the predictive validity of personality measures: using observer (i.e., supervisor and coworker) ratings and work‐specific self‐ratings of Big Five personality factors. Results indicated that among general self‐ratings of Big Five personality dimensions, Conscientiousness was the best predictor of in‐role performance, and Agreeableness and Emotional Stability were the best predictors of organizational citizenship behavior (OCB). Observer ratings of personality accounted for incremental variance in job performance (in‐role performance and OCB) beyond that accounted for by general self‐ratings. However, contrary to our expectations, work‐specific (i.e., contextual) self‐ratings of personality generally did not account for incremental variance in job performance beyond that accounted for by general self‐ratings.     

Cross‐cultural Measurement Invariance of the Employment Opportunity Index (EOI) in Mexican and Brazilian Professionals

Perceived employment alternatives have been shown to add incremental variance in understanding voluntary turnover as well as factors related to recruiting and staffing. The Employment Opportunity Index (EOI) is a multidimensional assessment of perceived employment alternatives that has exhibited predictive validity across multiple job types and populations. This study assessed the measurement invariance of the EOI in American, Mexican, and Brazilian professionals. We found support for configural and metric invariance in all five dimensions of the EOI with these populations, and support for scalar invariance three of the five EOI dimensions. The construct validity of the EOI appears to be relatively robust in Latin American populations, although cultural and macroeconomic factors may impose some response bias in these groups.     

The Ecological Validity of Neuropsychological Tests: A Review of the Literature on Everyday Cognitive Skills

Evaluating the ecological validity of neuropsychological tests has become an increasingly important topic over the past decade. In this paper, we provide a comprehensive review of the research on the ecological validity of neuropsychological tests, as it pertains to everyday cognitive skills. This review is presented in the context of several theoretical issues facing ecological validity research. Overall, the research suggests that many neuropsychological tests have a moderate level of ecological validity when predicting everyday cognitive functioning. The strongest relationships were noted when the outcome measure corresponded to the cognitive domain assessed by the neuropsychological tests. Several other factors that may moderate the degree of ecological validity established for neuropsychological tests are in need of further exploration. These factors include the effects of the population being tested, the approach utilized (verisimilitude vs. veridicality), the person completing the outcome measure (significant other vs. clinician), illness severity, and time from injury until evaluation. In addition, a standard measurement of outcome for each cognitive domain is greatly needed to allow for comparison across studies.     

Replication of the factor structure of the fear of AIDS schedule (FAIDSS) across samples

Abstract

The stability of the Fear of AIDS Scale (FAIDSS) was studied using three samples (two samples of health workers and one of social work students) on Cattel's s index. Using hyperplane cutoffs between 0.35 and 0.45, on the five factor solution of the FAIDSS, there were significant correlations between factors across samples and with few exceptions each factor correlated significantly with only one factor in its comparison samples. Concurrent validity was demonstrated by comparing scores on the Attitudes toward AIDS scale with the FAIDSS. Data indicated that the FAIDSS structure was stable across samples and is an appropriate instrument for measurement of fear of AIDS in the helping professions.     

Personality Correlates of Assessment Center Consensus Competency Ratings: Evidence from Russia

Controversy has revolved around whether assessment center ratings have construct validity to measure intended dimensions of managerial performance. In contrast to much recent research on the internal structure of assessment center ratings, the present studies investigated the relationship of final competency ratings derived by consensus discussion with external questionnaire measures of personality characteristics. Expanding on previous studies showing correlations of dimension scores in relation to individual trait measures, this study investigated the relationship of complex competencies with both single personality traits and with composites of personality traits. Evidence from two samples of managers in Russia shows that final competency ratings are related to predicted composites of personality factors more consistently than to single factors. Taken together, these findings provide evidence that assessment center ratings derived by consensus discussion show construct validity in relationship with predicted composites of personality characteristics.     

The relationship between productive and counterproductive work behaviour across four European countries

Research suggesting a similar but different relationship between Organizational Citizenship Behaviour (OCB) and Counterproductive Work Behaviour (CWB) is dominated by North American samples. Little evidence exists on whether these findings are replicated in other countries. To assess if a similar pattern emerged, we used the Social Axioms model (Bond et al., 2004) as a cultural framework and surveyed employees in the UK (105), The Netherlands (203), Turkey (185), and Greece (70) on the relationship between OCB and CWB, and the relationship between these behaviours and personality, justice, and commitment. Analysis supported a multidimensional structure to OCB and CWB and indicated a nonbipolar relationship between these behaviours. Culturally, somewhat different to OCB research in general, we find support for a convergence perspective across countries. Conceptually, linguistically and structurally the scale assessing OCB/CWB was shown to be equivalent across countries and a nonbipolar pattern of relationships was consistent across countries. Overall, findings imply a universal nature to the relationship between OCB and CWB across societal cultural groups.     

Organizational citizenship behavior and counterproductive work behavior: Do males and females differ?

While there has been considerable research on gender differences in core task performance, gender differences in organizational citizenship behavior (OCB) and counterproductive work behavior (CWB) have been largely ignored. Gender-consistent roles of females as being supportive would lead to the prediction that females engage in more OCB and less CWB than males. Using meta-analytic data gathered across 395 samples, we found several (weak) gender differences in CWB but none in OCB. We explain null and weak relationships from the perspective of social role theory, especially regarding males' roles being career-focused.     

A COMPARISON OF CRITERIA FOR TEST VALIDATION: A META-ANALYTIC INVESTIGATION

Meta-analyses of validity coefficients from tests of clerical abilities for five criteria—supervisor ratings, supervisor rankings, work samples, production quantity, and production quality—were conducted, and the resulting expected true validities were compared. Ratings, rankings, work samples, and production quantity all resulted in high test validities. Validities resulting from ratings and quantity-of-production criteria were highly similar across tests. Validities resulting from rankings and work samples were on the average higher than those from ratings and quantity of production. The fifth criterion, quality of production, had low predictability and did not generalize across situations.     

What sources contribute to variance in observer ratings? Using generalizability theory to assess construct validity of psychological measures

Cronbach and Meehl (Psychol. Bull. 1955; 52:281–302) stated that the key question to be addressed when assessing construct validity is ‘What sources contribute to variance in test performance?’ We illustrate the utility of generalizability theory (GT) as a conceptual framework that encourages psychological researchers to address this question and as a flexible set of analytic tools that can provide answers to inform both substantive theory and measurement practice. To illustrate these capabilities, we analyze observer ratings of 27 caregiver–child dyads, focusing on the importance of situational (contextual) factors as sources of variance in observer ratings of caregiver–child behaviors. Cross‐situational consistency was relatively low for the categories of behavior analyzed, indicating that dyads vary greatly in their interactional patterns from one situation to the next, so that it is difficult to predict behavioral frequencies in one context from behaviors observed in a different context. Our findings suggest that single‐situation behavioral measures may have limited generalizability, either to behavior in other contexts or as measures of global interaction tendencies. We discuss the implications of these findings for research and measurement design in developmental psychology. Copyright © 2008 John Wiley & Sons, Ltd.     

LEADERSHIP AND PROCEDURAL JUSTICE CLIMATE AS ANTECEDENTS OF UNIT-LEVEL ORGANIZATIONAL CITIZENSHIP BEHAVIOR

Despite an abundance of research conducted on organizational citizenship behavior (OCB) at the individual level of analysis, relatively little is known about unit-level OCB. To investigate the antecedents of unit-level OCB, data were collected from employees of 249 grocery store departments. Structural equation modeling techniques were used to test a model in which procedural justice climate was hypothesized to partially mediate the relationship between leadership behavior (servant-leadership) and unit-level OCB. Models were tested using both employee ratings and manager ratings of unit-level OCB. The results gave general support for the hypotheses, although there were some differences depending on the source of the OCB ratings (supervisor or subordinate), whether the type of department was controlled for, and whether a common method variance factor was included. Overall, the evidence generally supported the association of both servant-leadership and procedural justice climate with unit-level OCB. Building on the current study, a multilevel framework for the study of OCB is presented in conjunction with a discussion of future research directions in four specific areas.     

Conspiracy suspicions as a proxy for beliefs in conspiracy theories: Implications for theory and measurement

Research on the psychology of conspiracy theories has shown recent steps towards a standardization of measures. The present article seeks to continue that trend by presenting the Flexible Inventory of Conspiracy Suspicions (FICS), a questionnaire template that can be adapted to measure suspicions of a conspiracy around nearly any topic of public interest. Compared to conspiracy belief measures that ask about specific theories on a given topic, the FICS is worded in such a way as to provide relatively stable validity across time and cultural context. Using a hybrid approach incorporating classical test theory and Rasch scaling, three questionnaire studies on Mechanical Turk demonstrate the validity of the FICS in measuring conspiracy suspicions regarding 9/11, vaccine safety, and US elections, with good psychometric properties in most situations. However, the utility of the FICS is limited in the case of climate change due to the existence of two opposing conspiracy theories that share essentially no common assumptions (‘climate change is a hoax’ vs. ‘there is a conspiracy to make people believe that climate change is a hoax’). The results indicate that the FICS is a reliable and valid measure of conspiracy suspicions within certain parameters, and suggest a three‐level model that differentiates general conspiracist ideation, relatively vague conspiracy suspicions, and relatively specific conspiracy beliefs.     

Leader-Member Exchange and Employee Creativity: Test of a Multilevel Moderated Mediation Model

Using multisource data, this article examined the relationship of two types of employee perceptions of overall fairness (i.e., fairness of the organization and fairness of the department), with supervisor ratings of in-role task performance and organizational citizenship behaviors (OCB). We also examined whether high-quality leader-member exchange (LMX) relationships compensate for negative effects of fairness on job performance and citizenship behaviors. Focusing on a sample of 154 health care workers, we found that only employee perceptions of overall fairness regarding the department were related to supervisory ratings of OCB and in-role task performance. Moreover, LMX moderated the relationships between fairness perceptions and (a) task performance, (b) organization-focused OCB, but not (c) individual-focused OCB. We discuss implications for research and the potential for reducing negative effects of low fairness perceptions on job performance through positive LMX.     

Social validity of changes in informativeness and efficiency of aphasic discourse following linguistic specific treatment (LST).

This study presents the results of an analysis of pragmatic aspects of language samples obtained from five agrammatic aphasic individuals prior to and following Linguistic Specific Treatment (LST). Clinically and statistically significant positive changes in informativeness and efficiency were documented in posttreatment samples. Analysis of communicative competence (i.e., na?ve listeners' ratings of pre- and posttreatment audiotaped samples), was undertaken to examine the social validity of quantified changes. Mean ratings across listeners were computed to evaluate their subjective perceptions of general communicative constructs. Results indicated that objectively measured changes in pragmatic aspects were perceptible to na?ve listeners, however, to varying degrees across participants and constructs.     

Levels of measurement and analysis issues in organizational citizenship behaviour research

The majority of work in organizational behaviour has been conducted at the individual level of theory, measurement and analysis. While the field of organizational behaviour may be viewed as largely mixed‐level, incorporating individual, group and organizational‐level phenomena, relatively few group‐level and mixed‐level theories with corresponding levels of research exist. This is particularly problematic for organizational citizenship behaviour (OCB) research, since OCB is theorized to exist at multiple levels. Several individual‐level and group‐level antecedents affect OCB, but it is OCB in the aggregate (i.e. group and organizational level) which impacts organizational effectiveness, which is another organizational‐level phenomenon. A brief review of group‐level and mixed‐level research methodology is presented and these methods are then applied to a burgeoning area of organizational behaviour research, one which is inherently group and mixed‐level, organizational citizenship behaviour (OCB). Implications of the different levels of measurement and analysis for OCB theory development and testing are discussed.     

主管认知信任和情感信任对员工行为及绩效的影响

采用问卷调查法,检验主管认知信任与情感信任对员工行为及绩效影响的双路径模型。通过结构方程模型分析563份上下级配对数据,结果显示：主管认知信任通过注意聚焦的完全中介作用正向影响员工的任务绩效和组织公民行为。主管情感信任则通过情感承诺的部分中介作用正向影响员工的任务绩效和组织公民行为,即一方面直接影响员工的任务绩效和个体指向公民行为,另一方面通过情感承诺间接影响员工的任务绩效和组织公民行为。并且,主管情感信任对员工行为和绩效的影响作用要大于认知信任的作     

The positivity scale

Five studies document the validity of a new 8-item scale designed to measure positivity, defined as the tendency to view life and experiences with a positive outlook. In the first study (N = 372), the psychometric properties of Positivity Scale (P Scale) were examined in accordance with classical test theory using a large number of college participants. In Study 2, the unidimensionality of the P Scale was corroborated with confirmatory factor analysis in 2 independent samples (N1 = 322; N2 = 457). In Study 3, P Scale invariance across sexes and its relations with self-esteem, life satisfaction, optimism, positive negative affect, depression, and the Big Five provided further evidence of the internal and construct validity of the new measure in a large community sample (N = 3,589). In Study 4, test-retest reliability of the P Scale was found in a sample of college students (N = 262) who were readministered the scale after 5 weeks. In Study 5, measurement invariance and construct validity of P Scale were further supported across samples in different countries and cultures, including Italy (N = 689), the United States (N = 1,187), Japan (N = 281), and Spain (N = 302). Psychometric findings across diverse cultural context attest to the robustness of the P Scale and to positivity as a basic disposition. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

The Construct Validity of the Bioenergetic–Analytic Character Typology: A Multi-Method Investigation of a Humanistic Approach to Personality

The construct validity of the five bioenergetic–analytic character types was explored through a multitrait multi-method approach. Twenty-five college students had full-body photographs and Thematic Apperception Test (TAT) data rated using this typology, and these were compared to psychometric data, including from the Adjective Check List (ACL). The photographic ratings by two bioenergetic experts correlated significantly across all 5 types, evidencing satisfactory inter-rater reliability, but ratings on the TAT and ACL did not show adequate reliability and a comparison of the photographic ratings with the psychometrics did not evidence validity for the typology. The importance of developing multi-method approaches for holistic research across different domains within humanistic psychology, such as mind (psychological self-report) and body (photographic ratings) based methods, is emphasized.     

Instrumental values of organizational citizenship behavior for promotion: a field quasi-experiment

The present study examined the relationship between promotion, perceived instrumentality of organizational citizenship behavior (OCB) for promotion, and employees' OCB before and after promotion. A field quasi-experiment involving 293 tellers of a multinational bank was conducted. Both supervisors and employees provided OCB ratings 3 months before and 3 months after the promotion decision was announced. The authors found employees who perceived OCB as instrumental to their promotion and who were promoted were more likely to decline in their OCB after the promotion.