Managerial Experience and the Measurement Equivalence of Performance Ratings

ABSTRACT: Establishing the measurement equivalence of instruments is a prerequisite to making meaningful comparisons between individuals or within individuals over time. Whereas previous research has investigated the effects of rater characteristics on the measurement equivalence of performance ratings, the current study investigated a ratee characteristic—ratee job experience. Using confirmatory factor analysis and item response theory methods with replication, the measurement equivalence of supervisor ratings of 7,200 managers with differing levels of managerial experience was assessed. Overall, results indicated a high degree of measurement equivalence suggesting that meaningful comparison may be made across ratees with different levels of job experience.     

Construct validation of the Narcissistic Admiration and Rivalry Questionnaire in Spanish-speaking countries: Assessment of the reliability,structural and external validity and cross-cultural equivalence

A recent re-operationalisation of grandiose narcissism has resulted in the distinction of two narcissistic strategies based on the cognitive, affective-motivational and behavioural dynamics: admiration (assertive self-enhancement) and rivalry (antagonistic self-protection). The Narcissistic Admiration and Rivalry Questionnaire (NARQ) was developed to assess this model with two higher-order dimensions. However, cross-validations of the NARQ have not been extensively conducted across diverse population groups and languages. This study aimed to test the internal and external validity (through the relation with envy and self-esteem), reliability and cross-cultural equivalence of the Spanish version of the NARQ. The psychometric properties were evaluated in a Spanish sample (N = 310), and cross-cultural equivalence was tested in participants from Chile (N = 234) and Colombia (N = 256). The results supported the reliability and validity of the Spanish NARQ, as well as the cross-cultural equivalence across Spanish-speaking countries. In addition, we discuss obtained differences across Spanish, Chilean and Colombian sample within two narcissistic strategies.     

Measurement Equivalence Across Racial/Ethnic Groups of the Mood and Feelings Questionnaire for Childhood Depression

As research continues to document differences in the prevalence of mental health problems such as depression across racial/ethnic groups, the issue of measurement equivalence becomes increasingly important to address. The Mood and Feelings Questionnaire (MFQ) is a widely used screening tool for child and adolescent depression. This study applied a differential item functioning (DIF) framework to data from a sample of 6th and 8th grade students in the Seattle Public School District (N = 3,593) to investigate the measurement equivalence of the MFQ. Several items in the MFQ were found to have DIF, but this DIF was associated with negligible individual- or group-level impact. These results suggest that differences in MFQ scores across groups are unlikely to be caused by measurement non-equivalence.     

Meaning in life of young and elderly adults: An examination of the factorial validity and invariance of the Life Regard Index

The main question of the present investigation was whether the Life Regard Index (LRI) is an adequate instrument to study possible differences between young and elderly adults with regard to experienced meaning in life. Participants in this study were a group of 206 young adults (M= 17.8 years, 49.0% female) and a group of 373 elderly adults (M = 65.90 years, 58.8% female). Respondents completed a Dutch paper-and-pencil version of the LRI. The LISREL confirmatory factor analytic model was used to test for the equivalence of measurement and of structure of the instrument. Results show that in both age groups the items of the LRI were found to be distributed to the two a priori dimensions of meaning in life, Framework and Fulfillment. Only the factor loadings of the Framework items were invariant across both groups. Neither the error of measurement, nor the structure of the underlying concept were equivalent for young and elderly adults. Young adults were found to experience less meaning in life than the elderly.     

Factorial invariance,differential item functioning,and norms of the Orgasm Rating Scale

Background/Objective: Orgasm Rating Scale (ORS) assess the subjective orgasm experience in context of sexual relationship. It is composed of four dimensions attributed to the orgasm (Affective, Sensory, Intimacy, and Rewards). The purpose is to analyse the factorial invariance of the ORS across groups, to examine the metric equivalence across sex, and to present the standard scores. Method: A total of 1,472 Spanish adults (715 men and 757 women) were evaluated. They were distributed across age groups (18-34, 35-49 and 50 years old and older). Factorial invariance across different groups and the differential functioning of the items across sex were analyzed, internal consistency was examined, and the standard scores were developed. Results: The structure of the ORS showed strict measurement invariance across sex, relationship status, sexual orientation and education level. It also reached a scalar measurement invariance across age range and duration of the relationship. Some items showed a differential functioning between sexes. Conclusions: The Spanish version of the ORS is invariant across different groups at a factorial level, and it shows equivalence across sex in most of its items at a metric level. The standard scores allow a more accurate assessment of the subjective orgasm experience in context of sexual relationship.     

Ecocultural bias in culture-specific intelligence tests in an ecologically diverse culture: The Philippines

In an ecologically diverse society (the Philippines), measurement equivalence across ecological settings was investigated with two culture-specific intelligence tests. Comparative reliability and factor analyses indicated that the tests (one for preschoolers, one for adults) were measuring the same ability dimensions equally reliably within each setting. Across settings, however, delta-plot analyses revealed considerable bias at the item level in tests of crystallized abilities, and the preschool tests tended to favor children from the setting where the tests were developed. The results are interpreted in terms of an ecocultural framework in which group differences in intellectual performance can reflect not only actual ability differences but differences in practice with, and exposure to, particular environmental stimuli and/or cognitive skills. The study demonstrates: (a) the need, particularly in ecologically diverse and segmented societies, to inspect tests for item bias; (b) the difficulty of culturally “equating” cognitive measures, even for very similar cultural subgroups who differ primarily in ecological setting.     

Are assessment center behaviors' meanings consistent across exercises? A measurement invariance approach

To examine the appropriateness of a Multi‐Trait–Multi‐Method framework for testing construct validity of Assessment Centers (ACs) and get practical implications for the improved AC design, degree to which the AC dimension‐related performance behaviors consistently manifest across multiple AC rating situations was investigated. The present study used a large sample (N = 5,006) to apply a measurement invariance analysis. AC rating situations generally produced consistent factor loadings for items on AC dimensions, item residuals, dimension factor variances, and covariance between dimensions. The AC rating situation of interview tended to produce higher ratings and less item residuals. These findings support the consistency in constructs assessed across different AC rating situations, while some exercises may be better for teasing apart particular dimensions than others.     

Measurement Equivalence/Invariance of the Abusive Supervision Measure Across Workers From Taiwan and the United States

Growing international research interest in negative-leadership behaviors prompts the need to examine whether measures of ineffective leadership developed in the United States are equivalent across countries outside the United States. B. J. Tepper's (2000) abusive supervision measure has been used widely inside and outside the United States and merits research attention on its construct equivalence across different cultural settings. The authors conducted a series of multigroup confirmatory factor analyses to investigate the measurement equivalence of this measure across Taiwan (N = 256) and the United States (N = 389). Configural invariance was established, suggesting that both U.S. and Taiwanese samples perceive abusive supervision as a single-factor concept. Furthermore, the establishment of partial metric invariance and partial scalar invariance suggests that the abusive supervision measure is applicable to crosscultural comparisons in latent means, construct variance, construct covariances, and unstandardized path coefficients with the caution that workers from different cultures calibrate their responses differently when answering some items.     

Equivalence tests for comparing correlation and regression coefficients

Equivalence tests are an alternative to traditional difference‐based tests for demonstrating a lack of association between two variables. While there are several recent studies investigating equivalence tests for comparing means, little research has been conducted on equivalence methods for evaluating the equivalence or similarity of two correlation coefficients or two regression coefficients. The current project proposes novel tests for evaluating the equivalence of two regression or correlation coefficients derived from the two one‐sided tests (TOST) method (Schuirmann, 1987, J. Pharmacokinet. Biopharm, 15, 657) and an equivalence test by Anderson and Hauck (1983, Stat. Commun., 12, 2663). A simulation study was used to evaluate the performance of these tests and compare them with the common, yet inappropriate, method of assessing equivalence using non‐rejection of the null hypothesis in difference‐based tests. Results demonstrate that equivalence tests have more accurate probabilities of declaring equivalence than difference‐based tests. However, equivalence tests require large sample sizes to ensure adequate power. We recommend the Anderson–Hauck equivalence test over the TOST method for comparing correlation or regression coefficients.     

Confirmatory Validation and Measurement Equivalence of the Eating Loss of Control Scale in Binge Eating and Non-Clinical Samples

Although Loss of Control (LOC) is a transdiagnostic factor in eating pathology, there are few standalone assessments of LOC. The objective of this study was to evaluate the uni-dimensionality and measurement equivalence of the Eating Loss of Control Scale (ELOCS). Confirmatory factor analyses were used to achieve a well-fitting uni-dimensional model in clinical (N?=?226) and non-clinical (N?=?476) samples. Measurement equivalence was tested in a factor analytic framework, and effect sizes were computed to evaluate the impact of non-equivalence. A well-fitting model was achieved in both samples after the removal of 4 items. The instrument showed configural equivalence but not metric equivalence. Results suggest that the ELOCS is a reliable and valid measure of LOC in clinical and non-clinical samples. However, while the nature of the LOC construct is similar across binge eating and non-clinical participants, comparisons of ELOCS across these groups are affected by measurement non-equivalence. This research also revealed novel insights into the relative sensitivity of model fitting and effect size approaches to investigating measurement equivalence.     

Evaluating the Psychometric and Measurement Characteristics of a Measure of Sexual Orientation Harassment

We use classical test theory (CTT) and item response theory (IRT) methodologies to examine the psychometric and measurement properties of an instrument designed to assess sexual orientation harassment among military personnel (N?=?71,989). CTT analyses indicated that items were unidimensional and exhibited adequate levels of reliability. IRT analyses demonstrated that the items functioned similarly and exhibited appropriate levels of item discrimination. However, the analyses also suggested that the sensitivity of the items may be limited. Differential test functioning analyses provided evidence of the measurement equivalence of the instrument across male and female respondents. The findings provide support for the psychometric properties and measurement equivalence of the instrument for measuring sexual orientation harassment among male and female military personnel. We discuss the implications of our findings for future research on sexual orientation harassment in the workplace.     

Couple Adjustment and Differentiation of Self in the United States,Italy, and Spain: A Cross-Cultural Study

This study comprises a first attempt to explore a cross-cultural application of Bowen family systems theory and examines the relationship between differentiation of self and couple adjustment using measurement invariance. A sample of 2,141 individuals (n = 915 from the United States; n = 635 from Italy; n = 591 from Spain) completed measures of differentiation of self (DSI-R; Skowron & Schmitt, J Marital Fam Ther, 29, 2003, 209) and couple adjustment (DAS; Spanier, J Marriage Fam, 38, 1976, 15). Results showed evidence of partial measurement equivalence for the DSI-R across the three samples. Additionally, results from latent regression showed that less emotional cutoff predicted greater couple adjustment in the United States, Spain, and Italy, whereas greater emotional reactivity predicted poorer couple adjustment only in the more collectivistic cultures, Spain and Italy. Our findings are consistent with prior empirical studies, which suggest that emotional cutoff is a strong predictor of relationship adjustment. The use of measurement invariance provides a foundation for future studies to continue employing rigorous statistical methods when examining constructs across different cultures. The findings highlight culture-specific similarities and differences in differentiation that may benefit practitioners by informing psychotherapy with individuals, couples, or families from diverse populations.     

Life Satisfaction of Gifted and Average Adolescents in Hong Kong: Validation of the Chinese Brief Multidimensional Students’ Life Satisfaction Scale (BMSLSS)

Despite the popularity of the Brief Multidimensional Students’ Life Satisfaction Scale (BMSLSS) for use in research, this instrument has not yet been validated with adolescents of different intellectual ability levels and across different cultures. This study examined the psychometric properties and measurement equivalence of BMSLSS with a sample of 676 Chinese adolescents who are deemed ‘gifted’ and attending the Hong Kong Academy of Gifted Education, and 702 students with average achievement in mainstream high schools. Results indicated high internal consistency for the scale and a one-factor solution for BMSLSS with this population. Tests of configural, full metric and partial scalar invariance suggest that gifted students attending an academy conceptualize life satisfaction in a same way as their average attainment peers in regular schools. The findings support the applicability and measurement equivalence of BMSLSS for use with Chinese gifted and average-ability students.

     

Measurement equivalence in the conduct of a global organizational survey across countries in six cultural regions

This study examined the measurement equivalence of a global organizational survey measuring six work climate factors as administered across 25 countries (N = 31,315) in all regions of the world (West Europe, East Europe, North America, Latin America, South America, Middle East, Africa and Asia‐Pacific). Across all countries, the survey instrument exhibited ‘form equivalence’ and ‘metric equivalence’, suggesting that respondents completed the survey using the same frame‐of‐reference and interpreted the rating scale intervals similarly. Schwartz's (1994, 1999, 2004) cultural value theory was then used for grouping the countries in cultural regions, and to anticipate measurement equivalence of the data from the survey within and between these regions. Results showed partial support for Schwartz's theory. The English‐speaking region was the only region where empirical evidence for ‘scalar equivalence’ was found. No support was found for the prediction that measurement equivalence would be higher among countries that are part of cultural regions with a small cultural distance than among countries that are part of cultural regions with a large cultural distance. However, the use of a common language in a particular cultural region reduced the bias present in the cross‐country comparison within that region.     

Evaluation of the Religiosity in 5-D Scale

ABSTRACT

The goal of this study was to evaluate the factor structure and measurement equivalence of the Religiosity in 5-D Scale (Faulkner & DeJong, 1966). Using 262 participants from the Georgia Centenarian Study (Poon et al., 1992), analyses included exploratory factor analyses without rotation and using rotation, confirmatory factor analyses of a multiple and single factor religiosity construct, the establishment of measurement equivalence across age groups. The 5-D scale maintained a one-factor construct of religiosity. Dimensions of this factor included ideology, experience, ritual, and consequence. A measurement model of the religiosity construct was then established using summary scores. Confirmatory factor analysis revealed a satisfactory model of fit (GFI = .99; CFI = 1.00). Finally, measurement equivalence was maintained across age groups including sexagenarians (GFI = .95), octogenarians (GFI = .89), and centenarians (GFI = .95). The 5-D scale is an appropriate measure for use in aging research.     

Borderline Personality Disorder in 3-D: Dimensions,Symptoms, and Measurement Challenges

Borderline personality disorder (BPD) is a pervasive pattern of psychopathology characterized by unstable affect, suicidal behaviors, and identity problems (Diagnostic and statistical manual of mental disorders. Washington, DC: Author; 2000). BPD is heterogeneous in nature, highly prevalent in clinical settings, and increasingly studied by clinical and social psychologists. This review highlights affective instability, impulsivity, and interpersonal hypersensitivity, the interaction of which we believe accounts for the symptoms of BPD, the BPD criteria most associated with these features, and the interrelationships among these underlying dimensions. We also discuss difficulties in measurement of these dimensions. Real-world assessment methods will assist in the measurement of time-dependent processes and identify causes, covariates, or consequences of these processes to determine how these features manifest themselves in real life.     

TRAINING TEACHERS IN GENERALIZED WRITING OF BEHAVIOR MODIFICATION PROGRAMS FOR MULTIHANDICAPPED DEAF CHILDREN

In contrast to previous studies where teachers were instructed how to implement behavior modification programs designed by an experimenter, teachers in the present experiment were taught how to write as well as implement behavior modification programs. The generalized effects of two training conditions on teacher and pupil behaviors were assessed by a multiple baseline design where, following baseline, two teachers of multihandicapped deaf children were taught to set objectives and measure pupil performance (measurement training). Later, through a training manual, they learned a general problem-solving approach to writing behavior modification programs (programming training). After both training conditions, experimenter feedback was given for teachers' application of training to a target behavior for one pupil and generalization was measured across target behaviors for the same pupil and across pupils. It was found that measurement training had little general effect on either teacher behavior or pupil behavior. However, after programming training, teachers increased their program writing and correct use of behavior modification procedures and generalized this training across pupils and target behaviors. Along with these effects, there was improvement in pupil behaviors. Possible explanations for generalized effects of teacher training were considered.     

Temporal and spatial concepts in child language: conventional and configurational

The purpose of this research was to study the temporal and spatial systems of child language. Configurational time/space refers to the location of events/objects relative to referent events/objects (e.g., before/after or across/along) and conventional time/space refers to the specification of cycles/directions and the measurement of duration/distance (e.g., winter/fall or North/South). The research was designed to determine how configurational and conventional notions of time and space emerge in the child's language. The average age of the children in the cross-sectional design was: 3;9, 6;7, and 8;7. The children received a comprehension test that required them to match a sentence with a picture. In general, configurational contrasts were easier to comprehend than conventional contrasts, and, within the configurational problems, location requiring a single referent was easier than location requiring multiple referents. There was no overall difference between temporal and spatial dimensions. The role of conceptual development was discussed within the context of a comparison between first and second language acquisition.     

Cross-national applicability of a game-based cognitive assessment

New technology has had a discernable impact on how organizations recruit and select potential employees. Game-based assessment has emerged as a potential technology that can be used to enhance the assessment of individual differences and applicants' views of the selection process. However, studies investigating the psychometric properties and predictive validity of game-based assessments are still lacking. This study investigated the structural equivalence of a game-based assessment of cognitive ability across 228 Australians and 239 South Africans. A smaller sample of 115 South Africans also received work performance ratings to investigate the predictive validity of the cognitive assessment. Results of factor analysis supported a strong general factor of cognitive ability across the entire sample but only partial metric and scalar invariance across the two nations. The general factor of the game-based assessment further revealed promising results in terms of its predictive validity for five broad dimensions of individual work performance.     

The inconsistency with which raters weight and combine information across targets

To assign an overall performance rating to a target, a rater must weight and combine various pieces of specific performance information about that target. Policy‐capturing research has demonstrated that individual differences in raters can influence the way raters combine specific performance information. The current study examined information processing from a different perspective by exploring the possibility that target differences may also influence the way raters weight and combine performance information. Raters (N = 146) rated each of six targets on six specific performance dimensions and on overall performance. Sequential moderation analyses indicated that targets influenced the way raters, as a group, combined information across targets. These results lend support to the inference that overall performance ratings may not be comparable across targets, that is, they may not reflect the same underlying performance across targets.