Ill-structured measurement designs in organizational research: implications for estimating interrater reliability

Organizational research and practice involving ratings are rife with what the authors term ill-structured measurement designs (ISMDs)--designs in which raters and ratees are neither fully crossed nor nested. This article explores the implications of ISMDs for estimating interrater reliability. The authors first provide a mock example that illustrates potential problems that ISMDs create for common reliability estimators (e.g., Pearson correlations, intraclass correlations). Next, the authors propose an alternative reliability estimator--G(q,k)--that resolves problems with traditional estimators and is equally appropriate for crossed, nested, and ill-structured designs. By using Monte Carlo simulation, the authors evaluate the accuracy of traditional reliability estimators compared with that of G(q,k) for ratings arising from ISMDs. Regardless of condition, G(q,k) yielded estimates as precise or more precise than those of traditional estimators. The advantage of G(q,k) over the traditional estimators became more pronounced with increases in the (a) overlap between the sets of raters that rated each ratee and (b) ratio of rater main effect variance to true score variance. Discussion focuses on implications of this work for organizational research and practice.     

Not yet human: implicit knowledge, historical dehumanization, and contemporary consequences

Historical representations explicitly depicting Blacks as apelike have largely disappeared in the United States, yet a mental association between Blacks and apes remains. Here, the authors demonstrate that U.S. citizens implicitly associate Blacks and apes. In a series of laboratory studies, the authors reveal how this association influences study participants' basic cognitive processes and significantly alters their judgments in criminal justice contexts. Specifically, this Black-ape association alters visual perception and attention, and it increases endorsement of violence against Black suspects. In an archival study of actual criminal cases, the authors show that news articles written about Blacks who are convicted of capital crimes are more likely to contain ape-relevant language than news articles written about White convicts. Moreover, those who are implicitly portrayed as more apelike in these articles are more likely to be executed by the state than those who are not. The authors argue that examining the subtle persistence of specific historical representations such as these may not only enhance contemporary research on dehumanization, stereotyping, and implicit processes but also highlight common forms of discrimination that previously have gone unrecognized.     

Multisource Feedback and Leaders' Goal Performance: Moderating effects of rating purpose,rater perspective,and performance dimension

The main purposes of this study are to examine whether multisource feedback ratings predict leaders' organizational goal performance, and whether the relationships are consistent across the two rating purposes (developmental, administrative), two leadership dimensions behaviors (Consideration, Initiating Structure), and three rating perspectives (supervisor, self, and ‘other’ raters, i.e., peers and subordinates). Leaders (n=396) in a large organization in the transportation industry participated in two multisource feedback programs, the first for developmental purposes and the second 8 months later for administrative purposes. Approximately 1 month later, they were rated by their supervisor on their effectiveness in attaining five organizational performance goals (financial, safety, customer satisfaction, employee satisfaction, diversity). Results revealed that both developmental ratings and administrative ratings uniquely predicted leaders' goal performance. However, both leadership dimension and rater perspective moderated these relationships. Leadership behaviors associated with Consideration were stronger predictors of goal performance for supervisor ratings, whereas behaviors associated with Initiating Structure were stronger predictors of goal performance for self and other ratings.     

Acculturative dissonance, ethnic identity, and youth violence

Studies suggest that the process of acculturation for immigrant youth, particularly for second-generation youth, is significantly associated with delinquency and violence. This study explored the acculturation-violence link with respect to acculturative dissonance and ethnic identity. The results revealed in a sample of 329 Chinese, Cambodian, Mien/Laotian, and Vietnamese youth that acculturative dissonance was significantly predictive of serious violence, with full mediation through peer delinquency. Ethnic identity was not significantly associated with peer delinquency or serious violence. Although acculturative dissonance and ethnic identity accounted for a small percentage of variance in violence compared with peer delinquency, it cannot be discounted as trivial. Structural equation analyses provided support for both measurement and structural invariance across the four ethnic groups, lending support for cross-cultural comparisons. The results also lend support for the inclusion of cultural factors in youth violence prevention and intervention efforts.     

Subtyping adolescents with bulimia nervosa

Cluster analyses of eating disorder patients have yielded a "dietary-depressive" subtype, typified by greater negative affect, and a "dietary" subtype, typified by dietary restraint. This study aimed to replicate these findings in an adolescent sample with bulimia nervosa (BN) from a randomized controlled trial and to examine the validity and reliability of this methodology. In the sample of BN adolescents (N=80), cluster analysis revealed a "dietary-depressive" subtype (37.5%) and a "dietary" subtype (62.5%) using the Beck Depression Inventory, Rosenberg Self-Esteem Scale and Eating Disorder Examination Restraint subscale. The "dietary-depressive" subtype compared to the "dietary" subtype was significantly more likely to: (1) report co-occurring disorders, (2) greater eating and weight concerns, and (3) less vomiting abstinence at post-treatment (all p's<.05). The cluster analysis based on "dietary" and "dietary-depressive" subtypes appeared to have concurrent validity, yielding more distinct groups than subtyping by vomiting frequency. In order to assess the reliability of the subtyping scheme, a larger sample of adolescents with mixed eating and weight disorders in an outpatient eating disorder clinic (N=149) was subtyped, yielding similar subtypes. These results support the validity and reliability of the subtyping strategy in two adolescent samples.     

Comparing persistent juvenile delinquents and normative peers with the Five-Factor Model of Personality

Recent research suggests that personality traits are associated with delinquency. T-tests were run to identify which traits and facets of the Five-Factor Model of Personality contributed to differentiate persistent juvenile delinquents (n = 48) from normative peers (n = 48). Results showed that two traits, namely Agreeableness and Neuroticism, and 12 facets differed significantly between the groups. Observed effect sizes varied from medium to large.     

When head is tempered by heart: Heart rate variability modulates perception of other-blame reducing anger

This study examined the relationship between heart rate variability (HRV), as an index of individual differences in emotion regulation, and perception of other-blame, as a mechanism of anger induction. The physiological responses were recorded while subjects read a story from a computer screen. The story narrated a negative event in self-referred way -dismissal from a job as the result of a colleague’s action- under conditions of intentionality versus non-intentionality. Cognitive and emotional variables were assessed by questionnaire immediately after the physiological test. The resulting structural model supports the conclusion that HRV exerts its regulatory influence directly on perception of other-blame rather than on emotion. In situations of intentionality, individuals with higher HRV make less extreme evaluation of the offender’s blame, versus those with lower HRV, thus leading to a reduction in anger reaction. These results suggest that HRV is a direct index of cognitive rather than emotional regulation.     

The Relationship Between Adolescents’ Attitudes Toward Their Fathers and Loneliness: A Cross-Cultural Study

Loneliness can be regarded as an epidemic of modern society that is becoming increasingly problematic for millions of people. Adolescents are particularly susceptible to the development of this kind of emotional distress. Although many causes of loneliness can be identified, some researchers are of the opinion that it can also be attributed to unpleasant childhood experiences that result in unfulfilled needs relating to intimacy. In this research study, the relationship between loneliness and adolescents’ attitudes towards their fathers was investigated. In order to achieve this goal, information was gathered from 1068 South African respondents. Biographical questionnaires were administered and a Loneliness Scale was applied. In addition, adolescents’ attitudes towards their fathers were measured by means of a Fathers Questionnaire. Steps were taken to determine which of the independent variables, namely gender, age, race, language and socio-economic status, contributed significantly to the prediction of loneliness. In order to process the acquired information, a stepwise multiple regression analysis was conducted. The most important finding of this study is that adolescents’ attitudes towards their fathers comprise the most prominent predictor of loneliness during adolescence. The results also indicated that all of the independent variables had a statistically significant influence on loneliness.