Interrater reliability and agreement of performance ratings: A methodological comparison

This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, theT index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearsonr, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.This paper is based, in part, on a thesis submitted to East Carolina University by the second author. Portions of this study were presented at the American Psychological Association meeting in New Orleans, LA, August, 1989. The authors would like to thank Michael Campion and two anonymous reviewers for their comments on earlier drafts of this paper.     

Eye movements as a window into real-time spoken language comprehension in natural contexts

When listeners follow spoken instructions to manipulate real objects, their eye movements to the objects are closely time locked to the referring words. We review five experiments showing that this time-locked characteristic of eye movements provides a detailed profile of the processes that underlie real-time spoken language comprehension. Together, the first four experiments showed that listerners immediately integrated lexical, sublexical, and prosodic information in the spoken input with information from the visual context to reduce the set of referents to the intended one. The fifth experiment demonstrated that a visual referential context affected the initial structuring of the linguistic input, eliminating even strong syntactic preferences that result in clear garden paths when the referential context is introduced linguistically. We argue that context affected the earliest moments of language processing because it was highly accessible and relevant to the behavioral goals of the listener.We thank D. Ballard and M. Hayhoe for the use of their laboratory (National Resource Laboratory for the Study of Brain and Behavior). We also thank J. Pelz for his assistance in learning how to use the equipment and K. Kobashi for assisting in the data collection. Finally, we thank Janet Nicol and an anonymous reviewer for their comments and suggestions. The research was supported by NIH resource grant 1-P41-RR09283; NIH HD27206 (M.K.T.); an NSF graduate fellowship (M.J.S.-K.); and a Canadian SSHRC fellowship (J.C.S.).     

Initial examination of a multidimensional model of trauma-related guilt: Applications to combat veterans and battered women

Guilt is conceptualized as a multidimensional construct consisting of negative affect and a set of interrelated cognitions. Guilt magnitude is thought to be a function of the magnitudes of six variables posited as primary components of guilt: a negative event, distress, perceptions of responsibility, lack of justification, wrongdoing, and false beliefs about preoutcome knowledge. The model was tested with samples of Vietnam veterans and battered women. Participants rated their reactions to and perceived roles in trauma-related events. Among Vietnam veterans, distress ratings were highly correlated with guilt severity. Cognitive guilt-component variables were significantly correlated with guilt in both groups. In multiple regression, guilt components accounted for 61% of variance in veterans' guilt and 44% of variance in women's guilt. Among veterans, distress ratings were highly correlated with measures of PTSD and depression. In both groups, cognitive guilt-component variables were positively correlated with psychopathology. Results support the view that beliefs about one's role in trauma are important factors in posttrauma adjustment.     

Competing schools or stylistic variation in careers guidance interviewing

We know very little about how careers officers conduct careers guidance interviews. Research was carried out to discover how careers officers view interviewing, the factors that lead them to vary their interview style, and how they modify their approach with experience. There was little evidence that officers vary their style by using radically opposed models in different circumstances. Instead, variations in approach are best regarded as different combinations of ideas and techniques. Careers officers adapted their interview styles largely in response to the perceived needs of clients. Initial training course attended appeared to have a limited but enduring effect on interview style, but there was also evidence that with experience careers officers develop more confidence, coherence and flexibility. Careers officers judged their interviews to be most effective when they were able to establish a rapport with the client, to spend time drawing up a contract, to challenge clients in a sensitive way, and to structure the interview appropriately.     

The Relationship Between Aggression and Employment Integrity

The relationship between aggression and employment integrity was investigated, in order to assess the construct validity of a employment screening device. Participants (155 male, 151 female) completed an employment screening test and two aggression scales. For both men and women, general aggression negatively correlated with employment integrity. A negative relationship was also found between employment integrity and physical aggression, anger, and verbal aggression. No significant relationship was found between hostility and employment integrity.     

Receiving Help,Reciprocation, and Interpersonal Attraction1

In a 3 × 3 factorial experiment with 3 additional cells, 329 male or female subjects working with two same-sex confederates (A and B), received either voluntary, involuntary, or no help from A after which attraction to A and B were measured for one-fourth of the subjects. The remaining subjects were assigned to a second set of tasks and were induced to help A, help B, or were not given an opportunity to help, after which they completed attraction and mood measures. Subjects were more likely to volunteer help after the second task when they had received prior help. Receipt of help led to more liking for the helper and the helper was liked more when subjects were able to reciprocate. Attraction and mood results are discussed in relation to attractiveness mediation, modeling, and indebtedness explanations for reciprocity.     

FAMILY AND WORK PREDICTORS OF PSYCHOLOGICAL WELL-BEING AMONG HISPANIC WOMEN PROFESSIONALS

This study examines the relative contributions of job- and gender-related variables to the mental health of Hispanic women professionals. Hispanic women professionals, managers and business owners ( n = 303) were surveyed using the mailing list of a Hispanic professional women's organization. Results of multiple regression analyses indicate that income and Hispanic group were consistently related to mental health measures. Spouse support and ethnicity of spouse were associated with measures of stress in balancing roles and psychological distress symptoms. Marital status was related to personal life satisfaction (married women reported more satisfaction), and having young children was negatively associated with personal and professional satisfaction. Experience of discrimination, job stress and peer support were also significantly related to mental health measures. Although preliminary, this study underscores the importance of both job- and gender-related factors in research on the effects of employment on Hispanic women's mental health.     

Lexical projection and the interaction of syntax and semantics in parsing

A series of self-paced reading studies utilized an embedded anomaly technique to investigate long-distance dependencies with dative verbs. Previous research i our lab demonstrated that argument structure influences the gap-filling process. Experiment I extended that work by demonstrating that dative verbs pattern with other complex transitive verbs (i.e., a fronted filler that is implausible as the direct object will not be interpreted as the direct object until the absence of a noun phrase after the verb forces the postulation of a direct object gap. This pattern contrasts with that of transitive verbs that subcategorize for a single internal argument position, where fronted fillers are obligatorily interpreted as the direct object). Experiments 2 and 3 investigate the prediction that semantic analyses precede syntactic analyses in dative questions. It is argued that the lexical information about argument structure and thematic roles can guide semantic interpretation.This research was supported in part by NSF grant BNS-8617738 and NIH grant HD-22271.