What sources contribute to variance in observer ratings? Using generalizability theory to assess construct validity of psychological measures

Cronbach and Meehl (Psychol. Bull. 1955; 52:281–302) stated that the key question to be addressed when assessing construct validity is ‘What sources contribute to variance in test performance?’ We illustrate the utility of generalizability theory (GT) as a conceptual framework that encourages psychological researchers to address this question and as a flexible set of analytic tools that can provide answers to inform both substantive theory and measurement practice. To illustrate these capabilities, we analyze observer ratings of 27 caregiver–child dyads, focusing on the importance of situational (contextual) factors as sources of variance in observer ratings of caregiver–child behaviors. Cross‐situational consistency was relatively low for the categories of behavior analyzed, indicating that dyads vary greatly in their interactional patterns from one situation to the next, so that it is difficult to predict behavioral frequencies in one context from behaviors observed in a different context. Our findings suggest that single‐situation behavioral measures may have limited generalizability, either to behavior in other contexts or as measures of global interaction tendencies. We discuss the implications of these findings for research and measurement design in developmental psychology. Copyright © 2008 John Wiley & Sons, Ltd.     

Age effects on the employability-career success relationship

This study investigated the similarity of the factor structure for self-reported versus supervisor-rated employability for two age groups of workers, and then validated a career success enhancing model of employability across the two age groups. The results confirmed a two-factor model including self-reported and supervisor-rated employability as underlying factors. Moreover, Multi-Group Structural Equation Modeling (SEM) indicated that for the youngsters both self- and supervisor ratings of employability related significantly to objective career success outcomes. However, for the over-forties self-rated employability related positively to promotions throughout the career, while the supervisor ratings related negatively to overall promotions. The findings have important implications for performance appraisal practices aimed at increasing life-long employability and career success.     

Political skill as neutralizer of felt accountability—job tension effects on job performance ratings: A longitudinal investigation

This study examined the effects of felt accountability, political skill, and job tension on job performance ratings. Specifically, we hypothesized that felt accountability would lead to higher (lower) job performance ratings when coupled with high (low) levels of political skill, and that these relationships would be mediated by job tension. Data were gathered at multiple times over a one-year period (i.e., baseline performance, attitudinal variables one month later, supervisor reports of subordinate performance six months and one year after baseline performance was measured). Strong support was shown for the total effects model, whereby political skill moderated felt accountability—job performance ratings, felt accountability—job tension, and job tension—job performance ratings relationships. However, more focused analyses demonstrated that political skill most strongly moderated the job tension—job performance ratings linkage. Implications, strengths and limitations, and directions for future research are provided.     

Combining Data from Parent Ratings and Parent Interview When Assessing ADHD

This study examined different methods of combining data from a well established rating scale – the Disruptive Behavior Disorders Rating Scale (DBDRS) – with data from a well established interview – the computerized self report version of the Diagnostic Interview Schedule for Children for DSM-IV (P-DISC-IV) – when examining parent report of ADHD. These measures were completed for 80 children who participated in a summer camp for children with disruptive behavior. Results showed significant overlap between the two diagnostic measures but parent report on the DBDRS was more highly associated with children’s camp behaviors than was parent report on the P-DISC-IV. In addition, combining ratings and interview data using an “either/or” rule was more highly associated with camp behavior than was combining data using a “both/and” rule. Results suggest there may be little additional value of adding data from parent self-report on the computerized version of the P-DISC-IV to behavior rating data.     

Why convene rater teams: An investigation of the benefits of anticipated discussion,consensus, and rater motivation

This study explores the importance of anticipated group discussion, the consensus decision rule, and rater motivation in determining how well rater teams identify ratee behaviors, i.e., behavioral accuracy. Results, based on 382 raters in 111 teams, suggest that the anticipation of group discussion can improve behavioral accuracy, but it appears that the benefits of discussion-only teams are limited to this anticipation effect. Furthermore, it also appears that rater motivation plays an important role in this type of team. Rater teams required to reach consensus, however, appear to show improved behavioral accuracy, regardless of whether raters can anticipate the consensus discussion and regardless of rater motivation levels. Implications, especially for assessment centers, are discussed.     

Contextual control of human fear associations in a renewal paradigm

The original model of behavior change suggests that extinction is context dependent whereas fear acquisition is context independent [Bouton, M. E. & Ricker, S. T. (1994). Renewal of extinguished responding in a second context. Animal Learning and Behavior, 22, 317-324]. Supportive evidence stems mainly from animal studies, showing that after acquisition (conditioned stimulus-unconditioned stimulus (CS-US)) in Context A and extinction in Context B, fear is renewed by presenting the CS in acquisition Context A (ABA renewal) or in a novel Context C (ABC renewal). By implication, the model predicts equal ABA and ABC renewal. However, there is also evidence to suggest that the context dependency of extinction and the context independency of acquisition may be less stringent than originally proposed. The present study investigated renewal in humans using a differential fear conditioning paradigm with a shock US and online shock expectancy ratings and electrodermal responses as dependent variables. Experiment 1 compared an ABA condition with an AAA condition. Experiment 2 compared three conditions: ABA, ABC, and AAA. Both experiments demonstrated ABA renewal. Most importantly, Experiment 2 showed larger ABA than ABC renewal. Overall, results for expectancy ratings were more convincing than for electrodermal responses. In line with the extinction model, the present findings support the context dependency of extinction in humans. In contrast to the model, the findings suggest that in humans not only extinction learning, but also fear acquisition is controlled by its current context.     

Assessing Children's Homework Performance: Development of Multi-Dimensional, Multi-Informant Rating Scales

Efforts to develop interventions to improve homework performance have been impeded by limitations in the measurement of homework performance. This study was conducted to develop rating scales for assessing homework performance among students in elementary and middle school. Items on the scales were intended to assess student strengths as well as deficits in homework performance. The sample included 163 students attending two school districts in the Northeast. Parents completed the 36-item Homework Performance Questionnaire — Parent Scale (HPQ-PS). Teachers completed the 22-item teacher scale (HPQ-TS) for each student for whom the HPQ-PS had been completed. A common factor analysis with principal axis extraction and promax rotation was used to analyze the findings. The results of the factor analysis of the HPQ-PS revealed three salient and meaningful factors: student task orientation/efficiency, student competence, and teacher support. The factor analysis of the HPQ-TS uncovered two salient and substantive factors: student responsibility and student competence. The findings of this study suggest that the HPQ is a promising set of measures for assessing student homework functioning and contextual factors that may influence performance. Directions for future research are presented.     

Interpersonal perception and personality disorders: Utilization of a thin slice approach

Study 1 expands upon previous research by looking at the ability of untrained raters to detect pathological traits within a normal population of college students. In Study 1, 30-s video clips of 81 target persons were shown to 42 raters. Ratings of traits of personality disorders made by thin slice raters reliably predicted scores on the personality pathology measures obtained from the targets themselves and from close peers. Study 2 is a preliminary examination of how pathological rater traits impacts thin slice accuracy. In Study 2, peer and self-report data were examined regarding 87 thin slice raters. Raters who exhibited traits of narcissistic personality disorder were significantly less accurate in making personality predictions regarding targets. Three clusters of personality items were identified based on rater characteristics related to accuracy in predicting behavior.     

Reconsidering the Factor Structure of the Beavers Interactional Scales: Do External Ratings Mirror Self-Reports?

Factor analytic studies indicate that family relationships assessed by self-reports may be most clearly and parsimoniously differentiated along three dimensions, namely, Affective Expression, Behavioral Control, and Shared Activity. The present study explored the question of whether this were true of family ratings by outside parties. Confirmatory factor analysis was used with three data sets (n = 97, 91, and 95; N = 283) employing the Beavers Interactional Scales of family functioning. Multicollinearity of the observed variables was a pervasive problem. When the redundancy of the observed variables was pared down so that there were two observed variables for each latent factor, these three factors in oblique relationship provided a good fit to the observed data.