A multitrait-multimethod analysis of variance of teachers' ratings of aggression,hyperactivity, and inattention

The convergent and discriminant validities of three teacher rating scale measures of the traits of hyperactivity, aggression, and inattention were explored, using the multitrait-multimethod matrix approach of Campbell and Fiske (1959), as well as an analysis of variance procedure (Stanley, 1961). In the present study teachers rated children from their elementary school classrooms on the above traits. The results provided strong evidence for convergent validity. Data also indicated that these traits can be reliably differentiated by teachers, suggesting that research aimed at better understanding the unique contributions of hyperactivity, aggression, and inattention is warranted. The respective benefits of analyzing multitrait-multimethod matrices by employing the ANOVAprocedure or by using the Campbell and Fiske (1959) criteria were discussed.     

Relationships between job evaluation ratings and self-ratings of job characteristics

Self-ratings by 308 shop workers of their job variety, autonomy, task identity, and skill challenge correlate significantly with ratings by observers of the jobs' skill requirements. Self-ratings, however, are unrelated to observers' estimates of the physical demands of the jobs or to environmental conditions of the work. The self-ratings and observer measures provide complementary, not substitute, information. Self-ratings can signal person-job dysfunctions, and more objective job measures can identify possible sources of the dysfunctions. Modifications to job characteristics measures are discussed and suggestions are made to begin a new phase of programmatic job characteristics research.     

A comparison of female supervisors in business and government organizations

This exploratory study compared the life style and interpersonal need orientation, leadership style, and perception of the organization structure of female supervisors in business and government organizations. The government group had a significantly higher formalistic and lower sociocentric and personalistic life style orientation; lower needs for expressed control, expressed affection, wanted inclusion, wanted affection, and a higher need for wanted control; were lower on consideration and higher on structure leadership dimensions; and perceived their organization as being more bureaucratic and less collaborative and coordinative than the business group. The findings were discussed in terms of the processes of individual-organization congruity and organizational socialization.     

Environments as reinforcer systems in the study of job satisfaction

This paper examined differences in the degree to which fifteen specific sources of job satisfaction were related to and predictive of the overall satisfaction of department chairmen in the six model environments proposed by Holland (1973). The fifteen specific sources of job satisfaction were regressed on overall satisfaction, separately for chairmen in each environment, through the use of incremental stepwise multiple regression procedures. The results indicated that the overall satisfaction of chairmen in these six environments was differentially related to unique weightings and combinations of the fifteen predictor variables. It was concluded that environments, composed of essentially similar groups of people in different organizations, serve as job satisfaction reinforcer systems in a manner similar to organizations as illustrated by Dawis.     

Understanding the latent structure of job performance ratings

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n = 2,350 and n = 2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%.     

Describing organizations as reinforcer systems: A new use for job satisfaction and employee attitude surveys

Job satisfaction data on managerial personnel were used to illustrate how organizations can be differentiated and described as reinforcer systems. Twenty-seven specific satisfaction scales were regressed on overall job satisfaction, separately for each of five groups of managers from five organizations. For all groups, overall satisfaction was found to be determined mainly by scales concerning challenge of the job and prospects of one's chosen career. Application of decremental stepwise multiple regression procedures to obtain minimum-variable-set equations resulted in different, though overlapping, sets of variables in the equations for the different organizations. These results suggest a way of describing organizations as reinforcer systems, in terms of the specific determinants of overall satisfaction for the organization.     

Effects of analyst training and amount of available job related information on job analysis ratings

The present study investigated the effects of job analysis training and the amount of information available to the job analyst on job analysis ratings. Forty subjects were asked to complete the Position Analysis Questionnaire (PAQ) and the Job Components Inventory (JCI) for the position of College Resident Assistant. Results indicated that when using the PAQ as the job analysis instrument, there was no significant effect for either training or the amount of information received by the job analyst. For the JCI, the results indicated that trained analysts were more reliable and accurate than untrained analysts. Also, it was found that the more information received by the job analysts, the more reliable and accurate were their JCI ratings.     

The value of relative-time-spent ratings in task-oriented job analysis

Recent studies have called for the abandonment of the relative-time-spent scale in task inventories. This recommendation is based on findings that the job profile created with the scale data is highly correlated with the profile created from a much simpler “Do you perform this task?” checklist. We examined this issue using 3 inventories and 42 jobs (N=2252). Profile correlations were computed on only the tasks actually performed by incumbents to avoid possibly inflated rs due to including irrelevant tasks. The specificity of task inventory items was proposed as an explanation for the high correlation between the two job profiles. Specificity of items was examined by looking at both the type (job duties versus tasks) and the amount (number of items in job profile and average number of items relevant to each job) of items used in the inventory. Correlations between time spent and checklist profiles were in the .80's and .90's regardless of the number of irrelevant tasks or the specificity of tasks. We agree with previous military research and conclude that the relative-time-spent scale has limited incremental utility beyond a dichotomous checklist.     

Interrater reliability and agreement of performance ratings: A methodological comparison

This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, theT index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearsonr, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.This paper is based, in part, on a thesis submitted to East Carolina University by the second author. Portions of this study were presented at the American Psychological Association meeting in New Orleans, LA, August, 1989. The authors would like to thank Michael Campion and two anonymous reviewers for their comments on earlier drafts of this paper.     

Political skill as neutralizer of felt accountability—job tension effects on job performance ratings: A longitudinal investigation

This study examined the effects of felt accountability, political skill, and job tension on job performance ratings. Specifically, we hypothesized that felt accountability would lead to higher (lower) job performance ratings when coupled with high (low) levels of political skill, and that these relationships would be mediated by job tension. Data were gathered at multiple times over a one-year period (i.e., baseline performance, attitudinal variables one month later, supervisor reports of subordinate performance six months and one year after baseline performance was measured). Strong support was shown for the total effects model, whereby political skill moderated felt accountability—job performance ratings, felt accountability—job tension, and job tension—job performance ratings relationships. However, more focused analyses demonstrated that political skill most strongly moderated the job tension—job performance ratings linkage. Implications, strengths and limitations, and directions for future research are provided.     

Activity level: A comparison between actometer scores and observer ratings

Activity level of 13 boys (aged 9–13) from a day hospital program was measured using actometers in classroom, gym, woodshop, and group therapy settings. Ratings of Ss' activity were obtained from mothers using the Werry scale, and from six clinical staff familiar with the Ss using the Davids scale. It was predicted that activity ratings would have situationally specific relationships with actometermeasured activity level according to the rater's opportunities for observation. Comparisons between measures indicated that all clinical staff ratings correlated significantly with actometer activity in the classroom (r=.49 to r=.73), while mothers' ratings correlated significantly with actometer activity in gym (r=.67), and woodshop (r=.77), and with overall activity (r=.65), a combined measure derived from actometer scores in the four conditions tested. Five of six clinical staff raters showed significant interrater reliability (r=.58 to r=.83). Results are discussed in terms of their implications for solution of current problems in assessment of activity level and hyperactivity.     

The effect of dimension content on observation and ratings of job performance

Recent theory and research in the performance appraisal area (e.g., Denisi, Cafferty, & Meglino, 1984; Feldman, 1986; Ilgen & Feldman, 1983; Williams, Denisi, & Blencoe, 1985) have suggested that providing information regarding the performance dimension to be rated will cause raters to select appropriate observational schemata and, as a result, produce higher quality ratings. A study was conducted to determine if giving raters dimension-relevant information prior to performance observation would affect their attention processes and rating quality. Prior to watching a videotape of an instructor giving a lecture, 156 subjects were given either: 1) correct information, 2) incorrect information, or 3) no information regarding dimensions of performance they would subsequently be asked to rate. The results indicated that giving prior information regarding dimension content affected subjects' attention processes. Further, raters receiving no information and those receiving misinformation prior to performance observation produced less accurate ratings compared to expert raters. Ratings produced by subjects receiving correct information did not differ significantly from experts' ratings. These results are discussed in terms of both their practical and theoretical implications.