Self-serving bias effects on job analysis ratings

The purpose of this study was to investigate whether worker-oriented job analysis importance ratings were influenced by subject matter experts' (SME) standing (as measured by self-rated performance) on a competency. This type of relationship (whereby SMEs indicate that the traits they have are important for successful job performance) is an example of the self-serving bias (which is widely described in the social cognition literature and rarely described in the industrial/organizational psychology literature). An archival dataset covering 57 clerical and technical occupations with 26,682 participants was used. Support was found for the relationship between self-rated performance and importance ratings. Significant relationships (typically in the .30s) were observed for all 31 competencies that were studied. Controls were taken to account for common method bias and differences in the competencies required for each of the 57 occupations. Past research has demonstrated the effects of the self-serving bias on personality-based job analysis ratings. This study was the first to extend these findings to traditional job analysis, which covers other competencies in addition to personality. In addition, this study is the first to use operational field data instead of laboratory data.     

Effects of analyst training and amount of available job related information on job analysis ratings

The present study investigated the effects of job analysis training and the amount of information available to the job analyst on job analysis ratings. Forty subjects were asked to complete the Position Analysis Questionnaire (PAQ) and the Job Components Inventory (JCI) for the position of College Resident Assistant. Results indicated that when using the PAQ as the job analysis instrument, there was no significant effect for either training or the amount of information received by the job analyst. For the JCI, the results indicated that trained analysts were more reliable and accurate than untrained analysts. Also, it was found that the more information received by the job analysts, the more reliable and accurate were their JCI ratings.     

Relationships between job evaluation ratings and self-ratings of job characteristics

Self-ratings by 308 shop workers of their job variety, autonomy, task identity, and skill challenge correlate significantly with ratings by observers of the jobs' skill requirements. Self-ratings, however, are unrelated to observers' estimates of the physical demands of the jobs or to environmental conditions of the work. The self-ratings and observer measures provide complementary, not substitute, information. Self-ratings can signal person-job dysfunctions, and more objective job measures can identify possible sources of the dysfunctions. Modifications to job characteristics measures are discussed and suggestions are made to begin a new phase of programmatic job characteristics research.     

Understanding the latent structure of job performance ratings

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n = 2,350 and n = 2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%.     

The effect of dimension content on observation and ratings of job performance

Recent theory and research in the performance appraisal area (e.g., Denisi, Cafferty, & Meglino, 1984; Feldman, 1986; Ilgen & Feldman, 1983; Williams, Denisi, & Blencoe, 1985) have suggested that providing information regarding the performance dimension to be rated will cause raters to select appropriate observational schemata and, as a result, produce higher quality ratings. A study was conducted to determine if giving raters dimension-relevant information prior to performance observation would affect their attention processes and rating quality. Prior to watching a videotape of an instructor giving a lecture, 156 subjects were given either: 1) correct information, 2) incorrect information, or 3) no information regarding dimensions of performance they would subsequently be asked to rate. The results indicated that giving prior information regarding dimension content affected subjects' attention processes. Further, raters receiving no information and those receiving misinformation prior to performance observation produced less accurate ratings compared to expert raters. Ratings produced by subjects receiving correct information did not differ significantly from experts' ratings. These results are discussed in terms of both their practical and theoretical implications.     

The statistical analysis of concurrent detection ratings

The concurrent detection task is a powerful method for assessing interactions in the processing of two sensory signals. On each trial, a stimulus is presented that is composed of one, both, or neither signal, and the observer makes a detection rating for each stimulus. A classical bivariate signal-detection analysis applies to these data, but is limited by its inability to differentiate certain types of sensory interactions from more cognitive components, and by the lack of an associated testing procedure. The present paper presents an alternative analysis, based on the contingency table of sensory ratings. Six classes of effect can be distinguished and tested: (1) simple response bias, (2) detection of the two signals, (3) interference of each signal on the response to the other signal, (4) sensory and response correlation, (5) bivariate response biases, and (6) higher order association. Complete computational detail is provided.     

Modeling error variance in job specification ratings: the influence of rater, job, and organization-level factors

The authors modeled sources of error variance in job specification ratings collected from 3 levels of raters across 5 organizations (N=381). Variance components models were used to estimate the variance in ratings attributable to true score (variance between knowledge, skills, abilities, and other characteristics [KSAOs]) and error (KSAO-by-rater and residual variance). Subsequent models partitioned error variance into components related to the organization, position level, and demographic characteristics of the raters. Analyses revealed that the differential ordering of KSAOs by raters was not a function of these characteristics but rather was due to unexplained rating differences among the raters. The implications of these results for job specification and validity transportability are discussed.     

The impact of job complexity and performance measurement on the temporal consistency, stability, and test-retest reliability of employee job performance ratings

Although research has shown that individual job performance changes over time, the extent of such changes is unknown. In this article, the authors define and distinguish between the concepts of temporal consistency, stability, and test-retest reliability when considering individual job performance ratings over time. Furthermore, the authors examine measurement type (i.e., subjective and objective measures) and job complexity in relation to temporal consistency, stability, and test-retest reliability. On the basis of meta-analytic results, the authors found that the test-retest reliability of these ratings ranged from .83 for subjective measures in low-complexity jobs to .50 for objective measures in high-complexity jobs. The stability of these ratings over a 1-year time lag ranged from .85 to .67. The analyses also reveal that correlations between performance measures decreased as the time interval between performance measurements increased, but the estimates approached values greater than zero.     

The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: unconfounding construct-level convergence and rating difficulty

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed.     

Political skill as neutralizer of felt accountability—job tension effects on job performance ratings: A longitudinal investigation

This study examined the effects of felt accountability, political skill, and job tension on job performance ratings. Specifically, we hypothesized that felt accountability would lead to higher (lower) job performance ratings when coupled with high (low) levels of political skill, and that these relationships would be mediated by job tension. Data were gathered at multiple times over a one-year period (i.e., baseline performance, attitudinal variables one month later, supervisor reports of subordinate performance six months and one year after baseline performance was measured). Strong support was shown for the total effects model, whereby political skill moderated felt accountability—job performance ratings, felt accountability—job tension, and job tension—job performance ratings relationships. However, more focused analyses demonstrated that political skill most strongly moderated the job tension—job performance ratings linkage. Implications, strengths and limitations, and directions for future research are provided.     

Operating characteristic analysis of attribute ratings

The analysis of receiver operating characteristics as employed in psychophysics is suggested as a way of obtaining several useful measures in the context of attribute ratings. These include the difference between two stimuli on the attribute, the tendency for Ss to favor one pole of the rating scale, width of rating categories, and equal interval properties of the rating scale. The underlying measurement model is described along with means of evaluating its basic assumptions.