Measurement Equivalence and Multisource Ratings for Non-managerial Positions: Recommendations for Research and Practice

ABSTRACT: The present investigation applies a comprehensive sequence of confirmatory factor analysis tests (Vandenberg and Lance, Organisational Research Methods, 3, 4–69, 2000) to the examination of the measurement equivalence of self, peer, and supervisor ratings of non-managerial targets across several performance dimensions. Results indicate a high degree of measurement equivalence across rater sources and performance dimensions. The paper illustrates how this procedure can identify very specific areas of non-equivalence and how the complexity of a multisource feedback system may be represented using such procedures. Implications of these results and recommendations for both research and practice are offered.     

Supervisors as SMEs: The Relationship Between Supervisor Job Experience and Ratings of Subordinate Skill Importance

This study investigated the relationship between supervisor job experience and ratings of the importance of eight skill components for performing subordinates' jobs. When examining these correlations, statistical control was utilized to ensure that supervisor and subordinate characteristics confounded with supervisor job experience were not affecting hypothesized relationships. As predicted, there were statistically significant positive correlations between supervisor job experience and ratings on 6 of the 8 components. Also as predicted, controlling for the characteristics confounded with supervisor job experience made a substantial difference in results; 13 of the 16 correlations using statistical control significantly differed from correlations that did not. This illustrates that future research on SME characteristics should consider controlling variables confounded with the focal characteristic(s) under study in order to more thoroughly understand characteristic–rating relationships. Implications of these findings for research and practice are discussed and suggestions for future research are offered.     

Explaining the Weak Relationship Between Job Performance and Ratings of Job Performance

Ratings of job performance are widely viewed as poor measures of job performance. Three models of the performance–performance rating relationship offer very different explanations and solutions for this seemingly weak relationship. One-factor models suggest that measurement error is the main difference between performance and performance ratings and they offer a simple solution—that is, the correction for attenuation. Multifactor models suggest that the effects of job performance on performance ratings are often masked by a range of systematic nonperformance factors that also influence these ratings. These models suggest isolating and dampening the effects of these nonperformance factors. Mediated models suggest that intentional distortions are a key reason that ratings often fail to reflect ratee performance. These models suggest that raters must be given both the tools and the incentive to perform well as measurement instruments and that systematic efforts to remove the negative consequences of giving honest performance ratings are needed if we hope to use performance ratings as serious measures of job performance.     

The impact of rater agreeableness and rating context on the evaluation of poor performance

We tested the effects of rater agreeableness on the rating of others’ poor performance in performance appraisal (PA). We also examined the interactions between rater agreeableness and two aspects of the rating context: ratee self‐ratings and the prospect of future collaboration with the ratee. Participants (n= 230) were allocated to one of six experimental groups (a 3 × 2 between‐groups design) or a control group (n= 20). Participants received accurate, low‐deviated, or high‐deviated self‐ratings from the ratee. Half were notified they would collaborate with the ratee in a future task. High rater agreeableness, positive deviations in self‐rating, and the prospect of future collaboration were all independent predictors of higher PA ratings. The interactions between rater agreeableness and rating context were very small. We argue that conflict avoidance is an important motivation in the PA process.     

Political skill as neutralizer of felt accountability—job tension effects on job performance ratings: A longitudinal investigation

This study examined the effects of felt accountability, political skill, and job tension on job performance ratings. Specifically, we hypothesized that felt accountability would lead to higher (lower) job performance ratings when coupled with high (low) levels of political skill, and that these relationships would be mediated by job tension. Data were gathered at multiple times over a one-year period (i.e., baseline performance, attitudinal variables one month later, supervisor reports of subordinate performance six months and one year after baseline performance was measured). Strong support was shown for the total effects model, whereby political skill moderated felt accountability—job performance ratings, felt accountability—job tension, and job tension—job performance ratings relationships. However, more focused analyses demonstrated that political skill most strongly moderated the job tension—job performance ratings linkage. Implications, strengths and limitations, and directions for future research are provided.     

Ratings of managerial skill requirements: comparison of age- and job-related factors

This article examines how individual characteristics (age, experience) and organizational characteristics (department, level) influence the skill requirements rated as being important for managerial jobs. One hundred ninety-seven managerial employees completed a survey composed of 20 skill dimensions pertinent to supervisory positions in the mining industry. Organizational level and departmental affiliation were correlated with job skill importance ratings. Ratings of skill importance were also correlated with the age of the person being rated, years of experience, and the age of the rater. As predicted, correlations with ratee age varied across different skill dimensions. This study has implications for fair employment practices to the degree that raters base evaluations of a job on the age of incumbents vs. job relevant characteristics.     

Do personality factors predict job satisfaction?

Two studies investigated the relationships between personality traits and aspects of job satisfaction. In Study 1, job applicants (n=250) completed the Eysenck Personality Profiler and the Work Values Questionnaire (WVQ), which requires respondents to rate various work-related facets according to the extent to which they contribute to their job satisfaction. These facets were combined into two composites (hygiene and motivator) based on previous research. The three personality superfactors accounted for a small percentage of the variance in importance ratings (about 5%). In Study 2, employees (n=82) completed a measure of the ‘Big Five’ personality traits and the Job Satisfaction Questionnaire (JSQ), which assesses both what respondents consider as important in their work environment as well as their satisfaction with their current job. Importance ratings were again combined into two composites while job satisfaction ratings were factor analyzed and three factors, differentiated along hygiene versus motivator lines, emerged. Personality traits again accounted for a small percentage of the total variance both in importance ratings and in levels of job satisfaction. It is concluded that personality does not have a strong or consistent influence either on what individuals perceive as important in their work environment or on their levels of job satisfaction.     

The Relationship Between SME Job Experience and Job Analysis Ratings: Findings with and without Statistical Control

This investigation examined the relationship between SME job experience and job analysis ratings. Managerial job incumbents (N = 209) provided job analysis ratings which measured 9 task frequency and 6 skill importance components. Job experience was operationalized as both job tenure and residualized job tenure (i.e., statistically-controlled for SME characteristics correlated with job tenure). Respondents were categorized into low, medium, and high groups for each tenure measure. MANOVA results indicated that task ratings differed across residualized tenure groups, while skill ratings did not. Statistical control of the job tenure measure made a substantial difference in results. For the non-residualized tenure groups, findings were reversed: skill ratings differed across non-residualized tenure groups, while task ratings did not. The results of this research are integrated with the existing literature and implications for future research and practice are offered.     

Optimism and Positivity Biases in Performance Appraisal Ratings: Empirical Evidence from Professional Soccer

Using unique assessment data for players from a German Bundesliga club’s youth academy, we tested four core hypotheses on how player ratings and rater or ratee-related characteristics reflect the (prospective) optimism bias and (retrospective) positivity bias. The results indicate not only that the ratings of predicted and remembered performance are indeed higher than the talents’ actual performance throughout a season, but that these differences depend positively on the rater’s organizational experience and negatively on the amount of ratee data available. They also suggest that (prospective) anticipation is even more positively biased than (retrospective) recollection of player performances, underscoring the asymmetry between looking forward and looking backward.     

True intercorrelation among job components, time delay in rating, and rater intelligence as determinants of accuracy in performance ratings

Ninety subjects viewed one of three sets of videotapes which presented simulated work performances by five ratees. Subjects viewed videotapes where the true intercorrelation among the three job components was either high, moderate, or low. Half of the subjects completed ratings immediately after viewing all five ratees and again, 24 h later. The remaining subjects completed ratings only once, 24 h after viewing the videotapes. Rater intelligence was assessed via the Wesman Personnel Classification Test. Performance ratings for each ratee on each of the three job components were completed using a magnitude estimation scale. Subjects' ratings were compared to ratee true scores (based on objective worker output) to obtain four components of rater accuracy: elevation, differential elevation, stereotype accuracy, and differential accuracy as well as a measure of overall accuracy. Results indicated that subjects' ratings were more accurate with respect to overall accuracy and differential accuracy when the true intercorrelation among job components was high rather than low. Rater intelligence was significantly correlated with stereotype accuracy. In addition, rater intelligence was significantly related to overall accuracy, differential elevation, and elevation, but only when the true intercorrelation among job components was moderate or high. Also, there was a curvilinear component to the relationship between intelligence and both differential accuracy and stereotype accuracy such that the most intelligent raters tended to be less accurate than more moderately intelligent raters who were, in turn, more accurate than the least intelligent raters. Finally, subjects' immediate ratings were not more accurate than ratings provided by subjects who completed only delayed ratings. However, repeated measures analyses found that for subjects who completed both immediate and delayed ratings, delayed ratings were less accurate than immediate ratings with respect to overall accuracy, differential accuracy, and differential elevation.     

A Concurrent Validity Study Applied to a Secretarial Position in a State Universities Civil Service System

In a concurrent validity study, a comprehensive job analysis of a mid-level secretarial position resulted in the development of highly valid employment selection instruments. Six hundred fifty-nine supervisors and 883 incumbents in 20 locations participated in the job analysis. Scores from the selection test correlated with composite ratings from a research performance appraisal (RPA) .41 (p<.001). Two forms of the test (A and B) were developed. Form A and Form B test scores and RPA composite ratings correlated .55 and .48 (p<.001) respectively. The unbiased estimate of equivalence reliability of Form A and Form B was .94. The two tests correlated .89 (p<.001).     

A Meta-Analytic Comparison of Managerial Ratings and Self-Evaluations

The purpose of this study was to examine the equivalence of self-reports and managerial ratings of salesperson job performance. A meta-analysis showed that the two measures exhibited low convergent validity (mean corrected correlation = 0.19), which indicated that they are not interchangeable. The predictive validity of self-reports and managerial ratings was compared. Managerial ratings had a corrected mean correlation of 0.44 with objective performance, whereas the corrected mean correlation between self-reports and objective measures was 0.34. Further meta-analytic investigations showed that the divergence between self-reports and managerial ratings was attributable to the performance effect. A discussion of the findings and avenues for further research are also provided.     

Decoding deficits and social competency: A continuous rating approach

Males who self-reported heterosocial difficulties and whose performance in a role-played interaction was judged to be of low social competence were compared to high-competent males on a continuous rating task. The rating task required that subjects make continual, ongoing social competency ratings of their own and six other males' performance in a role-play situation. The continuous measurement of performance provided a profile of ratings that was compared on frequency, latency-to-first ratings, profile elevation or level, scatter, and patterning. Low-competent males had longer latencies-to-first ratings than high-competent males and less scatter in their continuous ratings than did the high-competent group. The reduced scatter effect for continuous ratings replicates previous work done with global ratings and suggests that the low-competent group may not be able to discriminate among social stimuli as well as the high-competent group. The research also suggests that the continuous rating methodology may hold promise as a tool to investigate social perception processes.     

RELIABILITY IS NOT VALIDITY AND VALIDITY IS NOT RELIABILITY

Interrater correlations do provide an index of reliability of job performance ratings. We show that the arguments presented by Murphy and DeShon (2000) lead to the radical conclusion that traditional measurement models–both classical theory and generalizability theory models–can be used neither with job performance ratings nor with other measures used in I-O and other areas of psychology and the social sciences. We show that this untenable conclusion is based on confusion of validity issues and questions with reliability issues and questions. It is also based on the incorrect belief that classical measurement models are capable of addressing only random response measurement error and cannot address other forms of measurement error. We also show that the solution Murphy and DeShon offer to the problem of measurement error in ratings, as they define this problem, cannot work. Properly understood, the position taken by Murphy and DeShon leaves us with the nihilistic conclusion that no appropriate measurement models are possible in psychological research, thus making meaningful research impossible.     

Context Effects in Performance Evaluation: The Impact of Ratee Sex and Performance Level on Performance Ratings and Behavioral Recall

We extend the research on context effects in performance evaluation by examining the impact of ratee sex and context performance level as moderators of context effects in performance ratings and in the recall of performance information. Subjects (N= 269) rated the performance of an average performer (male or female) alone or following a low or high performing context (male or female). We found significant differences in the magnitude of contrast effects for an average target ratee as a function of both target ratee sex and performance level. These differences were found for both performance ratings and the evaluative content of performance information recalled.     

Cognitive and behavioral rating measures of executive function as predictors of academic outcomes in children

Interrelations of two measurement methods (cognitive versus behavioral ratings) for executive function (EF) were examined and related to reading comprehension and math calculations in fourth and fifth grade students (n = 93) in the context of a diverse urban student population. Relations among measures within four EF processes (working memory, planning, inhibition and shifting) were modest; relations to academics were stronger. EF measures contributed to both academic outcomes even in the context of relevant covariates (age, language and educational program). Working memory was particularly important for reading comprehension across measurement type. Cognitive measures from all EF processes, particularly inhibition and planning, and behavioral ratings of working memory were important for math.     

The Relation of Job Knowledge,Job Performance,and Supervisory Ratings as a Function of Ratee Race

Racial bias in performance ratings may be inferred when ratings hold differ- ent meanings for different racial subgroups. Operationally, this would be indi- cated by differences (by ratee race) in the correlation between performance ratings and objective indices of performance. In this study, the effects of rate race on the relations between supervisory ratings and more objective criteria of job knowledge and work performance were examined by aggregating corre- lations across 25 studies. The results indicated that supervisory ratings were more highly related to work-performance measures-and to a lesser extent to job-knowledge measures-for Black than for White ratees. Tivo theories were proposed that could account for such differences.     

Employees' Political Skill and Job Performance: An Inverted U‐Shaped Relation?

During the past decade, the construct of political skill has attracted a lot of attention. In particular, its relation to job performance has been examined. With regard to this link, it is typically proposed that political skill affects job performance in a positive linear manner. However, in this article it is suggested that intermediate levels of employees' political skill yield the highest job performance, implying that this association is in fact represented by an inverted U‐shape. Findings from two field studies (N₁ = 178, N₂ = 115 employee‐supervisor‐colleague triads) that incorporated different sample characteristics (early career employees, established employees), job performance dimensions (overall, task, contextual, and adaptive performance), and rating sources (supervisors and colleagues) supported this idea. Across different analyses, employees with intermediate levels of political skill received higher job performance ratings compared to those with lower and higher levels, respectively. In addition, the nature of the relationships between employees and their raters was found to moderate this curvilinear effect. Specifically, besides the fact that employees who had close working relationships with their raters generally received higher job performance ratings, the decreases in the job performance ratings of employees high in political skill were less pronounced when they had close relationships.     

Equivalence of mother and father ratings of ADHD in children

Obtaining data from multiple informants provides a more comprehensive diagnostic picture in the assessment of attention deficit hyperactivity disorder (ADHD). Differences in symptom ratings have been observed between parent- and teacher-report scales, though less information is available regarding differences between mothers and fathers. To address this gap, this study examines the rater agreement between mothers and fathers on the Diagnostic and Statistical Manual of Mental Disorders – Fourth Edition (DSM-IV) ADHD Symptom Rating Scale (DSM-ADHD-SRS). The participants consisted of 337 children diagnosed with ADHD who underwent comprehensive neuropsychological assessment. Confirmatory factor analysis indicates that a three-factor model comprising inattention, hyperactivity, and impulsivity symptoms provides the best fit for both mothers’ and fathers’ ratings. Mothers provided higher mean ratings for the inattention scale. These results suggest that the factor structure for the DSM-ADHD-SRS is the same, regardless of parent gender. However, symptoms of inattention may vary depending upon which parent completes the ratings. This discrepancy could lead to differences in diagnostic impressions in clinical evaluations.