The Impact of Contextual Self‐Ratings and Observer Ratings of Personality on the Personality–Performance Relationship1

This study examined 2 possible ways of increasing the predictive validity of personality measures: using observer (i.e., supervisor and coworker) ratings and work‐specific self‐ratings of Big Five personality factors. Results indicated that among general self‐ratings of Big Five personality dimensions, Conscientiousness was the best predictor of in‐role performance, and Agreeableness and Emotional Stability were the best predictors of organizational citizenship behavior (OCB). Observer ratings of personality accounted for incremental variance in job performance (in‐role performance and OCB) beyond that accounted for by general self‐ratings. However, contrary to our expectations, work‐specific (i.e., contextual) self‐ratings of personality generally did not account for incremental variance in job performance beyond that accounted for by general self‐ratings.     

Validity of observer ratings of the five-factor model of personality traits: a meta-analysis

Conclusions reached in previous research about the magnitude and nature of personality-performance linkages have been based almost exclusively on self-report measures of personality. The purpose of this study is to address this void in the literature by conducting a meta-analysis of the relationship between observer ratings of the five-factor model (FFM) personality traits and overall job performance. Our results show that the operational validities of FFM traits based on observer ratings are higher than those based on self-report ratings. In addition, the results show that when based on observer ratings, all FFM traits are significant predictors of overall performance. Further, observer ratings of FFM traits show meaningful incremental validity over self-reports of corresponding FFM traits in predicting overall performance, but the reverse is not true. We conclude that the validity of FFM traits in predicting overall performance is higher than previously believed, and our results underscore the importance of disentangling the validity of personality traits from the method of measurement of the traits.     

The construct and criterion validity of emotional intelligence and its potential utility for management studies

In this study, the authors reviewed the definition of emotional intelligence (EI) and argued that El is conceptually distinct from personality. In Study 1, the authors showed that EI was related to yet distinct from personality dimensions and that it had incremental predictive power on life satisfaction. The authors examined the construct validity of self-reports and others' ratings of EI using two samples in Study 2. In a student sample, parents' ratings explained additional variance in the students' life satisfaction and feelings of powerlessness after controlling for the Big Five personality dimensions. In the work sample, peer ratings were found to be significant predictors of job performance ratings provided by supervisors after controlling for the Big Five personality dimensions. Other implications for future research on EI are discussed.     

Integrity Testing and Job Performance Within High Performance Work Teams: A Short Note

Integrity tests have exhibited robust validity for predicting performance among traditional jobs. However, some organizations have shifted to self-managed work teams and the relationship between integrity and performance within these jobs is untested. One hundred fourteen high performance team members within a single department completed the Hogan Personality Inventory (HPI). The HPI Reliability scale was used as the integrity measure. The subjects' job performance was evaluated by their team leaders. Performance ratings were transformed to equalize ratings across the eight teams. Results indicate that integrity was related to transformed performance ratings (r = .25) within a high performance team environment. Limitations and implications are discussed.     

Examining the reliability and predictive validity of performance assessments by soccer coaches and scouts: The influence of structured collection and mechanical combination of information

Soccer coaches and scouts typically assess in-game soccer performance to predict players’ future performance. However, there is hardly any research on the reliability and predictive validity of coaches’ and scouts’ performance assessments, or on strategies they can use to optimize their predictions. In the current study, we examined whether robust principles from psychological research on selection – namely structured information collection and mechanical combination of predictor information through a decision-rule – improve soccer coaches’ and scouts’ performance assessments. A total of n = 96 soccer coaches and scouts participated in an elaborate within-subjects experiment. Participants watched soccer players’ performance on video, rated their performance in both a structured and unstructured manner, and combined their ratings in a holistic and mechanical way. We examined the inter-rater reliability of the ratings and assessed the predictive validity by relating the ratings to players’ future market values. Contrary to our expectations, we did not find that ratings based on structured assessment paired with mechanical combination of the ratings showed higher inter-rater reliability and predictive validity. In contrast, unstructured-holistic ratings yielded the highest reliability and predictive validity, although differences were marginal. Overall, reliability was poor and predictive validities small-to-moderate, regardless of the approach used to rate players’ performance. The findings provide insights into the difficulty of predicting future performance in soccer.     

Unanswered questions in assessment center research

This study compared the predictive validity of two types of overall assessment center ratings (clinical and mechanical) and two types of performance ratings (supervisor and subordinate). Data were gathered on approximately 50 law enforcement agency managers over a four-year time span. Results indicated that clinical predictions were not significantly better than mechanical ones and that prior performance ratings were better predictors of future performance ratings than were assessment centers. Further, dimension ratings within the assessment center were found to be unrelated to ratings of the same dimensions on the job four years later, supporting the notion of subtle criterion contamination (Klimoski & Strickland, 1977) and raising questions about the central assumptions underlying the assessment center approach.     

Are observer ratings of applicants’ personality also faked? Yes,but less than self‐reports

Although self‐report personality tests are a comparatively cheap and easy‐to‐administer personnel selection tool, researchers have criticized them for not predicting enough criterion‐related variance. Researchers have suggested using observer‐ratings of personality (e.g., as part of a reference check from a supervisor) because observer‐ratings have been reported to be more predictive. However, it is theoretically and empirically unclear whether supervisors also engage in faking (the intentional distortion of responses). Study 1 explored faking among managers who were first asked to imagine that a subordinate had to leave his/her job for private reasons and then to rate the personality of the subordinate. A week later, managers rated their subordinates honestly. A repeated‐measures MANOVA indicated that managers did fake. Study 2 (among supervisors of working students) replicated the above findings but also showed that there is less faking in supervisor‐ratings than in self‐ratings. Furthermore, we found no evidence that the validity of personality scales for predicting academic performance depends on self‐ versus observer‐ratings or on an applicant versus an honest condition. These two studies thus show that practitioners should not equate personality ratings obtained from observers in a selection context with honest personality ratings.     

Prediction of job satisfactoriness for workers with severe handicaps from aptitudes,personality, and training ratings

The predictive validity of General Aptitude Test Battery and Sixteen Personality Factor Questionnaire scores were compared to standard training ratings made by vocational instructors against the criterion of work performance measured by the Minnesota Satisfactoriness Scales for a sample of 106 employees with severe handicaps. The psychometric test variables were not correlated with the criterion; however, the training ratings were consistently predictive of the job satisfactoriness scores. These results suggest that the employment potential of job applicants with disabilities can be assessed more accurately using situational training ratings, as opposed to standardized psychometric test scores.     

The team role test: development and validation of a team role knowledge situational judgment test

The main objectives in this research were to introduce the concept of team role knowledge and to investigate its potential usefulness for team member selection. In Study 1, the authors developed a situational judgment test, called the Team Role Test, to measure knowledge of 10 roles relevant to the team context. The criterion-related validity of this measure was examined in 2 additional studies. In a sample of academic project teams (N = 93), team role knowledge predicted team member role performance (r = .34). Role knowledge also provided incremental validity beyond mental ability and the Big Five personality factors in the prediction of role performance. The results of Study 2 revealed that the predictive validity of role knowledge generalizes to team members in a work setting (N = 82, r = .30). The implications of the results for selection in team environments are discussed.     

Team effectiveness: beyond skills and cognitive ability.

On the basis of job analysis results, the validity of using measures of general cognitive ability, job-specific skills, and personality traits jointly at both the individual level and the group level to predict the performance of 79 four-person, human resource work teams was evaluated. Team member trait and job skill scores were aggregated with a conjunctive model of task performance. At the individual level of analysis, measures of personality (i.e., Agreeableness and Conscientiousness) predicted peer ratings of team member performance beyond measures of job-specific skills and general cognitive ability. Similarly, at the group level of analysis, both Agreeableness and Conscientiousness predicted supervisor ratings of work team performance, objective measures of work team accuracy, and work completed. At both the individual and group levels, the trait of Agreeableness predicted Interpersonal Skills.     

Personality and the prediction of team performance

Although much is known about personality and individuals' job performance, only a few studies have considered the effects of team‐level personality on team performance. Existing research examining the effects of personality on team performance has found that, of the Big Five factors of personality, Conscientiousness is often the most important predictor. Accordingly, we investigated the criterion validity of lower‐level Conscientiousness traits to determine whether any one trait is particularly predictive of team performance. In addition to Conscientiousness, we examined the criterion validity of the other Big Five personality factors. We found that Conscientiousness and its facets predicted team performance. Agreeableness, Extraversion and Neuroticism were not predictive of team performance, whereas Openness had a modest negative relation with team performance. Copyright © 2010 John Wiley & Sons, Ltd.     

Cross-national applicability of a game-based cognitive assessment

New technology has had a discernable impact on how organizations recruit and select potential employees. Game-based assessment has emerged as a potential technology that can be used to enhance the assessment of individual differences and applicants' views of the selection process. However, studies investigating the psychometric properties and predictive validity of game-based assessments are still lacking. This study investigated the structural equivalence of a game-based assessment of cognitive ability across 228 Australians and 239 South Africans. A smaller sample of 115 South Africans also received work performance ratings to investigate the predictive validity of the cognitive assessment. Results of factor analysis supported a strong general factor of cognitive ability across the entire sample but only partial metric and scalar invariance across the two nations. The general factor of the game-based assessment further revealed promising results in terms of its predictive validity for five broad dimensions of individual work performance.     

Predicting Training and Job Performance for Transit Operators

Although considerable meta‐analytic research has validated the use of cognitive ability tests, structured interviews, and personality tests with training and job performance criteria, few studies have investigated the validity of these measures with transit operators. There are the only two single studies of concurrent validation research specifically with transit operators. This article presents the results of a predictive validation study conducted with transit operator applicants for a large urban transit authority in Canada. Key knowledge, skills, and abilities were determined for the role and used as a basis for the design and choice of predictors and criteria. Four predictors were used in the study: education, cognitive ability, personality assessment, and structured interview. Criteria included training performance (formative and summative), probationary performance, preventable accidents, and lost time injuries. Validation results supported cognitive ability, structured interview, and several personality factors as predictors of training performance, but less so for job performance. The use of formative training ratings greatly augmented the evidence supporting the predictors beyond typical organizational criteria.     

Designing Selection Systems for Medicine: The importance of balancing predictive and political validity in high‐stakes selection contexts

Selection into medical education and training is a high‐stakes process. A key unanswered issue is the effectiveness of measuring noncognitive predictors via both low‐fidelity and high‐fidelity selection approaches in this high‐stakes context. We review studies investigating the effectiveness of multiple selection instruments in terms of predictive validity, incremental validity, and applicant reactions in both entry‐level and advanced‐level medical selection. Our results show that the situational judgment test (SJT) is the best single predictor of performance, operationalized in multiple ways. In addition, the low‐fidelity SJT has incremental predictive power over cognitively oriented tests, and high‐fidelity assessment center (AC) exercises add incremental validity over the low‐fidelity (and less costly) selection methods. Concerning applicant reactions, results show that overall, the selection system is positively received. However, the method with the highest predictive validity – the SJT – received comparatively lower face validity ratings which may present a ‘justice dilemma’ for employers. Furthermore, various other stakeholders have a political interest in the selection methods used (e.g., government, the regulators and trade unions).     

Police Officer Selection validation project: The multijurisdictional Police Officer Examination

This study presents the results of a predictive validity study of the Multijurisdictional Police Officer Examination (MPOE). The MPOE is a paper-and-pencil test of cognitive abilities developed to select entry level police officers. Criterion measures for the study included training performance, supervisory performance ratings and a file composite measure of negative work related incidents (e.g., use of force). There were 159 (120 majority and 39 minority group) members in the final validation sample. Results indicated that the MPOE was predictive of the three performance indices. Cumulation of results across seven studies (including the present effort) revealed a weighted average correlation between the MPOE and performance ratings of .20 (N=913). The potential generalizability of the results to other police departments is discussed.     

A Meta-Analytic Comparison of Managerial Ratings and Self-Evaluations

The purpose of this study was to examine the equivalence of self-reports and managerial ratings of salesperson job performance. A meta-analysis showed that the two measures exhibited low convergent validity (mean corrected correlation = 0.19), which indicated that they are not interchangeable. The predictive validity of self-reports and managerial ratings was compared. Managerial ratings had a corrected mean correlation of 0.44 with objective performance, whereas the corrected mean correlation between self-reports and objective measures was 0.34. Further meta-analytic investigations showed that the divergence between self-reports and managerial ratings was attributable to the performance effect. A discussion of the findings and avenues for further research are also provided.     

Job performance ratings: The relative importance of mental ability,conscientiousness, and career adaptability

According to career construction theory, continuous adaptation to the work environment is crucial to achieve work and career success. In this study, we examined the relative importance of career adaptability for job performance ratings using an experimental policy-capturing design. Employees (N = 135) from different vocational backgrounds rated the overall job performance of fictitious employees in 40 scenarios based on information about their career adaptability, mental ability, conscientiousness, and job complexity. We used multilevel modeling to investigate the relative importance of each factor. Consistent with expectations, career adaptability positively predicted job performance ratings, and this effect was relatively smaller than the effects of conscientiousness and mental ability. Job complexity did not moderate the effect of career adaptability on job performance ratings, suggesting that career adaptability predicts job performance ratings in high-, medium-, and low-complexity jobs. Consistent with previous research, the effect of mental ability on job performance ratings was stronger in high- compared to low-complexity jobs. Overall, our findings provide initial evidence for the predictive validity of employees' career adaptability with regard to other people's ratings of job performance.     

An other perspective on personality: meta-analytic integration of observers' accuracy and predictive validity

The bulk of personality research has been built from self-report measures of personality. However, collecting personality ratings from other-raters, such as family, friends, and even strangers, is a dramatically underutilized method that allows better explanation and prediction of personality's role in many domains of psychology. Drawing hypotheses from D. C. Funder's (1995) realistic accuracy model about trait and information moderators of accuracy, we offer 3 meta-analyses to help researchers and applied psychologists understand and interpret both consistencies and unique insights afforded by other-ratings of personality. These meta-analyses integrate findings based on 44,178 target individuals rated across 263 independent samples. Each meta-analysis assessed the accuracy of observer ratings, as indexed by interrater consensus/reliability (Study 1), self-other correlations (Study 2), and predictions of behavior (Study 3). The results show that although increased frequency of interacting with targets does improve accuracy in rating personality, informants' interpersonal intimacy with the target is necessary for substantial increases in other-rating accuracy. Interpersonal intimacy improved accuracy especially for traits low in visibility (e.g., Emotional Stability) but only minimally for traits high in evaluativeness (e.g., Agreeableness). In addition, observer ratings were strong predictors of behaviors. When the criterion was academic achievement or job performance, other-ratings yielded predictive validities substantially greater than and incremental to self-ratings. These findings indicate that extraordinary value can gained by using other-reports to measure personality, and these findings provide guidelines toward enriching personality theory. Various subfields of psychology in which personality variables are systematically assessed and utilized in research and practice can benefit tremendously from use of others' ratings to measure personality variables.     

A team format for the Global Assessment Scale: reliability and validity on an inpatient unit

The Global Assessment Scale (GAS) is a 100-point rating instrument which purports to measure psychological adjustment on a continuum from self-actualization to severe regression. Its reliability and validity were examined on the admitting ward of a state hospital where GAS ratings are assigned to patients by computing a mean of the individual ratings assigned by a team of clinicians. Results showed such team GAS ratings to have good reliability. As for validity, team GAS predicted the outcomes of court hearings at two stages of the civil commitment process and showed construct validity in its relevant correlations with the Psychotic Inpatient Profile. A predicted association between team GAS and a ward atmosphere measure was not obtained. Taken as a whole, the results support the use of team GAS ratings in inpatient settings.