Identifying Competencies with Behavioral-Event Interviews

Coding competencies from behavioral-event interviews according to the principles of a new approach to assessment (McClelland, 1973) produces assessments that are reliable and validly associated with success as an executive. These assessments are not influenced by length of protocol or by performance in the preceding year. Bias is not a problem if both the interviewer and the coder are blind to executive success. In contrast to a traditional psychometric approach based on regression analysis, an algorithm based on competency scores predicted managerial success and improved performance across a number of samples. This algorithm identified potential outstanding performers as individuals whose scores reached designated tipping points within clusters of substitutable competencies. Experts' judgments of competencies needed or shown by executives in various positions agreed only moderately with competencies shown to be important by the data from behavioral-event interviews.     

EXPLORING BLACK-WHITE SUBGROUP DIFFERENCES OF MANAGERIAL COMPETENCIES

This study investigates whether different job-relevant competencies vary in terms of Black-White subgroup differences exhibited. There were 633 participants (545 Whites, 88 Blacks) who completed a managerial assessment center that evaluated 13 competency dimensions across 8 assessment exercises. Participants also completed a cognitive ability test. The results suggest that subgroup differences vary by the content domain of the competency. As predicted, significant subgroup differences emerged for a majority of the more cognitively loaded competencies (e.g., judgment) while nonsignificant differences were associated with a majority of the less cognitively loaded competencies (e.g., human relations). Furthermore, when cognitive ability was controlled, 12 of 13 competency scores demonstrated incremental validity in predicting supervisory job performance ratings. In addition, competencies with greater cognitive load tended to more strongly predict cognitive aspects of job performance as compared to noncognitive aspects. However, competencies with less cognitive load did not differentially predict cognitive and noncognitive aspects of job performance.     

Multisource Assessments of Behavioral Competencies and Selection Interview Performance

This study sought to determine whether and how ratings of leadership in a developmental multisource feedback program compare against those obtained by leadership selection decision makers. Seventy‐seven senior managers received feedback on eight competencies that also formed the basis of a subsequent promotional interview for senior executive positions. Each rating source displayed a unique perspective on the participants, but only the supervisor and peer ratings predicted performance on the interview. As well, peer, subordinate, and self‐ratings failed to improve upon the supervisors' predictions. The results also suggested a general performance factor underlying both the multisource assessments and the interview ratings.     

Personality Correlates of Assessment Center Consensus Competency Ratings: Evidence from Russia

Controversy has revolved around whether assessment center ratings have construct validity to measure intended dimensions of managerial performance. In contrast to much recent research on the internal structure of assessment center ratings, the present studies investigated the relationship of final competency ratings derived by consensus discussion with external questionnaire measures of personality characteristics. Expanding on previous studies showing correlations of dimension scores in relation to individual trait measures, this study investigated the relationship of complex competencies with both single personality traits and with composites of personality traits. Evidence from two samples of managers in Russia shows that final competency ratings are related to predicted composites of personality factors more consistently than to single factors. Taken together, these findings provide evidence that assessment center ratings derived by consensus discussion show construct validity in relationship with predicted composites of personality characteristics.     

Self-serving bias effects on job analysis ratings

The purpose of this study was to investigate whether worker-oriented job analysis importance ratings were influenced by subject matter experts' (SME) standing (as measured by self-rated performance) on a competency. This type of relationship (whereby SMEs indicate that the traits they have are important for successful job performance) is an example of the self-serving bias (which is widely described in the social cognition literature and rarely described in the industrial/organizational psychology literature). An archival dataset covering 57 clerical and technical occupations with 26,682 participants was used. Support was found for the relationship between self-rated performance and importance ratings. Significant relationships (typically in the .30s) were observed for all 31 competencies that were studied. Controls were taken to account for common method bias and differences in the competencies required for each of the 57 occupations. Past research has demonstrated the effects of the self-serving bias on personality-based job analysis ratings. This study was the first to extend these findings to traditional job analysis, which covers other competencies in addition to personality. In addition, this study is the first to use operational field data instead of laboratory data.     

Teachers' Ratings of Working Memory in English Language Learners: Do Laboratory Measures Predict Classroom Analogues?

This study investigated the relationship between teachers' classroom ratings of working memory (WM) and laboratory measures of WM of English language learners (ELLs) in the elementary grades. Multilevel modeling was used to identify whether teacher ratings accurately predicted ELL children's performance on WM tasks. The results indicated that teachers' ratings of WM were predicted by isolated components of WM even when measures of achievement, vocabulary and gender representation were entered into the analysis. Teacher ratings of WM were predictive of student performance on latent measures of Spanish short‐term memory and the executive component of WM. Entering inattention ratings into the analysis partialed out the influence of the executive component of WM, while leaving variance related to a language‐specific storage system as a significant predictor of teachers' classroom ratings of WM. The results suggest that classroom ratings of WM provide a valid analogue of student laboratory performance. Copyright © 2016 John Wiley & Sons, Ltd.     

AN EXAMINATION OF IMPRESSION MANAGEMENT USE AND EFFECTIVENESS ACROSS ASSESSMENT CENTER EXERCISES: THE ROLE OF COMPETENCY DEMANDS

We report 2 studies that examine how promotional candidates use verbal and nonverbal impression management (IM) tactics across several structured assessment center exercises that differ in the competency demands they place on candidates. Based on the competency-demand hypothesis ( Shoda, Mischel, & Wright, 1993a, 1993b ), it was predicted that IM use would occur most frequently and have the strongest effects on assessor evaluations in exercises that place greater demands on candidates' interpersonal skills than in exercises that depend primarily on technical skills. In both studies, IM tactics were generally used more frequently and there was more variability in IM use for those exercises requiring candidates to display interpersonal competencies (i.e., the role-plays and mock presentation) relative to the exercise that did not (i.e., the tactical exercise). The relationship between IM use and assessor evaluations was also influenced by the competencies assessed by the exercises, and IM use related to both interpersonal and noninterpersonal ratings of performance.     

A longitudinal study of the moderating role of extraversion: leader-member exchange, performance, and turnover during new executive development

Identifying factors that help or hinder new executives in "getting up to speed" quickly and remaining with an organization is vital to maximizing the effectiveness of executive development. The current study extends past research by examining extraversion as a moderator of relationships between leader-member exchange (LMX) and performance, turnover intentions, and actual turnover for an executive sample. The sample consisted of 116 new executives who were surveyed prior to starting their employment and at 3 months postentry. A total of 67 senior executives rated these new executives in terms of overall performance at 6 months postentry. Turnover data were gathered from company records 3 1/2 years later. Hierarchical regression results showed that LMX was not related to performance or turnover intentions for those high in extraversion; but for individuals low in extraversion, there was a relation between LMX, performance, and turnover intentions. Furthermore, survival analyses showed that LMX was only related to turnover-hazard rate for individuals low in extraversion.     

工作绩效的结构及其与胜任特征的关系

本研究旨在验证党政领导干部对下级工作绩效评定的结构,并探讨其与胜任特征之间的关系。两个省市的450名党政领导干部完成了对自己下级的工作绩效和胜任特征的评定。结果表明,评定下级干部工作绩效的两因素结构,即任务指向和个人素质,得到了验证;分别以工作绩效的两个二阶因素为因变量,以胜任特征的七个维度为预测变量进行逐步回归分析,胜任特征可以解释工作绩效的“才”和“德”成分54．5％和55．3％的变异。胜任特征和工作绩效中“德”和“才”的成分之间存在复杂的关系,既可以相互促进,也可以交互抑制。文中还对这一结果的含义及本土化研究的意义进行了深入讨论。     

THE RELATIONSHIP BETWEEN SUPERVISORY- AND SELF-APPRAISALS OF EXECUTIVE PERFORMANCE

There has been increased use of self-appraisal programs for evaluating the performance of executive personnel. Because objective measures of performance are rarely available or adequate for this level of personnel, ratings are often relied upon. The purpose of this study was to investigate the relationship between supervisory-perceptions and incumbent self-perceptions of the performance of executive personnel. The rationale for this study was that the relationship between supervisory-ratings and self-ratings has important implications for the usefulness of self-appraisal programs. If self-evaluations are to be effective in enlisting a person's cooperation and participation, it seems essential that his perceptions agree with those of his superior. This study was designed to identify those areas of executive performance where there might be significant amounts of disagreement. Knowledge of these problem areas would then be useful in conducting self-appraisal programs. Supervisory-ratings and self-ratings on 27 performance variables were obtained for the top executives of a large manufacturing corporation. A measure of promotability was used as the criterion and analysis of variance and correlation analyses were carried out to attain the objectives. The major conclusions of the study were: (1) Executives tended to rate themselves higher than they were rated by their supervisors, (2) Disagreement in the relationship of the two sets of ratings was found in several areas, (3) Executives who tended to overrate themselves were found to be the ones who were considered least promotable on the basis of a criterion measure of success in the organization.     

Clients' ratings of counselor multicultural competency

Differences in perceptions between Euro American and ethnic minority respondents were examined to compare the role of counselor multicultural competency in multicultural versus traditional counseling. Results showed a strong positive correlation between clients' ratings of counselors' multicultural competencies and clients' ratings of counselors' general competency and empathy. However, when comparisons were made between Euro American and ethnic minority clients' on satisfaction, counselor multicultural competency explained a large and significant amount of variance for the ethnic minority sample only, above and beyond counselor general competency and empathy. Results are discussed in the context of relevant literature and suggestions for future research.     

Competencies for infant mental health therapists: A survey of expert opinion

We report on data provided by a nationwide panel of 23 infant mental health (IMH) experts who provided numerical ratings of the relative importance of 143 competencies desirable for licensed mental health therapists working with infants/children birth to 5 years of age and their families/caregivers. The competencies were developed based on prior state and national efforts and our own experience in training IMH therapists. The competencies were grouped conceptually into seven areas: (a) Normal infant and toddler development; (b) Atypical development (perturbations in development); (c) Emotional/behavioral disorders in infants and young children; (d) Assessment; (e) Intervention; (f) Community resources and referrals; and (g) Organization, communication, and collaboration. We calculated means, SDs, 95% confidence intervals to rank order each competency. We conducted reliability analyses and tested for mean differences in ratings for the seven areas. Interrater and intraclass correlations were modest, likely the result of restriction of range in the ratings. The seven areas showed high levels of internal consistency and, with few exceptions, did not significantly differ in the means of their ratings. The utility of the competencies are discussed as a framework for training and professional development for IMH clinicians.     

The Great Eight competencies: a criterion-centric approach to validation

The author presents results of a meta-analysis of 29 validation studies (N=4,861) that uses the Great Eight competency factors (Kurz & Bartram, 2002) as the criterion measurement framework. Predictors of the Great Eight competencies based only on personality scales show moderate to good correlations with line-manager ratings for all 8 of the competencies. On their own, ability tests correlate with 4 of the 8 competencies, and together ability and personality data yield operational validities ranging from 0.20 to 0.44 for the 8 competencies. Operational validities for aggregated predictors with aggregated criteria were estimated to be 0.53. The value of differentiating the criterion space and of relating predictor variables to criterion variables in a one-to-one fashion is discussed.     

Practicum and On-the-job Ratings of School Counselors

Practicum ratings of 50 school counselors were examined in relation to administrators' subsequent on-the-job ratings of their performance. The relationship proved negligible. The practicum ratings did relate, however, to retention of cognitive material in guidance and counseling (scores on a comprehensive examination). Such a relationship did not exist between scores on the same comprehensive examination and the administrators' ratings. It was suggested that cognitive standards for counselors are probably best established and maintained by training institutions. Administrators seeking assurance of at least minimum cognitive competency in counselors ought to seek endorsement of candidates from those who observe their practicum performance.     

Multicultural competency in research: examining the relationships among multicultural competencies, research training and self-efficacy, and the multicultural environment

Research on multicultural competencies has mainly focused on the practice dimension of psychology training and practice. Little theoretical or empirical research has examined multicultural research training and self-efficacy. In this study, 119 psychology graduate students filled out a Web survey focusing on the research training environment, research self-efficacy, multicultural competency, the multicultural environment, and social desirability. Results showed that multicultural competency, research training, and the multicultural environment were related to multicultural research self-efficacy. Hierarchical regressions showed that multicultural competency predicted students' research anxiety; social desirability predicted multicultural research utility, multicultural competency, and the research training environment; and multicultural competency predicted students' confidence in research and perceptions that their graduate training programs were multicultural. Implications are discussed.     

The Impact of Competency Statements on Résumés for Short-listing Decisions

Sixty-two managers and human resource consultants rated a series of genuine résumés with covering letters. The résumés were manipulated to contain varying amounts of information about the candidate’s knowledge, skills and abilities (competency statements). This information appeared at different locations in the résumé and covering letter. In addition, half the managers were provided with extra job requirement details beyond the job advertisement. Managers rated the candidate résumés for candidate suitability, decision to interview and overall ranking. The inclusion of competency statements resulted in higher manager ratings. However, the location of the competency statements did not influence ratings given to résumés. Further, the extra information provided to managers did not influence their ratings. The results replicate and extend an earlier study by Earl, Bright and Adams (1998) and challenge the idea that selection decisions are largely based on the notion of applicant fit. The results suggest that the inclusion on the résumé of statements that address job competencies even in a general fashion will boost an applicant’s chances of being short-listed.     

Predictive Criterion-Related Validity of Observer Ratings of Personality and Job-Related Competencies Using Multiple Raters and Multiple Performance Criteria

The purpose of this study was to assess the predictive validity of observer ratings of personality and job-related competencies in a selection setting. Based on ratings from multiple raters of both the predictors and the criteria in a sample of MBA students, results indicated that observer ratings of Conscientiousness, Emotional Stability, leadership, and interpersonal skills predicted work performance, team performance, and academic performance. For work performance and team performance, a composite of the four predictors had incremental predictive validity over general mental ability, even after controlling for how well the rater knew the ratee.     

Book Review

Performance-based measures and ratings of executive functions were examined in a sample of adolescents with attention deficit/hyperactivity disorder (ADHD) and comparison controls. Performance-based measures of executive function included inhibition, working memory, set shifting, and planning, and ratings of these same executive functions were completed by parents and teachers. Adolescents with ADHD demonstrated lower executive function performance than controls and displayed elevated ratings on the executive function ratings by parents and teachers. Significant associations were obtained between the performance-based measures and the parent and teacher ratings, but each measure was not uniquely associated with its respective scale on the rating scales. When performance-based measures and ratings were examined as predictors of ADHD status, the parent and teacher ratings entered as significant predictors of ADHD status. Further commonality analyses indicated that performance-based measures accounted for little unique variance in predicting ADHD status and also displayed little overlap with the behavioral ratings. These findings highlight the diagnostic utility of behavioral ratings of executive function in predicting ADHD status; however, behavioral ratings should not be assumed to be a proxy for performance on measures of executive function in clinical practice.     

Socioanalytic theory and work behavior: Roles of work values and political skill in job performance and promotability assessment

Socioanalytic theory postulates that job performance ratings are predicted by basic social motives moderated by social competency. The two motives are the motive to get along with others and the motive to achieve status and power. The present two-study investigation assessed these motives as work values and collected supervisors' job performance and promotability assessments. Social competency was assessed as political skill at work. The results provided strong and consistent support for the hypotheses, thus providing a more direct test of socioanalytic theory and extending it to demonstrate effects beyond overall job performance ratings on contextual performance and promotability assessments. Contributions and implications of these results, strengths and limitations, directions for future research, and practical implications are discussed.