EFFECTS OF ASSESSORSHIP ON MANAGERs' PROFICIENCY IN ACQUIRING, EVALUATING, AND COMMUNICATING INFORMATION ABOUT PEOPLE

This study investigated the developmental effects of assessorship, i.e., serving as a trained member of an assessment center staff, on managers' proficiency in three categories of behavior: acquiring, evaluating, and communicating information about people. Eighty managers selected to serve as assessors in a management assessment center were assigned to one of two experimental groups. Immediately prior to assessor training, one group participated in three exercises designed to measure proficiency in the criterion behaviors. The other group participated in these exercises after they had accumulated at least 3 months of full-time experience as assessors. The results indicate that the experienced assessors were more proficient in interviewing another individual to obtain relevant information about job candidates, verbally presenting and defending information about others' managerial qualifications, and communicating this information in concise written reports. Also, the experienced assessors' ratings of videotaped ratees' managerial ability possessed somewhat superior psychometric properties.     

The predictive validity of individual psychological assessments in selecting UK public sector senior managers

Individual Psychological Assessment (IPA) is a very widely offered service in Organizational Psychology. It generally consists of a psychologist or HR practitioner using a combination of interview and psychometrics to arrive at a detailed assessment of an individual's capabilities in relation to a job they are being considered for. Although much used, this practice has limited supporting evidence of its validity—not least because of the methodological difficulties in conducting research on this subject—and has been criticized accordingly. The current study examines the use of IPAs with 115 middle and senior management level candidates in a civil service context. All candidates completed a set of psychometric measures and had an in‐depth interview with a psychologist as part of a standardized process. The ratings made by the assessors were correlated with a criterion measure of potential for promotion derived from multisource feedback ratings obtained on these candidates some months later. Analysis of the results indicated that three of the four attributes rated by assessors correlated significantly with the criterion measure. Further, assessors’ ratings were found to show incremental validity over that provided by psychometric test scores alone. These findings are discussed in terms of the use of IPAs in senior level assessment.     

A SURVEY OF ASSESSMENT CENTER PRACTICES IN ORGANIZATIONS IN THE UNITED STATES

Two hundred fifteen organizations in the United States provided information about multiple aspects of their assessment centers, including design, usage, and their adherence to professional guidelines and research-based suggestions for the use of this method. Results reveal that centers are usually conducted for selection, promotion, and development purposes. Supervisor recommendation plays a sizable role in choosing center participants. Most often, line managers act as assessors; they typically arrive at participant ratings through a consensus process. In general, respondents indicate close adherence to recommendations for center design and assessor training. Recommendations involving other practices (e.g., informing participants, evaluating assessors, validating center results) are frequently not followed. Furthermore, methods thought to improve predictive validity of center ratings are underutilized. Variability in center practices according to industry and center purpose was revealed. We encourage practitioners to follow recommendations for center usage, and researchers to work to better understand moderators of center validity.     

Do Assessors Have Too Much on their Plates? The Effects of Simultaneously Rating Multiple Assessment Center Candidates on Rating Quality

It has been suggested that the large cognitive demands during the observation of assessment center (AC) participants can impair the quality of the assessors' ratings. An aspect that is especially relevant in this regard is the number of candidates that assessors have to observe simultaneously during group discussions, which are one of the most commonly used AC exercises. The present research evaluated potential impairments of the quality of the assessors' ratings (construct‐ and criterion‐related validity and rating accuracy) related to the number of to‐be‐observed candidates. Study 1 (N=1046) was a quasi‐experimental field study and Study 2 (N=71) was an experimental laboratory study. Both studies found significant impairments of assessors' rating quality when a larger in comparison to a lower number of candidates had to be observed simultaneously. These results suggest that assessors should not have to observe too many candidates at the same time during AC group discussions.     

Analyzing Judgments of Ethnically Diverse Applicants During Personnel Selection: A study at the Dutch police

A judgment‐analysis study was used to investigate assessors' judgment processes, evaluating ethnic minority vs ethnic majority applicants. Sixteen ethnic majority assessors judged 5089 applicants during the Dutch police officer selection procedure, with each assessor judging 30 ethnic minority applicants minimally. Information from an employment interview, an assessment center, and a Big Five personality test were combined into a final selection advice. Results showed that as much as or more information sources were used to judge ethnic minority than ethnic majority applicants. Furthermore, a larger number of irrelevant cues were used for the judgment of ethnic minority applicants. Finally, when judging ethnic minority applicants, assessors based their decision to a lesser extent on their own ratings than on ratings of others.     

Assessor Constructs in Use as the Missing Component in Validation of Assessment Center Dimensions: A critique and directions for research

The basis for assessment center dimension ratings has been examined through the lens of various multitrait–multimethod approaches, with some researchers concluding that dimension ratings are not representative of meaningful constructs. This presents a serious challenge for those who would generalize predictor constructs, particularly those that are broadly articulated in management assessment center dimensions. This paper addresses the problems with applying such analyses to assessment center dimension ratings without first articulating expected construct relationships. Ackerman and Humphreys' analysis of constructs will be used to better frame assessment center dimensions and to articulate possible assessor constructs in use. Theories of performance held by the managers, who serve both as subject matter experts when dimensional constructs are initially defined, and as the assessors who impose these dimensions on assessees' behavior in assessment center ratings, are suggested as important bases for further construct validation. Approaches for generalization of assessor constructs in use to the managerial work domain will be discussed.     

The Measurement of Assessment Center Situations: Bem's Template Matching Technique for Examining Exercise Similarity

The present study was concerned with investigating the nature of assessment center exercises. Bem and Funder's (1978) technique of classifying and comparing situations in behavioral terms was applied to the measurement of exercises in an assessment center with ratings reflecting exercise factors. Six assessors created templates for each of the four exercises. Intercorrelations between the mean template ratings suggested that the exercises were not viewed as similar situations. Moreover, some relationship was found between exercise similarity and performance consistency. It is proposed that these differences in exercise content may be responsible for the inability to find cross-situational consistencies in candidate behavior in a typical assessment center. Practical implications and extensions of this study are discussed.     

CONSTRUCT VALIDITY OF TWO CATEGORIES OF ASSESSMENT CENTER DIMENSION RATINGS

The construct validity of assessment center final dimension ratings was examined within a nomological network of cognitive and personality measures. Four hundred forty-one employees of a large mid-western petroleum company were assessed on 11 dimensions in two broad categories and completed four tests. Results showed that several cognitive ability measures related more strongly to performance-style dimension ratings than to interpersonal-style dimension ratings, providing evidence for convergent and discriminant validity. Correlation analysis and factor analysis support the two a priori interpersonal- and performance-style categories. The results suggest that final dimension ratings possess construct validity and that assessors can differentiate between two broad categories of assessment dimensions.     

Assessment Center Procedures: Cognitive Load During the Observation Phase

This study explores the traditional procedure of observing assessment center exercises while taking notes vs. an alternative procedure where assessors merely observe and postpone note‐taking until immediately after the exercise. The first procedure is considered to be cognitively demanding due to the requirement of simultaneous note‐taking and observing. Also, dual task processing (concurrent observing and note‐taking) is considered to be especially demanding for assessors without rating experience. The procedures are evaluated using a 2 × 2 design (with note‐taking/without note‐taking × experienced/inexperienced). Some 121 experienced and inexperienced assessors rated videotaped candidates, observing either with or without taking notes. Results showed that experienced assessors yield significantly higher differential accuracy than inexperienced assessors. We did not find an effect of observation procedure on accuracy, interrater reliability or halo. Implications for future research are described.     

A new frame for frame-of-reference training: enhancing the construct validity of assessment centers

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (a) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (b) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (c) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice.     

Valid and more valid? Keys to understanding personal appraisal practices at work

Four empirical studies of the roles of self-report questionnaires, tests, and ratings in employment interviews, staff appraisal, and personnel validation surveys are presented in order to examine the number and relationships of psychological constructs derived from data provided by candidates, employees, and their assessors. Test results are related to different kinds of criteria, including job offers following interview, supervisor ratings on the job, and objective indices of work performance. From the practices reported and observed in these various contexts, a theory of personal appraisal mechanisms in occupational settings is derived using data reduction and linear regression methods. Applications of cognitive information processing theory to self-reports and ratings by individuals, and of exchange theory to the social context of panel discussions among assessors, provide a key to understanding why psychometric measures of personal qualities vary in their power to predict work performance in spite of consistent construct validity.     

A detection theory approach to the evaluation of assessors in assessment centres

The ratings given to job applicants in an assessment centre (AC) will be influenced both by an assessor's sensitivity to the evidence of suitability provided by the applicants, and by whether the assessor has a liberal or a conservative rating tendency. In this study we explore the usefulness of signal detection methodology for evaluating and comparing both aspects of performance. A group of eight managerial assessors in a real‐world AC rated 195 applicants for retail sales positions. The sensitivity and response bias of assessors was evaluated using receiver operating characteristic (ROC) analyses, and their performance was evaluated by means of critical operating characteristic (COC) analyses. RscorePlus software ( Harvey, 2002 ) was used for these purposes. We conclude that, in this and similar organizational contexts, such analyses are potentially useful for estimating and comparing the performances of assessors, thereby highlighting the need for, and subsequently evaluating the effectiveness of, any remedial intervention.     

EFFECTS OF CANDIDATE CHARACTERISTICS ON MANAGEMENT PROMOTION DECISIONS: AN EXPERIMENTAL STUDY

This study examined the effects of candidate characteristics on simulated management promotion decisions. Seventy-two managers from three organizational levels participated as decision makers in a half-day exercise. Four candidate characteristics were manipulated: potential for advancement, the availability of assessment center information, current position, and sex. Other characteristics such as tenure and past performance were controlled by randomly distributing levels of the attributes across candidates or holding the attributes constant. The results showed that potential, assessment center information, and position were important in selecting finalists and rating the extent to which a candidate was considered. Two interactions, potential by position and sex by position, indicated that configural relationships played a significant role in the promotion decisions. Participants' perceptions of the importance of the information paralleled the statistical importance of the information. The external validity of the study was supported by comparing the results to data on actual promotion decisions. The results are discussed in terms of the value of different information and how it is processed in evaluating and selecting candidates for promotion.     

A construct-driven investigation of gender differences in a leadership-role assessment center

This study examined gender differences in a large-scale assessment center for officer entry in the British Army. Subgroup differences were investigated for a sample of 1,857 candidates: 1,594 men and 263 women. A construct-driven approach was chosen (a) by examining gender differences at the construct level, (b) by formulating a priori hypotheses about which constructs would be susceptible to gender effects, and (c) by using both effect size statistics and latent mean analyses to investigate gender differences in assessment center ratings. Results showed that female candidates were rated notably higher on constructs reflecting an interpersonally oriented leadership style (i.e., oral communication and interaction) and on drive and determination. These results are discussed in light of role congruity theory and of the advantages of using latent mean analyses.     

Behavioral Assessment in the Schools:

This article discusses behavioral assessment as a way of obtaining and evaluating information used to individualize programming for special needs pupils. Behavioral assessment offers a range of techniques for gathering and for evaluating information. Strategies for obtaining information that are described are self-monitoring, direct observation, permanent product review, and retrospective ratings using behavior checklists. Strategies for evaluating information that are discussed are single-case research designs and social validation. These strategies assist in identifying pupil problems and selecting interventions, evaluating student progress, assessing treatment fidelity, determining causality, and revising individualized programs by providing a broad base for decision-making.     

Should assessment center dimension ratings be mechanically-derived?

This study examined the relationship between consensusderived and mechanically-derived assessment center ratings using a sample of 110 police candidates. It was determined that the two sets of dimension ratings were highly correlated (medianr=.83). Further, the mechanically-derived and the consensus-derived assessment ratings both correlated with the overall assessment ratings (OAR) at .71. It was estimated that the organization in this study could save $6137.92 a year, and $36.89 per assessee by eliminating the team meeting (consensus decision-making).     

Dimension and exercise variance in assessment center scores: a large-scale evaluation of multitrait-multimethod studies.

This study addresses 3 questions regarding assessment center construct validity: (a) Are assessment center ratings best thought of as reflecting dimension constructs (dimension model), exercises (exercise model), or a combination? (b) To what extent do dimensions or exercises account for variance? (c) Which design characteristics increase dimension variance? To this end, a large set of multitrait-multimethod studies (N = 34) were analyzed, showing that assessment center ratings were best represented (i.e., in terms of fit and admissible solutions) by a model with correlated dimensions and exercises specified as correlated uniquenesses. In this model, dimension variance equals exercise variance. Significantly more dimension variance was found when fewer dimensions were used and when assessors were psychologists. Use of behavioral checklists, a lower dimension-exercise ratio, and similar exercises also increased dimension variance.     

IMPLICATIONS OF TRAIT-ACTIVATION THEORY FOR EVALUATING THE CONSTRUCT VALIDITY OF ASSESSMENT CENTER RATINGS

Assessment centers have often been criticized for lacking evidence supporting the construct validity of dimension ratings. This study examines whether the poor convergence of assessment center ratings is a result of correlating ratings from exercises that differ in the extent that behavior relevant to personality traits can be observed. Using data from a promotional assessment center for law enforcement officers ( n = 79), the convergence of assessment center ratings was evaluated within the context of the five factor model by comparing the average within-dimension correlation of ratings from exercises that allowed for more opportunity to observe trait-relevant behavior to the average of those involving exercises where there was less opportunity. For each personality trait, ratings from exercises judged by experts to be high in trait-activation potential displayed stronger convergence (mean r = .30) than did ratings from exercises that were low in activation potential for that trait (mean r = .15). Implications for evaluating the construct validity of assessment centers are discussed along with future directions for classifying exercises based on situational similarity.     

Same‐gender and same‐race bias in assessment center ratings: A rating error approach to understanding subgroup differences

This study investigated leniency and similar‐to‐me bias as mechanisms underlying demographic subgroup differences among assessees in assessors’ initial dimension ratings from three assessment center (AC) simulation exercises used as part of high‐stakes promotional testing. It examined whether even small individual‐level effects can accumulate (i.e., “trickle‐up”) to produce larger subgroup‐level differences. Individual‐level analyses were conducted using cross‐classified multilevel modeling and conducted separately for each exercise. Results demonstrated weak evidence of leniency toward White assessees and similar‐to‐me bias among non‐White assessee–assessor pairs. Similar leniency was found toward female assessees, but no statistically significant effects were found for assessee or assessor gender or assessee–assessor gender similarity. Using traditional d effect size estimates, weak individual level assessee effects translated into small but consistent subgroup differences favoring White and female assessees. Generally small but less consistent subgroup differences indicated that non‐White and male assessors gave higher ratings. Moreover, analyses of overall promotion decisions indicate the absence of adverse impact. Findings from this AC provide some support for the “trickle‐up” effect, but the effect on subgroup differences is trivial. The results counter recent reviews of AC studies suggesting larger than previously assumed subgroup differences. Consequently, the findings demonstrate the importance of following established best practices when developing and implementing the AC method for selection purposes to minimize subgroup differences.