ASSESSMENT CENTERS AND SUBORDINATE APPRAISALS OF MANAGERS: A SEVEN-YEAR EXAMINATION OF PREDICTIVE VALIDITY

Assessment center ratings of 52 state law enforcement agency managers predicted both a contaminated promotions criterion and two uncontaminated ratings criteria, but they did significantly better in predicting the former. Criteria data were gathered two, four, and seven years after the assessment center, and a modest upward trend in validity coefficients was observed. Ratings by subordinates demonstrated promise as a near term predictor, outperforming the assessment center on uncontaminated criteria in the two- and four-year time frames. Last, regression analyses using Year 7 criteria revealed unique predictive power for both ratings by subordinates and assessment centers above and beyond that provided by supervisor ratings.
An earlier version of this paper was presented at the Academy of Management meeting, Anaheim, California, August 7-10, 1988.     

SIMULATION-BASED ASSESSMENT OF MANAGERIAL COMPETENCE: RELIABILITY AND VALIDITY

Prior work based on complexity theory has attempted, with some success, to predict general and managerial performance in complex, uncertain, and fluid task settings. The present paper evaluates a quasi-experimental simulation technique that was specifically designed to measure the impact of individual differences in a number of managerial styles (including a style reflecting cognitive complexity) on executive performance. Twenty simulation-based measures were tested for reliability and validity. On the basis of the data obtained from two separate samples, it was concluded that this quasi-experimental simulation technology may be useful for assessing a number of managerial styles that are not currently tapped by other measurement methods. Research results reported in this paper, as well as results obtained in other concurrent efforts, are summarized.     

EXERCISES AND DIMENSIONS ARE THE CURRENCY OF ASSESSMENT CENTERS

A novel assessment center (AC) structure that models broad dimension factors, exercise factors, and a general performance factor is proposed and supported in 4 independent samples of AC ratings. Consistent with prior research, the variance attributable to dimension and exercise factors varied widely across ACs. To investigate the construct validity of these empirically supported components of AC ratings, the nomological network of broad dimensions, exercises, and general performance was examined. Results supported the criterion‐related validity of broad dimensions and exercises as predictors of effectiveness and success criteria as well as the incremental validity of broad dimensions beyond exercises and general performance. Finally, the relationships between individual differences and AC factors supported the construct validity of broad dimension factors and provide initial insight as to the meaning of exercise specific variance and general AC performance.     

CONSTRUCTING PARALLEL SIMULATION EXERCISES FOR ASSESSMENT CENTERS AND OTHER FORMS OF BEHAVIORAL ASSESSMENT

Assessment centers rely on multiple, carefully constructed behavioral simulation exercises to measure individuals on multiple performance dimensions. Although methods for establishing parallelism among alternate forms of paper-and-pencil tests have been well researched (i.e., to equate tests on difficulty such that the scores can be compared), little research has considered the why and how of parallel simulation exercises. This paper extends established procedures for constructing parallel test forms to dimension-based behavioral simulations. We discuss reasons for establishing comparable, alternate simulation forms and discuss the issues raised when applying traditional procedures to simulation exercises. After proposing a set of guidelines for establishing alternate forms among simulations, we apply these guidelines to simulations used in an operational assessment center.     

评价中心的构想效度和结构模型

采用多质多法和验证性因素分析的方法,对以无领导小组讨论、文件筐和人格测验构成的一个评价中心的构想效度和结构模型进行了研究。通过对136名被试在四个测评维度上的施测,其结果表明,在评价中心中会聚效度低于区分效度,影响评价中心测评结果的主要因素是测评方法而不是测评维度,从而得到了一个以测评方法为潜变量的评价中心结构模型。从该结构模型来看,评价中心之所以起作用是由于其多个测评方法(情景)的结果。表明测评情景对于构建评价中心有着至关重要的作用。     

CORRESPONDENCE BETWEEN ESTIMATES OF CONTENT AND CRITERION-RELATED VALIDITY VALUES

The correspondence between inferences made using two validation strategies–content and criterion-related–were examined in a specific personnel selection application. Empirical validity values and Law-she's (1975) content validity ratios (CVR) were obtained for items from three structured interview guides used in the selection of insurance agents. Ratings of each item by over 300 field managers were used to calculate the CVR values. Statistically significant, yet modest correlations were found between empirical item validities and content validities for an interview guide used to select applicants with prior insurance sales experience. No significant differences were found among these correlations by comparing job experts of different levels of managerial experience and experience in selection. Data for the interview guide used to select experienced applicants also indicated that a content validity approach can be useful in developing a selection instrument with an empirically valid composite rating. The hypotheses were not confirmed for interview guides used to select applicants with no prior insurance sales experience. The practical importance of these results are discussed, as are plans for future research.     

EVALUATING DEVELOPMENTAL ASSESSMENT CENTERS AS INTERVENTIONS

This study addresses the question of whether developmental assessment centers are effective interventions. Career outcomes of assessment center participants ( n = 151) were evaluated with respect to center ratings, developmental activity, and acceptance of feedback. Although comparison between a subgroup of assessees ( n = 113) and a naturally occurring control group of non-assessees ( n = 167) showed no differences in terms of career advancement, findings within the assessee sample did show that developmental activity related to advancement. Findings also showed that assessors' ratings of career motivation, rather than ratings of knowledge or skill (KS) dimensions, predicted both developmental activity and career advancement.     

VALIDITY AND FAIRNESS OF SOME ALTERNATIVE EMPLOYEE SELECTION PROCEDURES1

Despite extensive evidence that tests are valid for employee selection, Federal Guidelines have urged employers to seek alternative selection procedures that are equally valid but have less adverse impact on minorities. Research on the validity, adverse impact and fairness of eight categories of alternatives was reviewed. Feasibility of operational use of each type of alternative in an employment setting was also discussed. Only biodata and peer evaluation were supported as having validities substantially equal to those for standardized tests. Previous reviews and more recent research indicated that interviews, self-assessments, reference checks, academic achievement, expert judgment and projective techniques had levels of validity generally below those reported for tests. Data, where available, offered no clear indication that any of the alternatives met the criterion of having equal validity with less adverse impact. Results are discussed and several additional promising alternatives are described.     

"CONTENT VALIDITY" IN MODERATION

"Content validity" has been widely but unwisely hailed as a solution to many problems in employee selection. The enthusiasm and its scope must be tempered. The arguments of this paper begin with the recognition that sampling from content domains cannot logically be substituted for criterion-related validity. Moreover, it must be recognized that the psychometric notion of validity ordinarily refers to scores, not to the stimulus content of a test. It is therefore suggested that evaluations of scores be based on the principle of construct validation; that is, that possible sources of contamination be considered as alternative hypotheses about what is measured by a content sample as it is scored.     

FROM LABORATORY TO PRACTICE: NEGLECTED ISSUES IN IMPLEMENTING FRAME-OF-REFERENCE RATER TRAINING

Data collected at two law enforcement agencies were used to address three specific issues concerning the development and implementation of frame-of-reference rater training. First, the prototype-anchored rating system was presented as a comprehensive method for generating an appropriate frame of reference in an organizational setting. Second, sensitivity and threshold analyses were used to demonstrate a method for identifying idiosyncratic raters (i.e., raters deviating from the appropriate frame of reference) in the rater population. Finally, areas of performance where supervisors and subordinates were likely to disagree on the frame of reference were identified. Concerning this latter issue, analyses indicated supervisors viewed poor-performance incidents more severely than did patrol officers on several dimensions of performance. To a lesser degree, supervisors and patrol officers also differed on their perceptions of the importance of poor-performance incidents. The implications of these findings are discussed in relation to the development and implementation of frame-of-reference rater training.     

ASSESSING CONTENT VALIDITY: STANDARDS SET BY THE COURT1

Thirty-one court cases were reviewed in order to determine the standards set by the court in their assessment of the content validity of paper-and-pencil tests. The findings indicate that the courts have not acted upon a uniform set of standards. In some instances, tests are judged solely on the basis of their “face validity;” in others, extensive evidence is required. The discussion centered on the failure of the courts to consistently apply professionally developed selection guidelines. The implications for practitioners and psychologists were discussed.     

DEVELOPMENT ENGAGEMENT WITHIN AND FOLLOWING DEVELOPMENTAL ASSESSMENT CENTERS: CONSIDERING FEEDBACK FAVORABILITY AND SELF–ASSESSOR AGREEMENT

This study sought to understand employees' level of behavioral engagement in response to feedback received in developmental assessment center (DAC) programs. Hypotheses were drawn from theories of self‐enhancement and self‐consistency and from findings in the multisource feedback and assessment center literatures regarding recipients' perceptions of feedback. Data were gathered from 172 U.S. middle managers participating in a DAC program. Results suggested that more favorable feedback was related to higher behavioral engagement. When discrepancies between self‐ and assessor ratings were examined, overraters (participants whose overall self‐ratings were higher than their assessor ratings) tended to show less engagement in the program compared to underraters. However, pattern agreement on the participant's dimension profile did not significantly correlate with behavioral engagement. Based on these findings, avenues for future research are presented and practical implications are discussed.     

A META-ANALYSIS OF WORK SAMPLE TEST VALIDITY: UPDATING AND INTEGRATING SOME CLASSIC LITERATURE

Work sample tests have been used in applied psychology for decades as important predictors of job performance, and they have been suggested to be among the most valid predictors of job performance. As we examined classic work sample literature, we found the narrative review by Asher and Sciarrino (1974) to be plagued by many methodological problems. Further, it is possible that data used in this study may have influenced the results (e.g., r = .54) reported by Hunter and Hunter in their seminal work in 1984. After integrating all of the relevant data, we found an observed mean correlation between work sample tests and measures of job performance of .26. This value increased to .33 when measures of job performance (e.g., supervisory ratings) were corrected for attenuation. Our results suggest that the level of the validity for work sample tests may not be as large as previously thought (i.e., approximately one third less than previously thought). Further, our work also summarizes the relationship of work sample exams to measures of general cognitive ability. We found that work sample tests were associated with an observed correlation of .32 with tests of general cognitive ability.     

WORKING THROUGH ENVY: SOME TECHNICAL ISSUES

The author argues that envy appears in an analysis as a hostile, life-destroying force and is directed at the good qualities of the object, often manifesting in the patient's need to devalue analytic work that has been helpful. An intractable negative therapeutic reaction can thus become a persistent feature of an analysis. Sometimes powerful envy is silently operating beneath the surface, with the patient feeling continually wronged and mistreated by the object. The author argues that bringing the projected envy into the patient's personality structure is an important aspect of successful analysis, but that various technical problems have to be considered before painful insight can be achieved into the division in the patient's self. Presenting material from a relatively long analysis, she discusses some technical issues in dealing with these processes. Interpretation of envy can be a delicate matter, requiring consideration not only of how to interpret, but also of the nature of the countertransference and the form of the interpretation. A transference interpretation directed solely at the patient's self-destructive behaviour and destructiveness towards the object carries the risk of further stimulating the destructiveness, even if it is correct; it is also important to bring the patient's libidinal aspect into the interpretation.     

SOME ISSUES IN CONVERSATION COHERENCE

This study examined respondent proficiency at reconstructing a sequence of utterances (discourse). Theories of text hold that participants in a discourse must construct and arrange the sense of the text; that is, they must make decisions about discourse coherence. We had respondents reconstruct a conversation and we made predictions about discourse coherence, the nature of various two-turn constructional units, unit differences, and individual differences. We found that the entire discourse as a processing unit affects the likelihood of certain two-turn pairs being resequenced correctly. Except for the fact that complex subjects were better able to resequence the initial interaction, there were no other individual differences. The research has implications for principles of coherence, text comprehension, and interpersonal behavior.