Evaluations of Tests Used for Making Selection and Promotion Decisions

Participants (N= 212) rated the fairness, job relevance, appropriateness, and invasiveness of 16 tests that could be used to select or promote people into production or management positions. Fairness, job relevance and appropriateness were highly correlated, and were combined to form a composite evaluation scale. Evaluations and invasiveness ratings varied among the 16 tests, with the most positive ratings given to interviews and work samples and the most negative ratings given to astrology, graphology and polygraphs. Evaluations of four tests were affected by the position (manager vs. production worker). Evaluations of 11 tests and invasiveness ratings of two tests were affected by respondent experience with the test. Respondents who had experienced the tests evaluated them more positively and considered them to entail a smaller invasion of privacy. Responses were not affected by whether the test was to be used for selection versus promotion decisions.     

Different subjective awareness measures demonstrate the influence of visual identification on perceptual awareness ratings

We compare four subjective awareness measures in the context of a visual identification task and investigate quantitative differences in terms of scale use and correlation with task performance. We also analyse the effect of identification task decisions on subsequent subjective reports. Results show that awareness ratings strongly predict accuracy for all scale types, although the type of awareness measure may influence the reported level of perceptual awareness. Surprisingly, the overall relationship between awareness ratings and performance was weaker when participants rated their awareness before providing identification responses. Furthermore, the Perceptual Awareness Scale was most exhaustive only when used after the identification task, whereas confidence ratings were most exhaustive when used before the identification task. We conclude that the type of subjective measure applied may influence the reports on visual awareness. We also propose that identification task decisions may affect subsequent awareness ratings.     

In fairness to future generations: The role of egocentrism, uncertainty, power, and stewardship in judgments of intergenerational allocations

In this paper, we found that fairness judgments in intergenerational allocation decisions depend on (1) individuals’ position in the intergenerational sequence (i.e., whether they are in the preceding or succeeding generation), (2) the amount of uncertainty about the effect of the preceding generation’s decisions on the succeeding one, and (3) whether the preceding generation is primed with power. We found that both power priming and greater levels of uncertainty about the future consequences of present decisions can elicit stewardship attitudes, which may temper self-interested behavior on the part of the preceding generation. Our data also indicate that the nature of the uncertainty is important; it is not just a matter of how much future generations will be affected, but also whether or not they will be affected at all by the decisions of earlier generations. Our findings help to explain (1) how intergenerational inequities can occur even when people are explicitly focused on achieving fair allocations of resources between generations, and (2) how social responsibility concerns can motivate intergenerational beneficence in the face of intergenerational conflict.     

Cognitive mechanisms mediating the biasing effects of leader sex on ratings of leader behavior

The present study examined the effects of the sex of the leader, the sex of the rater, and the temporal position of leader performance information on ratings of leader behavior. Male and female subjects listened to an audiotape of a male or female leader and then rated the leader's initiating structure and consideration behaviors. The position of the leader performance information was manipulated by describing the leader as successful before or after raters listened to the audiotape or not presenting raters with the performance information. Analyses indicated that the sex of the leader and/or sex of the rater biased ratings of initiating structure when the performance information was presented after the audiotape and when no leader performance information was presented. Consideration ratings were affected only by the position of the leader performance information. The results are interpreted within a cognitive/implicit sex theory framework. Implications for leadership research are also discussed.Portions of the data analyzed in this study were collected by the second author as part of a master's thesis directed by Joseph A. Sgro.     

不同促销方式对产品购买决策的影响—— 基于解释水平理论视角的研究

赠品促销与价格促销对消费者购买决策的影响研究结果尚存在分歧。根据解释水平理论, 通过三个实验探究了赠品促销产品组合与价格促销产品组合在不同任务类型、不同时间距离条件下的促销效果差异。结果表明：(1)不同时间距离下, 消费者对促销产品决策存在差异。赠品促销方式在近期将来购物场景下的促销效果比在远期将来购物场景下的促销效果更好; 价格促销方式在远期将来购物场景下的促销效果比在近期将来购物场景下的促销效果更好。价格支付意愿任务下, 消费者对远期将来场景下促销产品的感知价值比近期将来购物场景下促销产品的感知价值高; 购买可行性任务下, 消费者对近期将来购物场景下促销产品的感知价值比远期将来购物场景下促销产品的感知价值高。(2)时间距离对促销方式与任务类型的表征水平匹配关系有增强、扩大作用。购买可行性任务下, 与远期将来时间相比, 消费者更倾向于在近期将来时间条件下选择赠品促销产品组合。价格支付意愿任务下, 与近期将来时间相比, 消费者更倾向于在远期将来时间条件下选择价格促销产品组合。     

Incremental Validity of Leaderless Group Discussion Ratings Over and Above General Mental Ability and Personality in Predicting Promotion

Leaderless group discussions (LGDs) constitute one of the oldest assessment center exercises. In recent times, their added value has sometimes been questioned in light of trends to streamline assessment centers. The purpose of the present study is to examine the incremental validity of LGD ratings over cognitive ability scores and personality ratings for the prediction of extrinsic career success (i.e., promotion speed and number of promotions). We investigated this issue in the context of the promotion of French naval officers (N = 93) in an academy for high‐level executive positions over a 10‐year period. Results indicated that LGD ratings accounted for incremental variance in the prediction of promotion criterion measures, beyond cognitive ability and personality test scores. These results confirm that LGD ratings provide a unique contribution to the prediction of extrinsic career success in high‐level executive positions.     

Transparency of Assessment Centers: Lower Criterion‐related Validity but Greater Opportunity to Perform?

Assessment centers (ACs) are popular selection devices in which assessees are assessed on several dimensions during different exercises. Surveys indicate that ACs vary with regard to the transparency of the targeted dimensions and that the number of transparent ACs has increased during recent years. Furthermore, research on this design feature has put conceptual arguments forward regarding the effects of transparency on criterion‐related validity, impression management, and fairness perceptions. This study is the first to examine these effects using supervisor‐rated job performance data as the criterion. We conducted simulated ACs with transparency as a between‐subjects factor. The sample consisted of part‐time employed participants who would soon be applying for a new job. In line with our hypothesis, results showed that ratings from an AC with nontransparent dimensions were more criterion valid than ratings from an AC with transparent dimensions. Concerning impression management, our results supported the hypothesis that transparency moderates the relationship between self‐promotion and job performance, such that self‐promotion in the nontransparent AC was more positively related to job performance than self‐promotion in the transparent AC. The data lent no support for the hypothesis that participants’ perceptions of their opportunity to perform are higher in the transparent AC.     

Outcome effects: The impact of decision process and outcome controllability

An ‘outcome effect’ refers to the phenomenon whereby performance evaluations of decision makers are affected by the outcomes of those decisions. Although some consider such an effect to be a judgmental error, judgment by outcomes may not be dysfunctional when the evaluator does not know how the decision maker chose his or her action. In such situations, outcomes may provide some noisy information about decision quality. We test whether an outcome effect will still occur when the decision methodology and quality are more explicitly identified. Further, we test whether outcome controllability, a previously unexplored moderator variable, will have an impact on the outcome effect. Our first experiment, using undergraduates as subjects, shows that decision quality and controllability have an impact on performance evaluations but that the ubiquitous outcome effect still obtains. These results were replicated with experienced business people, except that controllability only affected their judgments in the case of negative outcomes. Implications of these results are discussed. © 1997 John Wiley & Sons, Ltd.     

Bias in Performance Ratings: Clarifying the Role of Positive Versus Negative Escalation

This study replicated and refined research on the effects of escalation of commitment in performance ratings. We utilized experimental manipulations in a laboratory setting to determine whether positive escalation or negative escalation (or both) could be responsible for the effect. In one session, participants (N = 210) were assigned to the perspective of the potential employee's supervisor and chose 1 of 2 candidates for a sales position. In a second session, participants rated the performance of (a) the individual they selected; (b) the individual they rejected; or (c) a third individual, whose preselection information they never viewed. Results replicated previous findings, such that ratings were biased upward when participants rated the performance of the salesperson they had originally selected. Results were not biased downward when individuals rated the performance of the salesperson they had rejected, however. Thus, our results suggest that positive escalation, and not negative escalation, was the cause of the bias.     

Faking under a nonlinear relationship between personality assessment scores and job performance

This study investigated whether faking in personality assessments affects the prediction of job performance and consequences of selection decisions when a nonlinear relationship was assumed between personality scores and job performance. It was also examined whether a selection strategy suggested for nonlinear relationships reduces the adverse impact of faking on consequences of selection decisions. The findings from the item response theory‐based simulations indicated that the prediction of job performance was affected by faking, whereas selection accuracy was not substantially affected by faking. The improvement in selection accuracy due to the strategy suggested for a nonlinear relationship was not noticeable. However, the suggested strategy reduced adverse impacts of faking on the estimation of selected applicants' performance scores.     

Structural aspects of face recognition and the other-race effect

The other-race effect was examined in a series of experiments and simulations that looked at the relationships among observer ratings of typicality, familiarity, attractiveness, memorability, and the performance variables ofd’ and criterion. Experiment 1 replicated the other-race effect with our Caucasian and Japanese stimuli for both Caucasian and Asian observers. In Experiment 2, we collected ratings from Caucasian observers on the faces used in the recognition task. A Varimax-rotated principal components analysis on the rating and performance data for the Caucasian faces replicated Vokey and Read’s (1992) finding that typicality is composed of two orthogonal components, dissociable via their independent relationships to: (1) attractiveness and familiarity ratings and (2) memorahility ratings. For Japanese faces, however, we fond that typicality was related only to memorahility. Where performance measures were concerned, two additional principal components dominated by criterion and byd’ emerged for Caucasian faces. For the Japanese faces, however, the performance measures ofd’ and criterion merged into a single component that represented a second component of typicality, one orthogonal to thememorability-dominated component. A measure offace representation quality extracted from an autoassociative neural network trained with a majority of Caucasian faces and a minority of Japanese faces was incorporated into the principal components analysis. For both Caucasian and Japanese faces, the neural network measure related both to memorability ratings and to human accuracy measures. Combined, the human data and simulation results indicate that the memorahility component of typicality may be related to small, local, distinctive features, whereas the attractiveness/familiarity component may be more related to the global, shape-based properties of the face.     

Differences in Inter‐Rater Reliability and Accuracy for a Treatment Adherence Scale

Inter‐rater reliability and accuracy are measures of rater performance. Inter‐rater reliability is frequently used as a substitute for accuracy despite conceptual differences and literature suggesting important differences between them. The aims of this study were to compare inter‐rater reliability and accuracy among a group of raters, using a treatment adherence scale, and to assess for factors affecting the reliability of these ratings. Paired undergraduate raters assessed therapist behavior by viewing videotapes of 4 therapists' cognitive behavioral therapy sessions. Ratings were compared with expert‐generated criterion ratings and between raters using intraclass correlation (2,1). Inter‐rater reliability was marginally higher than accuracy (p = 0.09). The specific therapist significantly affected inter‐rater reliability and accuracy. The frequency and intensity of the therapists' ratable behaviors of criterion ratings correlated only with rater accuracy. Consensus ratings were more accurate than individual ratings, but composite ratings were not more accurate than consensus ratings. In conclusion, accuracy cannot be assumed to exceed inter‐rater reliability or vice versa, and both are influenced by multiple factors. In this study, the subject of the ratings (i.e. the therapist and the intensity and frequency of rated behaviors) was shown to influence inter‐rater reliability and accuracy. The additional resources needed for a composite rating, a rating based on the average score of paired raters, may be justified by improved accuracy over individual ratings. The additional time required to arrive at a consensus rating, a rating generated following discussion between 2 raters, may not be warranted. Further research is needed to determine whether these findings hold true with other raters and treatment adherence scales.     

Measuring consciousness: Is one measure better than the other?

What is the best way of assessing the extent to which people are aware of a stimulus? Here, using a masked visual identification task, we compared three measures of subjective awareness: The Perceptual Awareness Scale (PAS), through which participants are asked to rate the clarity of their visual experience; confidence ratings (CR), through which participants express their confidence in their identification decisions, and Post-decision wagering (PDW), in which participants place a monetary wager on their decisions. We conducted detailed explorations of the relationships between awareness and identification performance, looking to determine (1) which scale best correlates with performance, and (2) whether we can detect performance in the absence of awareness and how the scales differ from each other in terms of revealing such unconscious processing. Based on these findings we discuss whether perceptual awareness should be considered graded or dichotomous. Results showed that PAS showed a much stronger performance-awareness correlation than either CR or PDW, particularly for low stimulus intensities. In general, all scales indicated above-chance performance when participants claimed not to have seen anything. However, such above-chance performance only showed when we also observed a correlation between awareness and performance. Thus (1) PAS seems to be the most exhaustive measure of awareness, and (2) we find support for above-chance performance in the absence of subjective awareness, but such unconscious knowledge only contributes to performance when we observe conscious knowledge as well. Similarities and differences between scales are discussed in the light of consciousness theories and response strategies.     

Differences in inter-rater reliability and accuracy for a treatment adherence scale

Inter-rater reliability and accuracy are measures of rater performance. Inter-rater reliability is frequently used as a substitute for accuracy despite conceptual differences and literature suggesting important differences between them. The aims of this study were to compare inter-rater reliability and accuracy among a group of raters, using a treatment adherence scale, and to assess for factors affecting the reliability of these ratings. Paired undergraduate raters assessed therapist behavior by viewing videotapes of 4 therapists' cognitive behavioral therapy sessions. Ratings were compared with expert-generated criterion ratings and between raters using intraclass correlation (2,1). Inter-rater reliability was marginally higher than accuracy (p = 0.09). The specific therapist significantly affected inter-rater reliability and accuracy. The frequency and intensity of the therapists' ratable behaviors of criterion ratings correlated only with rater accuracy. Consensus ratings were more accurate than individual ratings, but composite ratings were not more accurate than consensus ratings. In conclusion, accuracy cannot be assumed to exceed inter-rater reliability or vice versa, and both are influenced by multiple factors. In this study, the subject of the ratings (i.e. the therapist and the intensity and frequency of rated behaviors) was shown to influence inter-rater reliability and accuracy. The additional resources needed for a composite rating, a rating based on the average score of paired raters, may be justified by improved accuracy over individual ratings. The additional time required to arrive at a consensus rating, a rating generated following discussion between 2 raters, may not be warranted. Further research is needed to determine whether these findings hold true with other raters and treatment adherence scales.     

EFFECTS OF CANDIDATE CHARACTERISTICS ON MANAGEMENT PROMOTION DECISIONS: AN EXPERIMENTAL STUDY

This study examined the effects of candidate characteristics on simulated management promotion decisions. Seventy-two managers from three organizational levels participated as decision makers in a half-day exercise. Four candidate characteristics were manipulated: potential for advancement, the availability of assessment center information, current position, and sex. Other characteristics such as tenure and past performance were controlled by randomly distributing levels of the attributes across candidates or holding the attributes constant. The results showed that potential, assessment center information, and position were important in selecting finalists and rating the extent to which a candidate was considered. Two interactions, potential by position and sex by position, indicated that configural relationships played a significant role in the promotion decisions. Participants' perceptions of the importance of the information paralleled the statistical importance of the information. The external validity of the study was supported by comparing the results to data on actual promotion decisions. The results are discussed in terms of the value of different information and how it is processed in evaluating and selecting candidates for promotion.     

Same‐gender and same‐race bias in assessment center ratings: A rating error approach to understanding subgroup differences

This study investigated leniency and similar‐to‐me bias as mechanisms underlying demographic subgroup differences among assessees in assessors’ initial dimension ratings from three assessment center (AC) simulation exercises used as part of high‐stakes promotional testing. It examined whether even small individual‐level effects can accumulate (i.e., “trickle‐up”) to produce larger subgroup‐level differences. Individual‐level analyses were conducted using cross‐classified multilevel modeling and conducted separately for each exercise. Results demonstrated weak evidence of leniency toward White assessees and similar‐to‐me bias among non‐White assessee–assessor pairs. Similar leniency was found toward female assessees, but no statistically significant effects were found for assessee or assessor gender or assessee–assessor gender similarity. Using traditional d effect size estimates, weak individual level assessee effects translated into small but consistent subgroup differences favoring White and female assessees. Generally small but less consistent subgroup differences indicated that non‐White and male assessors gave higher ratings. Moreover, analyses of overall promotion decisions indicate the absence of adverse impact. Findings from this AC provide some support for the “trickle‐up” effect, but the effect on subgroup differences is trivial. The results counter recent reviews of AC studies suggesting larger than previously assumed subgroup differences. Consequently, the findings demonstrate the importance of following established best practices when developing and implementing the AC method for selection purposes to minimize subgroup differences.     

The Influence of Indirect Knowledge of Previous Performance on Ratings of Present Performance: The Effects of Job Familiarity and Rater Training

In a 2 × 2 × 2 crossed factorial design, trained or untrained subjects viewed a videotape and evaluated performance on either a familiar (college lecturer) or unfamiliar (salesperson) job. Prior to viewing the videotape, some subjects reviewed positive information about the ratee's prior performance, whereas other subjects did not review any prior performance information. To determine whether assimilation or contrast effects occurred, we compared ratings provided by subjects who reviewed positive information about prior performance with ratings provided by subjects who did not review any prior performance information. A three-way interaction was obtained. Ratings of performance on the familiar job by untrained or trained subjects revealed only a small assimilation effect. However, when rating performance on the unfamiliar job, a large assimilation effect was observed among untrained subjects, whereas a large contrast effect was observed among trained subjects. The results indicate that rater error training may reverse, rather than reduce or eliminate, rating errors that arise from knowledge of a ratee's prior performance. Implications for further understanding and reducing assimilation and contrast effects are discussed.     

Incremental Validity of Assessment Center Ratings Over Cognitive Ability Tests: A Study at the Executive Management Level

Both tests of cognitive ability and assessment center (AC) ratings of various performance attributes have proven useful in personnel selection and promotion contexts. To be of theoretical or practical value, however, the AC method must show incremental predictive accuracy over cognitive ability tests given the cost disparities between the two predictors. In the present study, we investigated this issue in the context of promotion of managers in German police departments into a training academy for high‐level executive positions. Candidates completed a set of cognitive ability tests and a 2‐day AC. The criterion measure was the final grade at the police academy. Results indicated that AC ratings of managerial abilities were important predictors of training success, even after accounting for cognitive ability test scores. These results confirm that AC ratings provide unique contribution to the understanding and prediction of training performance of high‐level executive positions beyond cognitive ability tests.     

The distortion of criteria after decision-making

The results of two experiments indicate that decision-makers separately distort their interpretations of decision criteria and evaluations of information in a manner that justifies their decisions. In Study 1, participants reported a yes/no decision either before or after they rated the applicable decision criterion and problem information. Participants distorted their criterion and information ratings more when they reported their decisions before, rather than after, they rated the decision criterion and problem information. Study 2 demonstrated that distortion of decision criterion ratings occurred after a decision had been made whereas distortion of information evaluations occurred during the process of reaching a decision. These results are consistent with a two-stage model of decision-making in which the distortion of decision criteria is distinct from the distortion of decision information.     

Realism in confidence judgements of performance based on implicit learning

In two experiments we employed calibration methods to investigate the realism of participants' confidence ratings of their own classification performance based on knowledge acquired after training on an artificial grammar. In Experiment 1 participants showed good realism (but overconfidence) for grammatical strings but very poor realism for non-grammatical strings. Method of training (string repetition in writing or mere exposure) did not affect the realism. Furthermore, the participants underestimated their overall performance. In Experiment 2, using a more complex grammar and controlling for two types of associative chunk-strength, participants showed good realism (but still overconfidence) for both letter and symbol strings, irrespective of grammaticality. Together, these experiments show that implicit learning can give rise to knowledge products that are associated with fairly realistic meta-knowledge. It is argued that both the zero-correlation criterion and the guessing criterion are misplaced when used to define implicit knowledge; two reasons being that confidence judgements may be affected both by implicit knowledge and by inferences.