EXPERT AND NAIVE RATERS USING THE PAQ: DOES IT MATTER?

A finding by Smith and Hakel (1979) is that job expert ratings on the PAQ correlate highly with ratings obtained from college students who are given no information about jobs other than their titles. One possible explanation for this finding is that the PAQ measures only trivial or common knowledge about work that both experts and naive observers possess. This particular view, of course, has serious implications for the use of the PAQ. In this paper we point out a problem in the way Smith and Hakel calculated the convergent validity between expert raters and naive raters. We also present the results from a replication study that indicate convergent validities are much lower than those reported by Smith and Hakel. Additional points are presented in order to caution against the interpretation that the PAQ measures only common knowledge about jobs.     

Accuracy of the Situational Interview in Rating Multiple Job Candidates

The importance of perceptual accuracy to human resource decision processes is evidenced by better than 40 years of accuracy research. This article extends such research by presenting an examination of the situational interview format (Latham, Saari, Pursell, & Campion, 1980) as a means of inducing and maintaining high levels of rating accuracy in a context in which interviewers must evaluate candidates in rapid succession. Situational interview ratings of 48 police sergeants and lieutenants showed consistently high levels of perceptual accuracy in both their overall ratings and differential accuracy scores in rating 6 videotaped sergeant/lieutenant job candidates who were of varying quality and appeared in 4 different multiple candidate sequences. Extensive experimental controls for cueing effects and ratings gained from a separate sample of 48 sergeants and lieutenants using a highly structured conventional format and familiar rating practices indicated that the accuracy of SI ratings were due to the SI itself.     

Do Raters Agree More on Observable Items?

When providing performance ratings, it is commonly assumed that raters agree more on rating items that are behaviorally based and observable than on items that are vague and less behaviorally based. This study empirically investigated the relationships between agreement among raters, raters' perceptions regarding their difficulty in providing ratings, and expert assessments of the behavioral observability of each item. The results, based on 611 raters in two studies conducted in different locations, suggest that contrary to common expectations, rater agreement can increase as raters' reported rating difficulty increases and as behavioral observability decreases. Explanations and implications are discussed.     

Can Nonverbal Cues be Used to Make Meaningful Personality Attributions in Employment Interviews?

Purpose  This study examines the role of personality attributions in understanding the relationships between nonverbal cues and interview performance ratings. Design/methodology/approach  A structured behavioral interview was developed for identifying management potential in a large, national company. Using a concurrent design to validate the interview, managers were interviewed and the interviews were videotaped (n = 110). These videotapes were used as stimuli for raters in this study. Findings  Results indicate that raters can make personality attributions using only one channel of information and these attributions partly explain the relationships between nonverbal cues and performance measures. Furthermore, conscientiousness attributions explain the relationship between visual cues and interview ratings, extraversion attributions mediate the relationship between vocal cues and interview ratings. Neuroticism attributions had a suppressing effect for both visual and vocal cues. Implications   No matter how much an interview is structured, nonverbal cues cause interviewers to make attributions about candidates. If we face this fact, rather than consider information from cues as bias that should be ignored, interviewers can do a better job of focusing on job-related behavior and information in the interview, while realizing that the cues are providing information that must be attended to. Originality/value  This study isolated the sources of information provided to raters to either the vocal or the visual channel to examine their impact individually. A Brunswik lens model shows the potential impact of personality attributions predicting both job and interview performance ratings when both channels of information are used.     

An applied examination of the computerized adaptive rating scale for assessing performance

In the present research, we developed and conducted a field test of the computerized adaptive rating scale (CARS) for assessing military officer performance. Participants completed the CARS and a behaviorally anchored rating scale (BARS) which were both designed to assess five leadership competencies (action orientation/initiative, communication, developing self and others, behavioral flexibility, and teamwork). We obtained data from 116 supervisors and 207 peers who provided ratings on 126 officer ratees. Although interrater reliability estimates were lower for CARS ratings on some competencies, there was a 20–25% improvement in standard error of measurement, the measurement precision in CARS ratings compared to the BARS. Results support findings from a previous lab study.     

Behavioral Expression of Job Interview Anxiety

Purpose

The aim of this study was to investigate (a) the behavioral cues that are displayed by, and trait judgments formed about, anxious interviewees, and (b) why anxious interviewees receive lower interview performance ratings. The Behavioral Expression of Interview Anxiety Model was created as a conceptual framework to explore these relations.

Design/Methodology/Approach

We videotaped and transcribed mock job interviews, obtained ratings of interview anxiety and interview performance, and trained raters to assess several verbal and nonverbal cues and trait judgments.

Findings

The results indicated that few behavioral cues, but several traits were related to interviewee and interviewer ratings of interview anxiety. Two factors emerged from our factor analysis on the trait judgments—Assertiveness and Interpersonal Warmth. Mediation analyses were performed and indicated that Assertiveness and Interpersonal Warmth mediated the relation between interview anxiety and interview performance. Speech rate (words spoken per minute) and Assertiveness were found to mediate the relation between interviewee and interviewer ratings of interview anxiety.

Implications

Overall, the results indicated that interviewees should focus less on their nervous tics and more on the broader impressions that they convey. Our findings indicate that anxious interviewees may want to focus on how assertive and interpersonally warm they appear to interviewers.

Originality/Value

To our knowledge, this is the first study to use a validated interview anxiety measure to examine behavioral cues and traits exhibited by anxious interviewees. We offer new insight into why anxious interviewees receive lower interview performance ratings.

     

The Relationship Quality Interview: evidence of reliability, convergent and divergent validity, and incremental utility

Relationship satisfaction and adjustment have been the target outcome variables for almost all couple research and therapies. In contrast, far less attention has been paid to the assessment of relationship quality. The present study introduces the Relationship Quality Interview (RQI), a semistructured, behaviorally anchored individual interview. The RQI was designed to provide a more objective assessment of relationship quality as a dynamic, dyadic construct across 5 dimensions: (a) quality of emotional intimacy in the relationship, (b) quality of the couple's sexual relationship, (c) quality of support transactions in the relationship, (d) quality of the couple's ability to share power in the relationship, and (e) quality of conflict/problem-solving interactions in the relationship. Psychometric properties of RQI ratings were examined through scores obtained from self-report questionnaires and behavioral observation data collected cross-sectionally from a sample of 91 dating participants and longitudinally from a sample of 101 married couples. RQI ratings demonstrated strong reliability (internal consistency, interrater agreement, interpartner agreement, and correlations among scales), convergent validity (correlations between RQI scale ratings and questionnaire scores assessing similar domains of relationship quality), and divergent validity (correlations between RQI scale ratings and (a) behavioral observation codes assessing related constructs, (b) global relationship satisfaction scores, and (c) scores on individual difference measures of related constructs). Clinical implications of the RQI for improving couple assessment and interventions are discussed.     

ACCURACY OF INTERVIEWER JUDGMENTS OF JOB APPLICANT PERSONALITY TRAITS

This study investigated whether interviewers can assess Big Five personality traits during a job interview. Four raters (self, interviewer, friend, and stranger) assessed the applicant's personality. Results from ratings for 73 applicants demonstrated that interviewer ratings of applicant personality correlate higher with self-ratings ( = .28) than do stranger ratings ( = .09) but less than ratings from close friends ( = .39). However, correlations between interviewer ratings and self-ratings were smaller for the two job-relevant personality traits, Conscientiousness ( = .16, n.s.) and Emotional Stability ( = .17, n.s.) than for the other three personality traits. Variance in ratings suggested the applicants managed their self-presentation on these two traits during the interview. Thus, although interviewers can and do assess personality during the interview, they are not able to assess those traits that would best predict later job success. Finally, the moderating effect of interview design (i.e., structure and content) was assessed. The results revealed that job-relevant interviews, situational interviews, and behavioral interviews did not affect an interviewer's ability to assess personality. Although there was a small (positive) effect for more structured interviews, this affect was modest. The results of this study suggest that future research should examine whether the interview can be designed to assess personality directly, and what the gains to predictive validity are by doing so.     

基于行为事件的履历资料评估

行为事件的履历资料评估能有效克服传统履历数据构思效度弱、情景限制多等问题。本研究结合某通信企业招聘工作,选取250名应聘者数据,结合关键事件法,通过对履历事件的行为锚定来评估应聘者的胜任特征。研究在多重比较行为履历资料、履历表数据、认知能力等多个指标组合对录用结果预测的回归模型基础上,检验新增指标的预测效度。结果表明,行为事件的履历资料评估具有效标关联效度,与其他工具组合使用时具有增量效度。     

Psychometric Properties of a Structured Behavioral Interview to Hire Private Security Personnel

This paper reports on a study about the reliability and validity of a structured behavioral interview to assess private security personnel. Reliability was estimated using interrater coefficients. Two independent interviewers were used to rate each interviewee. Results show a reliability coefficient of .81 (N = 43) and .89 with Spearman-Brown correction for two raters. Validity was estimated using a content validation approach. This strategy was suggested by Lawshe (1975) to estimate the content validity of selection tests. So far, only two studies carried out by Schmitt and Ostroff (1986) and Carrier et al. (1990) have used Lawshe's strategy in the structured behavioral interview case. The interview consisted of seven questions and each was rated by 11 experts in the job. Results show a significant content validity ratio (CVR) for majority of the questions in the interview and a content validity index (CVI) of .89. Implications of these findings for the practice of the structured behavioral interview are discussed and future research is suggested.     

Rater Personality,Rating Format,and Social Context: Implications for Performance Appraisal Ratings

This study investigates the effects of rater personality (Conscientiousness and Agreeableness), rating format (graphic rating scale vs. behavioral checklist), and the rating social context (face‐to‐face feedback vs. no face‐to‐face feedback) on rating elevation of performance ratings. As predicted, raters high on Agreeableness showed more elevated ratings than those low on Agreeableness when they expected to have the face‐to‐face feedback meeting. Furthermore, rating format moderated the relationship between Agreeableness and rating elevation, such that raters high on Agreeableness provided less elevated ratings when using the behavioral checklist than the graphic rating scale, whereas raters low on Agreeableness showed little difference in elevation across different rating formats. Results also suggest that the interactive effects of rater personality, rating format, and social context may depend on the performance level of the ratee. The implications of these findings will be discussed.     

SMALL GROUP BEHAVIORAL TRAINING TO IMPROVE THE JOB INTERVIEW SKILLS REPERTOIRE OF MILDLY RETARDED ADOLESCENTS

Four retarded adolescents, enrolled in a short-term residential treatment program, received behavioral job interview skills training. Although potentially employable, each was unable to present himself effectively in standard employment interviews. Treatment consisted of a series of behavioral group sessions using instructions, modeling and rehearsal procedures to increase, in multiple baseline fashion, such skills as the adolescents' ability to disclose positive information about their experience and background, convey interest in the position and direct relevant questions to an interviewer. Effectiveness of treatment for each client was assessed by: (1) Objective ratings of performance during individual, structured role-play job interviews following each treatment group; (2) objective ratings of pre- and posttraining performance during tape recorded in vivo generalization job interviews at a fast-food restaurant; and (3) global evaluations of pre- and posttraining in vivo generalization interviews made by experienced personnel interviewers unfamiliar with the nature of the treatment. The results indicated that potentially employable retarded citizens can be successfully taught appropriate job interview behavior using a small group behavioral procedure. The need for such techniques in community and rehabilitation centers for retarded citizens and other clinical populations is discussed.     

Correspondence between personality trait ratings and behavioral events

To investigate the validity of the trait, Energetic, two studies were conducted in which trait ratings of Energetic and objective measures of performance were recorded. The first study used naive raters (N= 55) who observed pairs of subjects perform in a laboratory setting. There was 84% agreement (p < .001) among the raters as to which person was more Energetic, and people rated as more Energetic performed more energetically on the objectively measured tasks (p < .001). The second study used peer ratings of Energetic and objective measures of energetic behavior in a laboratory setting. There was high agreement among the peers in ratings of Energetic, r = .95 (N = 30) and r = .93 (N = 31) for two groups. Nine men from the upper-quartile and nine men from the lower-quartile of the Energetic scores were tested in the laboratory and were given a structured interview regarding their physical activities. The correlation between peer ratings and laboratory performance was .56 and between peer ratings and the interview data, .64. The results support the position that trait ratings can be valid indicators of regularities in behavior. The procedures used in these studies could be a generally useful model for investigations of the validity of trait ratings.     

True intercorrelation among job components, time delay in rating, and rater intelligence as determinants of accuracy in performance ratings

Ninety subjects viewed one of three sets of videotapes which presented simulated work performances by five ratees. Subjects viewed videotapes where the true intercorrelation among the three job components was either high, moderate, or low. Half of the subjects completed ratings immediately after viewing all five ratees and again, 24 h later. The remaining subjects completed ratings only once, 24 h after viewing the videotapes. Rater intelligence was assessed via the Wesman Personnel Classification Test. Performance ratings for each ratee on each of the three job components were completed using a magnitude estimation scale. Subjects' ratings were compared to ratee true scores (based on objective worker output) to obtain four components of rater accuracy: elevation, differential elevation, stereotype accuracy, and differential accuracy as well as a measure of overall accuracy. Results indicated that subjects' ratings were more accurate with respect to overall accuracy and differential accuracy when the true intercorrelation among job components was high rather than low. Rater intelligence was significantly correlated with stereotype accuracy. In addition, rater intelligence was significantly related to overall accuracy, differential elevation, and elevation, but only when the true intercorrelation among job components was moderate or high. Also, there was a curvilinear component to the relationship between intelligence and both differential accuracy and stereotype accuracy such that the most intelligent raters tended to be less accurate than more moderately intelligent raters who were, in turn, more accurate than the least intelligent raters. Finally, subjects' immediate ratings were not more accurate than ratings provided by subjects who completed only delayed ratings. However, repeated measures analyses found that for subjects who completed both immediate and delayed ratings, delayed ratings were less accurate than immediate ratings with respect to overall accuracy, differential accuracy, and differential elevation.     

Moderating Effects of Rater Personality on the Relation Between Candidate Self‐Monitoring and Selection Interview Ratings

The present study examined the moderating effect of rater personality – extroversion and sensitivity to others – on the relations between selection interview ratings and measures of candidate self‐monitoring (SM) and social anxiety (SA). In a real‐life military selection procedure setting in which 445 candidates and 93 raters participated, rater extroversion moderated the relation between candidate SM and selection interview ratings so that this relation was negative for raters low on extroversion and positive for raters high on extroversion. Rater extroversion was also found to moderate the negative relation between candidate SA and selection interview ratings. No support was found for the moderating effect of rater sensitivity to others. An explanation of the moderating effect of rater extroversion based on the assumption that extroversion is negatively related to critical interpersonal sensitivity was suggested.     

Differences in inter-rater reliability and accuracy for a treatment adherence scale

Inter-rater reliability and accuracy are measures of rater performance. Inter-rater reliability is frequently used as a substitute for accuracy despite conceptual differences and literature suggesting important differences between them. The aims of this study were to compare inter-rater reliability and accuracy among a group of raters, using a treatment adherence scale, and to assess for factors affecting the reliability of these ratings. Paired undergraduate raters assessed therapist behavior by viewing videotapes of 4 therapists' cognitive behavioral therapy sessions. Ratings were compared with expert-generated criterion ratings and between raters using intraclass correlation (2,1). Inter-rater reliability was marginally higher than accuracy (p = 0.09). The specific therapist significantly affected inter-rater reliability and accuracy. The frequency and intensity of the therapists' ratable behaviors of criterion ratings correlated only with rater accuracy. Consensus ratings were more accurate than individual ratings, but composite ratings were not more accurate than consensus ratings. In conclusion, accuracy cannot be assumed to exceed inter-rater reliability or vice versa, and both are influenced by multiple factors. In this study, the subject of the ratings (i.e. the therapist and the intensity and frequency of rated behaviors) was shown to influence inter-rater reliability and accuracy. The additional resources needed for a composite rating, a rating based on the average score of paired raters, may be justified by improved accuracy over individual ratings. The additional time required to arrive at a consensus rating, a rating generated following discussion between 2 raters, may not be warranted. Further research is needed to determine whether these findings hold true with other raters and treatment adherence scales.     

INTERVIEW NOTES AND VALIDITY

Three studies were conducted investigating the effects of notetaking behavior and the content of notes on validity in a selection interview. Overall, results indicate that when notetaking was voluntary, notetakers made more valid ratings than non-notetakers. But, when note-taking was manipulated non-notetakers made ratings that were just as valid as raters told to take behavioral notes, and actually more valid than those raters instructed to take general notes. With regard to the content of notes, behavioral-type notes were found to contribute to greater validity. In addition, in the first and third study, procedural type notes resulted in lower validity. These results may have important implications for interviewer training for notetaking.     

Sex effects in evaluating applicant qualifications: A reexamination

It has been hypothesized that female applicants for male sex-typed jobs are evaluated using different selection criteria than males for the same jobs (E. A. Cecil, R. J. Paul, and R. A. Olins, Perceived Importance of Selected Variables Used to Evaluate Male and Female Job Applicants, Personnel Psychology, 1973, 26, 397–404). The present study examined this hypothesis in two separate experiments, in an attempt to replicate the findings and conclusions of Cecil et al. Results do not support the hypothesis that applicant sex affects the ratings of importance of applicant qualifications for sex-typed jobs, although raters do appear to stereotype the kinds of jobs for which males and females are likely to apply. Type of job for which applicants apply is a stronger determinant of what qualifications are evaluated as important in an interview situation. Use of selection criteria (e.g., tests) that validly predict later job performance and that are standardized for all job applicants, might prevent sex discrimination more than training raters to be wary of sex stereotyping of required applicant qualifications.The authors express their gratitude to Amanda Patacsil for her assistance in data collection and analysis.