VIDEO-BASED SITUATIONAL TESTING

In two separate studies, video-based tests of situational judgment were developed and validated against measures of performance for hourly service workers. In the first study, 684 employees were used to develop a test of retail associate judgment and 787 newly hired employees were used to cross-validate the instrument. In the second study, 412 current employees were used to develop a test of nursing home caregiver judgment and 148 newly hired caregivers were used to cross-validate this video-based test. In both studies, responses to video-based situational vignettes were empirically keyed against supervisory ratings of performance. The resulting keys produced uncorrected cross-validities in the low .20s. The video-based test scores were also found to be related to measures of cognitive ability and, to a lesser extent, experience. These results suggest that cognitive ability and possibly experience account for some but not all of the predictiveness of video-based situational tests. Video-based situational tests demonstrated score differences between Whites and non-Whites of roughly one-half a standard deviation, indicating that the use of such tests could produce adverse impact against non-Whites. In the second study, customer preferences regarding desired behavior were also used to develop a rational scoring key. This customer driven key was significantly related to performance ( r = .33) in the cross-validation sample, demonstrating that customers' preferences can provide the basis for developing valid predictors. Potential benefits of video-based situational tests in selection and directions for future research are discussed.     

FURTHER STUDIES OF SITUATIONAL TESTS

In the present paper, results are reported for two different situational judgment tests (SJT) used in validation studies with almost 4,000 employees in 7 different organizations. Across the 2 studies, it was shown that situational test scores were significantly related to performance (weighted average r = .19), cognitive ability (weighted average r = .45), and experience (weighted average r = .20). In one study, there was a slight tendency for experience and cognitive ability to interact in the prediction of situational judgment, such that cognitive ability became less predictive as experience increased. Situational judgment fully mediated the effects of cognitive ability in one study, but not in the other. Finally, SJT race effect sizes were consistent with past research and, while not trivial, smaller than those typically observed for cognitive ability tests. The studies yielded very similar results despite the use of different keying methods (empirical vs. subject matter expert opinion). The preponderance of the evidence indicates that situational judgment measures mediate avariety of job relevant skills. Limitations to these studies and directions for future research are reviewed.     

THE EFFECTS OF INTERRATER AND SELF-OTHER AGREEMENT ON PERFORMANCE IMPROVEMENT FOLLOWING UPWARD FEEDBACK

This study investigated (a) whether performance improvement following upward feedback is related to self-other rating discrepancies, (b) how self-ratings change after feedback, and (c) whether agreement among raters influences performance improvement. Self-ratings and subordinate ratings were collected from 1,888 managers at 2 points in time 1 year apart. Using polynomial regression equations, we determined that managers who overrated themselves relative to how others rated them tended to improve their performance from 1 year to the next, and underraters tended to decline. This is consistent with what would be predicted by self-consistency theory (Korman, 1976). Self-ratings tended to decrease for overraters and increase for underraters, but this effect was not constant throughout the range of self-ratings. Agreement among subordinate raters was negatively related to performance improvement, but this is likely a reflection of differential regression to the mean.     

METHODOLOGICAL AND PSYCHOMETRIC ISSUES IN SETTING CUTOFF SCORES USING THE ANGOFF METHOD

The present study examined several psychometric issues relevant to the use of a favored technique (the Angoff method) used to set standards in criterion referenced testing. The research was conducted within a setting which allowed (a) confident identification of expert and non-expert judges, and (b) estimation of "true" scores for items judged so that accuracy of judgments in addition to reliability could be examined. Results suggested that expertise of judges does make a difference in producing more accurate and reliable data, underscoring the importance of using true subject matter experts (SMEs) in the judgment process. A rater analysis technique (rater-total correlations) was illustrated, which might prove useful in improving the quality of data obtained using the Angoff method, particularly when there is some question regarding the internal consistency of ratings and expertise of some of the raters. Finally, a rater accuracy adjustment/calibration technique was examined and proved to be a potentially useful method to maximize accuracy of a standard derived using the Angoff method in settings where archival normative test data can be obtained. Other methods that could potentially be used to improve Angoff data were discussed.     

Exploring the Effects of the Naturalistic Fallacy: Evidence That Genetic Explanations Increase the Acceptability of Killing and Male Promiscuity

The naturalistic fallacy is the erroneous belief that what is natural is morally acceptable. Two studies assessed whether people commit the naturalistic fallacy by testing whether genetic explanations for killing and male promiscuity, as compared to experiential explanations (i.e., learning/“nurture” explanations) increase acceptance of these behaviors. In Study 1, participants who read a genetic explanation for why people kill bugs viewed bug killing as more morally acceptable than participants who read an experiential explanation, although they did not reliably kill more bugs. In Study 2, men who read a genetic explanation for why men are more promiscuous than women reported decreased interest in long‐term romantic commitment compared with men who read experiential explanations and women who read either explanation.     

VALIDATING SYNTHETIC VALIDATION: COMPARING TRADITIONAL AND SYNTHETIC VALIDITY COEFFICIENTS

We describe a unique application of a synthetic validation technique to a selection system development project in a large organization. Job analysis data were collected from 4,725 job incumbents and 619 supervisors, and were used to identify 11 job families and 27 job components. We developed 12 tests to predict performance on these job components and conducted a concurrent validation study to collect test and job component data for 1,926 incumbents. We created a test composite for each job component and then chose a test battery for each job family based on its relevant job components. Synthetic validity coefficients were computed for each test battery and compared to traditional validity coefficients that were computed within job families with large sample sizes. The synthetic validity coefficient was very close to the within-family validity coefficient for most job families and was within the bounds of sampling error for all job families. Validities tended to be highest when predictors were weighted according to the number of job components to which they were relevant and job component criterion measures were unit weighted.     

COGNITIVE DISABILITY AND COGNITIVE ENHANCEMENT

Abstract: This essay explores problems of consistency among commonsense beliefs about the comparative moral status of animals, fetuses, and human beings congenitally endowed with cognitive capacities and potential no higher than those of higher animals. The possibility of genetic cognitive enhancement exacerbates some of these problems, but also offers new resources for understanding the basis of our moral status as inviolable.