Incremental Validity of Interactive Multimedia Simulations in Two Organizations

Interactive multimedia simulations are conceptually distinct from other simulations commonly used as selection tools, such as assessment centers and situational judgment tests, and represent a potentially cost‐effective and job‐related candidate assessment tool. This study investigated the predictive validity of these simulations with a combined sample of call center employees from two organizations. Results indicate that customized interactive multimedia simulations demonstrate substantial criterion‐related validity and significant incremental validity over other noncognitive measures such as biodata and personality.     

CONSTRUCTING PARALLEL SIMULATION EXERCISES FOR ASSESSMENT CENTERS AND OTHER FORMS OF BEHAVIORAL ASSESSMENT

Assessment centers rely on multiple, carefully constructed behavioral simulation exercises to measure individuals on multiple performance dimensions. Although methods for establishing parallelism among alternate forms of paper-and-pencil tests have been well researched (i.e., to equate tests on difficulty such that the scores can be compared), little research has considered the why and how of parallel simulation exercises. This paper extends established procedures for constructing parallel test forms to dimension-based behavioral simulations. We discuss reasons for establishing comparable, alternate simulation forms and discuss the issues raised when applying traditional procedures to simulation exercises. After proposing a set of guidelines for establishing alternate forms among simulations, we apply these guidelines to simulations used in an operational assessment center.     

Transparency of Assessment Centers: Lower Criterion‐related Validity but Greater Opportunity to Perform?

Assessment centers (ACs) are popular selection devices in which assessees are assessed on several dimensions during different exercises. Surveys indicate that ACs vary with regard to the transparency of the targeted dimensions and that the number of transparent ACs has increased during recent years. Furthermore, research on this design feature has put conceptual arguments forward regarding the effects of transparency on criterion‐related validity, impression management, and fairness perceptions. This study is the first to examine these effects using supervisor‐rated job performance data as the criterion. We conducted simulated ACs with transparency as a between‐subjects factor. The sample consisted of part‐time employed participants who would soon be applying for a new job. In line with our hypothesis, results showed that ratings from an AC with nontransparent dimensions were more criterion valid than ratings from an AC with transparent dimensions. Concerning impression management, our results supported the hypothesis that transparency moderates the relationship between self‐promotion and job performance, such that self‐promotion in the nontransparent AC was more positively related to job performance than self‐promotion in the transparent AC. The data lent no support for the hypothesis that participants’ perceptions of their opportunity to perform are higher in the transparent AC.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.     

Constructs and developmental assessment centers: Further troubling empirical findings

Research indicates that assessment center (AC) ratings typically demonstrate poor construct validity; that is, they do not measure the intended dimensions of managerial performance (e.g., Sackett & Harris, 1988). The purpose of this study was to investigate the construct validity of dimension ratings from a developmental assessment center (N=102), using multitrait-multimethod analysis and factor analysis. The relationships between AC ratings, job performance ratings, and personality measures also were investigated. Results indicate that the AC ratings failed to demonstrate construct validity. The ratings did not show the expected relationships with the job performance and personality measures. Additionally, the factors underlying these ratings were found to be the AC exercises, rather than the managerial dimensions as expected. Potentially, this lack of construct validity of the dimension ratings is a serious problem for a developmental assessment center. There is little evidence that the managerial weaknesses identified by the AC are the dimensions that actually need to be improved on the job. Methods are discussed for improving the construct validity of AC ratings, for example, by decreasing the cognitive demands on the assessors.This study is based on a dissertation submitted to North Carolina State University. Portions of this paper were presented at the meeting of the Society for Industrial and Organizational Psychology in Montreal, Quebec, May, 1992. I am grateful to Paul Thayer, Bert Westbrook, James W. Cunningham, and Patrick Hauenstein for their contributions to this research. I also thank several anonymous reviewers for their comments on this article.     

Self-monitoring and Sex As Related to Assessment Center Rating and Job Performance

Snyder's Self-Monitoring Test was included in a 1-day assessment center for computer salespersons. It was predicted that self-monitoring would be related to success at the assessment center (where impression management is crucial for employment) and to success in the boundary-spanning role of computer sales. The relationships with self-monitoring were predicted to be stronger for women than for men because these computer sales jobs have been less traditional for women than for men. Results indicate that there were significant correlations between self-monitoring scores and the overall assessment rating only for women. Furthermore, self-monitoring was significantly related to job retention after 1 year only for women, but self-monitoring scores predicted job retention as well as did the assessment center ratings for both men and women. It was suggested that impression management ability (self-monitoring) is more strongly related to job auditions and job retention when the role is nontraditional to gender.     

Editorial overview: Analytic and methodological innovations for suicide‐focused research

This editorial overview provides an introduction to the Suicide and Life‐Threatening Behaviors Special Issue: “Analytic and Methodological Innovations for Suicide‐Focused Research.” We outline several challenges faced by modern suicidologists, such as the need to integrate different analytical and methodological techniques from other fields with the unique data problems in suicide research. Therefore, the overall aim of this issue was to provide up‐to‐date methodological and analytical guidelines, recommendations, and considerations when conducting suicide‐focused research. The articles herein present this information in an accessible way for researchers/clinicians and do not require a comprehensive background in quantitative methods. We introduce the topics covered in this special issue, which include how to conduct power analyses using simulations, work with large data sets, use experimental therapeutics, and choose covariates, as well as open science considerations, decision‐making models, ordinal regression, machine learning, network analysis, and measurement considerations. Many of the topics covered in this issue provide step‐by‐step walkthroughs using worked examples with the accompanied code in free statistical programs (i.e., R). It is our hope that these articles provide suicidologists with valuable information and strategies that can help overcome some of the past limitations of suicide research, and improve the methodological rigor of our field.     

A SURVEY OF ASSESSMENT CENTER PRACTICES IN ORGANIZATIONS IN THE UNITED STATES

Two hundred fifteen organizations in the United States provided information about multiple aspects of their assessment centers, including design, usage, and their adherence to professional guidelines and research-based suggestions for the use of this method. Results reveal that centers are usually conducted for selection, promotion, and development purposes. Supervisor recommendation plays a sizable role in choosing center participants. Most often, line managers act as assessors; they typically arrive at participant ratings through a consensus process. In general, respondents indicate close adherence to recommendations for center design and assessor training. Recommendations involving other practices (e.g., informing participants, evaluating assessors, validating center results) are frequently not followed. Furthermore, methods thought to improve predictive validity of center ratings are underutilized. Variability in center practices according to industry and center purpose was revealed. We encourage practitioners to follow recommendations for center usage, and researchers to work to better understand moderators of center validity.     

The Role of Job Relatedness and Self‐efficacy in Applicant Perceptions of Fairness in a High‐stakes Selection Setting

This article presents results from two samples of applicants (total N = 368) for general practitioner posts in the United Kingdom. The roles of job relatedness and self‐efficacy in fairness perceptions were explored, with data gathered at two time points: immediately after testing and one month later following outcome (pass/fail) feedback. Overall, results indicated that in two samples, job relatedness perceptions measured at the time of testing predicted fairness perceptions measured following outcome feedback. In addition, the stage in the selection process (shortlisting vs. assessment center) was important in determining the extent to which job relatedness perceptions predicted fairness. Findings also suggest that self‐efficacy may be a predictor, rather than an outcome variable, in applicant fairness perceptions in this high‐stakes setting. Results are discussed in relation to their practical and theoretical implications.     

Assessing Individual Differences in Experiential (Intuitive) and Rational (Analytical) Cognitive Styles

Even though it is argued that intuition has ‘come of age’ in the behavioral sciences, there are still unresolved issues regarding self‐report assessment of intuitive and analytical styles of information processing (cognitive styles). The unitary view proposes that intuition and analysis are opposite ends of a single continuum. The dual view proposes that intuition and analysis are independent (orthogonal) constructs. Moreover, within the dual view, it has been proposed that intuition and analysis can be further subdivided into ability and engagement subcomponents. The aim of this article was to test both of these claims and thereby move discussions regarding the assessment of cognitive styles further forward. This research is important given that much intuition research in organizations is predicated on self‐report methods of assessment. Using data from a sample of police officers and police staff from a large police organization in the United Kingdom, we found that the adoption of a dual (rather than unitary) perspective when assessing experiential and rational cognitive styles is warranted, whereas adopting an ability and engagement refinement is not (hence a simpler formulation is to be preferred). We also observed a number of main effects and interactions with respect to job type, job level, gender, and experience. We offer guidelines for the self‐report assessment of intuition and analysis cognitive styles and discuss a typology of styles. The article concludes by outlining a number of practical implications for cognitive styles assessment in organizational settings.     

Navigating the Black Hole: Explicating Layers of Job Search Context and Adaptational Responses

Via a qualitative study, we introduce and elucidate 5 layers of context‐related job search demands (omnibus, organizational, social, task, and personal) that are encountered by both employed and unemployed job seekers. We develop a process model to portray the mechanisms (managing mood and motivation, feedback/help seeking, and self‐reflection/learning) through which these context‐related demands are related to several important job search outcomes. We provide new insight into employed job seekers, showing that they report the job search as full of difficulties, obstacles, and challenges, and that some of these demands parallel those that unemployed job seekers face.     

Disregarding Preliminary Information When Rating Job Applicants' Performance: Mission Impossible?1

Firms allocate considerable resources to test job applicants' skills and to ensure that hiring decisions are nondiscriminatory. Interpreting selection output in light of preliminary information may undermine the impartiality of personnel selection decisions. In this study, human resource managers were presented preliminary information about a candidate's performance and asked to rate him according to a detailed assessment‐center report of his performance. We examined several interventions for reducing reliance on preliminary information: retrieving assessment‐center information, generating a rating model, or both. Participants (N = 167) excluded preliminary information from the interventions, but relied on it when rating the candidate. The documented failure to control for the effects of preliminary information on subsequent judgments can contribute to improved selection procedures.     

Unanswered questions in assessment center research

This study compared the predictive validity of two types of overall assessment center ratings (clinical and mechanical) and two types of performance ratings (supervisor and subordinate). Data were gathered on approximately 50 law enforcement agency managers over a four-year time span. Results indicated that clinical predictions were not significantly better than mechanical ones and that prior performance ratings were better predictors of future performance ratings than were assessment centers. Further, dimension ratings within the assessment center were found to be unrelated to ratings of the same dimensions on the job four years later, supporting the notion of subtle criterion contamination (Klimoski & Strickland, 1977) and raising questions about the central assumptions underlying the assessment center approach.     

Using Emotional Intelligence to help College Students Succeed in the Workplace

Research related to the characteristics that are sought by recruiters and prospective employers in graduating college students suggests that emotional intelligence (EI) skills are as important as, if not more important than, job‐related skills. Counselors working with college students, however, usually focus on career management and job search skills and neglect the development of EI skills. EI seems to be an excellent framework to use in helping college students find a job and succeed in the workplace. By using the Confluence Counseling model, counselors working with college students can effectively combine career counseling with personal counseling in a holistic manner.     

Teaching social‐learning procedures to paraprofessionals working with individuals with severe mental illness in a maximum‐security forensic hospital

This study describes an integrated technical method for teaching learning‐based interventions to paraprofessionals working in a maximum‐security psychiatric facility as a social‐learning program was implemented. This training approach combines brief didactic instruction with live demonstration and guided practice in specific procedures. Comparisons between baseline and post‐training functioning of staff were made using ongoing direct observational assessment data. Results revealed that following training and program implementation staff were much more active overall, interacted with clients far more often, and engaged in much lower rates of job‐irrelevant activity. Moreover, following training direct observational data of the patterns of staff–client interactions showed that staff were much more likely to demonstrate interactions reflective of learning‐based interventions than at baseline. These results have important implications for clinicians and administrators attempting to implement behavioral rehabilitation programs. Copyright © 2001 John Wiley & Sons, Ltd.     

ASSESSMENT CENTERS AND CONTENT VALIDITY: SOME NEGLECTED ISSUES

Literature on the use of content validity to establish the job-relatedness of assessment centers is reviewed. Most approaches to content validity are shown to focus on job-analysis-based exercise construction and dimension selection as the bases for determining the content validity of an assessment center. The article attempts to show that content validity requires more than the careful construction of stimulus materials (i.e., exercises); how these stimulus materials are presented to candidates and how responses to these stimuli are evaluated are also critical considerations in making judgments about content validity. Variability in the way these issues are dealt with is documented.     

IMPACT OF REFLECTIVE SUPERVISION ON EARLY CHILDHOOD INTERVENTIONISTS’ PERCEPTIONS OF SELF‐EFFICACY,JOB SATISFACTION,AND JOB STRESS

Despite widespread recognition of the importance of reflective practice in the field of infant mental health, little quantitative research exists regarding the impact of reflective supervision on professionals’ self‐efficacy, job satisfaction, and work‐related stress. Thirty‐three early childhood interventionists received approximately 9 months of reflective supervision and completed pre‐ and post‐assessments of their job‐related stress and self‐efficacy using a modified version of the Reflective Supervision Self‐Efficacy Scales for Supervisees (S. Shea, S. Goldberg, & D.J. Weatherston, 2012). At the post‐assessment, participants also rated their overall experience of reflective supervision. Reported self‐efficacy increased significantly from pre‐ to post‐assessment, and participants reported a positive overall impact of reflective supervision with respect to a variety of indices, such as job satisfaction, professional development, and the ability to cope with job‐related stress. These results provide necessary quantitative data demonstrating a positive impact of reflective supervision on early childhood professionals and suggest the value of reflective supervision for supporting workforce development.