Alternative Predictors for Dealing with the Diversity–Validity Dilemma in Personnel Selection: The constructed response multimedia test

In the context of the diversity–validity dilemma in personnel selection, the present field study compared ethnic subgroup differences on an innovative constructed response multimedia test to other commonly used selection instruments. Applicants (N = 245, 27% ethnic minorities) for entry‐level police jobs completed a constructed response multimedia test, cognitive ability test, language proficiency test, personality inventory, structured interview, and role play. Results demonstrated minor ethnic subgroup differences on constructed response multimedia test scores as compared to other instruments. Constructed response multimedia test scores were related to the selection decision, and no evidence for predictive bias was found. Subgroup differences were also examined on the dimensional level, with cognitively loaded dimension scores displaying larger differences.     

Differences between multimedia and text-based assessments of emotion management: An exploration with the multimedia emotion management assessment (MEMA)

People process emotional information using visual, vocal, and verbal cues. However, emotion management is typically assessed with text based rather than multimedia stimuli. This study (N?=?427) presents the new multimedia emotion management assessment (MEMA) and compares it to the text-based assessment of emotion management used in the MSCEIT. The text-based and multimedia assessment showed similar levels of cognitive saturation and similar prediction of relevant criteria. Results demonstrate that the MEMA scores have equivalent evidence of validity to the text-based MSCEIT test scores, demonstrating that multimedia assessment of emotion management is viable. Furthermore, our results inform the debate as to whether cognitive saturation in emotional intelligence (EI) measures represents “noise” or “substance”. We find that cognitive ability associations with EI represent substantive variance rather than construct-irrelevant shared variance due to reading comprehension ability required for text-based items.     

What do we know about how best to assess counsellor competence?

Issues of assessment of counsellor competence are becoming increasingly important as counselling becomes more professionalized and recognizable as a distinct profession. There is, however, a lack of systematic research into competence assessment in this field. The existing literature is discussed in terms of the issues involved in employing different sources of assessment, such as trainers, tutors, managers, supervisors, external assessors, clients, self and peers. Techniques of assessment are reviewed, including questionnaires, video and audiotapes, role play, examinations, learning diaries and computer simulations. Some suggestions are offered for increasing the validity of assessment judgements.     

In‐basket Validity: A systematic review

In‐baskets are high‐fidelity simulations often used to predict performance in a variety of jobs including law enforcement, clerical, and managerial occupations. They measure constructs not typically assessed by other simulations (e.g., administrative and managerial skills, and procedural and declarative job knowledge). We compiled the largest known database (k = 31; N = 3,958) to address the criterion‐related validity of in‐baskets and possible moderators. Moderators included features of the in‐basket: content (generic vs. job specific) and scoring approach (objective vs. subjective) and features of the validity studies: design (concurrent vs. predictive) and source (published vs. unpublished). Sensitivity analyses assessed how robust the results were to the influence of various biases. Results showed that the operational criterion‐related validity of in‐baskets was sufficiently high to justify their use in high‐stakes settings. Moderator analyses provided useful guidance for developers and users regarding content and scoring.     

Incremental validity in the psychological assessment of children and adolescents

Incremental validity in the process of psychological assessment of children and adolescents is explored. The authors highlight the dependence of the incremental validity of assessment information on factors such as goal of assessment, other information available, base rate of the problem or outcome, age or gender of the child, and type of problem being assessed. The authors discuss the incremental validity of assessment information from alternate sources, methods, and constructs. In view of the limited number of studies directly relevant to incremental validity in child clinical assessments, the authors call for more clinically relevant research. To have the greatest impact on child and adolescent services, this research must be readily generalized and immediately relevant to actual clinical practice.     

Classical and Modern Methods of Psychological Scale Construction

Scale construction is a growth enterprise in the psychological literature. Unfortunately, many measures promise much but are severely limited by the inadequacies of their conceptualization and execution. In this paper, a model for developing psychological scales is presented that is rooted in the traditions of construct validity and classical test theory but informed by modern psychometric methods. Construct validity is conceptualized as a guiding principle in each of three phases of scale development, focused on (i) construct conceptualization and development of the initial item pool, (ii) item selection and structural validity, and (iii) assessment of external validity vis‐à‐vis other measures and relevant nontest criteria.     

Validity and Adverse Impact Potential of Predictor Composite Formation

Previous research on the validity and adverse impact (AI) of predictor composite formation focused on the merits of regression‐based or ad hoc composites. We argue for a broader focus. Ad hoc chosen composites are usually not Pareto‐optimal, whereas the regression‐based composite represents only one element from the total set of Pareto‐optimal composites and can, therefore, provide only limited information on the potential for validity and AI reduction of forming predictor composites when both validity and AI are of concern. In that case, other Pareto‐optimal composites may provide a better benchmark to decide on the merits of the predictor composite formation. We summarize a method to determine the set of Pareto‐optimal composites and apply the method to a representative collection of selection predictors. The application shows that the assessment of the AI and validity of predictor composite formation can differ substantially from the one arrived at when considering only regression‐based composites.     

A replication of the response‐restriction preference assessment

We replicated the response‐restriction (RR) preference assessment and compared results in terms of preference hierarchies to those from free‐operant and multiple stimulus without replacement (MSWO) formats with six children with autism spectrum disorders (ASDs). We also assessed social validity of each format with teachers and clinicians who work with children with ASDs. Complete hierarchies were produced in four of 18 assessments and with MSWO and RR formats only. Results of the social validity assessment varied across raters, with each preference assessment format receiving the highest rating from at least one rater. Results are discussed in terms of practical recommendations and relative to the preference assessment literature as a whole.     

Psychometric evaluation and normative data for a Swedish version of the Patient Health Questionnaire 15‐Item Somatic Symptom Severity Scale

The lack of an available Swedish version of the Patient Health Questionnaire 15‐Item Somatic Symptom Severity Scale (PHQ‐15) motivated the present psychometric evaluation of such a version as well as providing normative data for the PHQ‐15. Data from 3,406 individuals who took part in the Västerbotten Environmental Health Study in Sweden were used. The respondents constitute a random sample, aged 18 to 79 years, stratified for age and sex. They responded to a Swedish translation of the PHQ‐15 as well as the Hospital Anxiety and Depression Scale, the Perceived Stress Scale, and the Shirom Melamed Burnout Questionnaire for assessment of convergent validity. The distribution of PHQ‐15 scores was positively skewed and mesokurtic in shape, and the internal consistency of the PHQ‐15 was satisfactory. Correlation coefficients between PHQ‐15 score and the measures of anxiety, depression, stress and mental/physical exhaustion indicate satisfactory validity. Normative data for PHQ‐15 scores as well as for categories of somatic symptom severity are provided. The favorable psychometric properties of the Swedish version of the PHQ‐15 suggest use of this instrument for quantification of somatization in Swedish and similar populations, and has the advantage of available normative data.     

An Examination of the Validity and Incremental Value of Needed‐at‐Entry Ratings for a Customer Service Job

We examined needed‐at‐entry ratings of knowledge, skills, abilities, and other characteristics (KSAOs) provided by job experts (N= 285) during a job analysis for a customer service manager position. To serve as an external referent, a group of industrial and organisational psychologists (N= 31) rated the perceived trainability of each KSAO. Analyses revealed only limited support for the validity of inferences drawn from job experts' needed‐at‐entry ratings. Consistent with our hypotheses, less validity evidence was found for ratings of the more person‐oriented and abstract “AO” attributes than for the more job‐oriented and concrete “KS” attributes. In general, job experts tended to rate attributes as needed‐at‐entry that psychologists thought could be developed on the job. We also found that failing to collect needed‐at‐entry ratings, and relying on only the more common importance‐to‐the‐job ratings, would have resulted in a rather different set of critical KSAOs identified for assessment during the selection process.     

TOWARD BETTER META‐ANALYTIC MATRICES: HOW INPUT VALUES CAN AFFECT RESEARCH CONCLUSIONS IN HUMAN RESOURCE MANAGEMENT SIMULATIONS

Simulations and analyses based on meta‐analytic matrices are fairly common in human resource management and organizational behavior research, particularly in staffing research. Unfortunately, the meta‐analytic values estimates for validity and group differences (i.e., ρ and δ, respectively) used in such matrices often vary in the extent to which they are affected by artifacts and how accurately the values capture the underlying constructs and the appropriate population. We investigate how such concerns might influence conclusions concerning key issues such as prediction of job performance and adverse impact of selection procedures, as well as noting wider applications of these issues. We also start the process of building a better matrix upon which to base many such simulations and analyses in staffing research. Finally, we offer guidelines to help researchers/practitioners better model human resources processes, and we suggest ways that researchers in a variety of areas can better assemble meta‐analytic matrices.     

Use of static and dynamic media to simulate environments: a meta-analysis

This article is a review of studies on how strongly responses from static media (such as color slides) and dynamic media (such as virtual reality models) compared to onsite evaluations or to each other. Eighty-four empirical findings were found. 6,323 participants and 967 environments were included in the review. The overall correlation of subjective responses obtained either onsite or by viewing static color simulations was r = .86. The overall correlation of subjective responses obtained either onsite or by viewing dynamic simulations was r = .83. The overall correlation of subjective responses obtained either from static or dynamic media was r = .82. It is suggested that both types of simulation generate statistically equivalent results and so a choice of simulation media should be based on efficacy rather than concerns about validity.     

Predicting Training Success with General Mental Ability,Specific Ability Tests,and (Un)Structured Interviews: A meta‐analysis with unique samples

Several meta‐analyses combine an extensive amount of research concerned with predicting training success. General mental ability is regarded as the best predictor with specific abilities or tests explaining little additional variance. However, only few studies measured all predictors within one sample. Thus, intercorrelations were often estimated based on other studies. Moreover, new methods for correcting range restriction are now available. The present meta‐analyses used samples derived from a German company in which applicants for different apprenticeships were tested with an intelligence structure test, specific ability tests as well as a structured and an unstructured interview. Therefore, intercorrelations between different assessment tools did not have to be estimated from other data. Results in the final examination, taking place at least 2 years after the original assessment, served as criterion variable. The dominant role of general mental ability was confirmed. However, specific abilities were identified that can be used as valuable additions. Job complexity moderated some of the relationships. Structured interviews were found to have good incremental validity over and above general mental ability. Unstructured interviews, on the other hand, performed poorly. Practical implications are discussed.     

Reliability and Validity of the Vicarious Trauma Organizational Readiness Guide (VT‐ORG)

The Vicarious Trauma Organizational Readiness Guide (VT‐ORG) is an assessment of an organization's readiness to address vicarious trauma (VT), which is exposure to the traumatic experiences of people served. This study reports on the psychometric properties of the VT‐ORG. Employees of first responder agencies (e.g., law enforcement, fire, emergency services) and victim assistance agencies are at a high risk for vicarious traumatization, which can lead to PTSD, substance use, and suicidal ideation, among other negative impacts. Organizations that do not address VT may see such effects as employee turnover, reduced efficiency, and negative work environments. The VT‐ORG is an assessment tool designed to help organizations complete the first step of organizational change—conducting a needs assessment. Study participants were 3,018 employees across 13 first responder and victim assistance agencies who completed the 67‐item VT‐ORG and additional measures for evaluation of its validity and reliability. The VT‐ORG was found to have excellent internal consistency (Cronbach's α = .98). A structural equation model demonstrated that the subscales of the VT‐ORG predicted criterion measures of turnover intention, compassion satisfaction, and organizational resilience, with an overall model fit of CFI = .99, RMSEA = .053. This study found the VT‐ORG to be a reliable and valid assessment of organizational responses to vicarious trauma.     

Designing Selection Systems for Medicine: The importance of balancing predictive and political validity in high‐stakes selection contexts

Selection into medical education and training is a high‐stakes process. A key unanswered issue is the effectiveness of measuring noncognitive predictors via both low‐fidelity and high‐fidelity selection approaches in this high‐stakes context. We review studies investigating the effectiveness of multiple selection instruments in terms of predictive validity, incremental validity, and applicant reactions in both entry‐level and advanced‐level medical selection. Our results show that the situational judgment test (SJT) is the best single predictor of performance, operationalized in multiple ways. In addition, the low‐fidelity SJT has incremental predictive power over cognitively oriented tests, and high‐fidelity assessment center (AC) exercises add incremental validity over the low‐fidelity (and less costly) selection methods. Concerning applicant reactions, results show that overall, the selection system is positively received. However, the method with the highest predictive validity – the SJT – received comparatively lower face validity ratings which may present a ‘justice dilemma’ for employers. Furthermore, various other stakeholders have a political interest in the selection methods used (e.g., government, the regulators and trade unions).     

The efficiency of multimedia learning into old age

Background: On the basis of a multimodal model of working memory, cognitive load theory predicts that a multimedia‐based instructional format leads to a better acquisition of complex subject matter than a purely visual instructional format. Aims: This study investigated the extent to which age and instructional format had an impact on training efficiency among both young and old adults. It was hypothesised that studying worked examples that are presented as a narrated animation (multimedia condition) is a more efficient means of complex skill training than studying visually presented worked examples (unimodal condition) and solving conventional problems. Furthermore, it was hypothesised that multimedia‐based worked examples are especially helpful for elderly learners, who have to deal with a general decline of working‐memory resources, because they address both mode‐specific working‐memory stores. Sample: The sample consisted of 60 young (mean age = 15.98 years) and 60 old adults (mean age = 64.48 years). Methods: Participants of both age groups were trained in either a conventional, a unimodal, or a multimedia condition. Subsequently, they had to solve a series of test problems. Dependent variables were perceived cognitive load during the training, performance on the test, and efficiency in terms of the ratio between these two variables. Results: Results showed that for both age groups multimedia‐based worked examples were more efficient than the other training formats in that less cognitive load led to at least an equal performance level. Conclusion: Although no difference in the beneficial effect of multimedia learning was found between the age groups, multimedia‐based instructions seem promising for the elderly.     

Comparing personality disorder models: cross-method assessment of the FFM and DSM-IV-TR

The current edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV-TR; American Psychiatric Association, 2000) defines personality disorders as categorical entities that are distinct from each other and from normal personality traits. However, many scientists now believe that personality disorders are best conceptualized using a dimensional model of traits that span normal and abnormal personality, such as the Five-Factor Model (FFM). However, if the FFM or any dimensional model is to be considered as a credible alternative to the current model, it must first demonstrate an increment in the validity of the assessment offered within a clinical setting. Thus, the current study extended previous research by comparing the convergent and discriminant validity of the current DSM-IV-TR model to the FFM across four assessment methodologies. Eighty-eight individuals receiving ongoing psychotherapy were assessed for the FFM and the DSM-IV-TR personality disorders using self-report, informant report, structured interview, and therapist ratings. The results indicated that the FFM had an appreciable advantage over the DSM-IV-TR in terms of discriminant validity and, at the domain level, convergent validity. Implications of the findings and directions for future research are discussed.     

履历数据测评的效度分析

履历数据是人员测评的重要手段, 但其组织情景性限制使得国外同类工具不能适用于国内人事选拔, 国内也缺乏实证研究验证其信度、效度。本研究结合某通信企业的招募选拔工作, 开发履历数据分析工具, 选取250名应聘者的履历数据、一般认知能力、大五个性和面试结果数据, 通过效标关联效度、增量效度思想, 分析履历数据有效性。结果表明, 以面试结果为效标, 履历数据具有良好的效标关联效度, 与其他测评工具组合使用时有良好的增量效度。     

Consensual Assessment of Creativity in Teaching Design by Supportive Peers—Its Validity,Practicality, and Benefit

Creativity in teaching is a significant and complex construct. However, in the local educational context, creativity in teaching has received little attention. This study aimed to investigate the validity, practicality, and benefits of applying a modified consensual assessment technique (CAT) to assess creativity in teaching design. Four hundred and eighty‐five written teaching designs were collected from 167 in‐service and pre‐service primary school teachers in Hong Kong. Instead of expert teachers, “supportive” peers, who had shown support, interest, and initiative in creative teaching were recruited as judges. A warm‐up exercise, with no definition of creativity, was given to the judges before beginning their assessments. The results indicated overall consistency in the judges' assessments of creativity, and that creativity factor could be distinguished from pedagogical skills and other technical factors. Most of the peer judges reported personal gains in creative teaching by engaging in the assessment process. On average, each judge spent approximately only 2 minutes rating each written teaching design. The findings confirm that the modified CAT is a valid and economical assessment method with learning benefits for the judges. The special values and implications of using supportive peer judges in consensual assessment are further discussed.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.