Employment Interview Reliability: New meta‐analytic estimates by structure and format

This study sought to provide an update on evidence regarding the interrater reliability of employment interviews. Using a final dataset of 125 coefficients with a total sample size of 32,428, our results highlight the importance of taking all three sources of measurement error (random response, transient, and conspect) into account. For instance, the mean interrater reliability was considerably higher for panel interviews than for separate interviews conducted by different interviewers (.74 vs. .44). A strong implication of our findings is that interview professionals should not base perceptions of the psychometric properties of their interview process on interrater estimates that do not include all three sources. A number of directions for future research were identified, including the influence of cues in medium structure panel interviews (e.g., changes in tone or pitch) and the lower than expected reliability for highly structured interviews conducted separately by different interviewers.     

Feasibility of a Simplified and Dynamic Bayesian System for Use in Structured Diagnostic Interviews

An important goal of clinical assessment is to balance cost-effectiveness, administration demands, and accuracy (G. Young, J. O'Brien, E. Gutterman, & P. Cohen, 1987). The incorporation of Bayesian logic into diagnostic interviewing may assist with this goal, but in previous examinations, such methods have been prohibitively complex. In this study, analysis of a simplified Bayesian system showed overall classification error rates as good or better than traditional structured interviewing, and reduction in error was positively related to the psychometric properties of the predictor used in the actuarial functions. A dynamic system using simplified Bayesian logic appears to function well in the context of a structured interview and requires comparatively less data than previously tested Bayesian approaches. This type of system appears suitable for further research with clinical populations to determine its performance in applied settings.     

STRUCTURED INTERVIEWING: RAISING THE PSYCHOMETRIC PROPERTIES OF THE EMPLOYMENT INTERVIEW

A highly structured employment interviewing technique is proposed, which includes the following steps: (1) develop questions based on a job analysis, (2) ask the same questions of each candidate, (3) anchor the rating scales for scoring answers with examples and illustrations, (4) have an interview panel record and rate answers, (5) consistently administer the process to all candidates, and (6) give special attention to job relatedness, fairness, and documentation in accordance with testing guidelines. Examination of psychometric properties for hiring entry-level production employees ( n = 149) reveals high interrater reliability ( r = .88) and predictive validity (uncorrected r = .34, corrected r = .56), as well as evidence for test fairness and utility. The levels of these properties are comparable to those of a comparison battery of typical employment tests, and correlations with the tests suggest that the interview has a strong cognitive aptitude component. Potential explanations for the effectiveness of this structured interviewing technique are discussed.     

Understanding the Academic–Practitioner Gap for Structured Interviews: ‘Behavioral’ interviews diffuse, ‘structured’ interviews do not

Recent studies about the academic–practitioner gap suggest that the nonadoption of evidenced‐based practices can be explained by their diffusion through practitioner‐oriented literature. This study extends these findings by investigating the case of the structured interview, which has not been widely adopted by practitioners despite its good psychometric properties. Using a social representations approach, we investigate how the academic concepts of ‘structured’ and ‘behavioral’ interviewing are diffused to practitioners in advice books. Results show that ‘behavioral’ interviews diffuse while ‘structured’ interviews do not, and that different arguments are used to describe these concepts. Facilitating the diffusion of structured interview practices requires academics to rethink their ways of communicating with practitioners through media.     

Comparing the Psychometric Characteristics of Ratings of Face‐to‐Face and Videotaped Structured Interviews

Videotaped interviews are used for both research and for making selection decisions in organizations. However, little research has examined the extent to which the psychometric characteristics of ratings of videotaped interviews are comparable with those of ratings made on the basis of face‐to‐face (FTF) interviews. Within a simulated selection setting, we compared ratings of interviewers who conducted FTF structured interviews to ratings of interviewers who viewed videotapes of those interviews. Results revealed that FTF ratings were significantly higher than video ratings of the same interviewees. We also found that the two sets of interviewers rated the relative performance of interviewees differently. For example, the correlation between FTF and video ratings (r=.31) was significantly smaller than the correlation between ratings of interviewers who conducted FTF panel interviews with the same interviewees (r=.73). Overall results suggest that researchers and practitioners should be cautious about generalizing research findings and selection decisions made on the basis of videotaped interviews to FTF interviews.     

Development of Standard Anchors for Scoring Generic Past-Behaviour Questions in Structured Interviews

A structured interviewing system was developed which included an item bank of pre-written past-behaviour questions. Scoring anchors for each of 168 questions in the item bank were created from analysis of protocols from individual assessments. A concurrent validation study was conducted, focusing on the job of highway patrol sergeant. Job knowledge experts used a job analysis tool to select a subset of twelve questions from the item bank. Panel interviews were conducted with 32 highway patrol sergeants as subjects. Average inter-rater reliability was 0.65 and interview validity was 0.42 for group consensus scores and 0.44 for panel average scores. Discussion focused on practical issues affecting the interviewing system.     

How Interviewers Judge Job Applicants: A Qualitative Study

The aim of a job interview is to allow an interviewer to judge whether an applicant will fit into the job at stake. This study investigated how various constructs (for instance, preinterview impressions, professional competencies, and attraction to applicants) influence interviewers’ judgment of applicants. The data were obtained from 49 qualitative interviews with job interviewers about their judgment of applicants. Attraction to the applicant was considered the most important construct in interviewers’ judgment processes, despite the fact that this has little to do with an applicant's job performance. The study argues that interviewers’ judgment processes are problematic, and that this is not due to poor interviewers. Instead, the very concept of the job interview leads to interviewers’ use of problematic constructs. The qualitative approach in the study provides insight into the way in which specific job interviewers judge applicants. Such insight cannot be gained from a quantitative approach, which has traditionally been used in job interview research. Finally, the study will provide a critical discussion of the psychological reasons for job interviewers’ use of problematic constructs in their decision making process.     

Acceptance of Structured Diagnostic Interviews for Mental Disorders in Clinical Practice and Research Settings

The objective of this study was to investigate the acceptance of structured diagnostic interviews in clinical practice, as well as research, settings. Using the Diagnostisches Interview bei Psychischen Störungen (the modified and extended German version of the Anxiety Disorders Interview Schedule for DSM-IV), 10 certified interviewers conducted 183 structured interviews in different inpatient, outpatient, and research settings in Switzerland and Germany. After each interview, patients and interviewers filled out a questionnaire asking for their evaluation of the interview. Patients' and interviewers' reactions to the interview were highly positive. On a scale measuring overall satisfaction with the interview (0 = not at all satisfied, 100 = totally satisfied) the mean patient rating was 86.55 (SD = 13.18), and the mean interviewer rating was 85.82 (SD = 12.84). The procedure used by the interviewer was rated by 142 (78.5%) patients as being helpful, and 176 (96.7%) rated the relationship as being positive. Less than 16% of the interviews were described as exhausting by the patients and interviewers. A majority of the interviewers (92.6%) indicated that during the interview they could respond adequately to the patient. The results of this study indicate that structured diagnostic interviews are highly accepted by interviewers and patients in a variety of settings. These findings, together with the existing evidence of the reliability and validity of structured interviews, should encourage their use in the diagnostic process, in outpatient and inpatient clinical settings as well as in research studies.     

Assessing personality disorders using a systematic clinical interview: evaluation of an alternative to structured interviews

The aim of this study was to assess interrater reliability and provide initial data bearing on the validity of a method of assessing personality disorders (PDs) that does not presume that patients can accurately self-report personality pathology. In a sample of 24 outpatients, two clinician-judges independently applied the Shedler-Westen Assessment Procedure-200 (SWAP-200; Westen & Shedler, 1999a, 2000), a 200-item Q-sort procedure for assessing personality pathology, to data from the Clinical Diagnostic Interview (Westen, 2002), a systematic clinical interview that mirrors and standardizes methods used by experienced clinicians to diagnose personality. In 16 of the 24 cases, the treating clinician also independently described the patient using the SWAP-200 Q-sort, based on longitudinal knowledge of the patient over the course of treatment, blind to the interview data. Interrater reliability was uniformly high, with median correlations between interviewers at r > .80. Interviewer-treating clinician correlations were also high, with median convergent validity coefficients at r > .80. Diagnostic overlap (discriminant validity) was moderate for dimensional DSM-IV diagnoses, reflecting extensive comorbidity among disorders, but minimal for empirically derived diagnoses identified in prior research. Treating clinicians' dimensional PD diagnoses using this method also strongly predicted interviewer-rated measures of adaptive functioning. The findings provide preliminary support for the reliability and validity of an alternative to structured interviews for diagnosing personality pathology, and suggest that the way to improve validity of personality diagnosis may not be to minimize clinical inference but to quantify it using psychometric instruments.     

Psychometric Properties of a Structured Behavioral Interview to Hire Private Security Personnel

This paper reports on a study about the reliability and validity of a structured behavioral interview to assess private security personnel. Reliability was estimated using interrater coefficients. Two independent interviewers were used to rate each interviewee. Results show a reliability coefficient of .81 (N = 43) and .89 with Spearman-Brown correction for two raters. Validity was estimated using a content validation approach. This strategy was suggested by Lawshe (1975) to estimate the content validity of selection tests. So far, only two studies carried out by Schmitt and Ostroff (1986) and Carrier et al. (1990) have used Lawshe's strategy in the structured behavioral interview case. The interview consisted of seven questions and each was rated by 11 experts in the job. Results show a significant content validity ratio (CVR) for majority of the questions in the interview and a content validity index (CVI) of .89. Implications of these findings for the practice of the structured behavioral interview are discussed and future research is suggested.     

A missing link in suggestibility research: what is known about the behavior of field interviewers in unstructured interviews with young children?

Despite suggestibility researchers' focus on adult behaviors that distort children's reports, whether behaviors examined in experimental work are used in the field is unknown. The current study presents a mutually exclusive and exhaustive hierarchical coding system that reflects interview questioning behaviors of concern in experimental work. The study examined 80 unstructured interviews conducted by 41 field interviewers with 40 children ages 3 to 7 about known events. Data on the use of leading and neutral questions are presented and include distinctions between accurate and inaccurate suggested information. In addition, analyses show that interviewers are consistent in their style of questioning and that a preinterview measure of interviewers' preference for a qualitative versus a quantitative interviewing style predicted the introduction of novel information into the interview.     

Automatic Generation of Rasch-Calibrated Items: Figural Matrices Test GEOM and Endless-Loops Test EC

The future of test construction for certain psychological ability domains that can be analyzed well in a structured manner may lie—at the very least for reasons of test security—in the field of automatic item generation. In this context, a question that has not been explicitly addressed is whether it is possible to embed an item response theory (IRT) based psychometric quality control procedure directly into the process of automatic item generation. Research in this area was conducted using 2 item generators (for the 2 domains of reasoning and spatial ability) that were developed and based on relevant models of cognitive psychology. During the course of the 4 studies reported here, those parts of the generators that check for possible violations of psychometric quality ("constraints") were improved. The main findings indicate that quality control procedures can be embedded in automatic item generators depending on (a) the degree to which the domain to be measured can be structured; (b) item-specific, content-based analyses; and (c) the degree to which the constraints can be implemented in software. Furthermore, beyond the global check of given model fit via IRT, the content-based analysis of items may be valuable in terms of finding such item properties that may lead to violations of psychometric quality.     

Towards understanding the effects of interviewer training in evaluating the cognitive interview

The effectiveness of the cognitive interview (CI) technique has been demonstrated in experimental studies of eyewitness memory conducted in the USA, Germany, and the UK. Much of the research, however, has used college students as interviewers rather than professionals who regularly interview witnesses (e.g. police officers). The aim of this research was to examine the effects of such CI training using experienced police officers with established interviewing styles. An armed robbery was staged and information was elicited in witness interviews. The CI did not significantly increase correct recall but tended to produce more suppositions and errors. A qualitative analysis of interviewer behaviour showed that many officers had difficulty in applying the new cognitive techniques and in avoiding the use of closed and leading questions. The paper addresses two issues: first, that of training experienced police officers who have received relatively little prior training in investigative interviewing, and second, the extent to which we can generalize from the laboratory data elicited in previous studies of CI.     

EFFECT OF SITUATIONAL INTERVIEWS, CONVENTIONAL STRUCTURED INTERVIEWS, AND TRAINING ON INTERVIEW RATING AGREEMENT: AN EXPERIMENTAL ANALYSIS

The present study examines the degree to which agreement in interviewer ratings may be influenced by training, use of structured conventional interviews, or situational interviews. Results of an experimental study conducted among 42 managers who were experienced as interviewers revealed no training effect on rating agreement, but the impact of the situational format on consistency in assessments of applicant suitability was significant. Implications of these findings for usefulness of situational employment interviews and future research investigating the reliability of situational interview formats are discussed.     

Hypnotic interviewing: the best way to interview eyewitnesses?

It has been suggested that hypnosis techniques may have the potential to enhance eyewitness memory in forensic investigations. However, laboratory research shows that increases in recall with hypnosis techniques are often associated with decreases in accuracy, false confidence in incorrect information, and increased suggestibility to leading questions and misleading post-event information. These problems limit the usefulness of hypnosis as an interviewing procedure. However, in practical investigations, many factors associated with hypnosis, apart from the hypnotic induction itself, might lead to memory enhancement compared with standard police interviews. For example, hypnotic interviewers, because of their psychological, clinical, and interpersonal skills, may be better interviewers than police officers. They may use effective interviewing strategies such as those associated with the “cognitive interview”; a procedure which has the potential to enhance recall by approximately 35% without the problems of memory distortion associated with hypnosis. It is concluded, therefore, that a cognitive interview procedure should be used in preference to hypnosis. © 1998 John Wiley & Sons, Ltd.     

Evaluation of video-based assessment in transit operator selection

The use of video testing is relatively new in personnel selection. Initial research has shown that the visual presentation of behaviorial incidents to job applicants may be a practical alternative to paper-and-pencil selection tests. Little research, however, has been conducted on the psychometric properties of video testing. To address this shortcoming in the personnel selection literature, this study reports the results of a validation study conducted on a video test for transit operators conducted in a large Canadian transit authority. The test, the Metro Seattle Video Test (MSVT), was designed to assess the interpersonal skills required of transit operators. The results show that, although content validation evidence was supportive, other psychometric evidence (i.e., reliability, criterion-related validation evidence, and construct-oriented validation evidence) is not consistent with an adequate selection test of interpersonal skills. Recommendations are made for future development and use of video testing in transit operator selection, as well as for improving the reliability and validity of video-based assessment of interpersonal skills.We gratefully acknowledge the assistance of Kevin Kelloway and Maury Getkate for their assistance with this paper, as well as Rick Hackett and Peter Hausdorf for their involvement in the data collection phase of this study.     

Investigation of nonverbal discrimination against women in simulated initial job interviews

This study investigated nonverbal sex discrimination in simulated initial job interviews with women applicants. It was hypothesized that experienced interviewers would exhibit more negative behaviors while interviewing a woman for a “masculine” job (an incongruent interview), but more positive behaviors while interviewing a woman for a “feminine” job (a congruent interview). It was further hypothesized that the behavior of inexperienced interviewers would remain the same across interviews. Mock initial job interviews were videotaped and nonverbal behaviors were coded. As hypothesized, experienced interviewers exhibited more negative and fewer positive behaviors in the incongruent interviews, whereas inexperienced interviewers did not. Unexpectedly, inexperienced interviewers exhibited more positive and fewer negative behaviors during incongruent interviews. Implications for training to reduce nonverbal discrimination are discussed.     

Revising Test Items to Increase Validity

Eight female interviewers varied their interview styles in interviewing 207 adults who had been involved in automobile accidents during a four-year period. A “professional” and a rapport or “interpersonal” style was used. Dependent variables consisted of reporting of events related to an automobile accident in which the respondent was the driver, and for which independent criterion data were available. In addition, the amount of health information reported was also analyzed. In spite of the respondents' more favorable attitudes towards the interpersonal interviews, accuracy was not significantly greater in these rapport interviews. In fact, significant interaction effects between interviewer style and recency of the accident were obtained. Respondents, however, reported more completely under rapport conditions. The results are interpreted within a cognitive, role-learning framework.     

Computerized Neurocognitive Testing in the Management of Sport-Related Concussion: An Update

Since the late nineties, computerized neurocognitive testing has become a central component of sport-related concussion (SRC) management at all levels of sport. In 2005, a review of the available evidence on the psychometric properties of four computerized neuropsychological test batteries concluded that the tests did not possess the necessary criteria to warrant clinical application. Since the publication of that review, several more computerized neurocognitive tests have entered the market place. The purpose of this review is to summarize the body of published studies on psychometric properties and clinical utility of computerized neurocognitive tests available for use in the assessment of SRC. A review of the literature from 2005 to 2013 was conducted to gather evidence of test-retest reliability and clinical validity of these instruments. Reviewed articles included both prospective and retrospective studies of primarily sport-based adult and pediatric samples. Summaries are provided regarding the available evidence of reliability and validity for the most commonly used computerized neurocognitive tests in sports settings.     

The factor structure of the Autobiographical Memory Test in recent trauma survivors

The objective of this study was to examine the psychometric properties of the Autobiographical Memory Test (AMT), which is widely used to measure overgeneral autobiographical memory in individuals with depression and a trauma history. Its factor structure and internal consistency have not been explored in a clinical sample. This study examined the psychometric properties of the AMT in a sample of recent trauma survivors (N = 194), who completed the AMT 2 weeks after a trauma. Participants were also assessed with structured clinical interviews for current acute stress disorder and current and past major depressive disorder. Confirmatory factor analysis and item response theory were used to analyze the AMT in the whole sample. The factor structure of the AMT was also compared for (a) individuals with and without lifetime major depressive disorder and (b) individuals with current (posttrauma) major depressive disorder and/or acute stress disorder versus those with neither disorder. In all of these analyses, the AMT with cues of positive and negative valence had a 1-factor structure, which replicates work in nonclinical samples. Based on analyses of the whole sample, scores from the AMT had a reliability estimate of .72, and standard error of measurement was lowest for people who scored low on memory specificity. In conclusion, the AMT measures 1 factor of memory specificity in a clinical sample and can yield reliable scores for memory specificity. More psychometric studies of the AMT are needed to replicate these results with similar and other clinical populations.