The Effects of Coaching and Speeding on Big Five and Impression Management Scale Scores

We examined the effects of coaching and speeding on personality scale scores in a faking context (N = 192). A completely crossed 2 × 2 experimental design was used in which instructions (no coaching or coaching) and speeding (with or without a time limit) were manipulated. No statistically significant effects on scale scores were evidenced for speeding. Coaching participants significantly elevated scores (average d = .76) for each of the Big Five personality factors but did not significantly elevate the scores on the Impression Management scale (d = .06). Cognitive ability was significantly positively related to impression management for uncoached participants but not for coached participants. An exploratory simulation suggests that coaching would have an effect on who would be selected for a job.     

Differential Daily Writing Conditions and Performance on Major Multiple-Choice Exams

Students (N=153) in three equivalent sections of an undergraduate human development course compared pairs of related concepts via either written or oral discussion at the beginning of most class sessions. A writing-for-random-credit section achieved significantly higher ratings on the writing activities than did a writing-for-no-credit section. The writing-for-random-credit section also obtained significantly higher correlations between the writing ratings and scores on multiple-choice exam items related to the conceptual pairs (r=.74) and on total exam scores (r=.78) than did the writing-for-no-credit section (rs=.45 and .43, respectively). Finally, the writing-for-random-credit condition produced significantly higher scores on exam items related to the conceptual pairs than did the oral-discussion-only condition. Plus, effect-size comparisons showed weak to medium differences between exam scores under the writing-for-random-credit and the writing-for-no-credit condition.     

Measuring Procedural Knowledge More Simply with a Single-Response Situational Judgment Test

Purpose  This paper describes the development of a situational judgment test (SJT) based on single-response options developed directly from critical incidents and reports a study that tested the SJT’s concurrent validity against ratings of job performance. Design/Methodology/Approach  Situational judgment test items were developed from critical incidents provided by administrators of volunteer agencies. Volunteers who worked at another agency completed the SJT and a self-report personality test. Supervisors rated their job performance on three dimensions. Findings  Situational judgment test scores representing procedural knowledge about work effort were significantly correlated with ratings of work effort performance (r = .28). Conscientiousness was correlated with work effort knowledge (r = .26), but not with work effort performance (r = −.02). Implications  These results provide some preliminary evidence that a single-response SJT format based upon critical incidents can produce valid measures of procedural knowledge and might be a useful alternative to the traditional multiple-response format. Originality/Value  This paper presents a novel way to construct SJTs using single-response options that are less laborious to develop than the multiple-response options in traditional SJT formats. Results of the validity study suggest that this new single-response format can predict job performance and encourage further research on the viability of this approach.     

Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations

Assessing reliability of situational judgment tests (SJTs) in high‐stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of α = .46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT test–retest reliability was r = .82, compared with internal consistency, α = .46, and stratified alpha, α = .45 at Time 1 and α = .52 and stratified α = .51 at Time 2. We used a student sample (Time 1: n = 185; Time 2: n = 132) with items from a credentialing exam with ‘should do’ instructions. The SJT correlated significantly with cognitive ability, r = .30, and agreeableness, r = .24. In Study 2, we assessed test–retest reliability with Human Resource professionals (Time 1: n = 94; Time 2: n = 32) who had been recently credentialed and who participated in a pilot test of new SJT items with ‘most likely/least likely do’ response options. The SJT test–retest reliability was r = .66 compared with internal consistency, α = .43 and stratified α = .47 at Time 1 and α = .61 and stratified α = .67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.     

Meta-analyses of Big Six Interests and Big Five Personality Factors

This article presents a series of meta-analyses examining the 24 samples to date that have revealed the overlap of the three most widely used measures of Holland's Big Six domains of vocational interest, namely the Self-Directed Search (Holland, 1985a), the Strong Interest Inventory (Hansen & Campbell, 1985; Harmon, Hansen, Borgen, & Hammer, 1994), and the Vocational Preference Inventory (Holland, 1985b), with the most widely accepted measure of the Big Five personality factors, namely the Revised NEO Personality Inventory (Costa & McCrae, 1992). The meta-analyses showed the mean effect sizes for each of the 30 correlations between the six interest dimensions and the five personality dimensions. Of the 30 correlations, 5 appeared to be substantial for both women and men and across the interest measures. They are Artistic-Openness (r=.48), Enterprising-Extraversion (r=.41), Social-Extraversion (r=.31), Investigative-Openness (r=.28), and Social-Agreeableness (r=.19).     

Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis

In this article, we describe a systematic review of mean race and sex differences in situational judgment test (SJT) performance. On average, White test takers perform better on SJTs than Black, Hispanic, and Asian test takers. Female examinees perform slightly better than male test takers on SJTs. We investigate two moderators of these differences: loading of g or personality on the SJT, and response instructions. Mean race differences between Black, Hispanic, Asian and White examinees in SJT performance are largely explained by the cognitive loading of the SJT such that the larger the cognitive load, the larger the mean race differences. Regarding the effect of personality loadings of SJTs on race differences, Black–White and Asian–White differences are smaller to the extent that the SJT is correlated with emotional stability and Hispanic–White differences are smaller to the extent that SJTs are correlated with conscientiousness and agreeableness. Cognitive loading has minimal effect on male–female SJT score differences; however, SJT score differences are larger, favoring women, when SJTs are correlated with conscientiousness and agreeableness. Concerning response instructions, knowledge response instructions showed greater race differences than behavioral tendency instructions. The mean correlations show that these differences are largely because of the greater g loading of knowledge instructions. A second study showed that when used in hiring, SJTs are likely to have adverse impact by race but not by sex.     

Faking on a situational judgment test in a medical school selection setting: Effect of different scoring methods?

We examined the occurrence of faking on a rating situational judgment test (SJT) by comparing SJT scores and response styles of the same individuals across two naturally occurring situations. An SJT for medical school selection was administered twice to the same group of applicants (N = 317) under low‐stakes (T1) and high‐stakes (T2) circumstances. The SJT was scored using three different methods that were differentially affected by response tendencies. Applicants used significantly more extreme responding on T2 than T1. Faking (higher SJT score on T2) was only observed for scoring methods that controlled for response tendencies. Scoring methods that do not control for response tendencies introduce systematic error into the SJT score, which may lead to inaccurate conclusions about the existence of faking.     

SITUATIONAL JUDGMENT TESTS, RESPONSE INSTRUCTIONS, AND VALIDITY: A META-ANALYSIS

Situational judgment tests (SJTs) are personnel selection instruments that present job applicants with work-related situations and possible responses to the situations. There are typically 2 types of instructions: behavioral tendency and knowledge. Behavioral tendency instructions ask respondents to identify how they would likely behave in a given situation. Knowledge instructions ask respondents to evaluate the effectiveness of possible responses to a given situation. Results showed that response instructions influenced the constructs measured by the tests. Tests with knowledge instructions had higher correlations with cognitive ability. Tests with behavioral tendency instructions showed higher correlations with personality constructs. Results also showed that response instructions had little moderating effect on criterion-related validity. Supplemental analyses showed that the moderating effect of response instructions on construct validity was not due to systematic differences in item content. SJTs have incremental validity over cognitive ability, the Big 5, and over a composite of cognitive ability and the Big 5.     

Be Careful What You Ask For: Effects of Response Instructions on the Construct Validity and Reliability of Situational Judgment Tests

The aim of this study was to examine how six different types of situational judgment test (SJT) instructions, used frequently in practice, influence the psychometric characteristics of SJTs. The six SJT versions used the exact same items and differed only in their instructions; these versions were administered in two phases. Phase I was a between–subjects design (n= 486) that had participants complete one version of the SJTs. Phase II was a within–subjects design (n= 231) held several weeks later that had participants complete all six versions of the SJTs. Further, 146 of these individuals completed both phases, allowing for an assessment of test–retest reliability. A variety of objective and subjective criteria were collected, including self and peer ratings. Results indicated that instructions had a large effect on SJT responses, reliability, and validity. In general, instructions asking what one ‘would do’ showed more favorable characteristics than those that asked what one ‘should do’. Correlations between these two types were relatively low despite the fact that the same items were used, and criterion–related validities differed substantially in favor of the ‘would do’ instructions. Overall, this study finds that researchers and practitioners must give careful consideration to the types of SJT instructions used; failing to do so could influence criterion–related validity and cloud inferences of construct validity.     

Better Late than Never? The Relationship between Ratings of Attendance,Punctuality, and Overall Job Performance among Nursing Home Employees

This study examined the relationship between lateness, absenteeism, and overall job performance among nursing home employees. We addressed four issues: (a) agreement between subordinate and superior in their ratings, (b) the stability of rated lateness and rated absenteeism over 6 months, (c) the correlation between ratings of lateness and absenteeism assigned by the worker to himself or herself and the corresponding ratings from supervisors, and (d) the relationship of rated lateness and rated absenteeism to impressions of overall job performance. Neither attendance nor punctuality showed inter‐rater agreement between the worker and his/her supervisor on either of the two occasions. Ratings of attendance were not stable across time for either the subordinate (r=.16) or the supervisor (r=.04), but punctuality ratings showed moderate temporal stability over 6 months for both subordinate (r=.59) and supervisor (r=. 41) impressions. On the first administration of the rating form, attendance and punctuality were not significantly correlated to each other in either the workers' self‐ratings (r=.17) nor the supervisors' ratings of the workers (r=.08). On the second administration, however, these two facets were moderately correlated with each other in both workers' (r=.54) and supervisors' (r=. 39) impressions of employee behavior. Compared with attendance, punctuality was somewhat more highly correlated with impressions of overall job performance from the perspective of the worker herself/himself as well as the supervisor.     

Personality and Intelligence in Business People: A Study of Two Personality and Two Intelligence Measures

This study examined the overlap and correlations among two well-known personality measures (NEO-PI–R; Myers-Briggs Type Indicator, MBTI) and two widely used intelligence tests (the Graduate Management Assessment (GMA), Watson-Glaser Critical Thinking Appraisal (WGCTA)). The GMA measures both fluid intelligence (Gf) and crystallized intelligence (Gc), whereas WGCTA mainly assess Gc. A total of over 3,500 participants completed the four measures in a middle management assessment event. Correlational analysis showed that Extraversion on the MBTI tended to be associated with Openness and Stability on the NEO. Intuition was associated with Openness and Introversion. Feeling types tended to be both Agreeable and Neurotic while perceiving types were high on Openness but low on Agreeableness. The NEO Big Five factor of Openness was most consistently and significantly associated with both measures of intelligence (r = .09 to r =.12). Results from the MBTI showed that Intuition and Perceiving scores were positively and significantly associated with both intelligence test scores which were intercorrelated (r = .38). Regressional analysis showed that personality traits are logically and coherently related to intelligence test scores. Implications for selection and assessment are considered.     

Emotional Intelligence in Selection Contexts: Measurement method,criterion‐related validity,and vulnerability to response distortion

Two studies examined aspects of the validity of self‐report and performance‐based measures of emotional intelligence (EI) relevant to their use in personnel selection. In Study 1, structural equation modeling indicated that a two‐factor model with separate factors for the two types of EI measures fit better than a one‐factor solution. The performance‐based EI factor was more related to cognitive ability (R=.38) than personality (R=.26), whereas the self‐report EI factor was more related to personality (R=.85) than cognitive ability (R=.09). Although the performance‐based EI factor correlated more strongly with job performance (ρ=.24) than did that of the self‐report (ρ=.05), it provided little incremental validity beyond cognitive ability and conscientiousness. In Study 2, participants were asked to complete the measures as if applying for a job, and mean scores were then compared with those of Study 1. Results indicated that self‐report EI measures were more vulnerable to distortion than were the performance‐based measures. Implications for the assessment of EI in personnel selection contexts are discussed.     

The Effects of Warnings and Speeding on Scale Scores and Convergent Validity of Conscientiousness

We examined the effects of warnings and speeding on scale scores and convergent validity of a measure of Conscientiousness in a faking context (N = 329). A completely crossed 2 × 2 experimental design was used in which instructions (no warning or a warning) and speeding (with or without a time limit) were manipulated. No statistically significant effects on scale scores or convergent validity were evidenced for speeding. Warning participants did decrease Conscientiousness scores by almost 1 standard deviation (d = .91). Warnings also moderated the relationship between self- and observer-ratings of Conscientiousness such that the relationship between self- and observer-ratings was statistically significant and positive in the warned conditions (r _partial = .29, p < .01, n = 136), but this same relationship was statistically nonsignificant (r _partial = ?.14, p > .05, n = 148) in the unwarned conditions.     

Increasing Validity with Forced‐Choice Criterion Measurement Formats

The relative validities of forced‐choice (ipsative) and Likert rating‐scale item formats as criterion measures are examined. While there has been much debate about the relative technical and psychometric merits and demerits of ipsative instruments, the present research focused on the crucial question of whether the use of this format has any practical benefit – in terms of improved validity. An analysis is reported from a meta‐analysis data set. This demonstrates that higher operational validity coefficients (prediction of line‐manager ratings of competencies) are associated with the use of forced‐choice (r=.38) rather than rating scale (r=.25) item formats for the criterion measurement instrument when performance is rated by the same line managers on both formats and where the predictor is held constant. Thus the apparent criterion‐related validity of a predictor can increase by 50% simply by changing the format of the criterion measurement instrument. The implications of this for practice are discussed.     

Assessment of the life stress-depression relationship: The use of social support as a moderator variable

Two hundred and fifty-two undergraduates completed a modified version of the Life Experience Survey (LES), the Beck Depression Inventory (Revised), and a measure of social support. Correlations between negative and total life changes on the LES and the Beck were significant for individuals with high social support (r=.36 and .29), but these correlations were significantly higher for individuals with low social support (r=.67 and .66). In addition, positive LES scores were significantly correlated with the Beck for individuals with low social support (r=.37), and individuals with low social support had higher mean scores on the Beck,F(1,250)=20.72,p<.0001. These findings were interpreted as supporting the theory of Holmes and Rahe and Sarason and Johnson that aversive life changes are related to depression, and provide evidence for the view the positive life changes can also be stressful. These data also support the concept of individual differences in vulnerability to stress and the role of social support in this vulnerability. The use of social support as a moderator variable appears to enhance correlations between the LES and depression to the level of clinical utility.     

Development and Validation of a Situational Judgment Test of Emotional Intelligence

The present study describes the development and validation of a situational judgment test (SJT) of emotional intelligence (EI). Initially, 80 situations and three response alternatives for each situation were created based on the available theoretical models. Principal component factor analysis with direct oblimin rotation of data (N = 213) yielded a three‐factor structure with 46 items. These factors were (1) utilizing own emotion, (2) sensing other's emotion, and (3) understanding emotional context. Additional studies showed that the measure had good internal consistency and test‐retest reliability. None of the three factors strongly correlated with the Big Five factors of personality (NEO Five‐Factor Inventory, thus establishing its identity as a construct distinct from personality. Findings of confirmatory factor analysis on secondary data reconfirm the three‐factor model for a 46‐item SJT of EI. The second study also found no correlation among these three factors, intelligence scores measured using Raven's Matrices, and trait EI score measured using the Trait Emotional Intelligence Questionnaire. The third study was conducted in order to determine the relationship of SJT of EI with academic achievement and life satisfaction. All three factors of SJT‐based EI measure were significantly associated with academic achievement and life satisfaction.     

Do Applicants Fake Their Personality Questionnaire Responses and How Successful are Their Attempts? A Case of Military Pilot Cadet Selection

This paper presents the results of three interrelated studies investigating the occurrence of response distortion on personality questionnaires within selection and the success of applicants in faking situations. In Study 1, comparison of the Big Five personality scores obtained from applicants in a military pilot cadet selection procedure with participants responding honestly, faking good, and faking an ideal candidate revealed that applicants responded more desirable than participants responding honestly but less desirable than respondents under fake instructions. The occurrence of faking within the military pilot selection process was replicated in Study 2 using the Eysenck Personality Questionnaire and another comparison group. Finally, in Study 3, comparison of personality profiles obtained in selection and ‘fake job’ situations with experts' estimates indicated that participants were partially successful in faking the desirable profile.     

Childhood Physical Abuse and Sociopathy: Is This Link Magnified among Firstborn Children?

The Psychopathic Deviate scale of the Minnesota Multiphasic Personality Inventory–2 (MMPI–2) provides a valid psychometric index of sociopathic tendencies in both clinical and nonclinical samples. Childhood physical abuse has provided a robust predictor of sociopathic penchants. The link between childhood physical abuse and MMPI–2 Psychopathic Deviate scores was examined (N = 322) as a function of birth order. A significant childhood physical abuse by birth order interaction was found (p < .0001, η² = .62) with a 6-fold increase in relative risk of a Psychopathic Deviate elevation (T > 70) found for firstborn participants. Childhood physical abuse predicted Psychopathic Deviate scores for firstborn (r = .50, p < .0001, d = 1.15) and middle-born (r = .24, p = .006, d = .49) offspring. Harris–Lingoes subscale scores suggested firstborns felt selectively alienated from self and others in response to childhood physical abuse experiences.     

The depression proneness rating scale: Reliability, validity, and factor structure

This study describes the development of the Depression Proneness Rating Scale (DPRS), a brief, self-administered measure of the tendency to experience frequent, long-lasting, and severe depressions, and three investigations into the scale’s reliability, validity, and factor structure. Study 1, using 100 university students, found a stability coefficient of .82 for the DPRS over a test-retest interval of nine weeks. Further, Time 1 (T1) DPRS scores predicted Time 2 (T2) symptoms of depression, even after adjusting for Time 1 symptoms (R² Change=.03). Study 2, using 440 university students, found the DPRS to be a better predictor of past depressive episodes (r=.41 to .47) than was the Beck Depression Inventory (r=.32). Study 3, using 1101 university students, found that all 13 items of the DPRS loaded .40 or greater on a single factor for both males and females. Overall, results provide substantial evidence for the DPRS as a valid, unidimensional, and practical measure of depression proneness.     

The depression proneness rating scale: Reliability,validity, and factor structure

This study describes the development of the Depression Proneness Rating Scale (DPRS), a brief, self-administered measure of the tendency to experience frequent, long-lasting, and severe depressions, and three investigations into the scale’s reliability, validity, and factor structure. Study 1, using 100 university students, found a stability coefficient of .82 for the DPRS over a test-retest interval of nine weeks. Further, Time 1 (T1) DPRS scores predicted Time 2 (T2) symptoms of depression, even after adjusting for Time 1 symptoms (R² Change=.03). Study 2, using 440 university students, found the DPRS to be a better predictor of past depressive episodes (r=.41 to .47) than was the Beck Depression Inventory (r=.32). Study 3, using 1101 university students, found that all 13 items of the DPRS loaded .40 or greater on a single factor for both males and females. Overall, results provide substantial evidence for the DPRS as a valid, unidimensional, and practical measure of depression proneness.