Factor analytic approaches to personality item-level data

Factor analysis models have played a central role in formulating conceptual models in personality and personality assessment, as well as in empirical examinations of personality measurement instruments. Yet, the use of item-level data presents special problems for factor analysis, applications. In this article, we review recent developments in factor analysis that are appropriate for the type of item-level data often collected in personality. Included in this review are discussions of how these developments have been addressed in the context of two different (but formally related) statistical models item response theory (IRT: Hambleton, Swaminathan, & Rogers, 1991) and structural, equation modeling (Bollen 1989) for item-level data. We also discuss the relevance of item scaling in the context of these models. Using the restandardization data for the Minnesota Multiphasic Personality Inventory-2 Scale (cf. Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer, 1989), we show brief examples of the utility of these approaches to address basic questions about responses to personality scale items regarding: (a) scale, dimensionality and general item properties, (b) the "appropriateness" of the observed responses, and (c) differential item functioning across subsamples. implications for analyses of personality item-level data in the IRT and factor analytic traditions are discussed.     

Comparing applicants and incumbents: Effects of response distortion on mean scores and validity of personality measures

This study examined the negative effect of likely applicant distortion on mean scores and validity of personality measures. The personality test scores and performance ratings of applicants were directly compared to those of incumbents with the same occupation in four different samples. The results showed that applicant mean scores were higher and validity coefficients were lower than for incumbents.     

Intentional response distortion on personality tests: using eye-tracking to understand response processes when faking

Intentional response distortion or faking among job applicants completing measures such as personality and integrity tests is a concern in personnel selection. The present study aimed to investigate whether eye-tracking technology can improve our understanding of the response process when faking. In an experimental within-participants design, a Big Five personality test and an integrity measure were administered to 129 university students in 2 conditions: a respond honestly and a faking good instruction. Item responses, response latencies, and eye movements were measured. Results demonstrated that all personality dimensions were fakeable. In support of the theoretical position that faking involves a less cognitively demanding process than responding honestly, we found that response times were on average 0.25 s slower and participants had less eye fixations in the fake good condition. However, in the fake good condition, participants had more fixations on the 2 extreme response options of the 5-point answering scale, and they fixated on these more directly after having read the question. These findings support the idea that faking leads to semantic rather than self-referenced item interpretations. Eye-tracking was demonstrated to be potentially useful in detecting faking behavior, improving detecting rates over and beyond response extremity and latency metrics.     

Adolescent personality and two measures of response inconsistency

This study examined the relationship between the Response Inconsistency Scale (RINC) of the Adolescent Personality Questionnaire and the Variable Response Inconsistency Scale (VRIN) of the Minnesota Multiphasic Personality Inventory for Adolescents (MMPI-A) in a group of 62 participants. RINC and VRIN were positively correlated .30 (p < .01). Additionally, both scales were correlated with five broad personality scales available from the Adolescent Personality Questionnaire. Scores on both inconsistency scales correlated negatively with those on Extraversion, Independence, and Self-control (but the latter two not significantly).     

Using response times to detect aberrant responses in computerized adaptive testing

A lognormal model for response times is used to check response times for aberrances in examinee behavior on computerized adaptive tests. Both classical procedures and Bayesian posterior predictive checks are presented. For a fixed examinee, responses and response times are independent; checks based on response times offer thus information independent of the results of checks on response patterns. Empirical examples of the use of classical and Bayesian checks for detecting two different types of aberrances in response times are presented. The detection rates for the Bayesian checks outperformed those for the classical checks, but at the cost of higher false-alarm rates. A guideline for the choice between the two types of checks is offered.This study received funding from the Law School Admission Council (LSAC). The opinions and conclusions contained in this paper are those of the authors and do not necessarily reflect the policy and position of LSAC. The authors are most indebted to Wim M. M. Tielen for his computational assistance and to the US Defense Manpower Data Center for the permission to use the ASVAB data set in the empirical examples.     

Item characteristics that predict response latency in personality measures

This study assesses some item characteristics that are expected to affect response latencies in the computerized administration of a personality questionnaire. First, some previous empirical results are discussed, and a theoretical framework from which predictions can be made is proposed. Second, the predictions are empirically assessed using two data-sets, one of them based on binary items, and the other one based on graded-response items. Some of the results obtained are new, whereas the remaining agree with previous empirical results. The implications of the present results in applied research, and particularly for the use of latencies, is discussed.     

Using response time measures to assess "guilty knowledge"

How can a suspect's guilt or innocence be reliably tested? The validity of the polygraph, which measures changes in physiological arousal during a "guilty knowledge" test, is controversial (e.g., T. R. Bashore & P. E. Rapp, 1993; T. P. Cross & L. Saxe, 1992; D. T. Lykken, 1998; J. P. Rosenfeld, 1995; R. Steinbrook, 1992). One alternative to the polygraph examines event-related potentials recorded during a memory interference task (L. A. Farwell & E. Donchin, 1991). The present study extended this paradigm to determine whether response times (RTs) can accurately identify participants possessing specific guilty knowledge. Results from Experiment 1 showed that RT alone can reliably discriminate "guilty" from "innocent" participants. Experiments 2a and 2b indicated that an RT-based paradigm is more resistant to strategic manipulation than previously suggested (Farwell & Donchin, 1991). This RT-based paradigm may be a viable alternative to the polygraph for detecting guilty knowledge.     

The utility of the NEO-PI-R validity scales to detect response distortion: a comparison with the MMPI-2

In this psychometric study, we compared the recently developed Validity Scales from the Revised NEO Personality Inventory (NEO PI-R; Costa & McCrae, 1992b) with the MMPI-2 (Butcher, Dahstrom, Graham, Tellegen, & Kaemmer, 1989) Validity Scales. We collected data from clients (n = 74) who completed comprehensive psychological evaluations at a university-based outpatient mental health clinic. Correlations between the Validity Scales of the NEO-PI-R and MMPI-2 were significant and in the expected directions. The relationships provide support for convergent and discriminant validity of the NEO-PI-R Validity Scales. The percent agreement of invalid responding on the two measures was high, although the diagnostic agreement was modest (kappa = .22-.33). Finally, clients who responded in an invalid manner on the NEO-PI-R Validity Scales produced significantly different clinical profiles on the NEO-PI-R and MMPI-2 than clients with valid protocols. These results provide additional support for the clinical utility of the NEO-PI-R Validity Scales as indicators of response bias.     

Beyond Impression Management: Evaluating three measures of response distortion and their relationship to job performance

Three measures of response distortion (i.e., social desirability, covariance index, and implausible answers) were examined in both applicant and incumbent samples. Performance data, including supervisor ratings of task and contextual performance as well as objective performance criteria such as tardiness, work‐related accidents, and a customized work simulation, were obtained for the incumbent sample. Results provided further support for the existence of applicant faking behavior and shed light into the relationship between faking and job performance, largely depending on how one defines and measures faking as well as the performance criteria evaluated. Implications for future research and practice in personality assessment for selection purposes were discussed.     

Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data

A 2-step approach for obtaining internal consistency reliability estimates with item-level missing data is outlined. In the 1st step, a covariance matrix and mean vector are obtained using the expectation maximization (EM) algorithm. In the 2nd step, reliability analyses are carried out in the usual fashion using the EM covariance matrix as input. A Monte Carlo simulation examined the impact of 6 variables (scale length, response categories, item correlations, sample size, missing data, and missing data technique) on 3 different outcomes: estimation bias, mean errors, and confidence interval coverage. The 2-step approach using EM consistently yielded the most accurate reliability estimates and produced coverage rates close to the advertised 95% rate. An easy method of implementing the procedure is outlined.     

Interpersonal violence and its relationship to some personality measures

One hundred and fifteen patients in a psychiatric hospital for criminal offenders volunteered to be tested with the overcontrolled hostility scale, the self focus sentence completion, and an emotional empathy measure. The subjects were divided into five groups on the basis of criminal offenses (murder, assault with a deadly weapon, rape, pedophilia, and nonviolent property offenses). It was also possible to divide the sample on the basis of diagnostic features (psychosis, personality disorder, and brain syndrome). The results failed to support the hypothesis that more violent (by nature of their offense) individuals would-as a group-score higher on the overcontrolled hostility scale than less aggressive counterparts. There was some evidence that psychotics showed more self focus at the expense of external focus. The empathy measure failed to reveal any meaningful differences. It is suggested that more validation data are called for to establish the generality of the overcontrolled hostility scale and that, in view of the therapeutic importance which empathy training may have for violent individuals, further effort should be expanded to find a generally valid empathy measure.     

A covariance structure analysis of employees' response to performance feedback

This longitudinal study used D. R. Ilgen, C. D. Fisher, and M. S. Taylor's (1979) feedback process model as a theoretical framework to determine whether a sequential chain of cognitive variables mediates an individual's response to performance feedback. One hundred two employees were surveyed 2 weeks after their performance appraisal, and performance was assessed 11 months later at the end of the review cycle. Covariance structure analysis supported the convergent and discriminant validity of the constructs underlying the model and the constellation of structural relationships. A set of cognitive variables was found to completely mediate the relationship between an individual's receipt and response to feedback. Implications for the feedback process and future research are discussed.     

Deliberate faking on personality and emotional intelligence measures

This study examined the extent the Big Five personality traits and emotional intelligence can be faked. Using a student sample, the equivalence of measurement and theoretical structure of models in a faking and honest condition was tested. Comparisons of the models for the honest and faking groups showed the data fit better in the faking condition. These results suggest that faking does change the rank orders of high scoring participants. The personality dimensions most affected by faking were emotional stability and conscientiousness within the Big Five and the general mood and stress management dimensions of Bar-On's Emotional Quotient Inventory-Short Form (1997) measure of emotional intelligence.     

Defensive style and performance on objective personality measures

Two studies on the relations between various styles of defense and the power or validity of self-reports on objective personality questionnaires. The power of individual items on the Rotter Internal-External Locus of Control scale to discriminate between subjects defined as internals or externals by a median-cut procedure using the total score was not affected by the presence Or absence of a projective defensive style. However, the presence of repressive tendencies did appear to restrict item power. Prediction from two achievement scales taken from Gough's California Psychological Inventory was not influenced by level of repression, was affected somewhat adversely by a projective style, and was most clearly contaminated by a rationalizing defense. It was proposed that the differential effects of defensive styles upon the validity of personality tests depended upon the convergence or divergence of the cognitive operations involved in the type of defense and in the type of personality measure.     

Constructing personality scales under the assumptions of an ideal point response process: toward increasing the flexibility of personality measures

The main aim of this article is to explicate why a transition to ideal point methods of scale construction is needed to advance the field of personality assessment. The study empirically demonstrated the substantive benefits of ideal point methodology as compared with the dominance framework underlying traditional methods of scale construction. Specifically, using a large, heterogeneous pool of order items, the authors constructed scales using traditional classical test theory, dominance item response theory (IRT), and ideal point IRT methods. The merits of each method were examined in terms of item pool utilization, model-data fit, measurement precision, and construct and criterion-related validity. Results show that adoption of the ideal point approach provided a more flexible platform for creating future personality measures, and this transition did not adversely affect the validity of personality test scores.     

Examining the item response process to personality measures in high-stakes situations: Issues of measurement validity and predictive validity

We conducted two experimental studies with between-subjects and within-subjects designs to investigate the item response process for personality measures administered in high- versus low-stakes situations. Apart from assessing measurement validity of the item response process, we examined predictive validity; that is, whether or not different response models entail differential selection outcomes. We found that ideal point response models fit slightly better than dominance response models across high- versus low-stakes situations in both studies. Additionally, fitting ideal point models to the data led to fewer items displaying differential item functioning compared to fitting dominance models. We also identified several items that functioned as intermediate items in both the faking and honest conditions when ideal point models were fitted, suggesting that ideal point model is “theoretically” more suitable across these contexts for personality inventories. However, the use of different response models (dominance vs. ideal point) did not have any substantial impact on the validity of personality measures in high-stakes situations, or the effectiveness of selection decisions such as mean performance or percent of fakers selected. These findings are significant in that although prior research supports the importance and use of ideal point models for measuring personality, we find that in the case of personality faking, though ideal point models seem to have slightly better measurement validity, the use of dominance models may be adequate with no loss to predictive validity.     

Using machine learning to detect events in eye-tracking data

Event detection is a challenging stage in eye movement data analysis. A major drawback of current event detection methods is that parameters have to be adjusted based on eye movement data quality. Here we show that a fully automated classification of raw gaze samples as belonging to fixations, saccades, or other oculomotor events can be achieved using a machine-learning approach. Any already manually or algorithmically detected events can be used to train a classifier to produce similar classification of other data without the need for a user to set parameters. In this study, we explore the application of random forest machine-learning technique for the detection of fixations, saccades, and post-saccadic oscillations (PSOs). In an effort to show practical utility of the proposed method to the applications that employ eye movement classification algorithms, we provide an example where the method is employed in an eye movement-driven biometric application. We conclude that machine-learning techniques lead to superior detection compared to current state-of-the-art event detection algorithms and can reach the performance of manual coding.