A Comparison of the Nomological Networks Associated With Forced-Choice and Likert Formats of the Narcissistic Personality Inventory

The Narcissistic Personality Inventory (NPI) is one of the most popular measures of narcissism. However, its use of a forced-choice response set might negatively affect some of its psychometric properties. The purpose of this research was to compare a Likert version of the NPI, in which only the narcissistic response of each pair was given, to the original NPI, in 3 samples of participants (N = 1,109). To this end, we compared the nomological networks of the forced-choice and Likert formats of the NPI in relation to alternative measures of narcissism, narcissistic personality disorder, entitlement, self-esteem, general personality traits (reported by self and informants), interpersonal styles, and general pathological traits included in the DSM–5. The Likert format NPI—total and subscales—manifested similar construct validity to the original forced-choice format across all criteria with only minor differences that seem to be due mainly to the increased reliability and variability found in the Likert NPI Entitlement/Exploitativeness subscale. These results provide evidence that a version of the NPI that employs a Likert format can justifiably be used in place of the original.     

The Impact of Faking on Employment Tests: Does Forced Choice Offer a Solution?

We evaluated the effects of faking on mean scores and correlations with self-reported counterproductive behavior of integrity-related personality items administered in single-stimulus and forced-choice formats. In laboratory studies, we found that respondents instructed to respond as if applying for a job scored higher than when given standard or "straight-take" instructions. The size of the mean shift was nearly a full standard deviation for the single-stimulus integrity measure, but less than one third of a standard deviation for the same items presented in a forced-choice format. The correlation between the personality questionnaire administered in the single-stimulus condition and self-reported workplace delinquency was much lower in the job applicant condition than in the straight-take condition, whereas the same items administered in the forced-choice condition maintained their substantial correlations with workplace delinquency.     

Fitting a Thurstonian IRT model to forced-choice data using Mplus

To counter response distortions associated with the use of rating scales (a.k.a. Likert scales), items can be presented in a comparative fashion, so that respondents are asked to rank the items within blocks (forced-choice format). However, classical scoring procedures for these forced-choice designs lead to ipsative data, which presents psychometric challenges that are well described in the literature. Recently, Brown and Maydeu-Olivares (Educational and Psychological Measurement 71: 460–502, 2011a) introduced a model based on Thurstone’s law of comparative judgment, which overcomes the problems of ipsative data. Here, we provide a step-by-step tutorial for coding forced-choice responses, specifying a Thurstonian item response theory model that is appropriate for the design used, assessing the model’s fit, and scoring individuals on psychological attributes. Estimation and scoring is performed using Mplus, and a very straightforward Excel macro is provided that writes full Mplus input files for any forced-choice design. Armed with these tools, using a forced-choice design is now as easy as using ratings.     

The role of unconscious memory errors in judgments of confidence for sentence recognition

The present experiment tested the hypothesis that unconscious reconstructive memory processing can lead to the breakdown of the relationship between memory confidence and memory accuracy. Participants heard deceptive schema-inference sentences and nondeceptive sentences and were tested with either simple or forced-choice recognition. The nondeceptive items showed a positive relation between confidence and accuracy in both simple and forced-choice recognition. However, the deceptive items showed a strong negative confidence/accuracy relationship in simple recognition and a low positive relationship in forced choice. The mean levels of confidence for erroneous responses for deceptive items were inappropriately high in simple recognition but lower in forced choice. These results suggest that unconscious reconstructive memory processes involved in memory for the deceptive schema-inference items led to inaccurate confidence judgments and that, when participants were made aware of the deceptive nature of the schema-inference items through the use of a forced-choice procedure, they adjusted their confidence accordingly.     

From ABLE to TAPAS: A New Generation of Personality Tests to Support Military Selection and Classification Decisions

This article discusses 3 modern forced-choice personality tests developed for the U.S Armed Services to provide resistance to faking and other forms of response distortion: the Assessment of Individual Motivation, the Navy Computerized Adaptive Personality Scales, and the Tailored Adaptive Personality Assessment System. These tests represent the transition from Likert to forced-choice formats and from static to computerized adaptive item selection to meet the challenges of large-scale, high-stakes testing environments. For each test, we briefly describe the personality constructs that are assessed, the response format and scoring methods, and selected ongoing research and development efforts. We also highlight the potential of these tests for personnel selection, classification, and diagnostic screening purposes.     

Statement Desirability Ratings in Forced-Choice Personality Measure Development: Implications for Reducing Score Inflation and Providing Trait-Level Information

Recent studies have produced mixed results regarding the effectiveness of forced-choice formats in reducing faking on personality measures. The present research examined a potentially important factor influencing forced-choice measure effectiveness—namely, the manner in which statement desirability ratings used in constructing forced-choice items are obtained—with the goal of clarifying previous findings and providing guidance for those interested in developing these measures. Two studies indicated that different desirability rating instructions produce substantially different forced-choice measures that are differentially effective in reducing score inflation and retaining trait-level information under faking conditions. These studies suggest that the manner in which statement desirability ratings are obtained may be an important consideration in forced-choice measure development.     

Further evidence of the Narcissistic Personality Inventory’s validity problems: A meta-analytic investigation—Response to Miller, Maples, and Campbell (this issue)

Miller, Maples, and Campbell (this issue) present evidence that Rosenthal and Hooley’s (2010) concerns regarding the Narcissistic Personality Inventory’s (NPI) relation to psychological health may be unwarranted. To resolve this issue empirically, we conducted a meta-analysis (k = 54, N = 38,932). Meta-analytic results revealed that a subset of NPI items were indeed problematic; items that function poorly at differentiating narcissists from non-narcissists accounted entirely for the NPI’s connection to psychological health. These items were also strongly associated with self-esteem, but unrelated to aggression/anger. In contrast, the remaining NPI items were unrelated to psychological health, but associated with aggression/anger. We conclude that although the NPI measures narcissism, its poorest functioning items also link it to outcomes unrelated to narcissism.     

迫选式人格测验的传统计分与IRT计分模型

迫选测验的传统计分方式会产生自模式数据, 不能进行传统的信效度检验、因素分析和方差分析等。近年来研究者提出了一些基于项目反应理论的计分模型, 如瑟斯顿IRT模型和MUPP模型等, 它们可以规避自模式数据的弊端。瑟斯顿IRT模型方便进行参数估计, 模型定义灵活; 而MUPP模型的拓展性较差, 参数估计的方法有待提高。另一方面, 已有研究者基于MUPP模型开发了一些抗作假的迫选测验, 而瑟斯顿IRT模型距离这种应用还比较远。此外, 两个模型的适用性和有效性都有待更多的实证研究来检验。     

A dual process item response theory model for polytomous multidimensional forced-choice items

The use of multidimensional forced-choice (MFC) items to assess non-cognitive traits such as personality, interests and values in psychological tests has a long history, because MFC items show strengths in preventing response bias. Recently, there has been a surge of interest in developing item response theory (IRT) models for MFC items. However, nearly all of the existing IRT models have been developed for MFC items with binary scores. Real tests use MFC items with more than two categories; such items are more informative than their binary counterparts. This study developed a new IRT model for polytomous MFC items based on the cognitive model of choice, which describes the cognitive processes underlying humans' preferential choice behaviours. The new model is unique in its ability to account for the ipsative nature of polytomous MFC items, to assess individual psychological differentiation in interests, values and emotions, and to compare the differentiation levels of latent traits between individuals. Simulation studies were conducted to examine the parameter recovery of the new model with existing computer programs. The results showed that both statement parameters and person parameters were well recovered when the sample size was sufficient. The more complete the linking of the statements was, the more accurate the parameter estimation was. This paper provides an empirical example of a career interest test using four-category MFC items. Although some aspects of the model (e.g., the nature of the person parameters) require additional validation, our approach appears promising.     

适用于多维迫选测验的IRT计分模型

迫选(forced-choice, FC)测验由于可以控制传统李克特方法带来的反应偏差, 被广泛应用于非认知测验中, 而迫选测验的传统计分方式会产生自模式数据, 这种数据由于不适合于个体间的比较, 一直备受批评。近年来, 多种迫选IRT模型的发展使研究者能够从迫选测验中获得接近常模性的数据, 再次引起了研究者与实践人员对迫选IRT模型的兴趣。首先, 依据所采纳的决策模型和题目反应模型对6种较为主流的迫选IRT模型进行分类和介绍。然后, 从模型构建思路、参数估计方法两个角度对各模型进行比较与总结。其次, 从参数不变性检验、计算机化自适应测验(computerized adaptive testing, CAT)和效度研究3个应用研究方面进行述评。最后提出未来研究可以在模型拓展、参数不变性检验、迫选CAT测验和效度研究4个方向深入。     

Egos inflating over time: a cross-temporal meta-analysis of the Narcissistic Personality Inventory

ABSTRACT A cross-temporal meta-analysis found that narcissism levels have risen over the generations in 85 samples of American college students who completed the 40-item forced-choice Narcissistic Personality Inventory (NPI) between 1979 and 2006 (total n =16,475). Mean narcissism scores were significantly correlated with year of data collection when weighted by sample size (β=.53, p <.001). Since 1982, NPI scores have increased 0.33 standard deviation. Thus, almost two-thirds of recent college students are above the mean 1979–1985 narcissism score, a 30% increase. The results complement previous studies finding increases in other individualistic traits such as assertiveness, agency, self-esteem, and extraversion.     

An evaluation of forced-choice and true-false item formats in personality assessment

The reliabilities and validities of true-false and forced-choice formats in personality assessment were compared. Subjects from college residential units were assigned randomly to groups receiving the Personality Research Form (PRF) in either forced-choice or standard true-false form. Reliabilities were substantially higher for the true-false form. Peer rating validities for each format were in a comparable range, but correlations with self-ratings were higher for the true-false form. Results do not support the contention that a forced-choice format is consistently more valid than a standard format. Subjects well acquainted with ratees manifested more highly differentiated judgments, showed consistently higher validity, but were more prone to show a bias to attribute more salient traits, like dominance and exhibition, to ratees.     

Introduction to the Special Issue: Personality and the Prediction of Job Performance: More Than the Big Five

The effects of motivated distortion on forced-choice (FC) and normative inventories were examined in three studies. Study 1 examined the effects of distortion on the construct validity of the two item formats in terms of convergent and discriminant validity. The results showed that both types of measures were susceptible to motivated distortion, however the FC items were better indicators of personality and less related to socially desirable responding when participants were asked to respond as if applying for a job. Study 2 considered the criterion-related validity of the inventories in terms of predicting supervisors' ratings of job performance, finding that distortion had a more deleterious effect on the validity of the normative inventory with some enhancement of the validity of the FC inventory being observed. Study 3 investigated whether additional constructs are introduced into the measurement process when motivated respondents attempt to increase scores on FC items. Results of Study 3 indicated that individuals higher in cognitive ability tend to have more accurate theories about which traits are job-related and therefore are more successful at improving scores on FC inventories. Implications for using personality inventories in personnel selection are discussed.     

Ipsative and Normative Scales in Adjectival Measurement of Personality: Problems of Bias and Discrepancy

887 respondents completed ipsative and normative versions of the PAL-TOPAS personality questionnaire. Data were analysed to test for (1) systematic bias in scores associated with the two response formats and (2) predictors of the magnitude of the discrepancy in the individual's ipsative and normative scores. Discrepancy was assessed for both item responses and scale scores. Sources of biases investigated included ipsative scaling artifact, extremeness of scores on the normative scales and response variability. Results showed that systematic bias in scale scores and magnitude of discrepancy were predicted by different factors. One source of systematic bias was associated with ipsative scaling artifact: the ipsative scales measure both the scale itself and rejection of other alternatives. A second source of systematic bias was acquiescence in response to normative items. A confirmatory factor analysis showed that a good but imperfect fit to the data may be obtained by constructing a structural model of the inter-relationship between normative and ipsative scores which accommodates both sources of bias. The strongest influence on discrepancy in scale scores was extremeness of normative scoring, associated with a bias towards either general acceptance or rejection of trait adjectives. It is concluded that both normative and ipsative response formats have limitations, and it may often be desirable to assess both.     

Normative Scoring of Multidimensional Pairwise Preference Personality Scales Using IRT: Empirical Comparisons With Other Formats

In this article, we offer some suggestions as to why tetrads and pentads have become the dominant formats for administering multidimensional forced choice (MFC) items but, in turn, raise questions regarding the underlying psychometric model and means of addressing item quality and scoring accuracy. We then focus our attention on multidimensional pairwise preference (MDPP) items and present an item response theory–based approach to constructing and modeling MDPP responses directly, assessing information at the item and scale levels, and a way of computing standard errors for trait scores and estimating scale reliability. To demonstrate the viability of this method for applied use, we show that the correspondence between MDPP scores derived from direct modeling with those obtained using single statement and unidimensional pairwise preference measures administered in a laboratory setting. Trait score correlations and criterion related validities are compared across testing formats and rating sources (i.e., self and other), and the usefulness of our model-based approach is further demonstrated by some illustrative results involving computerized adaptive tests (CAT).     

Conceptual and empirical approaches to developing family-based assessment procedures: resolving the case of the Family Environment Scale

This article focuses on the reliability and validity of the Family Environment Scale (FES). The FES subscales generally show adequate internal consistency reliability and stability over time when applied in samples that are diverse; the items also have good content and face validity. An extensive body of research supports the construct, concurrent, and predictive validity of the FES. More generally, reliability and validity are a joint function of scale items and response formats and of the characteristics and diversity of specific samples. To contribute to further advances in family assessment, researchers need to use both conceptual and psychometric criteria rather than rely too heavily on the pursuit of internal consistency reliability and factor analytic approaches to scale construction and validation.     

Fast pairs: a visual word recognition paradigm for measuring entrenchment, top-down effects, and subjective phenomenology

When word pairs having a familiar order are sequentially flashed on a computer in their non-familiar order, (code zip), observers have a strong phenomenology of seeing them in familiar order (zip code). Reversal errors remained frequent even when participants obtained perceptual experience of reverse-display items by beginning with a block of longer-duration trials. A forced-choice order-detection procedure reduced but did not eliminate reversal errors, showing that "fast pairs" is a robust perceptual illusion. Even adjective + noun pairs (green skirt) showed reversal errors, and reversal errors increased with the log frequency of the word pair, consistent with a strong role for statistical processing at the level of multi-word units.     

Toward an understanding of situational judgment item validity and group differences

This paper evaluates 2 adjustments to common scoring approaches for situational judgment tests (SJTs). These adjustments can result in substantial improvements to item validity, reductions in mean racial differences, and resistance to coaching designed to improve scores. The first adjustment, applicable to SJTs that use Likert scales, controls for elevation and scatter (Cronbach & Gleser, 1953). This adjustment improves item validity. Also, because there is a White-Black mean difference in the preference for extreme responses on Likert scales (Bachman & O'Malley, 1984), these adjustments substantially reduce White-Black mean score differences. Furthermore, this adjustment often eliminates the score elevation associated with the coaching strategy of avoiding extreme responses (Cullen, Sackett, & Lievens, 2006). Item validity is shown to have a U-shaped relationship with item means. This holds both for SJTs with Likert score response formats and for SJTs where respondents identify the best and worst response option. Given the U-shaped relationship, the second adjustment is to drop items with midrange item means. This permits the SJT to be shortened, sometimes dramatically, without necessarily harming validity.