Evidence for Context Switching in the Effects of Average Item Length and Item-Length Variability on Internal Consistency

This investigation examined the effects of 3 item characteristics—the average number of words per item, within-scale variability in item length, and item “direction”—on internal consistency reliability and interitem correlation. In Study 1, we examined the effects of these variables on overall scale-level reliability using 444 subscales from 9 personality scales. In Study 2, we examined interitem correlation at the paired-item level using 477 nonredundant item pairs from 14 personality scales. Lower scale reliability was associated with more average words per item, greater within-scale variability in item length, and a greater percentage of reverse-keyed items. Similarly, smaller interitem correlations were associated with a greater degree of mismatch in item length between the paired items and with a mismatch (vs. match) in the items' respective “directions.” The pattern of results across both studies supports our notion that lower internal consistency results from increased context switching; that is, from the confusion that occurs when respondents must switch back and forth between the interpretive frames pertaining to short versus long items, or between items pertaining to one pole of a personality dimension and its “opposite” pole. Suggestions for maximizing the internal consistency of personality scales are proposed.     

Separating method factors and higher order traits of the Big Five: a meta-analytic multitrait-multimethod approach

Though most personality researchers now recognize that ratings of the Big Five are not orthogonal, the field has been divided about whether these trait intercorrelations are substantive (i.e., driven by higher order factors) or artifactual (i.e., driven by correlated measurement error). We used a meta-analytic multitrait-multirater study to estimate trait correlations after common method variance was controlled. Our results indicated that common method variance substantially inflates trait correlations, and, once controlled, correlations among the Big Five became relatively modest. We then evaluated whether two different theories of higher order factors could account for the pattern of Big Five trait correlations. Our results did not support Rushton and colleagues' (Rushton & Irwing, 2008; Rushton et al., 2009) proposed general factor of personality, but Digman's (1997) α and β metatraits (relabeled by DeYoung, Peterson, and Higgins (2002) as Stability and Plasticity, respectively) produced viable fit. However, our models showed considerable overlap between Stability and Emotional Stability and between Plasticity and Extraversion, raising the question of whether these metatraits are redundant with their dominant Big Five traits. This pattern of findings was robust when we included only studies whose observers were intimately acquainted with targets. Our results underscore the importance of using a multirater approach to studying personality and the need to separate the causes and outcomes of higher order metatraits from those of the Big Five. We discussed the implications of these findings for the array of research fields in which personality is studied.     

Probing item social desirability by correlating personality items with Balanced Inventory of Desirable Responding (BIDR): A validity examination

Researchers often include a social desirability measure in personality measures, commonly the Balanced Inventory of Desirable Responding (BIDR), and correlate it with personality items to probe for social desirability of the items. A strong correlation between BIDR scores and a personality item would indicate high item social desirability. The current research assesses the validity of this practice. Results showed that these correlations have high validity only when BIDR scores are calculated as a continuous variable rather than as dichotomized item scores. In addition, self-deception scores have higher validity for detecting item social desirability than do impression management scores. The current research supported the use of the self-deception scores, in particular, to detect highly desirable or undesirable items.     

Cohort differences in Big Five personality factors over a period of 25 years

The notion of personality traits implies a certain degree of stability in the life span of an individual. But what about generational effects? Are there generational changes in the distribution or structure of personality traits? This article examines cohort changes on the Big Five personality factors Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience, among first-year psychology students in The Netherlands, ages 18 to 25 years, between 1982 and 2007. Because measurement invariance of a personality test is essential for a sound interpretation of cohort differences in personality, we first assessed measurement invariance with respect to cohort for males and females separately on the Big Five personality factors, as measured by the Dutch instrument Five Personality Factors Test. Results identified 11 (females) and 2 (males) biased items with respect to cohort, out of a total of 70 items. Analyzing the unbiased items, results indicated small linear increases over time in Extraversion, Agreeableness, and Conscientiousness and small linear decreases over time in Neuroticism. No clear patterns were found on the Openness to Experience factor. Secondary analyses on students from 1971 to 2007 of females and males of different ages together revealed linear trends comparable to those in the main analyses among young adults between 1982 onward. The results imply that the broad sociocultural context may affect personality factors.     

Measuring Response Styles Across the Big Five: A Multiscale Extension of an Approach Using Multinomial Processing Trees

This study shows how to address the problem of trait-unrelated response styles (RS) in rating scales using multidimensional item response theory. The aim is to test and correct data for RS in order to provide fair assessments of personality. Expanding on an approach presented by Böckenholt (2012), observed rating data are decomposed into multiple response processes based on a multinomial processing tree. The data come from a questionnaire consisting of 50 items of the International Personality Item Pool measuring the Big Five dimensions administered to 2,026 U.S. students with a 5-point rating scale. It is shown that this approach can be used to test if RS exist in the data and that RS can be differentiated from trait-related responses. Although the extreme RS appear to be unidimensional after exclusion of only 1 item, a unidimensional measure for the midpoint RS is obtained only after exclusion of 10 items. Both RS measurements show high cross-scale correlations and item response theory-based (marginal) reliabilities. Cultural differences could be found in giving extreme responses. Moreover, it is shown how to score rating data to correct for RS after being proved to exist in the data.     

Assessing the Structural and Psychometric Properties of a New Personality Measure for Use With Military Personnel in the French Armed Forces

We evaluated the measurement properties of a new personality scale for use with military personnel in the French Navy (N = 1,266). Principal components analyses indicated a five-factor structure assessing emotional stability, ascendancy, openness, agreeableness, and conscientiousness. Each factor comprised two facets that included 7 items per facet for a total of 70 items. Reliability analyses indicate that the factors and corresponding facets were internally consistent and temporally stable. Validity analyses indicate that the TAMI-P was significantly correlated with comparable measures of personality—correlations ranged from .53 to .68 with the Revised NEO Personality Inventory and the Big Five Questionnaire.     

Person‐factors in the California Adult Q‐Set: closing the door on personality trait types?

To investigate recent hypotheses of replicable personality types, we examined data from 1540 self‐sorts on the California Adult Q‐Set (CAQ). Conventional factor analysis of the items showed the expected Five‐Factor Model (FFM). Inverse factor analysis across random subsamples showed that none of the previously reported person‐factors were replicated. Only two factors were replicable, and, most importantly, these factors were contaminated by mean level differences in item endorsement. Results were not due to sample size or age heterogeneity. Subsequent inverse factor analysis of standardized items revealed at least three replicable factors; when five person‐factors were extracted, they could be aligned precisely with the dimensions of the FFM. The major factors of person similarity can be accounted for entirely in terms of the FFM, consistent with the hypothesis that there are no replicable personality types in the CAQ. Published in 2006 John Wiley & Sons, Ltd.     

Qualitative variations in personality inventories: subjective understandings of items in a personality inventory

Most studies using personality inventories do not take individual, subjective understandings of the items into account. The present study is one of the few to have investigated the quality of individuals’ psychological processes when making the Likert-like responses often used in psychological inventories. Respondents were asked to elaborate verbally on their Likert item responses to the 10-item short version of the Big Five Inventory. A common assumption about personality inventories is that there is a relatively homogenous understanding of the items and, in particular, the rating scales across respondents. However, our results suggest that the same item responses to a given item can reflect a variety of qualities across individuals’ understandings. At the same time, similar understandings and ways of relating to an item can lead to different item responses. Such findings have substantial implications for quantitative personality studies as well as quantitative survey or questionnaire studies, in general.     

‘Grandpa,Do You Like Roller Coasters?’: Identifying Age‐Appropriate Personality Indicators

Personality development research heavily relies on the comparison of scale means across age. This approach implicitly assumes that the scales are strictly measurement invariant across age. We questioned this assumption by examining whether appropriate personality indicators change over the lifespan. Moreover, we identified which types of items (e.g. dispositions, behaviours, and interests) are particularly prone to age effects. We reanalyzed the German Revised NEO Personality Inventory normative sample (N = 11,724) and applied a genetic algorithm to select short scales that yield acceptable model fit and reliability across locally weighted samples ranging from 16 to 66 years of age. We then examined how the item selection changes across age points and item types. Emotion‐type items seemed to be interchangeable and generally applicable to people of all ages. Specific interests, attitudes, and social effect items—most prevalent within the domains of Extraversion, Agreeableness, and Openness—seemed to be more prone to measurement variations over age. A large proportion of items were systematically discarded by the item‐selection procedure, indicating that, independent of age, many items are problematic measures of the underlying traits. The implications for personality assessment and personality development research are discussed. © 2019 European Association of Personality Psychology     

Differential item functioning in a Spanish translation of the PTSD checklist: detection and evaluation of impact

This study demonstrated the application of an innovative item response theory (IRT) based approach to evaluating measurement equivalence, comparing a newly developed Spanish version of the Posttraumatic Stress Disorder Checklist-Civilian Version (PCL-C) with the established English version. Basic principles and practical issues faced in the application of IRT methods for instrument evaluation are discussed. Data were derived from a study of the mental health consequences of community violence in both Spanish speakers (n = 102) and English speakers (n = 284). Results of differential item functioning (DIF) analyses revealed that the 2 versions were not fully equivalent on an item-by-item basis in that 6 of the 17 items displayed uniform DIF. No bias was observed, however, at the composite PCL-C scale score, indicating that the 2 language versions can be combined for scale-level analyses.     

Use of item ratings to examine personality test item cognitive response processes

This research examines the processes respondents use to answer personality test items. A total of 158 true/false items from four scales of the Personality Research Form and the California Psychological Inventory were used as stimuli. University students (N = 120) responded to each item and indicated one of nine strategies used in deciding on a response. Obtained response strategy ratings for items were reliable and their frequencies corresponded closely to previous findings with other items. Subsequently, the associations between item response strategy frequencies and item-total correlations were computed. Congruent with previous research, better items avoided behaviours or experiences and evoked responding based on traits and on referring to the statements of others. The associations between item response strategies and other indices of item quality are discussed and implications regarding scale development are offered.     

Measurement Equivalence Between Applicant and Incumbent Groups: An IRT Analysis of Personality Scales

Incumbents are often used in the development and validation of a wide variety of personnel selection instruments, including noncognitive instruments such as personality tests. However, the degree to which assumed motivational factors impact the measurement equivalence and validity of tests developed using incumbents has not been adequately addressed. This study addressed this issue by examining the measurement equivalence of 6 personality scales between a group applying for jobs as sales managers in a large retail organization (N = 999) and a group of sales managers currently employed in that organization (N = 796). A graded item response theory model (Samejima, 1969) was fit to the personality scales in each group. Results indicated that moderately large differences existed in personality scale scores (approximately 1/2 standard deviation units) but only one of the six scales contained any items that evidenced differential item functioning and no scales evidenced differential test functioning. In addition, person-level analyses showed no apparent differences across groups in aberrant responding. The results suggest that personality measures used for selection retain similar psychometric properties to those used in incumbent validation studies.     

Nonverbal assessment of the Big Five personality factors

The Nonverbal Personality Questionnaire (NPQ) is an experimental, structured, nonverbal measure of 16 personality traits. Its items lack verbal content and, therefore, the inventory is useful for cross‐cultural research. Our goal is this research was to select a subset of the NPQ items to form a new nonverbal questionnaire based on the Five‐Factor Model of personality. We describe the construction of the Five‐Factor Nonverbal Personality Questionnaire (FF‐NPQ), and present data on its psychometric properties. These data include scale internal consistencies, intercorrelations, convergences with verbal measures of the Big Five factors, discriminant validity correlations, correlations with peer ratings, and ability to predict socially important behaviour criteria such as smoking and alcohol consumption. In a second study, we report on the psychometric properties of the FF‐NPQ in an independent sample of respondents from seven different countries. The utility of the new nonverbal inventory for cross‐cultural research is discussed. Copyright © 2001 John Wiley & Sons, Ltd.     

A successive approximation method of maximizing test validity

The ratio of item validity to item-total correlation can be used to select items which will tend to yield the maximum correlation with a criterion. Items to be retained are identified by comparing the ratio for each item with the validity of the original test. Further improvement of the validity in the experimental sample can be obtained by adding items to or removing items from the selected nucleus, according to recomputed ratios involving the correlations of the items with the nucleus and evaluated by means of a revised cut-off point. With slight variations, the method may be used for interest and personality tests as well as for aptitude material. The principal advantage over previous methods is that for any cycle of the analysis an exact cut-off point is provided.     

Examining the item response process to personality measures in high-stakes situations: Issues of measurement validity and predictive validity

We conducted two experimental studies with between-subjects and within-subjects designs to investigate the item response process for personality measures administered in high- versus low-stakes situations. Apart from assessing measurement validity of the item response process, we examined predictive validity; that is, whether or not different response models entail differential selection outcomes. We found that ideal point response models fit slightly better than dominance response models across high- versus low-stakes situations in both studies. Additionally, fitting ideal point models to the data led to fewer items displaying differential item functioning compared to fitting dominance models. We also identified several items that functioned as intermediate items in both the faking and honest conditions when ideal point models were fitted, suggesting that ideal point model is “theoretically” more suitable across these contexts for personality inventories. However, the use of different response models (dominance vs. ideal point) did not have any substantial impact on the validity of personality measures in high-stakes situations, or the effectiveness of selection decisions such as mean performance or percent of fakers selected. These findings are significant in that although prior research supports the importance and use of ideal point models for measuring personality, we find that in the case of personality faking, though ideal point models seem to have slightly better measurement validity, the use of dominance models may be adequate with no loss to predictive validity.     

Parent–teacher agreement on 7‐Year‐old children's personality

Agreement between multiple informants on child personality has received limited attention. Focusing on factor structure, gender differences and the influence of socially desirable responding (SDR), we compared parent and teacher Big Five personality ratings of around 600 7‐year olds. Although parent ratings were more desirable than teacher ratings, differential agreement was generally similar to that found for adults, and especially high for ratings of boys. The more evaluative the personality item, the larger the mean‐level difference between parents and teachers on that item. However, undesirable items showed the highest levels of differential agreement. In parent ratings, the two poles of Agreeableness formed separate factors. To view Pro‐sociality as independent of Antagonism could enable parents to view their child more positively. Copyright © 2010 John Wiley & Sons, Ltd.