Examining the item response process to personality measures in high-stakes situations: Issues of measurement validity and predictive validity

We conducted two experimental studies with between-subjects and within-subjects designs to investigate the item response process for personality measures administered in high- versus low-stakes situations. Apart from assessing measurement validity of the item response process, we examined predictive validity; that is, whether or not different response models entail differential selection outcomes. We found that ideal point response models fit slightly better than dominance response models across high- versus low-stakes situations in both studies. Additionally, fitting ideal point models to the data led to fewer items displaying differential item functioning compared to fitting dominance models. We also identified several items that functioned as intermediate items in both the faking and honest conditions when ideal point models were fitted, suggesting that ideal point model is “theoretically” more suitable across these contexts for personality inventories. However, the use of different response models (dominance vs. ideal point) did not have any substantial impact on the validity of personality measures in high-stakes situations, or the effectiveness of selection decisions such as mean performance or percent of fakers selected. These findings are significant in that although prior research supports the importance and use of ideal point models for measuring personality, we find that in the case of personality faking, though ideal point models seem to have slightly better measurement validity, the use of dominance models may be adequate with no loss to predictive validity.     

Examining the Process of Responding to Circumplex Scales of Interpersonal Values Items: Should Ideal Point Scoring Methods Be Considered?

The Circumplex Scales of Interpersonal Values (CSIV) is a 64-item self-report measure of goals from each octant of the interpersonal circumplex. We used item response theory methods to compare whether dominance models or ideal point models best described how people respond to CSIV items. Specifically, we fit a polytomous dominance model called the generalized partial credit model and an ideal point model of similar complexity called the generalized graded unfolding model to the responses of 1,893 college students. The results of both graphical comparisons of item characteristic curves and statistical comparisons of model fit suggested that an ideal point model best describes the process of responding to CSIV items. The different models produced different rank orderings of high-scoring respondents, but overall the models did not differ in their prediction of criterion variables (agentic and communal interpersonal traits and implicit motives).     

Examining assumptions about item responding in personality assessment: should ideal point methods be considered for scale development and scoring?

The present study investigated whether the assumptions of an ideal point response process, similar in spirit to Thurstone's work in the context of attitude measurement, can provide viable alternatives to the traditionally used dominance assumptions for personality item calibration and scoring. Item response theory methods were used to compare the fit of 2 ideal point and 2 dominance models with data from the 5th edition of the Sixteen Personality Factor Questionnaire (S. Conn & M. L. Rieke, 1994). The authors' results indicate that ideal point models can provide as good or better fit to personality items than do dominance models because they can fit monotonically increasing item response functions but do not require this property. Several implications of these findings for personality measurement and personnel selection are described.     

Individual differences in achieving tendency: Review of evidence bearing on a questionnaire measure

The Mehrabian Achieving Tendency Scale (MATS) provides a broad-scoped assessment of individual characteristics associated with achievement. The MATS is highly reliable and reasonably free of response bias. It correlated negatively with various measures of general (trait) anxiety and fear of success. The MATS successfully predicted performance in situations approximating real-life tests of work and school performance and academic attitudes (e.g., level of aspiration, interest and satisfaction, leadership and initiative). The MATS also predicted occupational choice and occupational/organizational commitment and attractive/appropriate self-presentation. Within a general model of personality and temperament, the MATS was shown to consist primarily of trait dominance and secondarily of trait pleasure. The model successfully predicted relationships of the MATS with a variety of personality measures.     

Constructing personality scales under the assumptions of an ideal point response process: toward increasing the flexibility of personality measures

The main aim of this article is to explicate why a transition to ideal point methods of scale construction is needed to advance the field of personality assessment. The study empirically demonstrated the substantive benefits of ideal point methodology as compared with the dominance framework underlying traditional methods of scale construction. Specifically, using a large, heterogeneous pool of order items, the authors constructed scales using traditional classical test theory, dominance item response theory (IRT), and ideal point IRT methods. The merits of each method were examined in terms of item pool utilization, model-data fit, measurement precision, and construct and criterion-related validity. Results show that adoption of the ideal point approach provided a more flexible platform for creating future personality measures, and this transition did not adversely affect the validity of personality test scores.     

Handedness in a virtual haptic environment: Assessments from kinematic behavior and modeling

This study evaluated hand asymmetries in performance of a dexterous, controlled task under haptic feedback. Participants punctured a virtual membrane with a pushing or pulling movement, using the left or right hand. For pulling movements, the dominant (right) hand exhibited faster average stopping latency and shorter skidding distance. When the kinematic data were fit to a three-phase model previously applied to this task (Klatzky et al., 2013), the right hand exhibited faster force decay attributable to biomechanical factors. Analyses of the aggregated performance measures and model parameters showed that the left and right hands are associated with two different distributions, supporting handedness effects. Furthermore, while the majority of participants expressed right-hand dominance, which was consistent with their self-reported hand preferences, others showed partial or no dominance. This approach could potentially be extended to quantify and differentiate individuals with difficulties in manual behavior due to abnormal motor control (e.g., dyspraxia), progressive deterioration (e.g., Parkinson's syndrome) or improvement (neural regrowth after transplant).     

Latent growth curve analysis with dichotomous items: Comparing four approaches

A Monte Carlo study was used to compare four approaches to growth curve analysis of subjects assessed repeatedly with the same set of dichotomous items: A two‐step procedure first estimating latent trait measures using MULTILOG and then using a hierarchical linear model to examine the changing trajectories with the estimated abilities as the outcome variable; a structural equation model using modified weighted least squares (WLSMV) estimation; and two approaches in the framework of multilevel item response models, including a hierarchical generalized linear model using Laplace estimation, and Bayesian analysis using Markov chain Monte Carlo (MCMC). These four methods have similar power in detecting the average linear slope across time. MCMC and Laplace estimates perform relatively better on the bias of the average linear slope and corresponding standard error, as well as the item location parameters. For the variance of the random intercept, and the covariance between the random intercept and slope, all estimates are biased in most conditions. For the random slope variance, only Laplace estimates are unbiased when there are eight time points.     

Alternative Approaches to Addressing Non-Normal Distributions in the Application of IRT Models to Personality Measures

It is generally assumed that the latent trait is normally distributed in the population when estimating logistic item response theory (IRT) model parameters. This assumption requires that the latent trait be fully continuous and the population homogenous (i.e., not a mixture). When this normality assumption is violated, models are misspecified, and item and person parameter estimates are inaccurate. When normality cannot be assumed, it might be appropriate to consider alternative modeling approaches: (a) a zero-inflated mixture, (b) a log-logistic, (c) a Ramsay curve, or (d) a heteroskedastic-skew model. The first 2 models were developed to address modeling problems associated with so-called quasi-continuous or unipolar constructs, which apply only to a subset of the population, or are meaningful at one end of the continuum only. The second 2 models were developed to address non-normal latent trait distributions and violations of homogeneity of error variance, respectively. To introduce these alternative IRT models and illustrate their strengths and weaknesses, we performed real data application comparing results to those from a graded response model. We review both statistical and theoretical challenges in applying these models and choosing among them. Future applications of these and other alternative models (e.g., unfolding, diffusion) are needed to advance understanding about model choice in particular situations.     

Do Applicants and Incumbents Respond to Personality Items Similarly? A Comparison of Dominance and Ideal Point Response Models

This study used an ideal point response model to examine the extent to which applicants and incumbents differ when responding to personality items. It was hypothesized that applicants' responses would exhibit less folding at high trait levels than incumbents' responses. We used sample data from applicants (N=1,509) and incumbents (N=1,568) who completed the 16 Personality Questionnaire Select. Differential item (DIF) and test functioning (DTF) analyses were conducted using the generalized graded unfolding model, which is based on ideal point model assumptions. Out of the 90 items, 50 showed DIF; however, only 11 were in the hypothesized direction. DTF was significant for 3 of the 12 scales; 2 were in the hypothesized direction.     

Multidimensional modeling with unidimensional approximations

This paper advances nonparametric multidimensional item response theory by reporting experimental results on the use of nonmetric multidimensional scaling (MDS) to synthesize a multidimensional model from several approximating one-dimensional models. A two-dimensional simulation data set contains items in which the two-component traits combine linearly (dominance model items) and items in which the two-component traits combine quadratically (ideal point items). Several unidimensional approximations of the two-dimensional model were obtained by running unidimensional estimation software on the simulated data set. The graphs reconstructed from MDS of the unidimensional approximations at selected points clearly separate dominance items from ideal point items, and also various types of dominance or ideal point models. MDS also succeeded in determining the dimensionality of the simulation model items from the observable item responses.     

When Middle Really Means “Top” or “Bottom”: An Analysis of the 16PF5 Using Bock's Nominal Response Model

When self-report items with a Likert-type scale include a middle response option (e.g., Unsure, Neither agree nor disagree, or ?), this middle option is assumed to measure a level of the trait intermediate between the high and low response categories. In this study, we tested this assumption in the 16 Personality Factor Questionnaire, Version 5 (16PF5) by fitting Bock's nominal response model in the U.S. and UK standardization samples of the 16PF5. We found that in many cases, the middle option was indicative of higher levels of the latent trait than the ostensibly highest response option. In certain other cases, it was indicative of lower levels of the latent trait than the ostensibly lowest response option. This undermines the use of a simple successive integer scoring scheme where responses in adjacent response categories are assigned scores of 0, 1, and 2. Recommendations for alternative scoring schemes are provided. Results also suggested that certain personality traits, especially neurotic traits, are associated with a tendency toward selecting the middle option.     

Testing an Adaptationist Theory of Trait Covariation: Relative Bargaining Power as a Common Calibrator of an Interpersonal Syndrome

This article provides the first test of an adaptationist ‘common calibration’ theory to explain the origins of trait covariation, which holds that (i) personality traits are often facultatively calibrated in response to cues that ancestrally predicted the reproductive payoffs of different trait levels and (ii) distinct traits that are calibrated on the basis of common input cues will exhibit consistent patterns of covariation. This theory is applied to explain the covariation within a ‘personality syndrome’ encompassing various interpersonal trait dimensions (e.g. extraversion, emotionality and attachment styles). Specifically, it is hypothesized that these traits are inter‐correlated because each is calibrated in response to relative bargaining power (RBP)—a joint function of one's ability to benefit others and harm others. Path analyses from a correlational study compellingly supported this theoretical model: Objective and self‐perceived measures of RBP‐enhancing phenotypic features (physical attractiveness and physical strength) influenced an internal regulatory variable indexing RBP (i.e. self‐perceived RBP), which in turn had robust effects on each of the focal personality traits. Moreover, in support of the theory's core postulate, controlling for self‐perceived RBP greatly reduced the covariation within the interpersonal syndrome. These novel findings illustrate the promise of an evolutionary psychological approach to elucidating trait covariation. Copyright © 2013 John Wiley & Sons, Ltd.     

The trade-off between accuracy and precision in latent variable models of mediation processes

Social psychologists place high importance on understanding mechanisms and frequently employ mediation analyses to shed light on the process underlying an effect. Such analyses can be conducted with observed variables (e.g., a typical regression approach) or latent variables (e.g., a structural equation modeling approach), and choosing between these methods can be a more complex and consequential decision than researchers often realize. The present article adds to the literature on mediation by examining the relative trade-off between accuracy and precision in latent versus observed variable modeling. Whereas past work has shown that latent variable models tend to produce more accurate estimates, we demonstrate that this increase in accuracy comes at the cost of increased standard errors and reduced power, and examine this relative trade-off both theoretically and empirically in a typical 3-variable mediation model across varying levels of effect size and reliability. We discuss implications for social psychologists seeking to uncover mediating variables and provide 3 practical recommendations for maximizing both accuracy and precision in mediation analyses.     

Nonlinear Indicator-Level Moderation in Latent Variable Models

Linear, nonlinear, and nonparametric moderated latent variable models have been developed to investigate possible interaction effects between a latent variable and an external continuous moderator on the observed indicators in the latent variable model. Most moderation models have focused on moderators that vary across persons but not across the indicators (e.g., moderators like age and socioeconomic status). However, in many applications, the values of the moderator may vary both across persons and across indicators (e.g., moderators like response times and confidence ratings). Indicator-level moderation models are available for categorical moderators and linear interaction effects. However, these approaches require respectively categorization of the continuous moderator and the assumption of linearity of the interaction effect. In this article, parametric nonlinear and nonparametric indicator-level moderation methods are developed. In a simulation study, we demonstrate the viability of these methods. In addition, the methods are applied to a real data set pertaining to arithmetic ability.     

Comparing time-accuracy curves: Beyond goodness-of-fit measures

The speed-accuracy trade-off (SAT) is a ubiquitous phenomenon in experimental psychology. One popular strategy for controlling SAT is to use the response signal paradigm. This paradigm produces time-accuracy curves (or SAT functions), which can be compared across different experimental conditions. The typical approach to analyzing time-accuracy curves involves the comparison of goodness-of-fit measures (e.g., adjusted-R2), as well as interpretation of point estimates. In this article, we examine the implications of this approach and discuss a number of alternative methods that have been successfully applied in the cognitive modeling literature. These methods include model selection criteria (the Akaike information criterion and the Bayesian information criterion) and interval estimation procedures (bootstrap and Bayesian). We demonstrate the utility of these methods with a hypothetical data set.     

Using the PRISM to compare the explanatory value of general and role-contextualized trait ratings

In an earlier work (Wood & Roberts, 2006), the Personality and Role Identity Structural Model (PRISM) was proposed as a model for organizing the relations between diverse self-perceptions, with a person's general identity ("how I am in general") organized above diverse role identities (e.g., "how I am as an employee"), which in turn is organized above role-specific behaviors and experiences (e.g., typical interactions with coworkers). In the present article, I argue that despite the fact that role trait measures are often much more related to role behaviors than general trait measures in cross-sectional analyses, general trait measures better capture the dispositional causes of a person's role behavior. In support of this, a brief study is presented illustrating how general traits may be better predictors than role-contextualized trait ratings of the evolution of an individual's experiences or behaviors within a given context. Finally, I contend that the basic framework of the PRISM, where a person's behaviors and identities within multiple contexts are assessed simultaneously and longitudinally alongside general personality ratings, is necessary to make strong statements concerning the nature of the relationships between personality traits and role experiences.     

人格测量：从累积式模型到展开式模型

当前大多数人格测量都采用的是累积式反应模型方法,该模型假设被试在测验上的得分随其能力或特质提高而增加,但是随着人格测量技术的不断发展,这一模型的实施效果遭到了质疑,研究者们开始关注展开式模型,该模型认为被试的反应取决于被试能力和项目阈值的匹配程度,当被试能力与项目阈值完全匹配时,被试做出肯定回答的概率达到最高点,称之为“理想点”,展开式模型的目的就是找到被试的理想点,从而寻找其真正的态度强度或人格特质水平。GGUM作为一种比较成熟的展开式模型,已经开始应用于人格测量的各个领域,但仍需要进行大规模的试测,在评估和预测效度方面积累经验,建立业界认可的心理测量学标准,不断探讨和开发相应的心理测量理论和简便易行的统计程序。     

Associations between power,stress, and dominance in romantic relationships during the COVID-19 pandemic: Examining curvilinear and within-person effects

Two key processes in romantic relationships—power and dominance—can contribute to relationship disruption, but the association between these variables is complex. Elucidating the association between power and dominance during the COVID-19 pandemic is particularly important given the economic, social, and health-related stressors that pose a risk to relationship health. We examined associations between power, stress, and dominance by recruiting 1813 participants to complete an initial online survey at the beginning of the COVID-19 pandemic. Participants were contacted 10 and 22 months later to complete follow-up surveys. Results revealed two main effects: individuals who had greater relationship power and experienced more COVID-19-related stressors than other people engaged in more dominance behaviors. A significant curvilinear effect revealed that at low levels of power, power was not associated with dominance behaviors. However, once power surpassed low levels, individuals with more power engaged in more dominance behaviors. Finally, people engaged in more dominance behaviors when they experienced more power and stress compared to their own average (i.e., within-person effects) during the pandemic. Implications for theories of power, dominance, and relationship disruption and distress are discussed.     

Consistency,self-consciousness,and the predictive validity of typical and maximal personality measures

Most personality measures which ask for subjects' typical performance have often not been effective as predictors of criterion behaviors. Based upon the maximal performance model of ability tests, predictors were constructed which required subjects to report both how dominant they typically are in a particular situation and how dominant they are capable of acting. The efficacy of these typical and maximal self-reports in predicting typical and maximal expressions of dominance in the laboratory was compared. Self-reports of maximal dominance tended to outpredict self-reports of typical dominance for both typical and maximal laboratory expressions of dominance. In addition, maximal self-reports were of equal predictive validity for both those subjects reporting themselves as consistent in expressing dominance and those reporting themselves as inconsistent (situationally variable). Finally, the extent to which subjects were self-reflective (private self-consciousness) and concerned about their appearance to others (public self-consciousness) mediated the validity of both their typical and maximal self-reports, although the predictive superiority of maximal self-reports was maintained.