Choice model analysis of the “pick any/n” type of binary data

In the “pick any/n” method, subjects are asked to choose any number of items from a list of n items according to some criterion. This kind of data can be analyzed as a special case of either multiple-choice data or successive categories data where the number of response categories is limited to two. An item response model was proposed for the latter case, which is a combination of an unfolding model and a choice model. The marginal maximum-likelihood estimation method was developed for parameter estimation to avoid incidental parameters, and an expectation-maximization algorithm used for numerical optimization. Two examples of analysis are given to illustrate the proposed method, which we call MAXSC.     

On Minimizing Guessing Effects on Multiple‐Choice Items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format

Multiple‐choice response formats are troublesome, as an item is often scored as solved simply because the examinee may be lucky at guessing the correct option. Instead of pertinent Item Response Theory models, which take guessing effects into account, this paper considers a psycho‐technological approach to re‐conceptualizing multiple‐choice response formats. The free‐response format is compared with two different multiple‐choice formats: a traditional format with a single correct response option and five distractors (‘1 of 6’), and another with five response options, three of them being distractors and two of them being correct (‘2 of 5’). For the latter format, an item is scored as mastered only if both correct response options and none of the distractors are marked. After the exclusion of a few items, the Rasch model analyses revealed appropriate fit for 188 items altogether. The resulting item‐difficulty parameters were used for comparison. The multiple‐choice format ‘1 of 6’ differs significantly from the multiple‐choice format ‘2 of 5’, while the latter does not differ significantly from the free‐response format. The lower difficulty of items ‘1 of 6’ suggests guessing effects.     

Latent Trait Modeling with the Common beliefs Survey III: Using Item Response Theory to Evaluate an Irrational Beliefs Inventory

Assessment of irrational beliefs by such measures as the Common Beliefs Survey III (CBS) has traditionally relied upon classical test theory assumptions, in which the properties of specific test items are less important than the total test score as the aggregate of all item responses. An alternative approach using item response theory (IRT) methodology allows one to specify the parameters of difficulty and discrimination for each test item. Difficulty levels of CBS items range along a continuum of irrationality, the implied latent trait measured by responses to the questionnaire as a whole. We evaluated the CBS responses of 605 individuals from clinical and college settings, drawing from current and archival data. The original Likert scale ratings were recoded into dichotomous scores. Fourteen of the 54 items were highly or very highly discriminating in distinguishing respondents with high and low irrationality levels. However, discriminating items exhibited a very narrow range of difficulty; most functioned at a point a little above the halfway mark on the continuum of irrationality. Item characteristic curves and test information curves were very similar for female (n = 424) and male (n = 179) respondents. We derived a 4-item screening test for irrationality from our IRT analyses of the 54 CBS items. Further test development, focused on the selection and scaling of items with a much broader range of difficulty, would facilitate evaluation of the hierarchical structure of irrational beliefs. Portions of this paper were presented at the 39th Annual Convention of the Association for Behavioral and Cognitive Therapies, Washington, DC, November, 2005.     

How do respondents construe ambiguous response formats of affect items?

Ambiguous response formats predict correlations from -.467 to -1 between opposite items, depending on whether the respondent's interpretation of the format is unipolar or bipolar. The authors present a procedure to investigate the proper interpretation in each case. It consists of applying nonparametric and parametric item response theory models (the Mokken and the graded response models) to pairs of opposite items in order to find the locations of the response options along the latent scale and, therefore, identify the response format construction. The authors tested this procedure on 4 samples (Ns=142-1,150) and 2 item pairs ("relaxed"-"tense" and "optimistic"-"pessimistic"). The results revealed that respondents constructed the formats as bipolar and supported the bipolarity of the item pairs.     

A Psychometric Analysis of the Trait Emotional Intelligence Questionnaire–Short Form (TEIQue–SF) Using Item Response Theory

Trait emotional intelligence refers to a constellation of emotional self-perceptions located at the lower levels of personality hierarchies. In 2 studies, we sought to examine the psychometric properties of the Trait Emotional Intelligence Questionnaire–Short Form (TEIQue–SF; Petrides, 2009) using item response theory (IRT). Study 1 (N= 1,119, 455 men) showed that most items had good discrimination and threshold parameters and high item information values. At the global level, the TEIQue–SF showed very good precision across most of the latent trait range. Study 2 (N= 866, 432 men) used similar IRT techniques in a new sample based on the latest version of the TEIQue–SF (version 1.50). Results replicated Study 1, with the instrument showing good psychometric properties at the item and global level. Overall, the 2 studies suggest the TEIQue-SF can be recommended when a rapid assessment of trait emotional intelligence is required.     

Use of item ratings to examine personality test item cognitive response processes

This research examines the processes respondents use to answer personality test items. A total of 158 true/false items from four scales of the Personality Research Form and the California Psychological Inventory were used as stimuli. University students (N = 120) responded to each item and indicated one of nine strategies used in deciding on a response. Obtained response strategy ratings for items were reliable and their frequencies corresponded closely to previous findings with other items. Subsequently, the associations between item response strategy frequencies and item-total correlations were computed. Congruent with previous research, better items avoided behaviours or experiences and evoked responding based on traits and on referring to the statements of others. The associations between item response strategies and other indices of item quality are discussed and implications regarding scale development are offered.     

Increasing Validity with Forced‐Choice Criterion Measurement Formats

The relative validities of forced‐choice (ipsative) and Likert rating‐scale item formats as criterion measures are examined. While there has been much debate about the relative technical and psychometric merits and demerits of ipsative instruments, the present research focused on the crucial question of whether the use of this format has any practical benefit – in terms of improved validity. An analysis is reported from a meta‐analysis data set. This demonstrates that higher operational validity coefficients (prediction of line‐manager ratings of competencies) are associated with the use of forced‐choice (r=.38) rather than rating scale (r=.25) item formats for the criterion measurement instrument when performance is rated by the same line managers on both formats and where the predictor is held constant. Thus the apparent criterion‐related validity of a predictor can increase by 50% simply by changing the format of the criterion measurement instrument. The implications of this for practice are discussed.     

The effect of being in a tip-of-the-tongue state on subsequent items

The tip-of-the-tongue state (TOT) is the feeling that an inaccessible item will be recalled. In the TOT induction paradigm, participants are given a list of general information questions or word definitions, and the participants indicate whether they are in a TOT for each item. The present study explored the effect that being in a TOT for one item (N) has on the recall and the likelihood of a TOT for the subsequent item (N + 1). Three experiments were conducted. All three experiments showed that TOTs do not affect the rate of recall for the next item but decrease the likelihood of a TOT for the next item. This effect extended to items occurring two items after the initial TOT (N + 2) in two experiments. Thus, TOTs are less likely to occur after another TOT than after an item not in a TOT. These data are interpreted within a metacognitive framework.     

Development and Initial Validation of the Five-Factor Model Adolescent Personality Questionnaire (FFM–APQ)

This research reports on the 4-phase development of the 25-item Five-Factor Model Adolescent Personality Questionnaire (FFM–APQ). The purpose was to develop and determine initial evidence for validity of a brief adolescent personality inventory using a vocabulary that could be understood by adolescents up to 18 years old. Phase 1 (N = 48) consisted of item generation and expert (N = 5) review of items; Phase 2 (N = 179) involved item analyses; in Phase 3 (N = 496) exploratory factor analysis assessed the underlying structure; in Phase 4 (N = 405) confirmatory factor analyses resulted in a 25-item inventory with 5 subscales.     

Unidimensionality of Abbreviated Proactive Personality Scales across Cultures

Vu la pertinence de la personnalité proactive et du comportement proactif pour l’efficacité des individus, des équipes et des organisations dans un environnement de plus en plus multiculturel, cette étude a examiné l’unidimensionnalité des formes abrégées de l’Echelle de Personnalité Proactive (PPS) de Bateman & Crant (1993) sur des données ni américaines, ni britanniques. L’unidimensionnalité des PPS à 10, 6, 5 et 4 items a été mise à l’épreuve grâce à une analyse factorielle et à une analyse de fidélité interne sur des échantillons indépendants provenant de trois pays: la Belgique (N= 822), la Finlande (N= 100) et l’Espagne (N= 100). Les résultats montrent que les versions de 4 et 5 items ne présentent aucune fidélité interne en Belgique, en Finlande et en Espagne, tandis que les deux autres formes abrégées sont satisfaisantes. L’analyse factorielle confirme qu’un modèle à facteur unique est une solution quasi optimale pour la PPS à 10 items. La PPS à 6 items mesure la personnalité proactive avec une fidélité interne cohérente à partir d’un facteur unique. Le score total sur l’échelle a été calculé par l’addition des scores sur les 6 items. On a obtenu sur un autre échantillon belge (N= 499) une corrélation des plus satisfaisantes (r= .92) entre la PPS de 6 items et la version originelle de 17 items. Given the relevance of proactive personality and proactive behavior for effectiveness of individuals, teams, and organisations in an increasingly multicultural context, this study investigated the unidimensionality of abbreviated forms of the Proactive Personality Scale (PPS; Bateman & Crant, 1993 ) beyond American and British data. The unidimensionality of the 10‐item, the 6‐item, the 5‐item, and the 4‐item PPS was tested through internal reliability analysis and factor analysis across independent samples in three countries (Belgium, N= 822; Finland, N= 100; Spain, N= 100). The results showed that the 5‐item and the 4‐item versions were not internally reliable in Belgium, Finland, and Spain, while the two other abbreviated forms of the PPS were. Factor analysis showed that a one‐factor solution for the 10‐item PPS was a sub‐optimal solution. The 6‐item PPS, however, measured the proactive personality in an internally consistent manner and through a single factor. The total score on the scale was calculated by adding up scores on the six items. In a separate Belgian sample (N= 499), correlations of the 6‐item PPS with the original 17‐item PPS proved satisfactory with r= .92.     

Clustering preference data in the presence of response-style bias

Preference data, such as Likert scale data, are often obtained in questionnaire-based surveys. Clustering respondents based on survey items is useful for discovering latent structures. However, cluster analysis of preference data may be affected by response styles, that is, a respondent's systematic response tendencies irrespective of the item content. For example, some respondents may tend to select ratings at the ends of the scale, which is called an ‘extreme response style’. A cluster of respondents with an extreme response style can be mistakenly identified as a content-based cluster. To address this problem, we propose a novel method of clustering respondents based on their indicated preferences for a set of items while correcting for response-style bias. We first introduce a new framework to detect, and correct for, response styles by generalizing the definition of response styles used in constrained dual scaling. We then simultaneously correct for response styles and perform a cluster analysis based on the corrected preference data. A simulation study shows that the proposed method yields better clustering accuracy than the existing methods do. We apply the method to empirical data from four different countries concerning social values.     

Items bundles

An item bundle is a small group of multiple choice items that share a common reading passage or graph, or a small group of matching items that share distractors. Item bundles are easily identified by paging through a copy of a test. Bundled items may violate the latent conditional independence assumption of unidimensional item response theory (IRT), but such a violation would not typically suggest the existence of a new fundamental human ability to read one specific reading passage or to interpret one specific graph. It is important, therefore, to have theoretical concepts and empirical checks that distinguish between, on the one hand, anticipated violations of latent conditional independence within item bundles, and, on the other hand, violations that cannot be attributed to idiosyncratic features of test format and instead suggest departures from unidimensionalty. To this end, two theorems on unidimensional IRT are extended to describe observable item response distributions when there is conditional independencebetween but not necessarilywithin item bundles.The author is grateful to Ivo Molenaar and the referees for many helpful suggestions, and to D. Thayer for assistance with computing.     

Reconceptualizing adult attachment relationships: A network perspective

This article explores attachment relationships from a network theory perspective: Correlations among behaviors, beliefs, and feelings related to attachment are hypothesized to stem from causal relations. The authors used two data sets that assessed relationships with four attachment figures (mother, father, romantic partner, and best friend) using the Relationship Structures Questionnaire. Separate networks (Gaussian Graphical Models) were estimated based on 10 items for each attachment figure. Across networks in Study 1 (N = 310), items related to anxiety, seeking support, and discomfort disclosing feelings clustered with other items from their respective domains; a trust‐related item bridged the clusters. Study 2 replicated these findings in a larger and more diverse sample (N = 3,710). The potential of network analysis for advancing the study of attachment is discussed.     

Syntax for calculation of discounting indices from the monetary choice questionnaire and probability discounting questionnaire

The 27‐item Monetary Choice Questionnaire (MCQ; Kirby, Petry, & Bickel, 1999) and 30‐item Probability Discounting Questionnaire (PDQ; Madden, Petry, & Johnson, 2009) are widely used, validated measures of preferences for immediate versus delayed rewards and guaranteed versus risky rewards, respectively. The MCQ measures delayed discounting by asking individuals to choose between rewards available immediately and larger rewards available after a delay. The PDQ measures probability discounting by asking individuals to choose between guaranteed rewards and a chance at winning larger rewards. Numerous studies have implicated these measures in addiction and other health behaviors. Unlike typical self‐report measures, the MCQ and PDQ generate inferred hyperbolic temporal and probability discounting functions by comparing choice preferences to arrays of functions to which the individual items are preconfigured. This article provides R and SPSS syntax for processing the MCQ and PDQ. Specifically, for the MCQ, the syntax generates k values, consistency of the inferred k, and immediate choice ratios; for the PDQ, the syntax generates h indices, consistency of the inferred h, and risky choice ratios. The syntax is intended to increase the accessibility of these measures, expedite the data processing, and reduce risk for error.     

Rural and Urban Elderly Construe Health Differently

Samples of people aged 65 or older (N = 396) living in the metropolitan Omaha area and in the rural Sandhills counties of central and western Nebraska completed an instrument to assess health satisfaction, health behaviors, and attitudes toward heath care. Few intergroup differences were found that could be attributed to the area of residence. However, factor analysis and item analysis of the attitudes toward health items indicated that older respondents in rural areas may have very different perceptions of health in general and of health care services in particular than those of elderly urban residents.     

Dissecting the expanded cognitive reflection test: an item response theory analysis

This study presents a psychometric evaluation of the Expanded Cognitive Reflection Test (CRT7) based on item response theory. The participants (N?=?1204) completed the CRT7 and provided self-reported information about their cognitive styles through the Preference for Intuition and Deliberation Scale (PID). A two-parameter logistic model was fitted to the data to obtain the item difficulty and discrimination parameters of the CRT7. The results showed that the items had good discriminatory power (αs?=?.80 ? 2.92), but the range of difficulty was restricted (βs ranged from ?.60 to .32). Moreover, the CRT7 showed a pattern of correlations with the PID which was similar to that of the original CRT. When taken together, these results are evidence of the adequacy of the CRT7 as an expanded tool for measuring cognitive reflection; however, one of the newer items (the pig item) was consistently problematic across analyses, and so it is recommended that in future studies it should be removed from the CRT7.     

Evaluating the Psychometric and Measurement Characteristics of a Measure of Sexual Orientation Harassment

We use classical test theory (CTT) and item response theory (IRT) methodologies to examine the psychometric and measurement properties of an instrument designed to assess sexual orientation harassment among military personnel (N?=?71,989). CTT analyses indicated that items were unidimensional and exhibited adequate levels of reliability. IRT analyses demonstrated that the items functioned similarly and exhibited appropriate levels of item discrimination. However, the analyses also suggested that the sensitivity of the items may be limited. Differential test functioning analyses provided evidence of the measurement equivalence of the instrument across male and female respondents. The findings provide support for the psychometric properties and measurement equivalence of the instrument for measuring sexual orientation harassment among male and female military personnel. We discuss the implications of our findings for future research on sexual orientation harassment in the workplace.     

On estimation of parameters in latent structure analysis

The latent structure model considered here postulates that a population of individuals can be divided intom classes such that each class is homogeneous in the sense that for the individuals in the class the responses toN dichotomous items or questions are statistically independent. A method is given for deducing the proportions of the population in each latent class and the probabilities of positive responses to each item for individuals in each class from knowledge of the probabilities of positive responses for individuals from the population as a whole. For estimation of the latent parameters on the basis of a sample, it is proposed that the same method of analysis be applied to the observed data. The method has the advantages of avoiding implicitly defined and unobservable quantities, and of using relatively simple computational procedures of conventional matrix algebra, but it has the disadvantages of using only a part of the available information and of using that part asymmetrically.Work supported by the RAND Corporation.