A tutorial on how to do a Mokken scale analysis on your test and questionnaire data

Over the past decade, Mokken scale analysis (MSA) has rapidly grown in popularity among researchers from many different research areas. This tutorial provides researchers with a set of techniques and a procedure for their application, such that the construction of scales that have superior measurement properties is further optimized, taking full advantage of the properties of MSA. First, we define the conceptual context of MSA, discuss the two item response theory (IRT) models that constitute the basis of MSA, and discuss how these models differ from other IRT models. Second, we discuss dos and don'ts for MSA; the don'ts include misunderstandings we have frequently encountered with researchers in our three decades of experience with real‐data MSA. Third, we discuss a methodology for MSA on real data that consist of a sample of persons who have provided scores on a set of items that, depending on the composition of the item set, constitute the basis for one or more scales, and we use the methodology to analyse an example real‐data set.     

Outlier Detection in Test and Questionnaire Data

Classical methods for detecting outliers deal with continuous variables. These methods are not readily applicable to categorical data, such as incorrect/correct scores (0/1) and ordered rating scale scores (e.g., 0, …, 4) typical of multi-item tests and questionnaires. This study proposes two definitions of outlier scores suited for categorical data. One definition combines information on outliers from scores on all the items in the test, and the other definition combines information from all pairs of item scores. For a particular item-score vector, an outlier score expresses the degree to which the item-score vector is unusual. For ten real-data sets, the distribution of each of the two outlier scores is inspected by means of Tukey's fences and the extreme studentized deviate procedure. It is investigated whether the outliers that are identified are influential with respect to the statistical analysis performed on these data. Recommendations are given for outlier identification and accommodation in test and questionnaire data.     

A method of optimal scaling for multivariate ordinal data and its extensions

This paper develops a method of optimal scaling for multivariate ordinal data, in the framework of a generalized principal component analysis. This method yields a multidimensional configuration of items, a unidimensional scale of category weights for each item and, optionally, a multidimensional configuration of subjects. The computation is performed by alternately solving an eigenvalue problem and executing a quasi-Newton projection method. The algorithm is extended for analysis of data with mixed measurement levels or for analysis with a combined weighting of items. Numerical examples and simulations are provided. The algorithm is discussed and compared with some related methods.Earlier results of this research appeared in Saito and Otsu (1983). The authors would like to acknowledge the helpful comments and encouragement of the editor.     

Stochastic search item selection for factor analytic models

In this paper we implement a Markov chain Monte Carlo algorithm based on the stochastic search variable selection method of George and McCulloch (1993) for identifying promising subsets of manifest variables (items) for factor analysis models. The suggested algorithm is constructed by embedding in the usual factor analysis model a normal mixture prior for the model loadings with latent indicators used to identify not only which manifest variables should be included in the model but also how each manifest variable is associated with each factor. We further extend the suggested algorithm to allow for factor selection. We also develop a detailed procedure for the specification of the prior parameters values based on the practical significance of factor loadings using ideas from the original work of George and McCulloch (1993). A straightforward Gibbs sampler is used to simulate from the joint posterior distribution of all unknown parameters and the subset of variables with the highest posterior probability is selected. The proposed method is illustrated using real and simulated data sets.     

Detecting Differential Item Functioning Using Multiple-Group Cognitive Diagnosis Models

This study proposes a multiple-group cognitive diagnosis model to account for the fact that students in different groups may use distinct attributes or use the same attributes but in different manners (e.g., conjunctive, disjunctive, and compensatory) to solve problems. Based on the proposed model, this study systematically investigates the performance of the likelihood ratio (LR) test and Wald test in detecting differential item functioning (DIF). A forward anchor item search procedure was also proposed to identify a set of anchor items with invariant item parameters across groups. Results showed that the LR and Wald tests with the forward anchor item search algorithm produced better calibrated Type I error rates than the ordinary LR and Wald tests, especially when items were of low quality. A set of real data were also analyzed to illustrate the use of these DIF detection procedures.     

A comparative study of the dimensionality of the self-concealment scale using principal components analysis and Mokken scale analysis

We discuss and contrast 2 methods for investigating the dimensionality of data from tests and questionnaires: the popular principal components analysis (PCA) and the more recent Mokken scale analysis (MSA; Mokken, 1971). First, we discuss the theoretical similarities and differences between both methods. Then, we use both methods to analyze data collected by means of Larson and Chastain's (1990) Self-Concealment Scale (SCS). We present the different results and highlight the instances in which the methods complement one another so as to obtain a stronger result than would be obtained using only 1 method. Finally, we discuss the implications of the results for the dimensionality of the SCS and provide recommendations for both the further development of the SCS and the future use of PCA and MSA in personality research.     

Robust canonical discriminant analysis

A method for robust canonical discriminant analysis via two robust objective loss functions is discussed. These functions are useful to reduce the influence of outliers in the data. Majorization is used at several stages of the minimization procedure to obtain a monotonically convergent algorithm. An advantage of the proposed method is that it allows for optimal scaling of the variables. In a simulation study it is shown that under the presence of outliers the robust functions outperform the ordinary least squares function, both when the underlying structure is linear in the variables as when it is nonlinear. Furthermore, the method is illustrated with empirical data.The research of the first author was supported by the Netherlands Organization of Scientific Research (NWO grant 560-267-029).     

A DIMENSIONAL ANALYSIS OF MMPI ITEMS

A method of analysis specifically designed for binary data was applied to 100 MMPI items. Sixty, items were carefully chosen to represent the nine major clinical scales with respect to direction of keying, social desirability scale value and endorsement frequency. The remaining 40 items were randomly chosen from items not appearing on any of these scales. Although a complete solution was obtained in five dimensions, only three were retained. The three dimensions were related to scale membership, gender of the respondent and various item characteristics. The results clearly support the two major MMPI factors obtained on a scale level and additionally show a strong gender dimension.     

Dimensionality Analysis of the Thought Suppression Inventory: Combining EFA, MSA, and CFA

The Thought Suppression Inventory (TSI; Rassin, European Journal of Personality 17: 285-298, 2003) was designed to measure thought intrusion, thought suppression and successful thought suppression. Given the importance to distinguish between these three aspects of thought control, the aim of this study was to scrutinize the dimensionality of the TSI. In a sample of 333 Dutch senior citizins, we examined (1) the dimensionality of the TSI using various procedures such as PAF, Mokken scale analysis (MSA) and CFA, and (2) the scale properties of the TSI. PAF favored a two factor solution, however, MSA and CFA suggested that three dimensions most adequately capture the structure of the TSI. Although all scales obtained at least medium scalability coefficients, several items were identified that are psychometrically unsound and may benefit from rewording or replacement. The findings suggest that the TSI is a three-dimensional questionnaire as originally proposed by Rassin (European Journal of Personality 17: 285-298, 2003) measuring thought intrusion, thought suppression, and successful thought suppression.     

A FORTRAN 77 program for a nonparametric item response model: The Mokken scale analysis

A nonparametric item response theory model—the Mokken scale analysis (a stochastic elaboration of the deterministic Guttman scale)—and a computer program that performs this analysis are described. Three procedures of scaling are distinguished: a search procedure, an evaluation of the whole set of items, and an extension of an existing scale. All procedures provide a coefficient of scalability for all items that meet the criteria of the Mokken model and an item coefficient of scalability for every item. Four different types of reliability coefficient are computed both for the entire set of items and for the scalable items. A test of robustness of the found scale can be performed to analyze whether the scale is invariant across different subgroups or samples. This robustness test serves as a goodness of fit test for the established scale. The program is written in FORTRAN 77. Two versions are available, an SPSS-X procedure program (which can be used with the SPSS-X mainframe package) and a stand-alone program suitable for both mainframe and microcomputers.     

Scale analysis and the measurement of social attitudes

This paper discusses and compares the methods of attitude scale construction of Thurstone (method of equal-appearing intervals), Likert (method of summated ratings), and Guttman (method of scale analysis), with special emphasis on the latter as one of the most recent and significant contributions to the field. Despite a certain lack of methodological precision, scale analysis provides a means of evaluating the uni-dimensionality of a set of items. If the criteria for uni-dimensionality are met, the interpretation of rank-order scores is made unambiguous, and efficiency of prediction from the set of items is maximized. The Guttman technique, however, provides no satisfactory means of selecting the original set of items for scale analysis. Preliminary studies indicated that both the Likert and the Thurstone methods tend to select scalable sets of items and that their functions in this respect are complementary. A method of combining the Likert and Thurstone methods in order to yield a highly scalable set of items is outlined. Sets of 14 items selected by the method have, in the two cases where the technique has been tried, yielded very satisfactory scalability.     

Model specification search using a genetic algorithm with factor reordering for a simple structure factor analysis model

Abstract:  Many techniques for automated model specification search based on numerical indices have been proposed, but no single decisive method has yet been determined. In the present article, the performance and features of the model specification search method using a genetic algorithm (GA) were verified. A GA is a robust and simple metaheuristic algorithm with great searching power. While there has already been some research applying metaheuristics to the model fitting task, we focus here on the search for a simple structure factor analysis model and propose a customized algorithm for dealing with certain problems specific to that situation. First, implementation of model specification search using a GA with factor reordering for a simple structure factor analysis is proposed. Then, through a simulation study using generated data with a known true structure and through example analysis using real data, the effectiveness and applicability of the proposed method were demonstrated.     

Clustering preference data in the presence of response-style bias

Preference data, such as Likert scale data, are often obtained in questionnaire-based surveys. Clustering respondents based on survey items is useful for discovering latent structures. However, cluster analysis of preference data may be affected by response styles, that is, a respondent's systematic response tendencies irrespective of the item content. For example, some respondents may tend to select ratings at the ends of the scale, which is called an ‘extreme response style’. A cluster of respondents with an extreme response style can be mistakenly identified as a content-based cluster. To address this problem, we propose a novel method of clustering respondents based on their indicated preferences for a set of items while correcting for response-style bias. We first introduce a new framework to detect, and correct for, response styles by generalizing the definition of response styles used in constrained dual scaling. We then simultaneously correct for response styles and perform a cluster analysis based on the corrected preference data. A simulation study shows that the proposed method yields better clustering accuracy than the existing methods do. We apply the method to empirical data from four different countries concerning social values.     

Beyond internal consistency reliability: Rationale and user’s guide for Multitrait Analysis Program on the microcomputer

A methodology for evaluating Likert-type scales is presented. Multitrait scaling is a straightforward approach to scale analysis that focuses on items as the unit of analysis and utilizes the logic of convergent and discriminant validity. Multitrait scaling is illustrated with the Multitrait Analysis Program, using patient satisfaction data from the Medical Outcomes Study.     

Effects of estimation methods on making trait-level inferences from ordered categorical items for assessing psychopathology

In assessments of attitudes, personality, and psychopathology, unidimensional scale scores are commonly obtained from Likert scale items to make inferences about individuals' trait levels. This study approached the issue of how best to combine Likert scale items to estimate test scores from the practitioner's perspective: Does it really matter which method is used to estimate a trait? Analyses of 3 data sets indicated that commonly used methods could be classified into 2 groups: methods that explicitly take account of the ordered categorical item distributions (i.e., partial credit and graded response models of item response theory, factor analysis using an asymptotically distribution-free estimator) and methods that do not distinguish Likert-type items from continuously distributed items (i.e., total score, principal component analysis, maximum-likelihood factor analysis). Differences in trait estimates were found to be trivial within each group. Yet the results suggested that inferences about individuals' trait levels differ considerably between the 2 groups. One should therefore choose a method that explicitly takes account of item distributions in estimating unidimensional traits from ordered categorical response formats. Consequences of violating distributional assumptions were discussed.     

Identification of Differential Item Functioning in Multiple-Group Settings: A Multivariate Outlier Detection Approach

We focus on the identification of differential item functioning (DIF) when more than two groups of examinees are considered. We propose to consider items as elements of a multivariate space, where DIF items are outlying elements. Following this approach, the situation of multiple groups is a quite natural case. A robust statistics technique is proposed to identify DIF items as outliers in the multivariate space. For low dimensionalities, up to 2–3 groups, a simple graphical tool is derived. We illustrate our approach with a reanalysis of data from Kim, Cohen, and Park (1995) on using calculators for a mathematics test.     

Choice model analysis of the “pick any/n” type of binary data

In the “pick any/n” method, subjects are asked to choose any number of items from a list of n items according to some criterion. This kind of data can be analyzed as a special case of either multiple-choice data or successive categories data where the number of response categories is limited to two. An item response model was proposed for the latter case, which is a combination of an unfolding model and a choice model. The marginal maximum-likelihood estimation method was developed for parameter estimation to avoid incidental parameters, and an expectation-maximization algorithm used for numerical optimization. Two examples of analysis are given to illustrate the proposed method, which we call MAXSC.     

Measurement of self-evaluative motives: a shopping scenario

To develop measures of consumers' self-evaluative motives of Self-verification, Self-enhancement, and Self-improvement within the context of a mall shopping environment, an initial set of 49 items was generated by conducting three focus-group sessions. These items were subsequently converted into shopping-dependent motive statements. 250 undergraduate college students responded on a 7-point scale to each statement as these related to the acquisition of recent personal shopping goods. An exploratory factor analysis yielded five factors, accounting for 57.7% of the variance, three of which corresponded to the Self-verification motive (five items), Self-enhancement motive (three items), and Self-improvement motive (six items). These 14 items, along with 9 reconstructed items, yielded 23 items retained and subjected to additional testing. In a final round of data collection, 169 college students provided data for exploratory factor analysis. 11 items were used in confirmatory factor analysis. Analysis indicated that the 11-item scale adequately captured measures of the three self-evaluative motives. However, further data reduction produced a 9-item scale with marked improvement in statistical fit over the 11-item scale.     

Odd-item search by pigeons: Method,instrumentation, and uses

In odd-item visual search, subjects confront a display on which a number of stimulus items appear. All but one of these items are identical; the subject must respond to the one item (the target) that in some way differs from all the others (the distractors). The time required to find the target reflects the similarity between the target form and the distractor form. A matrix of search times for all possible pairs of a set of 20 or more items can be obtained in a single session. Such similarity matrices may reflect stimulus features, dimensions, and categories, among other things. A method is described through which pigeons learn odd-item search rapidly and perform with high accuracy despite the appearance of each form as a target on some trials and as a distractor on others. The paper also describes the essential apparatus and exemplifies displays and data.