Review of Issues About Classical Change Scores: A Multilevel Modeling Perspective on Some Enduring Beliefs

Change scores obtained in pretest–posttest designs are important for evaluating treatment effectiveness and for assessing change of individual test scores in psychological research. However, over the years the use of change scores has raised much controversy. In this article, from a multilevel perspective, we provide a structured treatise on several persistent negative beliefs about change scores and show that these beliefs originated from the confounding of the effects of within-person change on change-score reliability and between-person change differences. We argue that psychometric properties of change scores, such as reliability and measurement precision, should be treated at suitable levels within a multilevel framework. We show that, if examined at the suitable levels with such a framework, the negative beliefs about change scores can be renounced convincingly. Finally, we summarize the conclusions about change scores to dispel the myths and to promote the potential and practical usefulness of change scores.     

Mokken Scale Analysis for Dichotomous Items Using Marginal Models

Scalability coefficients play an important role in Mokken scale analysis. For a set of items, scalability coefficients have been defined for each pair of items, for each individual item, and for the entire scale. Hypothesis testing with respect to these scalability coefficients has not been fully developed. This study introduces marginal modelling as a framework to derive the standard errors for the scaling coefficients and test hypotheses about these coefficients. Several examples demonstrate the possibilities of marginal modelling in Mokken scale analysis. These possibilities include testing whether Mokken’s criteria for a scale are satisfied, testing whether scalability coefficients of different items are equal, and testing whether scalability coefficients are equal across different groups.     

On the Usefulness of a Multilevel Logistic Regression Approach to Person-Fit Analysis

The logistic person response function (PRF) models the probability of a correct response as a function of the item locations. Reise (2000) Reise, S. P. 2000. Using multilevel logistic regression to evaluate person-fit in IRT models. Multivariate Behavioral Research, 35: 543–568. [Taylor &; Francis Online], [Web of Science ®] , [Google Scholar] proposed to use the slope parameter of the logistic PRF as a person-fit measure. He reformulated the logistic PRF model as a multilevel logistic regression model and estimated the PRF parameters from this multilevel framework. An advantage of the multilevel framework is that it allows relating person fit to explanatory variables for person misfit/fit. We critically discuss Reise's approach. First, we argue that often the interpretation of the PRF slope as an indicator of person misfit is incorrect. Second, we show that the multilevel logistic regression model and the logistic PRF model are incompatible, resulting in a multilevel person-fit framework, which grossly violates the bivariate normality assumption for residuals in the multilevel model. Third, we use a Monte Carlo study to show that in the multilevel logistic regression framework estimates of distribution parameters of PRF intercepts and slopes are biased. Finally, we discuss the implications of these results and suggest an alternative multilevel regression approach to explanatory person-fit analysis. We illustrate the alternative approach using empirical data on repeated anxiety measurements of cardiac arrhythmia patients who had a cardioverter-defibrillator implanted.     

A general solution to Mosier's oblique procrustes problem

Browne provided a method for finding a solution to the normal equations derived by Mosier for rotating a factor matrix to a best least squares fit with a specified structure. Cramer showed that Browne's solution is not always valid, and proposed a modified algorithm. Both Browne and Cramer assumed the factor matrix to be of full rank. In this paper a general solution is derived, which takes care of rank deficient factor matrices as well. A new algorithm is offered.     

Psychometrics in Psychological Research: Role Model or Partner in Science?

This is a reaction to Borsboom's (2006) discussion paper on the issue that psychology takes so little notice of the modern developments in psychometrics, in particular, latent variable methods. Contrary to Borsboom, it is argued that latent variables are summaries of interesting data properties, that construct validation should involve studying nomological networks, that psychological research slowly but definitely will incorporate latent variable methods, and that the role of psychometrics in psychology is that of partner, not role model. Requests for reprints should be sent to Klaas Sijtsma, Department of Methodology and Statistics, FSW, Tilburg University, PO Box 90153, 5000 LE, Tilburg, The Netherlands.     

Global, local, and graphical person-fit analysis using person-response functions

Person-fit statistics test whether the likelihood of a respondent's complete vector of item scores on a test is low given the hypothesized item response theory model. This binary information may be insufficient for diagnosing the cause of a misfitting item-score vector. The authors propose a comprehensive methodology for person-fit analysis in the context of nonparametric item response theory. The methodology (a) includes H. Van der Flier's (1982) global person-fit statistic U3 to make the binary decision about fit or misfit of a person's item-score vector, (b) uses kernel smoothing (J. O. Ramsay, 1991) to estimate the person-response function for the misfitting item-score vectors, and (c) evaluates unexpected trends in the person-response function using a new local person-fit statistic (W. H. M. Emons, 2003). An empirical data example shows how to use the methodology for practical person-fit analysis.