A General Family of Limited Information Goodness-of-Fit Statistics for Multinomial Data

Maydeu-Olivares and Joe (J. Am. Stat. Assoc. 100:1009–1020, 2005; Psychometrika 71:713–732, 2006) introduced classes of chi-square tests for (sparse) multidimensional multinomial data based on low-order marginal proportions. Our extension provides general conditions under which quadratic forms in linear functions of cell residuals are asymptotically chi-square. The new statistics need not be based on margins, and can be used for one-dimensional multinomials. We also provide theory that explains why limited information statistics have good power, regardless of sparseness. We show how quadratic-form statistics can be constructed that are more powerful than X ² and yet, have approximate chi-square null distribution in finite samples with large models. Examples with models for truncated count data and binary item response data are used to illustrate the theory.     

Assessing Approximate Fit in Categorical Data Analysis

A family of Root Mean Square Error of Approximation (RMSEA) statistics is proposed for assessing the goodness of approximation in discrete multivariate analysis with applications to item response theory (IRT) models. The family includes RMSEAs to assess the approximation up to any level of association of the discrete variables. Two members of this family are RMSEA₂, which uses up to bivariate moments, and the full information RMSEA_n. The RMSEA₂ is estimated using the M₂ statistic of Maydeu-Olivares and Joe (2005, 2006), whereas for maximum likelihood estimation, RMSEA_n is estimated using Pearson's X² statistic. Using IRT models, we provide cutoff criteria of adequate, good, and excellent fit using the RMSEA₂. When the data are ordinal, we find a strong linear relationship between the RMSEA₂ and the Standardized Root Mean Squared Residual goodness-of-fit index. We are unable to offer cutoff criteria for the RMSEA_n as its population values decrease as the number of variables and categories increase.     

How Should We Assess the Fit of Rasch-Type Models? Approximating the Power of Goodness-of-Fit Statistics in Categorical Data Analysis

We investigate the performance of three statistics, R ₁, R ₂ (Glas in Psychometrika 53:525–546, 1988), and M ₂ (Maydeu-Olivares & Joe in J. Am. Stat. Assoc. 100:1009–1020, 2005, Psychometrika 71:713–732, 2006) to assess the overall fit of a one-parameter logistic model (1PL) estimated by (marginal) maximum likelihood (ML). R ₁ and R ₂ were specifically designed to target specific assumptions of Rasch models, whereas M ₂ is a general purpose test statistic. We report asymptotic power rates under some interesting violations of model assumptions (different item discrimination, presence of guessing, and multidimensionality) as well as empirical rejection rates for correctly specified models and some misspecified models. All three statistics were found to be more powerful than Pearson’s X ² against two- and three-parameter logistic alternatives (2PL and 3PL), and against multidimensional 1PL models. The results suggest that there is no clear advantage in using goodness-of-fit statistics specifically designed for Rasch-type models to test these models when marginal ML estimation is used.     

Limited information estimation and testing of discretized multivariate normal structural models

Discretized multivariate normal structural models are often estimated using multistage estimation procedures. The asymptotic properties of parameter estimates, standard errors, and tests of structural restrictions on thresholds and polychoric correlations are well known. It was not clear how to assess the overall discrepancy between the contingency table and the model for these estimators. It is shown that the overall discrepancy can be decomposed into a distributional discrepancy and a structural discrepancy. A test of the overall model specification is proposed, as well as a test of the distributional specification (i.e., discretized multivariate normality). Also, the small sample performance of overall, distributional, and structural tests, as well as of parameter estimates and standard errors is investigated under conditions of correct model specification and also under mild structural and/or distributional misspecification. It is found that relatively small samples are needed for parameter estimates, standard errors, and structural tests. Larger samples are needed for the distributional and overall tests. Furthermore, parameter estimates, standard errors, and structural tests are surprisingly robust to distributional misspecification. This research was supported by the Department of Universities, Research and Information Society (DURSI) of the Catalan Government, and by grants BSO2000-0661 and BSO2003-08507 of the Spanish Ministry of Science and Technology.     

The effect of varying the number of response alternatives in rating scales: Experimental evidence from intra-individual effects

Despite a hundred years of questionnaire testing, no consensus has been reached on the optimal number of response alternatives in rating scales. Differences in prior research may have been due to the use of various psychometric models (classical test theory, item factor analysis, and item response theory) and different performance criteria (reliability, convergent/discriminant validity, and internal structure of the questionnaire). Furthermore, previous empirical studies on this issue have tackled the experimental design from a between-subjects perspective, thus ignoring intra-individual effects. In contrast with this approach, we propose a within-subjects experimental design and a comprehensive statistical methodology using structural equation models for studying all of these aspects simultaneously, therefore increasing statistical power. To illustrate the method, two personality questionnaires were examined using a repeated measures design. Results indicated that as the number of response alternatives increased, (1) internal consistency increased, (2) there was no effect on convergent validity, and (3) goodness of fit worsened. Finally, the article assesses the practical consequences of this research for the design of future personality questionnaires.     

Structural equation modeling of paired-comparison and ranking data

L. L. Thurstone's (1927) model provides a powerful framework for modeling individual differences in choice behavior. An overview of Thurstonian models for comparative data is provided, including the classical Case V and Case III models as well as more general choice models with unrestricted and factor-analytic covariance structures. A flow chart summarizes the model selection process. The authors show how to embed these models within a more familiar structural equation modeling (SEM) framework. The different special cases of Thurstone's model can be estimated with a popular SEM statistical package, including factor analysis models for paired comparisons and rankings. Only minor modifications are needed to accommodate both types of data. As a result, complex models for comparative judgments can be both estimated and tested efficiently.     

A Polychoric Instrumental Variable (PIV) Estimator for Structural Equation Models with Categorical Variables

This paper presents a new polychoric instrumental variable (PIV) estimator to use in structural equation models (SEMs) with categorical observed variables. The PIV estimator is a generalization of Bollen’s (Psychometrika 61:109–121, 1996) 2SLS/IV estimator for continuous variables to categorical endogenous variables. We derive the PIV estimator and its asymptotic standard errors for the regression coefficients in the latent variable and measurement models. We also provide an estimator of the variance and covariance parameters of the model, asymptotic standard errors for these, and test statistics of overall model fit. We examine this estimator via an empirical study and also via a small simulation study. Our results illustrate the greater robustness of the PIV estimator to structural misspecifications than the system-wide estimators that are commonly applied in SEMs. Kenneth Bollen gratefully acknowledges support from NSF SES 0617276, NIDA 1-RO1-DA13148-01, and DA013148-05A2. Albert Maydeu-Olivares was supported by the Department of Universities, Research and Information Society (DURSI) of the Catalan Government, and by grant BSO2003-08507 from the Spanish Ministry of Science and Technology. We thank Sharon Christ, John Hipp, and Shawn Bauldry for research assistance. The comments of the members of the Carolina Structural Equation Modeling (CSEM) group are greatly appreciated. An earlier version of this paper under a different title was presented by K. Bollen at the Psychometric Society Meetings, June, 2002, Chapel Hill, North Carolina.     

Asymptotically distribution-free (ADF) interval estimation of coefficient alpha

The point estimate of sample coefficient alpha may provide a misleading impression of the reliability of the test score. Because sample coefficient alpha is consistently biased downward, it is more likely to yield a misleading impression of poor reliability. The magnitude of the bias is greatest precisely when the variability of sample alpha is greatest (small population reliability and small sample size). Taking into account the variability of sample alpha with an interval estimator may lead to retaining reliable tests that would be otherwise rejected. Here, the authors performed simulation studies to investigate the behavior of asymptotically distribution-free (ADF) versus normal-theory interval estimators of coefficient alpha under varied conditions. Normal-theory intervals were found to be less accurate when item skewness >1 or excess kurtosis >1. For sample sizes over 100 observations, ADF intervals are preferable, regardless of item skewness and kurtosis. A formula for computing ADF confidence intervals for coefficient alpha for tests of any size is provided, along with its implementation as an SAS macro.     

IQ heritability estimation: analyzing genetically-informative data with structural equation models

When analyzing genetic data, Structural Equations Modeling (SEM) provides a straightforward methodology to decompose phenotypic variance using a model-based approach. Furthermore, several models can be easily implemented, tested, and compared using SEM, allowing the researcher to obtain valuable information about the sources of variability. This methodology is briefly described and applied to re-analyze a Spanish set of IQ data using the biometric ACE model. In summary, we report heritability estimates that are consistent with those of previous studies and support substantial genetic contribution to phenotypic IQ; around 40% of the variance can be attributable to it. With regard to the environmental contribution, shared environment accounts for 50% of the variance, and non-shared environment accounts for the remaining 10%. These results are discussed in the text.