CHull: A generic convex-hull-based model selection method

When analyzing data, researchers are often confronted with a model selection problem (e.g., determining the number of components/factors in principal components analysis [PCA]/factor analysis or identifying the most important predictors in a regression analysis). To tackle such a problem, researchers may apply some objective procedure, like parallel analysis in PCA/factor analysis or stepwise selection methods in regression analysis. A drawback of these procedures is that they can only be applied to the model selection problem at hand. An interesting alternative is the CHull model selection procedure, which was originally developed for multiway analysis (e.g., multimode partitioning). However, the key idea behind the CHull procedure—identifying a model that optimally balances model goodness of fit/misfit and model complexity—is quite generic. Therefore, the procedure may also be used when applying many other analysis techniques. The aim of this article is twofold. First, we demonstrate the wide applicability of the CHull method by showing how it can be used to solve various model selection problems in the context of PCA, reduced K-means, best-subset regression, and partial least squares regression. Moreover, a comparison of CHull with standard model selection methods for these problems is performed. Second, we present the CHULL software, which may be downloaded from http://ppw.kuleuven.be/okp/software/CHULL/, to assist the user in applying the CHull procedure.     

Bootstrap区间估计在认知诊断模型误设中的应用

认知诊断模型选择是认知诊断评估中重要研究问题之一。在实际应用中实践者并不知道真正拟合数据的模型,通常会用模型拟合指标检验模型与数据的拟合程度。从测量结果质量来看,除保证模型与数据拟合之外,还需要重点评价模型诊断结果的信度和效度等。考虑到以往研究大都采用基于信息量的拟合指标去判定模型与数据的匹配性,本研究提出综合考虑模型拟合指标与信度指标用于模型选择或评价模型误设。考虑实验因素为真实模型或分析模型（DINA模型、G-DINA模型、R-RUM模型）、样本量、题量和属性个数,在五因素（3×3×2×2×2）实验设计条件下,比较Bootstrap区间估计的属性分类一致性信度平均数与标准误和常用的拟合统计量-2LL、AIC、BIC对正确模型的选择率。结果表明：-2LL在题目数量多的情况下表现较好,而AIC、BIC在被试量较大的情况下表现较好,在不同的研究条件下,-2LL、AIC、BIC的模型选择率很不稳定,而用Bootstrap法估计的属性分类一致性信度平均数和标准误在不同研究条件的模型选择率较稳定,总体表现较好。     

Model selection and psychological theory: a discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC)

This article reviews the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) in model selection and the appraisal of psychological theory. The focus is on latent variable models, given their growing use in theory testing and construction. Theoretical statistical results in regression are discussed, and more important issues are illustrated with novel simulations involving latent variable models including factor analysis, latent profile analysis, and factor mixture models. Asymptotically, the BIC is consistent, in that it will select the true model if, among other assumptions, the true model is among the candidate models considered. The AIC is not consistent under these circumstances. When the true model is not in the candidate model set the AIC is efficient, in that it will asymptotically choose whichever model minimizes the mean squared error of prediction/estimation. The BIC is not efficient under these circumstances. Unlike the BIC, the AIC also has a minimax property, in that it can minimize the maximum possible risk in finite sample sizes. In sum, the AIC and BIC have quite different properties that require different assumptions, and applied researchers and methodologists alike will benefit from improved understanding of the asymptotic and finite-sample behavior of these criteria. The ultimate decision to use the AIC or BIC depends on many factors, including the loss function employed, the study's methodological design, the substantive research question, and the notion of a true model and its applicability to the study at hand.     

测验相对拟合检验方法CVLL法在认知诊断中的拓展及应用

本文将IRT中表现较好的CVLL法引入到认知诊断领域,同时比较并分析CVLL及认知诊断领域已有的测验相对拟合检验统计量的表现,为实际工作者在认知诊断模型选用上提供方法学支持和借鉴。结果表明:CVLL的表现比其它传统测验相对拟合统计量要好;且当对Q矩阵进行误设时,该统计量也能选择较优的Q矩阵,说明CVLL在Q矩阵侦查上有较好的应用前景。     

A comparison of statistical selection strategies for univariate and bivariate log‐linear models

In this study, eight statistical selection strategies were evaluated for selecting the parameterizations of log‐linear models used to model the distributions of psychometric tests. The selection strategies included significance tests based on four chi‐squared statistics (likelihood ratio, Pearson, Freeman–Tukey, and Cressie–Read) and four additional strategies (Akaike information criterion (AIC), Bayesian information criterion (BIC), consistent Akaike information criterion (CAIC), and a measure attributed to Goodman). The strategies were evaluated in simulations for different log‐linear models of univariate and bivariate test‐score distributions and two sample sizes. Results showed that all eight selection strategies were most accurate for the largest sample size considered. For univariate distributions, the AIC selection strategy was especially accurate for selecting the correct parameterization of a complex log‐linear model and the likelihood ratio chi‐squared selection strategy was the most accurate strategy for selecting the correct parameterization of a relatively simple log‐linear model. For bivariate distributions, the likelihood ratio chi‐squared, Freeman–Tukey chi‐squared, BIC, and CAIC selection strategies had similarly high selection accuracies.     

Regarding the complexity of additive clustering models: Comment on Lee (2001)

The additive clustering approach to modeling pairwise similarity of entities is a powerful tool for deriving featural stimulus representations. In a recent paper, Lee (2001) proposes a statistically principled measure for choosing between clustering models that accounts for model complexity as well as data fit. Importantly, complexity is understood to be a property, not merely of the number of clusters, but also their size and pattern of overlap. However, some caution is required when interpreting the measure, with regard to the applicability of the Hadamard inequality to the complexity matrix.     

Subspace K-means clustering

To achieve an insightful clustering of multivariate data, we propose subspace K-means. Its central idea is to model the centroids and cluster residuals in reduced spaces, which allows for dealing with a wide range of cluster types and yields rich interpretations of the clusters. We review the existing related clustering methods, including deterministic, stochastic, and unsupervised learning approaches. To evaluate subspace K-means, we performed a comparative simulation study, in which we manipulated the overlap of subspaces, the between-cluster variance, and the error variance. The study shows that the subspace K-means algorithm is sensitive to local minima but that the problem can be reasonably dealt with by using partitions of various cluster procedures as a starting point for the algorithm. Subspace K-means performs very well in recovering the true clustering across all conditions considered and appears to be superior to its competitor methods: K-means, reduced K-means, factorial K-means, mixtures of factor analyzers (MFA), and MCLUST. The best competitor method, MFA, showed a performance similar to that of subspace K-means in easy conditions but deteriorated in more difficult ones. Using data from a study on parental behavior, we show that subspace K-means analysis provides a rich insight into the cluster characteristics, in terms of both the relative positions of the clusters (via the centroids) and the shape of the clusters (via the within-cluster residuals).     

Evaluation of Model Fit in Cognitive Diagnosis Models

Cognitive diagnosis models (CDMs) estimate student ability profiles using latent attributes. Model fit to the data needs to be ascertained in order to determine whether inferences from CDMs are valid. This study investigated the usefulness of some popular model fit statistics to detect CDM fit including relative fit indices (AIC, BIC, and CAIC), and absolute fit indices (RMSEA2, ABS(fcor) and MAX(χ²_jj′)). These fit indices were assessed under different CDM settings with respect to Q-matrix misspecification and CDM misspecification. Results showed that relative fit indices selected the correct DINA model most of the times and selected the correct G-DINA model well across most conditions. Absolute fit indices rejected the true DINA model if the Q-matrix was misspecified in any way. Absolute fit indices rejected the true G-DINA model whenever the Q-matrix was under-specified. RMSEA2 could be artificially low when the Q-matrix was over-specified.     

Parameter recovery and model selection in mixed Rasch models

This study examines the precision of conditional maximum likelihood estimates and the quality of model selection methods based on information criteria (AIC and BIC) in mixed Rasch models. The design of the Monte Carlo simulation study included four test lengths (10, 15, 25, 40), three sample sizes (500, 1000, 2500), two simulated mixture conditions (one and two groups), and population homogeneity (equally sized subgroups) or heterogeneity (one subgroup three times larger than the other). The results show that both increasing sample size and increasing number of items lead to higher accuracy; medium-range parameters were estimated more precisely than extreme ones; and the accuracy was higher in homogeneous populations. The minimum-BIC method leads to almost perfect results and is more reliable than AIC-based model selection. The results are compared to findings by Li, Cohen, Kim, and Cho (2009) and practical guidelines are provided.     

Simple use of BIC to Assess Model Selection Uncertainty: An Illustration using Mediation and Moderation Models

Abstract

The Bayesian information criterion (BIC) has been used sometimes in SEM, even adopting a frequentist approach. Using simple mediation and moderation models as examples, we form posterior probability distribution via using BIC, which we call the BIC posterior, to assess model selection uncertainty of a finite number of models. This is simple but rarely used. The posterior probability distribution can be used to form a credibility set of models and to incorporate prior probabilities for model comparisons and selections. This was validated by a large scale simulation and results showed that the approximation via the BIC posterior is very good except when both the sample sizes and magnitude of parameters are small. We applied the BIC posterior to a real data set, and it has the advantages of flexibility in incorporating prior, addressing overfitting problems, and giving a full picture of posterior distribution to assess model selection uncertainty.     

Uniqueness of three-mode factor models with sparse cores: The 3 × 3 × 3 case

Three-Mode Factor Analysis (3MFA) and PARAFAC are methods to describe three-way data. Both methods employ models with components for the three modes of a three-way array; the 3MFA model also uses a three-way core array for linking all components to each other. The use of the core array makes the 3MFA model more general than the PARAFAC model (thus allowing a better fit), but also more complicated. Moreover, in the 3MFA model the components are not uniquely determined, and it seems hard to choose among all possible solutions. A particularly interesting feature of the PARAFAC model is that it does give unique components. The present paper introduces a class of 3MFA models in between 3MFA and PARAFAC that share the good properties of the 3MFA model and the PARAFAC model: They fit (almost) as well as the 3MFA model, they are relatively simple and they have the same uniqueness properties as the PARAFAC model.This research has been made possible by a fellowship from the Royal Netherlands Academy of Arts and Sciences to the first author. Part of this research has been presented at the first conference on ThRee-way methods In Chemistry (TRIC), a meeting of Psychometrics and Chemometrics, Epe, The Netherlands, August 1993. The authors are obliged to Age Smilde for stimulating this research, and two anonymous reviewers for many helpful suggestions.     

Interactive clustering

An interactive strategy for applying cluster-analytic techniques in behavioral research is presented. The two-part approach stresses the use of on-line computers for both data collection and analysis. In data collection, an extension of multidimensional unfolding to clustering reduces the number of judgments required of subjects by as much as 50%, During data analysis, the interactive procedures described permit the testing of multiple clustering models from an extensive family. With each selection, the goodness of fit of the model to the data can be tested. In addition to improving efficiency, the interactive strategy promoted here combines the advantages of the original nonmetric clustering procedures (e.g., Johnson, 1967) with those of the latest linear additive models (e.g., Sattath & Tversky, 1977; Shepard & Arabie, 1979).     

On the Complexity of Additive Clustering Models

Additive clustering provides a conceptually simple and potentially powerful approach to modeling the similarity relationships between stimuli. The ability of additive clustering models to accommodate similarity data, however, typically arises through the incorporation of large numbers of parameterized clusters. Accordingly, for the purposes of both model generation and model comparison, it is necessary to develop quantitative evaluative measures of additive clustering models that take into account both data-fit and complexity. Using a previously developed probabilistic formulation of additive clustering, the Bayesian Information Criterion is proposed for this role, and its application demonstrated. Limitations inherent in this approach, including the assumption that model complexity is equivalent to cluster cardinality, are discussed. These limitations are addressed by applying the Laplacian approximation of a marginal probability density, from which a measure of cluster structure complexity is derived. Using this measure, a preliminary investigation is made of the various properties of cluster structures that affect additive clustering model complexity. Among other things, these investigations show that, for a fixed number of clusters, a model with a strictly nested cluster structure is the least complicated, while a model with a partitioning cluster structure is the most complicated. Copyright 2001 Academic Press.     

基于类别水平的多级计分认知诊断Q矩阵修正：相对拟合统计量视角

多级计分认知诊断模型的开发对认知诊断的发展具有重要作用, 但对于多级计分模型下的Q矩阵修正还有待研究。本研究尝试对多级计分认知诊断Q矩阵修正进行研究, 并聚焦更具诊断价值的基于项目类别水平的Q矩阵修正。将相对拟合统计量应用于多级计分认知诊断Q矩阵修正, 并与已有方法Stepwise方法( Ma & de la Torre, 2019)进行比较。研究表明：BIC方法对多级计分认知诊断模型的Q矩阵修正具有较高的模式判准率和属性判准率, 其对Q矩阵的恢复率也高于Stepwise方法, BIC方法修正后的Q矩阵与数据更加拟合; 在复杂模型中, 相对拟合指标BIC比AIC和-2LL表现更好, 在实践中, 使用者可以选择BIC法进行测验Q矩阵修正; Q矩阵修正效果受到被试人数的影响, 增加被试人数可以提高Q矩阵修正的正确率。总之, 本研究为多级计分认知诊断Q矩阵修正提供了重要的方法支持。     

Latent variable selection in multidimensional item response theory models using the expectation model selection algorithm

The aim of latent variable selection in multidimensional item response theory (MIRT) models is to identify latent traits probed by test items of a multidimensional test. In this paper the expectation model selection (EMS) algorithm proposed by Jiang et al. (2015) is applied to minimize the Bayesian information criterion (BIC) for latent variable selection in MIRT models with a known number of latent traits. Under mild assumptions, we prove the numerical convergence of the EMS algorithm for model selection by minimizing the BIC of observed data in the presence of missing data. For the identification of MIRT models, we assume that the variances of all latent traits are unity and each latent trait has an item that is only related to it. Under this identifiability assumption, the convergence of the EMS algorithm for latent variable selection in the multidimensional two-parameter logistic (M2PL) models can be verified. We give an efficient implementation of the EMS for the M2PL models. Simulation studies show that the EMS outperforms the EM-based L₁ regularization in terms of correctly selected latent variables and computation time. The EMS algorithm is applied to a real data set related to the Eysenck Personality Questionnaire.     

MPTinR: Analysis of multinomial processing tree models in R

We introduce MPTinR, a software package developed for the analysis of multinomial processing tree (MPT) models. MPT models represent a prominent class of cognitive measurement models for categorical data with applications in a wide variety of fields. MPTinR is the first software for the analysis of MPT models in the statistical programming language R, providing a modeling framework that is more flexible than standalone software packages. MPTinR also introduces important features such as (1) the ability to calculate the Fisher information approximation measure of model complexity for MPT models, (2) the ability to fit models for categorical data outside the MPT model class, such as signal detection models, (3) a function for model selection across a set of nested and nonnested candidate models (using several model selection indices), and (4) multicore fitting. MPTinR is available from the Comprehensive R Archive Network at http://cran.r-project.org/web/packages/MPTinR/.     

The Importance of Complexity in Model Selection

Model selection should be based not solely on goodness-of-fit, but must also consider model complexity. While the goal of mathematical modeling in cognitive psychology is to select one model from a set of competing models that best captures the underlying mental process, choosing the model that best fits a particular set of data will not achieve this goal. This is because a highly complex model can provide a good fit without necessarily bearing any interpretable relationship with the underlying process. It is shown that model selection based solely on the fit to observed data will result in the choice of an unnecessarily complex model that overfits the data, and thus generalizes poorly. The effect of over-fitting must be properly offset by model selection methods. An application example of selection methods using artificial data is also presented. Copyright 2000 Academic Press.     

A tutorial on Bayesian nonparametric models

A key problem in statistical modeling is model selection, that is, how to choose a model at an appropriate level of complexity. This problem appears in many settings, most prominently in choosing the number of clusters in mixture models or the number of factors in factor analysis. In this tutorial, we describe Bayesian nonparametric methods, a class of methods that side-steps this issue by allowing the data to determine the complexity of the model. This tutorial is a high-level introduction to Bayesian nonparametric methods and contains several examples of their application.     

Model based clustering of large data sets: Tracing the development of spelling ability

There are two main theories with respect to the development of spelling ability: the stage model and the model of overlapping waves. In this paper exploratory model based clustering will be used to analyze the responses of more than 3500 pupils to subsets of 245 items. To evaluate the two theories, the resulting clusters will be ordered along a developmental dimension using an external criterion. Solutions for three statistical problems will be given: (1) an algorithm that can handle large data sets and only renders non-degenerate clusters; (2) a goodness of fit test that is not affected by the fact that the number of possible response vectors by far out-weights the number of observed response vectors; and (3) a new technique,data expunction, that can be used to evaluate goodness-of-fit tests if the missing data mechanism is known. Research supported by a grant (NWO 411-21-006) of the Dutch Organization for Scientific Research.