Local Optima in Mixture Modeling

It is common knowledge that mixture models are prone to arrive at locally optimal solutions. Typically, researchers are directed to utilize several random initializations to ensure that the resulting solution is adequate. However, it is unknown what factors contribute to a large number of local optima and whether these coincide with the factors that reduce the accuracy of a mixture model. A real-data illustration and a series of simulations are presented that examine the effect of a variety of data structures on the propensity of local optima and the classification quality of the resulting solution. We show that there is a moderately strong relationship between a solution that has a high proportion of local optima and one that is poorly classified.     

The likelihood function of additive learning models: Sufficient conditions for strict log-concavity and uniqueness of maximum

Luce introduced a family of learning models in which response probabilities are a function of some underlying continuous real variable. This variable can be represented as an additive function of the parameters of these learning models. Additive learning models have also been applied to signal-detection data. There are a wide variety of problems of contemporary psychophysics for which the assumption of a continuum of sensory states seems appropriate, and this family of learning models has a natural extension to such problems. One potential difficulty in the application of such models to data is that estimation of parameters requires the use of numerical procedures when the method of maximum likelihood is used. Given a likelihood function generated from an additive model, this paper gives sufficient conditions for log-concavity and strict log-concavity of the likelihood function. If a likelihood function is strictly log-concave, then any local maximum is a unique global maximum, and any solution to the likelihood equations is the unique global maximum point. These conditions are quite easy to evaluate in particular cases, and hence, the results should be quite useful. Some applications to Luce's beta model and to the signal-detection learning models of Dorfman and Biderman are presented.     

Trajectories of Depression and Anxiety Symptoms in Adolescent Girls: A Comparison of Parallel Trajectory Approaches

Longitudinal mixture models have become popular in the literature. However, modest attention has been paid to whether these models provide a better fit to the data than growth models. Here, we compared longitudinal mixture models to growth models in the context of changes in depression and anxiety symptoms in a community sample of girls from age 10 to 17. Model comparisons found that the preferred solution was a 5-class parallel process growth mixture model that differed in the course of depression and anxiety symptoms reflecting both ordering of symptoms and qualitative group differences. Comparisons between classes revealed substantive differences on a number of outcomes using this solution. Findings are discussed in the context of clinical assessment and implementation of growth mixture models.     

A Finite Mixture of Nonlinear Random Coefficient Models for Continuous Repeated Measures Data

Nonlinear random coefficient models (NRCMs) for continuous longitudinal data are often used for examining individual behaviors that display nonlinear patterns of development (or growth) over time in measured variables. As an extension of this model, this study considers the finite mixture of NRCMs that combine features of NRCMs with the idea of finite mixture (or latent class) models. The efficacy of this model is that it allows the integration of intrinsically nonlinear functions where the data come from a mixture of two or more unobserved subpopulations, thus allowing the simultaneous investigation of intra-individual (within-person) variability, inter-individual (between-person) variability, and subpopulation heterogeneity. Effectiveness of this model to work under real data analytic conditions was examined by executing a Monte Carlo simulation study. The simulation study was carried out using an R routine specifically developed for the purpose of this study. The R routine used maximum likelihood with the expectation–maximization algorithm. The design of the study mimicked the output obtained from running a two-class mixture model on task completion data.     

Estimating the π* goodness of fit index for finite mixtures of item response models

Testing the fit of finite mixture models is a difficult task, since asymptotic results on the distribution of likelihood ratio statistics do not hold; for this reason, alternative statistics are needed. This paper applies the π* goodness of fit statistic to finite mixture item response models. The π* statistic assumes that the population is composed of two subpopulations – those that follow a parametric model and a residual group outside the model; π* is defined as the proportion of population in the residual group. The population was divided into two or more groups, or classes. Several groups followed an item response model and there was also a residual group. The paper presents maximum likelihood algorithms for estimating item parameters, the probabilities of the groups and π*. The paper also includes a simulation study on goodness of recovery for the two‐ and three‐parameter logistic models and an example with real data from a multiple choice test.     

Regularized Finite Mixture Models for Probability Trajectories

Finite mixture models are widely used in the analysis of growth trajectory data to discover subgroups of individuals exhibiting similar patterns of behavior over time. In practice, trajectories are usually modeled as polynomials, which may fail to capture important features of the longitudinal pattern. Focusing on dichotomous response measures, we propose a likelihood penalization approach for parameter estimation that is able to capture a variety of nonlinear class mean trajectory shapes with higher precision than maximum likelihood estimates. We show how parameter estimation and inference for whether trajectories are time-invariant, linear time-varying, or nonlinear time-varying can be carried out for such models. To illustrate the method, we use simulation studies and data from a long-term longitudinal study of children at high risk for substance abuse. This work was supported in part by NIAAA grants R37 AA07065 and R01 AA12217 to RAZ.     

Estimation Methods for Mixed Logistic Models with Few Clusters

For mixed models generally, it is well known that modeling data with few clusters will result in biased estimates, particularly of the variance components and fixed effect standard errors. In linear mixed models, small sample bias is typically addressed through restricted maximum likelihood estimation (REML) and a Kenward-Roger correction. Yet with binary outcomes, there is no direct analog of either procedure. With a larger number of clusters, estimation methods for binary outcomes that approximate the likelihood to circumvent the lack of a closed form solution such as adaptive Gaussian quadrature and the Laplace approximation have been shown to yield less-biased estimates than linearization estimation methods that instead linearly approximate the model. However, adaptive Gaussian quadrature and the Laplace approximation are approximating the full likelihood rather than the restricted likelihood; the full likelihood is known to yield biased estimates with few clusters. On the other hand, linearization methods linearly approximate the model, which allows for restricted maximum likelihood and the Kenward-Roger correction to be applied. Thus, the following question arises: Which is preferable, a better approximation of a biased function or a worse approximation of an unbiased function? We address this question with a simulation and an illustrative empirical analysis.     

Bayesian estimation and testing of structural equation models

The Gibbs sampler can be used to obtain samples of arbitrary size from the posterior distribution over the parameters of a structural equation model (SEM) given covariance data and a prior distribution over the parameters. Point estimates, standard deviations and interval estimates for the parameters can be computed from these samples. If the prior distribution over the parameters is uninformative, the posterior is proportional to the likelihood, and asymptotically the inferences based on the Gibbs sample are the same as those based on the maximum likelihood solution, for example, output from LISREL or EQS. In small samples, however, the likelihood surface is not Gaussian and in some cases contains local maxima. Nevertheless, the Gibbs sample comes from the correct posterior distribution over the parameters regardless of the sample size and the shape of the likelihood surface. With an informative prior distribution over the parameters, the posterior can be used to make inferences about the parameters underidentified models, as we illustrate on a simple errors-in-variables model.We thank David Spiegelhalter for suggesting applying the Gibbs sampler to structural equation models to the first author at a 1994 workshop in Wiesbaden. We thank Ulf Böckenholt, Chris Meek, Marijtje van Duijn, Clark Glymour, Ivo Molenaar, Steve Klepper, Thomas Richardson, Teddy Seidenfeld, and Tom Snijders for helpful discussions, mathematical advice, and critiques of earlier drafts of this paper.     

Examining the effect of initialization strategies on the performance of Gaussian mixture modeling

Mixture modeling is a popular technique for identifying unobserved subpopulations (e.g., components) within a data set, with Gaussian (normal) mixture modeling being the form most widely used. Generally, the parameters of these Gaussian mixtures cannot be estimated in closed form, so estimates are typically obtained via an iterative process. The most common estimation procedure is maximum likelihood via the expectation-maximization (EM) algorithm. Like many approaches for identifying subpopulations, finite mixture modeling can suffer from locally optimal solutions, and the final parameter estimates are dependent on the initial starting values of the EM algorithm. Initial values have been shown to significantly impact the quality of the solution, and researchers have proposed several approaches for selecting the set of starting values. Five techniques for obtaining starting values that are implemented in popular software packages are compared. Their performances are assessed in terms of the following four measures: (1)?the ability to find the best observed solution, (2)?settling on a solution that classifies observations correctly, (3)?the number of local solutions found by each technique, and (4)?the speed at which the start values are obtained. On the basis of these results, a set of recommendations is provided to the user.     

A realistic perspective on pattern representation in growth data: comment on Bauer and Curran (2003)

D. J. Bauer and P. J. Curran (2003) cautioned that results obtained from growth mixture models may sometimes be inaccurate. The problem they addressed occurs when a growth mixture model is applied to a single, general population of individuals but findings incorrectly support the conclusion that there are 2 subpopulations. In an artificial sampling experiment, they showed that this can occur when the variables in the population have a nonnormal distribution. A realistic perspective is that although a healthy skepticism to complex statistical results is appropriate, there are no true models to discover. Consequently, the issue of model misspecification is irrelevant in practical terms. The purpose of a mathematical model is to summarize data, to formalize the dynamics of a behavioral process, and to make predictions. All of this is scientifically valuable and can be accomplished with a carefully developed model, even though the model is false.     

Analyses of multinomial mixture distributions: new tests for stochastic models of cognition and action

Mixture distributions are formed from a weighted linear combination of 2 or more underlying basis distributions [g(x) = sigma j alpha j fj(x); sigma alpha j = 1]. They arise frequently in stochastic models of perception, cognition, and action in which a finite number of discrete internal states are entered probabilistically over a series of trials. This article reviews various distributional properties that have been examined to test for the presence of mixture distributions. A new multinomial maximum likelihood mixture (MMLM) analysis is discussed for estimating the mixing probabilities alpha j and the basis distributions fj(x) of a hypothesized mixture distribution. The analysis also generates a maximum likelihood goodness-of-fit statistic for testing various mixture hypotheses. Stochastic computer simulations characterize the statistical power of such tests under representative conditions. Two empirical studies of mental processes hypothesized to involve mixture distributions are summarized to illustrate applications of the MMLM analysis.     

Robust Bayesian growth curve modelling using conditional medians

Growth curve models have been widely used to analyse longitudinal data in social and behavioural sciences. Although growth curve models with normality assumptions are relatively easy to estimate, practical data are rarely normal. Failing to account for non-normal data may lead to unreliable model estimation and misleading statistical inference. In this work, we propose a robust approach for growth curve modelling using conditional medians that are less sensitive to outlying observations. Bayesian methods are applied for model estimation and inference. Based on the existing work on Bayesian quantile regression using asymmetric Laplace distributions, we use asymmetric Laplace distributions to convert the problem of estimating a median growth curve model into a problem of obtaining the maximum likelihood estimator for a transformed model. Monte Carlo simulation studies have been conducted to evaluate the numerical performance of the proposed approach with data containing outliers or leverage observations. The results show that the proposed approach yields more accurate and efficient parameter estimates than traditional growth curve modelling. We illustrate the application of our robust approach using conditional medians based on a real data set from the Virginia Cognitive Aging Project.     

Nonlinear Random-Effects Mixture Models for Repeated Measures

A mixture model for repeated measures based on nonlinear functions with random effects is reviewed. The model can include individual schedules of measurement, data missing at random, nonlinear functions of the random effects, of covariates and of residuals. Individual group membership probabilities and individual random effects are obtained as empirical Bayes predictions. Although this is a complicated model that combines a mixture of populations, nonlinear regression, and hierarchical models, it is straightforward to estimate by maximum likelihood using SAS PROC NLMIXED. Many different models can be studied with this procedure. The model is more general than those that can be estimated with most special purpose computer programs currently available because the response function is essentially any form of nonlinear regression. Examples and sample code are included to illustrate the method.     

A Composite Likelihood Inference in Latent Variable Models for Ordinal Longitudinal Responses

The paper proposes a composite likelihood estimation approach that uses bivariate instead of multivariate marginal probabilities for ordinal longitudinal responses using a latent variable model. The model considers time-dependent latent variables and item-specific random effects to be accountable for the interdependencies of the multivariate ordinal items. Time-dependent latent variables are linked with an autoregressive model. Simulation results have shown composite likelihood estimators to have a small amount of bias and mean square error and as such they are feasible alternatives to full maximum likelihood. Model selection criteria developed for composite likelihood estimation are used in the applications. Furthermore, lower-order residuals are used as measures-of-fit for the selected models.     

Modeling Growth in Latent Variables Using a Piecewise Function

Latent growth curve models with piecewise functions for continuous repeated measures data have become increasingly popular and versatile tools for investigating individual behavior that exhibits distinct phases of development in observed variables. As an extension of this framework, this research study considers a piecewise function for describing segmented change of a latent construct over time where the latent construct is itself measured by multiple indicators gathered at each measurement occasion. The time of transition from one phase to another is not known a priori and thus is a parameter to be estimated. Utility of the model is highlighted in 2 ways. First, a small Monte Carlo simulation is executed to show the ability of the model to recover true (known) growth parameters, including the location of the point of transition (or knot), under different manipulated conditions. Second, an empirical example using longitudinal reading data is fitted via maximum likelihood and results discussed. Mplus (Version 6.1) code is provided in Appendix C to aid in making this class of models accessible to practitioners.     

A reliability coefficient for maximum likelihood factor analysis

Maximum likelihood factor analysis provides an effective method for estimation of factor matrices and a useful test statistic in the likelihood ratio for rejection of overly simple factor models. A reliability coefficient is proposed to indicate quality of representation of interrelations among attributes in a battery by a maximum likelihood factor analysis. Usually, for a large sample of individuals or objects, the likelihood ratio statistic could indicate that an otherwise acceptable factor model does not exactly represent the interrelations among the attributes for a population. The reliability coefficient could indicate a very close representation in this case and be a better indication as to whether to accept or reject the factor solution. This research was supported by the Personnel and Training Research Programs Office of the Office of Naval Research under contract US NAVY/00014-67-A-0305-0003. Critical review of the development and suggestions by Richard Montanelli were most helpful.     

Finite mixtures in confirmatory factor-analysis models

In this paper, various types of finite mixtures of confirmatory factor-analysis models are proposed for handling data heterogeneity. Under the proposed mixture approach, observations are assumed to be drawn from mixtures of distinct confirmatory factor-analysis models. But each observation does not need to be identified to a particular model prior to model fitting. Several classes of mixture models are proposed. These models differ by their unique representations of data heterogeneity. Three different sampling schemes for these mixture models are distinguished. A mixed type of the these three sampling schemes is considered throughout this article. The proposed mixture approach reduces to regular multiple-group confirmatory factor-analysis under a restrictive sampling scheme, in which the structural equation model for each observation is assumed to be known. By assuming a mixture of multivariate normals for the data, maximum likelihood estimation using the EM (Expectation-Maximization) algorithm and the AS (Approximate-Scoring) method are developed, respectively. Some mixture models were fitted to a real data set for illustrating the application of the theory. Although the EM algorithm and the AS method gave similar sets of parameter estimates, the AS method was found computationally more efficient than the EM algorithm. Some comments on applying the mixture approach to structural equation modeling are made.Note: This paper is one of the Psychometric Society's 1995 Dissertation Award papers.—EditorThis article is based on the dissertation of the author. The author would like to thank Peter Bentler, who was the dissertation chair, for guidance and encouragement of this work. Eric Holman, Robert Jennrich, Bengt Muthén, and Thomas Wickens, who served as the committee members for the dissertation, had been very supportive and helpful. Michael Browne is appreciated for discussing some important points about the use of the approximate information in the dissertation. Thanks also go to an anonymous associate editor, whose comments were very useful for the revision of an earlier version of this article.     

Modeling Incomplete Longitudinal Substance Use Data Using Latent Variable Growth Curve Methodology

Longitudinal data sets typically suffer from attrition and other forms of missing data. When this common problem occurs, several researchers have demonstrated that correct maximum likelihood estimation with missing data can be obtained under mild assumptions concerning the missing data mechanism. With reasonable substantive theory, a mixture of cross-sectional and longitudinal methods developed within multiple-group structural equation modeling can provide a strong basis for inference about developmental change. Using an approach to the analysis of missing data, the present study investigated developmental trends in adolescent (N = 759) alcohol, marijuana, and cigarette use across a 5-year period using multiple-group latent growth modeling. An associative model revealed that common developmental trends existed for all three substances. Age and gender were included in the model as predictors of initial status and developmental change. Findings discuss the utility of latent variable structural equation modeling techniques and missing data approaches in the study of developmental change.     

A Newton-Raphson algorithm for maximum likelihood factor analysis

This paper demonstrates the feasibility of using a Newton-Raphson algorithm to solve the likelihood equations which arise in maximum likelihood factor analysis. The algorithm leads to clean easily identifiable convergence and provides a means of verifying that the solution obtained is at least a local maximum of the likelihood function. It is shown that a popular iteration algorithm is numerically unstable under conditions which are encountered in practice and that, as a result, inaccurate solutions have been presented in the literature. The key result is a computationally feasible formula for the second differential of a partially maximized form of the likelihood function. In addition to implementing the Newton-Raphson algorithm, this formula provides a means for estimating the asymptotic variances and covariances of the maximum likelihood estimators. This research was supported by the Air Force Office of Scientific Research, Grant No. AF-AFOSR-4.59-66 and by National Institutes of Health, Grant No. FR-3.     

Missing data techniques for structural equation modeling

As with other statistical methods, missing data often create major problems for the estimation of structural equation models (SEMs). Conventional methods such as listwise or pairwise deletion generally do a poor job of using all the available information. However, structural equation modelers are fortunate that many programs for estimating SEMs now have maximum likelihood methods for handling missing data in an optimal fashion. In addition to maximum likelihood, this article also discusses multiple imputation. This method has statistical properties that are almost as good as those for maximum likelihood and can be applied to a much wider array of models and estimation methods.