Using Multilevel Logistic Regression to Evaluate Person-Fit in IRT Models

I describe how multilevel logistic regression can be used to assess the consistency of an individual's response pattern with an item response theory measurement model. Specifically, by treating item responses as being nested within individuals, multilevel logistic regression is used to estimate a person-response curve that models how an individual's item endorsement rate decreases as a function of item difficulty. The slope of an individual's person-response curve is used as an indicator of the degree of response consistency or person-fit. I argue that the proposed multilevel modeling approach to person-fit assessment has several potential advantages over traditional techniques. The most important advantage being that the multilevel modeling approach allows explanatory variables to be entered into the model so that the causes of response inconsistency or differential test functioning can be investigated.     

有调节的多层中介效应分析

多层中介和多层调节效应分析在社科领域已常有应用,但如果将多层中介和调节整合在一起,可以产生2（多层中介类型）×2（调节变量的层次）×3（调节的中介路径）共12种有调节的多层中介模型。面对有调节的多层中介效应分析,研究者往往束手无策。详述了基于多层线性模型的12种有调节的多层中介的分析方法和基于多层结构方程模型的4类有调节的多层中介分析方法,包括正交分割法,随机系数预测法,潜调节结构方程法和贝叶斯合理值法。这四类方法的核心议题在于如何处理潜调节项。当样本量足够大时,建议选择潜调节结构方程法;当样本量不足时,建议选择贝叶斯合理值法。随后用一个实际例子演示如何进行有调节的多层中介效应分析并有相应的Mplus程序。最后展望了有调节的多层中介效应分析研究的拓展方向。     

Studying Multivariate Change Using Multilevel Models and Latent Curve Models

In longitudinal research investigators often measure multiple variables at multiple points in time and are interested in investigating individual differences in patterns of change on those variables. In the vast majority of applications, researchers focus on studying change in one variable at a time. In this article we consider methods for studying relations1.1ips between patterns of change on different variables. We show how the multilevel modeling framework, which is often used to study univariate change, can be extended to the multivariate case to yield estimates of covariances of parameters representing aspects of change on different variables. We illustrate this approach using data from a study of physiological response to marital conflict in older married couples, showing a substantial correlation between rate of linear change on different stress-related hormones during conflict. We also consider how similar issues can be studied using extensions of latent curve models to the multivariate case, and we show how such models are related to multivariate multilevel models.     

Multilevel Mixture Factor Models

Factor analysis is a statistical method for describing the associations among sets of observed variables in terms of a small number of underlying continuous latent variables. Various authors have proposed multilevel extensions of the factor model for the analysis of data sets with a hierarchical structure. These Multilevel Factor Models (MFMs) have in common that—as in multilevel regression analysis—variation at the higher level is modeled using continuous random effects. In this article, we present an alternative multilevel extension of factor analysis which we call the Multilevel Mixture Factor Model (MMFM). It is based on the assumption that higher level units belong to latent classes that differ in terms of the parameters of the factor model specified for the lower level units. We demonstrate the added value of MMFM compared with MFM, both from a theoretical and applied perspective, and we illustrate the complementarity of the two approaches with an empirical application on students' satisfaction with the University of Florence. The multilevel aspect of this application is that students are nested within study programs, which makes it possible to cluster these programs based on their differences in students' satisfaction.     

Multilevel modeling: current and future applications in personality research

Traditional statistical analyses can be compromised when data are collected from groups or multiple observations are collected from individuals. We present an introduction to multilevel models designed to address dependency in data. We review current use of multilevel modeling in 3 personality journals showing use concentrated in the 2 areas of experience sampling and longitudinal growth. Using an empirical example, we illustrate specification and interpretation of the results of series of models as predictor variables are introduced at Levels 1 and 2. Attention is given to possible trends and cycles in longitudinal data and to different forms of centering. We consider issues that may arise in estimation, model comparison, model evaluation, and data evaluation (outliers), highlighting similarities to and differences from standard regression approaches. Finally, we consider newer developments, including 3-level models, cross-classified models, nonstandard (limited) dependent variables, multilevel structural equation modeling, and nonlinear growth. Multilevel approaches both address traditional problems of dependency in data and provide personality researchers with the opportunity to ask new questions of their data.     

密集追踪数据分析：模型及其应用

在心理学、教育学和临床医学等领域, 越来越多的研究者开始关注个体内部的行为、心理、临床效果等随时间而产生的动态变化, 重视针对个体的差异化建模。密集追踪是一种在短时间内对个体进行多个时间节点密集追踪测量的方法, 更适合用于研究个体内部心理过程等的动态变化及其作用机制。近年来, 密集追踪成为心理学研究的一大热点, 但许多密集追踪的研究分析仍停留在较为传统的方法。方法学领域已涌现出较多用于密集追踪数据分析的模型方法, 较为主流的模型包括以动态结构方程模型(Dynamic Structural Equation Model, DSEM)为代表的自上而下的建模方法, 以及以组迭代多模型估计(Group Iterative Multiple Model Estimation, GIMME)为代表的自下而上的建模方法。二者均可以方便地对密集追踪数据中的自回归及交叉滞后效应进行建模。     

Multilevel meta-analysis of single-subject experimental designs: A simulation study

One way to combine data from single-subject experimental design studies is by performing a multilevel meta-analysis, with unstandardized or standardized regression coefficients as the effect size metrics. This study evaluates the performance of this approach. The results indicate that a multilevel meta-analysis of unstandardized effect sizes results in good estimates of the effect. The multilevel meta-analysis of standardized effect sizes, on the other hand, is suitable only when the number of measurement occasions for each subject is 20 or more. The effect of the treatment on the intercept is estimated with enough power when the studies are homogeneous or when the number of studies is large; the power of the effect on the slope is estimated with enough power only when the number of studies and the number of measurement occasions are large.     

Multiple Group Analysis in Multilevel Structural Equation Model Across Level 1 Groups

This article introduces and evaluates a procedure for conducting multiple group analysis in multilevel structural equation model across Level 1 groups (MG1-MSEM; Ryu, 2014). When group membership is at Level 1, multiple group analysis raises two issues that cannot be solved by a simple extension of the standard multiple group analysis in single-level structural equation model. First, the Level 2 data are not independent between Level 1 groups. Second, the standard procedure fails to take into account the dependency between members of different Level 1 groups within the same cluster. The MG1-MSEM approach provides solutions to these problems. In MG1-MSEM, the Level 1 mean structure is necessary to represent the differences between Level 1 groups within clusters. The Level 2 model is the same regardless of Level 1 group membership. A simulation study examined the performance of MUML (Muthén's maximum likelihood) estimation in MG1-MSEM. The MG1-MSEM approach is illustrated for both a multilevel path model and a multilevel factor model using empirical data sets.     

Multilevel analysis of matching behavior

Multilevel modeling has been considered a promising statistical tool in the field of the experimental analysis of behavior and may serve as a convenient statistical analysis for matching behavior because it structures data in groups (or levels) to account simultaneously for the within‐subject and between‐subject variances. Heretofore, researchers have sometimes pooled data erroneously from different subjects in a single analysis by using average ratios, average response and reinforcer rates, aggregation of subjects, etc. Unfortunately, this leads to loss of information and biased estimations, which can severely undermine generalization of the results. Instead, a multilevel approach is advocated to combine several subjects' matching behavior. A reanalysis of previous data on matching behavior is provided to illustrate the method and point out its advantages. It illustrates that multilevel regression leads to better estimations, is more convenient, and offers more behavioral information. We hope this paper will encourage the use of multilevel modeling in the statistical practices of behavior analysts.     

Omitted Variables in Multilevel Models

Statistical methodology for handling omitted variables is presented in a multilevel modeling framework. In many nonexperimental studies, the analyst may not have access to all requisite variables, and this omission may lead to biased estimates of model parameters. By exploiting the hierarchical nature of multilevel data, a battery of statistical tools are developed to test various forms of model misspecification as well as to obtain estimators that are robust to the presence of omitted variables. The methodology allows for tests of omitted effects at single and multiple levels. The paper also introduces intermediate-level tests; these are tests for omitted effects at a single level, regardless of the presence of omitted effects at a higher level. A simulation study shows, not surprisingly, that the omission of variables yields bias in both regression coefficients and variance components; it also suggests that omitted effects at lower levels may cause more severe bias than at higher levels. Important factors resulting in bias were found to be the level of an omitted variable, its effect size, and sample size. A real data study illustrates that an omitted variable at one level may yield biased estimators at any level and, in this study, one cannot obtain reliable estimates for school-level variables when omitted child effects exist. However, robust estimators may provide unbiased estimates for effects of interest even when the efficient estimators fail, and the one-degree-of-freedom test helps one to understand where the problem is located. It is argued that multilevel data typically contain rich information to deal with omitted variables, offering yet another appealing reason for the use of multilevel models in the social sciences. This research was supported by the National Academy of Education/Spencer Foundation and the National Science Foundation, Grant Number SES-0436274.     

A Comparison of Multilevel Mediation Modeling Methods: Recommendations for Applied Researchers

Multilevel structural equation modeling (MSEM) has been proposed as a valuable tool for estimating mediation in multilevel data and has known advantages over traditional multilevel modeling, including conflated and unconflated techniques (CMM & UMM). Recent methodological research has focused on comparing the three methods for 2-1-1 designs, but in regards to 1-1-1 mediation designs, there are significant gaps in the published literature that prevent applied researchers from making educated decisions regarding which model to employ in their own specific research design. A Monte Carlo study was performed to compare MSEM, UMM, and CMM on relative bias, confidence interval coverage, Type I Error, and power in a 1-1-1 model with random slopes under varying data conditions. Recommendations for applied researchers are discussed and an empirical example provides context for the three methods.     

A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models

Group-level variance estimates of zero often arise when fitting multilevel or hierarchical linear models, especially when the number of groups is small. For situations where zero variances are implausible a priori, we propose a maximum penalized likelihood approach to avoid such boundary estimates. This approach is equivalent to estimating variance parameters by their posterior mode, given a weakly informative prior distribution. By choosing the penalty from the log-gamma family with shape parameter greater than 1, we ensure that the estimated variance will be positive. We suggest a default log-gamma(2,λ) penalty with λ→0, which ensures that the maximum penalized likelihood estimate is approximately one standard error from zero when the maximum likelihood estimate is zero, thus remaining consistent with the data while being nondegenerate. We also show that the maximum penalized likelihood estimator with this default penalty is a good approximation to the posterior median obtained under a noninformative prior. Our default method provides better estimates of model parameters and standard errors than the maximum likelihood or the restricted maximum likelihood estimators. The log-gamma family can also be used to convey substantive prior information. In either case—pure penalization or prior information—our recommended procedure gives nondegenerate estimates and in the limit coincides with maximum likelihood as the number of groups increases.     

Multilevel SEM with random slopes in discrete data using the pairwise maximum likelihood

Pairwise maximum likelihood (PML) estimation is a promising method for multilevel models with discrete responses. Multilevel models take into account that units within a cluster tend to be more alike than units from different clusters. The pairwise likelihood is then obtained as the product of bivariate likelihoods for all within-cluster pairs of units and items. In this study, we investigate the PML estimation method with computationally intensive multilevel random intercept and random slope structural equation models (SEM) in discrete data. In pursuing this, we first reconsidered the general ‘wide format’ (WF) approach for SEM models and then extend the WF approach with random slopes. In a small simulation study we the determine accuracy and efficiency of the PML estimation method by varying the sample size (250, 500, 1000, 2000), response scales (two-point, four-point), and data-generating model (mediation model with three random slopes, factor model with one and two random slopes). Overall, results show that the PML estimation method is capable of estimating computationally intensive random intercept and random slopes multilevel models in the SEM framework with discrete data and many (six or more) latent variables with satisfactory accuracy and efficiency. However, the condition with 250 clusters combined with a two-point response scale shows more bias.     

Multilevel Model Prediction

Multilevel models are proven tools in social research for modeling complex, hierarchical systems. In multilevel modeling, statistical inference is based largely on quantification of random variables. This paper distinguishes among three types of random variables in multilevel modeling—model disturbances, random coefficients, and future response outcomes—and provides a unified procedure for predicting them. These predictors are best linear unbiased and are commonly known via the acronym BLUP; they are optimal in the sense of minimizing mean square error and are Bayesian under a diffuse prior. For parameter estimation purposes, a multilevel model can be written as a linear mixed-effects model. In this way, parameters of the many equations can be estimated simultaneously and hence efficiently. For prediction purposes, we show that it is more convenient to retain the multiple equation feature of multilevel models. In this way, the efficient BLUPs are easy to compute and retain their intuitively appealing recursive form. We also derive explicit equations for standard errors of these different types of predictors. Prediction in multilevel modeling is important in a wide range of applications. To demonstrate the applicability of our results, this paper discusses prediction in the context of a study of school effectiveness. This research was supported by a grant from the Graduate School at the University of Wisconsin at Madision and the National Science Foundation, Grant number SES-0436274. We are grateful to Norman Webb at Wisconsin Center for Education Research for making available the data used in the reported application.     

A Computationally More Efficient and More Accurate Stepwise Approach for Correcting for Sampling Error and Measurement Error

Over the last decade or two, multilevel structural equation modeling (ML-SEM) has become a prominent modeling approach in the social sciences because it allows researchers to correct for sampling and measurement errors and thus to estimate the effects of Level 2 (L2) constructs without bias. Because the latent variable modeling software Mplus uses maximum likelihood (ML) by default, many researchers in the social sciences have applied ML to obtain estimates of L2 regression coefficients. However, one drawback of ML is that covariance matrices of the predictor variables at L2 tend to be degenerate, and thus, estimates of L2 regression coefficients tend to be rather inaccurate when sample sizes are small. In this article, I show how an approach for stabilizing covariance matrices at L2 can be used to obtain more accurate estimates of L2 regression coefficients. A simulation study is conducted to compare the proposed approach with ML, and I illustrate its application with an example from organizational research.     

A New Multilevel CART Algorithm for Multilevel Data with Binary Outcomes

The multilevel logistic regression model (M-logit) is the standard model for modeling multilevel data with binary outcomes. However, many assumptions and restrictions should be considered when applying this model for unbiased estimation. To overcome these limitations, we proposed a multilevel CART (M-CART) algorithm which combines the M-logit and single level CART (S-CART) within the framework of the expectation-maximization. Simulation results showed that the proposed M-CART provided substantial improvements on classification accuracy, sensitivity, and specific over the M-logit, S-CART, and single level logistic regression model when modeling multilevel data with binary outcomes. This benefit of using M-CART was consistently found across different conditions of sample size, intra-class correlation, and when relationships between predictors and outcomes were nonlinear and nonadditive.     

Testing Group Mean Differences of Latent Variables in Multilevel Data Using Multiple-Group Multilevel CFA and Multilevel MIMIC Modeling

Considering that group comparisons are common in social science, we examined two latent group mean testing methods when groups of interest were either at the between or within level of multilevel data: multiple-group multilevel confirmatory factor analysis (MG ML CFA) and multilevel multiple-indicators multiple-causes modeling (ML MIMIC). The performance of these methods were investigated through three Monte Carlo studies. In Studies 1 and 2, either factor variances or residual variances were manipulated to be heterogeneous between groups. In Study 3, which focused on within-level multiple-group analysis, six different model specifications were considered depending on how to model the intra-class group correlation (i.e., correlation between random effect factors for groups within cluster). The results of simulations generally supported the adequacy of MG ML CFA and ML MIMIC for multiple-group analysis with multilevel data. The two methods did not show any notable difference in the latent group mean testing across three studies. Finally, a demonstration with real data and guidelines in selecting an appropriate approach to multilevel multiple-group analysis are provided.     

基于结构方程模型的多层中介效应分析

近年社科领域常见使用多层线性模型进行多层中介研究。尽管多层线性模型区分了多层中介的组间和组内效应, 仍然存在抽样误差和测量误差。比较好的方法是, 将多层线性模型整合到结构方程模型中, 在多层结构方程模型框架下设置潜变量和多指标, 可有效校正抽样误差和测量误差、得到比较准确的中介效应值, 还能适用于更多种类的多层中介分析并提供模型的拟合指数。在介绍新方法后, 总结出一套多层中介的分析流程, 通过一个例子来演示如何用MPLUS软件进行多层中介分析。最后展望了多层结构方程和多层中介研究的拓展方向。     

Multilevel Factor Analysis: Reporting Guidelines and a Review of Reporting Practices

We provide reporting guidelines for multilevel factor analysis (MFA) and use these guidelines to systematically review 72 MFA applications in journals across a range of disciplines (e.g., education, health/nursing, management, and psychology) published between 1994 and 2014. Results are organized in terms of the (a) characteristics of the MFA application (e.g., construct measured), (b) purpose (e.g., measurement validation), (c) data source (e.g., number of cases at Level 1 and Level 2), (d) statistical approach (e.g., maximum likelihood), and (e) results reported (e.g., intraclass correlations for indicators and latent variables, standardized factor loadings, fit indices). Results from this review have implications for applied researchers interested in expanding their approaches to psychometric analyses and construct validation within a multilevel framework and for methodologists using Monte Carlo methods to explore technical and methodological issues grounded in realistic research design conditions.     

A Comparison of Inverse-Wishart Prior Specifications for Covariance Matrices in Multilevel Autoregressive Models

Multilevel autoregressive models are especially suited for modeling between-person differences in within-person processes. Fitting these models with Bayesian techniques requires the specification of prior distributions for all parameters. Often it is desirable to specify prior distributions that have negligible effects on the resulting parameter estimates. However, the conjugate prior distribution for covariance matrices—the Inverse-Wishart distribution—tends to be informative when variances are close to zero. This is problematic for multilevel autoregressive models, because autoregressive parameters are usually small for each individual, so that the variance of these parameters will be small. We performed a simulation study to compare the performance of three Inverse-Wishart prior specifications suggested in the literature, when one or more variances for the random effects in the multilevel autoregressive model are small. Our results show that the prior specification that uses plug-in ML estimates of the variances performs best. We advise to always include a sensitivity analysis for the prior specification for covariance matrices of random parameters, especially in autoregressive models, and to include a data-based prior specification in this analysis. We illustrate such an analysis by means of an empirical application on repeated measures data on worrying and positive affect.