A Comparison of Multilevel Imputation Schemes for Random Coefficient Models: Fully Conditional Specification and Joint Model Imputation with Random Covariance Matrices

Abstract

Literature addressing missing data handling for random coefficient models is particularly scant, and the few studies to date have focused on the fully conditional specification framework and “reverse random coefficient” imputation. Although it has not received much attention in the literature, a joint modeling strategy that uses random within-cluster covariance matrices to preserve cluster-specific associations is a promising alternative for random coefficient analyses. This study is apparently the first to directly compare these procedures. Analytic results suggest that both imputation procedures can introduce bias-inducing incompatibilities with a random coefficient analysis model. Problems with fully conditional specification result from an incorrect distributional assumption, whereas joint imputation uses an underparameterized model that assumes uncorrelated intercepts and slopes. Monte Carlo simulations suggest that biases from these issues are tolerable if the missing data rate is 10% or lower and the sample is composed of at least 30 clusters with 15 observations per group. Furthermore, fully conditional specification tends to be superior with intraclass correlations that are typical of crosssectional data (e.g., ICC?=?.10), whereas the joint model is preferable with high values typical of longitudinal designs (e.g., ICC?=?.50).     

三类多层中介效应分析方法比较

比较了贝叶斯法、Monte Carlo法和参数Bootstrap法在2-1-1多层中介分析中的表现。结果发现：1)有先验信息的贝叶斯法的中介效应点估计和区间估计都最准确;2)无先验信息的贝叶斯法、Monte Carlo法、偏差校正和未校正的参数Bootstrap法的中介效应点估计和区间估计表现相当,但Monte Carlo法在第Ⅰ类错误率和区间宽度指标上表现略优于其他三种方法,偏差校正的Bootstrap法在统计检验力上表现略优于其他三种方法,但在第Ⅰ类错误率上表现最差;结果表明,当有先验信息时,推荐使用贝叶斯法;当先验信息不可得时,推荐使用Monte Carlo法。     

A Comparison of ML,WLSMV, and Bayesian Methods for Multilevel Structural Equation Models in Small Samples: A Simulation Study

Multilevel structural equation models are increasingly applied in psychological research. With increasing model complexity, estimation becomes computationally demanding, and small sample sizes pose further challenges on estimation methods relying on asymptotic theory. Recent developments of Bayesian estimation techniques may help to overcome the shortcomings of classical estimation techniques. The use of potentially inaccurate prior information may, however, have detrimental effects, especially in small samples. The present Monte Carlo simulation study compares the statistical performance of classical estimation techniques with Bayesian estimation using different prior specifications for a two-level SEM with either continuous or ordinal indicators. Using two software programs (Mplus and Stan), differential effects of between- and within-level sample sizes on estimation accuracy were investigated. Moreover, it was tested to which extent inaccurate priors may have detrimental effects on parameter estimates in categorical indicator models. For continuous indicators, Bayesian estimation did not show performance advantages over ML. For categorical indicators, Bayesian estimation outperformed WLSMV solely in case of strongly informative accurate priors. Weakly informative inaccurate priors did not deteriorate performance of the Bayesian approach, while strong informative inaccurate priors led to severely biased estimates even with large sample sizes. With diffuse priors, Stan yielded better results than Mplus in terms of parameter estimates.     

A Comparison of Multilevel Mediation Modeling Methods: Recommendations for Applied Researchers

Multilevel structural equation modeling (MSEM) has been proposed as a valuable tool for estimating mediation in multilevel data and has known advantages over traditional multilevel modeling, including conflated and unconflated techniques (CMM & UMM). Recent methodological research has focused on comparing the three methods for 2-1-1 designs, but in regards to 1-1-1 mediation designs, there are significant gaps in the published literature that prevent applied researchers from making educated decisions regarding which model to employ in their own specific research design. A Monte Carlo study was performed to compare MSEM, UMM, and CMM on relative bias, confidence interval coverage, Type I Error, and power in a 1-1-1 model with random slopes under varying data conditions. Recommendations for applied researchers are discussed and an empirical example provides context for the three methods.     

At the Frontiers of Modeling Intensive Longitudinal Data: Dynamic Structural Equation Models for the Affective Measurements from the COGITO Study

With the growing popularity of intensive longitudinal research, the modeling techniques and software options for such data are also expanding rapidly. Here we use dynamic multilevel modeling, as it is incorporated in the new dynamic structural equation modeling (DSEM) toolbox in Mplus, to analyze the affective data from the COGITO study. These data consist of two samples of over 100 individuals each who were measured for about 100 days. We use composite scores of positive and negative affect and apply a multilevel vector autoregressive model to allow for individual differences in means, autoregressions, and cross-lagged effects. Then we extend the model to include random residual variances and covariance, and finally we investigate whether prior depression affects later depression scores through the random effects of the daily diary measures. We end with discussing several urgent—but mostly unresolved—issues in the area of dynamic multilevel modeling.     

Explained Variance and Intraclass Correlation in a Two-Level AR(1) Model

The total variance of a first-order autoregressive AR(1) time series is well known in time series literature. However, despite the increased use and interest in two-level AR(1) models, an equation for the total variance of these models does not exist. This paper presents an approximation of this total variance. It will be used to compute the unexplained and explained variance at each level of the model, the proportion of explained variance, and the intraclass correlation (ICC). The use of these variances and the ICC will be illustrated using an example concerning structured diary data about the positive affect of 96 married women.     

On the Multilevel Nature of Meta-Analysis: A Tutorial,Comparison of Software Programs,and Discussion of Analytic Choices

The term “multilevel meta-analysis” is encountered not only in applied research studies, but in multilevel resources comparing traditional meta-analysis to multilevel meta-analysis. In this tutorial, we argue that the term “multilevel meta-analysis” is redundant since all meta-analysis can be formulated as a special kind of multilevel model. To clarify the multilevel nature of meta-analysis the four standard meta-analytic models are presented using multilevel equations and fit to an example data set using four software programs: two specific to meta-analysis (metafor in R and SPSS macros) and two specific to multilevel modeling (PROC MIXED in SAS and HLM). The same parameter estimates are obtained across programs underscoring that all meta-analyses are multilevel in nature. Despite the equivalent results, not all software programs are alike and differences are noted in the output provided and estimators available. This tutorial also recasts distinctions made in the literature between traditional and multilevel meta-analysis as differences between meta-analytic choices, not between meta-analytic models, and provides guidance to inform choices in estimators, significance tests, moderator analyses, and modeling sequence. The extent to which the software programs allow flexibility with respect to these decisions is noted, with metafor emerging as the most favorable program reviewed.     

A Limited Information Estimator for Dynamic Factor Models

Structural equation modeling (SEM) is an increasingly popular method for examining multivariate time series data. As in cross-sectional data analysis, structural misspecification of time series models is inevitable, and further complicated by the fact that errors occur in both the time series and measurement components of the model. In this article, we introduce a new limited information estimator and local fit diagnostic for dynamic factor models within the SEM framework. We demonstrate the implementation of this estimator and examine its performance under both correct and incorrect model specifications via a small simulation study. The estimates from this estimator are compared to those from the most common system-wide estimators and are found to be more robust to the structural misspecifications considered.     

Double Decomposition of Level-1 Variables in Multilevel Models: An Analysis of the Flynn Effect in the NSLY Data

This paper introduces an extension of cluster mean centering (also called group mean centering) for multilevel models, which we call “double decomposition (DD).” This centering method separates between-level variance, as in cluster mean centering, but also decomposes within-level variance of the same variable. This process retains the benefits of cluster mean centering but allows for context variables derived from lower level variables, other than the cluster mean, to be incorporated into the model. A brief simulation study is presented, demonstrating the potential advantage (or even necessity) for DD in certain circumstances. Several applications to multilevel analysis are discussed. Finally, an empirical demonstration examining the Flynn effect (Flynn, 1987 Flynn, J. R. (1987). Massive IQ gains in 14 nations: What IQ tests really measure. Psychological Bulletin, 101(2), 171. https://doi.org/10.1037/h0090408.[Crossref], [Web of Science ®] , [Google Scholar]), our motivating example, is presented. The use of DD in the analysis provides a novel method to narrow the field of plausible causal hypotheses regarding the Flynn effect, in line with suggestions by a number of researchers (Mingroni, 2014 Mingroni, M. A. (2014). Future efforts in Flynn effect research: Balancing reductionism with holism. Journal of Intelligence, 2(4), 122. https://doi.org/10.3390/jintelligence2040122.[Crossref] , [Google Scholar]; Rodgers, 2015 Rodgers, J. L. (2015). Methodological issues associated with studying the Flynn effect: Exploratory and confirmatory efforts in the past, present, and future. Journal of Intelligence, 3(4), 111. https://doi.org/10.3390/jintelligence3040111. [Crossref] , [Google Scholar]).     

A Bayesian Approach to More Stable Estimates of Group-Level Effects in Contextual Studies

Multilevel analyses are often used to estimate the effects of group-level constructs. However, when using aggregated individual data (e.g., student ratings) to assess a group-level construct (e.g., classroom climate), the observed group mean might not provide a reliable measure of the unobserved latent group mean. In the present article, we propose a Bayesian approach that can be used to estimate a multilevel latent covariate model, which corrects for the unreliable assessment of the latent group mean when estimating the group-level effect. A simulation study was conducted to evaluate the choice of different priors for the group-level variance of the predictor variable and to compare the Bayesian approach with the maximum likelihood approach implemented in the software Mplus. Results showed that, under problematic conditions (i.e., small number of groups, predictor variable with a small ICC), the Bayesian approach produced more accurate estimates of the group-level effect than the maximum likelihood approach did.     

Modeling Nonstationary Emotion Dynamics in Dyads using a Time-Varying Vector-Autoregressive Model

Emotion dynamics are likely to arise in an interpersonal context. Standard methods to study emotions in interpersonal interaction are limited because stationarity is assumed. This means that the dynamics, for example, time-lagged relations, are invariant across time periods. However, this is generally an unrealistic assumption. Whether caused by an external (e.g., divorce) or an internal (e.g., rumination) event, emotion dynamics are prone to change. The semi-parametric time-varying vector-autoregressive (TV-VAR) model is based on well-studied generalized additive models, implemented in the software R. The TV-VAR can explicitly model changes in temporal dependency without pre-existing knowledge about the nature of change. A simulation study is presented, showing that the TV-VAR model is superior to the standard time-invariant VAR model when the dynamics change over time. The TV-VAR model is applied to empirical data on daily feelings of positive affect (PA) from a single couple. Our analyses indicate reliable changes in the male’s emotion dynamics over time, but not in the female’s—which were not predicted by her own affect or that of her partner. This application illustrates the usefulness of using a TV-VAR model to detect changes in the dynamics in a system.     

A primer for using multilevel models to meta-analyze single case design data with AB phases

Meta-analytic methods provide a way to synthesize data across treatment evaluation studies. However, these well-accepted methods are infrequent with behavior analytic studies. Multilevel models may be a promising method to meta-analyze single-case data. This technical article provides a primer for how to conduct a multilevel model with single-case designs with AB phases using data from the differential-reinforcement-of-low-rate behavior literature. We provide details, recommendations, and considerations for searching for appropriate studies, organizing the data, and conducting the analyses. All data sets are available to allow the reader to follow along with this primer. The purpose of this technical article is to minimally equip behavior analysts to complete a meta-analysis that will summarize a current state of affairs as it relates to the science of behavior analysis and its practice. Moreover, we aim to demonstrate the value of analyses of this sort for behavior analysis.     

Bayesian parameter estimation in the Expectancy Valence model of the Iowa gambling task

The purpose of the popular Iowa gambling task is to study decision making deficits in clinical populations by mimicking real-life decision making in an experimental context. Busemeyer and Stout [Busemeyer, J. R., & Stout, J. C. (2002). A contribution of cognitive decision models to clinical assessment: Decomposing performance on the Bechara gambling task. Psychological Assessment, 14, 253-262] proposed an “Expectancy Valence” reinforcement learning model that estimates three latent components which are assumed to jointly determine choice behavior in the Iowa gambling task: weighing of wins versus losses, memory for past payoffs, and response consistency. In this article we explore the statistical properties of the Expectancy Valence model. We first demonstrate the difficulty of applying the model on the level of a single participant, we then propose and implement a Bayesian hierarchical estimation procedure to coherently combine information from different participants, and we finally apply the Bayesian estimation procedure to data from an experiment designed to provide a test of specific influence.     

Evidence for Mixed Processes in Normal/Mirror Discrimination of Rotated Letters: A Bayesian Model Comparison Between Single- and Mixed-Distribution Models1

When determining whether a rotated letter is normal or mirrored, an observer mentally rotates the letter to its canonical orientation. To account for patterns of response times (RTs) for the normal/mirror discrimination of rotated letters, previous research formulated a model that postulated a mixture of trials with and without mental rotation. While this model could explain the curvilinear relationship that has been found between averaged RT and letter orientations, the curved RT function is still open to alternative explanations without assuming mixed processes. To address this issue and test the mixed-process hypothesis more directly, we analyzed trial-by-trial RT data instead of averaged RTs by employing a Bayesian model comparison technique. If rotation and non-rotation trials are mixed, trial-by-trial RTs for letters in a particular orientation should not follow a single distribution but a mixed one formed from the superposition of two separate distributions, one for rotation and one for non-rotation trials. In the present study, we compared single- and mixed-distribution models. Bayes-factor analysis showed decisive support for the mixed-distribution model over the single-distribution model. In addition, using the widely applicable information criterion (WAIC), the predictive accuracy of the mixed-distribution model was found to be as high as that of the single-distribution model. These results indicated the involvement of mixed processes in normal/mirror discrimination of rotated letters. The usefulness of statistical modeling in psychological study and necessary precautions to take in the interpretation of the parameters of unconfirmed models are also discussed.     

认知诊断模型中项目水平模型比较统计量的健壮性

使用模拟研究方法比较了以往研究中提出的基于观察信息矩阵、三明治矩阵的Wald（分别表示为W_Obs、W_Sw）、似然比（Likelihood Ratio）统计量以及新提出的基于经验交叉相乘信息矩阵的Wald统计量（W_XPD）在模型——数据失拟条件下进行项目水平上模型比较时的表现。结果显示：（1）W_Sw的一类错误控制率有很强的健壮性。（2）W_XPD在Q矩阵错误设定的大多数条件下的表现优于W_Sw。结论：模型—数据拟合良好时可以使用W_Sw进行项目水平上的模型比较,当模型与数据失拟时W_XPD可能是更好的选择。     

认知诊断模型中项目水平模型比较统计量的健壮性

使用模拟研究方法比较了以往研究中提出的基于观察信息矩阵、三明治矩阵的Wald（分别表示为W_Obs、W_Sw）、似然比（Likelihood Ratio）统计量以及新提出的基于经验交叉相乘信息矩阵的Wald统计量（W_XPD）在模型——数据失拟条件下进行项目水平上模型比较时的表现。结果显示：（1）W_Sw的一类错误控制率有很强的健壮性。（2）W_XPD在Q矩阵错误设定的大多数条件下的表现优于W_Sw。结论：模型—数据拟合良好时可以使用W_Sw进行项目水平上的模型比较,当模型与数据失拟时W_XPD可能是更好的选择。     

用于处理不努力作答的标准化残差系列方法和混合多层模型法的比较

文章采用模拟研究, 分别在混合多层模型假设满足和违背的情境下, 比较了混合多层模型方法与标准化残差系列方法在识别不努力作答和参数估计方面的表现。结果显示：(1)不存在不努力作答或其严重性低时, 各方法表现接近; (2)不努力作答严重性高时, 固定参数迭代标准化残差法普遍更优, 混合多层模型法仅在假设满足且两种作答反应时差异大的条件下表现较好。建议实际应用中优先选择固定参数迭代标准化残差法。     

A Hierarchical Rater Model for Longitudinal Data

Research studies in psychology and education often seek to detect changes or growth in an outcome over a duration of time. This research provides a solution to those interested in estimating latent traits from psychological measures that rely on human raters. Rater effects potentially degrade the quality of scores in constructed response and performance assessments. We develop an extension of the hierarchical rater model (HRM), which yields estimates of latent traits that have been corrected for individual rater bias and variability, for ratings that come from longitudinal designs. The parameterization, called the longitudinal HRM (L-HRM), includes an autoregressive time series process to permit serial dependence between latent traits at adjacent timepoints, as well as a parameter for overall growth. We evaluate and demonstrate the feasibility and performance of the L-HRM using simulation studies. Parameter recovery results reveal predictable amounts and patterns of bias and error for most parameters across conditions. An application to ratings from a study of character strength demonstrates the model. We discuss limitations and future research directions to improve the L-HRM.     

A Computational Evaluation of Two Models of Retrieval Processes in Sentence Processing in Aphasia

Can sentence comprehension impairments in aphasia be explained by difficulties arising from dependency completion processes in parsing? Two distinct models of dependency completion difficulty are investigated, the Lewis and Vasishth (2005) activation-based model and the direct-access model (DA; McElree, 2000). These models' predictive performance is compared using data from individuals with aphasia (IWAs) and control participants. The data are from a self-paced listening task involving subject and object relative clauses. The relative predictive performance of the models is evaluated using k-fold cross-validation. For both IWAs and controls, the activation-based model furnishes a somewhat better quantitative fit to the data than the DA. Model comparisons using Bayes factors show that, assuming an activation-based model, intermittent deficiencies may be the best explanation for the cause of impairments in IWAs, although slowed syntax and lexical delayed access may also play a role. This is the first computational evaluation of different models of dependency completion using data from impaired and unimpaired individuals. This evaluation develops a systematic approach that can be used to quantitatively compare the predictions of competing models of language processing.     

A New Model for Acquiescence at the Interface of Psychometrics and Cognitive Psychology

ABSTRACT

When measuring psychological traits, one has to consider that respondents often show content-unrelated response behavior in answering questionnaires. To disentangle the target trait and two such response styles, extreme responding and midpoint responding, Böckenholt (2012a Böckenholt, U. (2012a). Modeling multiple response processes in judgment and choice. Psychological Methods, 17, 665–678. doi:10.1037/a0028111[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]) developed an item response model based on a latent processing tree structure. We propose a theoretically motivated extension of this model to also measure acquiescence, the tendency to agree with both regular and reversed items. Substantively, our approach builds on multinomial processing tree (MPT) models that are used in cognitive psychology to disentangle qualitatively distinct processes. Accordingly, the new model for response styles assumes a mixture distribution of affirmative responses, which are either determined by the underlying target trait or by acquiescence. In order to estimate the model parameters, we rely on Bayesian hierarchical estimation of MPT models. In simulations, we show that the model provides unbiased estimates of response styles and the target trait, and we compare the new model and Böckenholt’s model in a recovery study. An empirical example from personality psychology is used for illustrative purposes.