Omitted Variables in Multilevel Models

Statistical methodology for handling omitted variables is presented in a multilevel modeling framework. In many nonexperimental studies, the analyst may not have access to all requisite variables, and this omission may lead to biased estimates of model parameters. By exploiting the hierarchical nature of multilevel data, a battery of statistical tools are developed to test various forms of model misspecification as well as to obtain estimators that are robust to the presence of omitted variables. The methodology allows for tests of omitted effects at single and multiple levels. The paper also introduces intermediate-level tests; these are tests for omitted effects at a single level, regardless of the presence of omitted effects at a higher level. A simulation study shows, not surprisingly, that the omission of variables yields bias in both regression coefficients and variance components; it also suggests that omitted effects at lower levels may cause more severe bias than at higher levels. Important factors resulting in bias were found to be the level of an omitted variable, its effect size, and sample size. A real data study illustrates that an omitted variable at one level may yield biased estimators at any level and, in this study, one cannot obtain reliable estimates for school-level variables when omitted child effects exist. However, robust estimators may provide unbiased estimates for effects of interest even when the efficient estimators fail, and the one-degree-of-freedom test helps one to understand where the problem is located. It is argued that multilevel data typically contain rich information to deal with omitted variables, offering yet another appealing reason for the use of multilevel models in the social sciences. This research was supported by the National Academy of Education/Spencer Foundation and the National Science Foundation, Grant Number SES-0436274.     

Multilevel Model Prediction

Multilevel models are proven tools in social research for modeling complex, hierarchical systems. In multilevel modeling, statistical inference is based largely on quantification of random variables. This paper distinguishes among three types of random variables in multilevel modeling—model disturbances, random coefficients, and future response outcomes—and provides a unified procedure for predicting them. These predictors are best linear unbiased and are commonly known via the acronym BLUP; they are optimal in the sense of minimizing mean square error and are Bayesian under a diffuse prior. For parameter estimation purposes, a multilevel model can be written as a linear mixed-effects model. In this way, parameters of the many equations can be estimated simultaneously and hence efficiently. For prediction purposes, we show that it is more convenient to retain the multiple equation feature of multilevel models. In this way, the efficient BLUPs are easy to compute and retain their intuitively appealing recursive form. We also derive explicit equations for standard errors of these different types of predictors. Prediction in multilevel modeling is important in a wide range of applications. To demonstrate the applicability of our results, this paper discusses prediction in the context of a study of school effectiveness. This research was supported by a grant from the Graduate School at the University of Wisconsin at Madision and the National Science Foundation, Grant number SES-0436274. We are grateful to Norman Webb at Wisconsin Center for Education Research for making available the data used in the reported application.     

Discrete Choice Models for Ordinal Response Variables: A Generalization of the Stereotype Model

In this paper I present a class of discrete choice models for ordinal response variables based on a generalization of the stereotype model. The stereotype model can be derived and generalized as a random utility model for ordered alternatives. Random utility models can be specified to account for heteroscedastic and correlated utilities. In the case of the generalized stereotype model this includes category-specific random effects due to individual differences in response style. But unlike standard random utility models the generalized stereotype model is better suited for ordinal response variables and can be interpreted as a kind of unidimensional unfolding model. This paper discusses the specification, interpretation, identification, and estimation of generalized stereotype models. Two applications are provided for illustration. This paper benefited significantly from the comments and suggestions of the editor, associate editor, and three anonymous reviewers. It is dedicated to my late colleague, peer, and friend Bradley D. Crouch.     

An alternative two stage least squares (2SLS) estimator for latent variable equations

The Maximum-likelihood estimator dominates the estimation of general structural equation models. Noniterative, equation-by-equation estimators for factor analysis have received some attention, but little has been done on such estimators for latent variable equations. I propose an alternative 2SLS estimator of the parameters in LISREL type models and contrast it with the existing ones. The new 2SLS estimator allows observed and latent variables to originate from nonnormal distributions, is consistent, has a known asymptotic covariance matrix, and is estimable with standard statistical software. Diagnostics for evaluating instrumental variables are described. An empirical example illustrates the estimator. I gratefully acknowledge support for this research from the Sociology Program of the National Science Foundation (SES-9121564) and the Center for Advanced Study in the Behavioral Sciences, Stanford, California. This paper was presented at the Interdisciplinary Consortium for Statistical Applications at Indiana University at Bloomington (March 2, 1994) and at the RMD Conference on Causal Modeling at Purdue University, West Lafayette, Indiana (March 3-5, 1994).     

基于结构方程模型的多层调节效应

使用多层线性模型进行调节效应分析在社科领域已常有应用。尽管多层线性模型区分了层1自变量的组间和组内效应、实现了多层调节效应的分解, 仍然存在抽样误差和测量误差。建议在多层结构方程模型框架下, 设置潜变量和多指标来有效校正抽样误差和测量误差。在介绍多层调节SEM分析的随机系数预测法和潜调节结构方程法后, 总结出一套多层调节的SEM分析流程, 通过一个例子来演示如何用Mplus软件进行多层调节SEM分析。随后评述了多层调节效应分析方法在国内心理学的应用现状, 并展望了多层结构方程和多层调节研究的拓展方向。     

On the asymptotic standard error of a class of robust estimators of ability in dichotomous item response models

In item response theory, the classical estimators of ability are highly sensitive to response disturbances and can return strongly biased estimates of the true underlying ability level. Robust methods were introduced to lessen the impact of such aberrant responses on the estimation process. The computation of asymptotic (i.e., large‐sample) standard errors (ASE) for these robust estimators, however, has not yet been fully considered. This paper focuses on a broad class of robust ability estimators, defined by an appropriate selection of the weight function and the residual measure, for which the ASE is derived from the theory of estimating equations. The maximum likelihood (ML) and the robust estimators, together with their estimated ASEs, are then compared in a simulation study by generating random guessing disturbances. It is concluded that both the estimators and their ASE perform similarly in the absence of random guessing, while the robust estimator and its estimated ASE are less biased and outperform their ML counterparts in the presence of random guessing with large impact on the item response process.     

Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study

When multisource feedback instruments, for example, 360-degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non-standard multilevel structural equation model that incorporates self-ratings (level-2 variables) and others’ ratings from different additional perspectives (level-1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self-ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self-ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade-off effects between the sample sizes of self-ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ² test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level-specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.     

A study of algorithms for covariance structure analysis with specific comparisons using factor analysis

Several algorithms for covariance structure analysis are considered in addition to the Fletcher-Powell algorithm. These include the Gauss-Newton, Newton-Raphson, Fisher Scoring, and Fletcher-Reeves algorithms. Two methods of estimation are considered, maximum likelihood and weighted least squares. It is shown that the Gauss-Newton algorithm which in standard form produces weighted least squares estimates can, in iteratively reweighted form, produce maximum likelihood estimates as well. Previously unavailable standard error estimates to be used in conjunction with the Fletcher-Reeves algorithm are derived. Finally all the algorithms are applied to a number of maximum likelihood and weighted least squares factor analysis problems to compare the estimates and the standard errors produced. The algorithms appear to give satisfactory estimates but there are serious discrepancies in the standard errors. Because it is robust to poor starting values, converges rapidly and conveniently produces consistent standard errors for both maximum likelihood and weighted least squares problems, the Gauss-Newton algorithm represents an attractive alternative for at least some covariance structure analyses.Work by the first author has been supported in part by Grant No. Da01070 from the U. S. Public Health Service. Work by the second author has been supported in part by Grant No. MCS 77-02121 from the National Science Foundation.     

A Composite Likelihood Inference in Latent Variable Models for Ordinal Longitudinal Responses

The paper proposes a composite likelihood estimation approach that uses bivariate instead of multivariate marginal probabilities for ordinal longitudinal responses using a latent variable model. The model considers time-dependent latent variables and item-specific random effects to be accountable for the interdependencies of the multivariate ordinal items. Time-dependent latent variables are linked with an autoregressive model. Simulation results have shown composite likelihood estimators to have a small amount of bias and mean square error and as such they are feasible alternatives to full maximum likelihood. Model selection criteria developed for composite likelihood estimation are used in the applications. Furthermore, lower-order residuals are used as measures-of-fit for the selected models.     

A revision of the response ratio hypothesis for magnitude estimation

The response ratio hypothesis for magnitude estimation proposes that the relevant variables involved in the subject's behavior are ratios of stimulus random variables, and ratios of response random variables. We add to these “errors of calculation.” Such error random variables add flexibility to the hypothesis, but do not enable it to explain effects of stimulus range.     

Determination of the Regression Coefficients and Their Associated Standard Errors in Hierarchical Regression Analysis

Hierarchical regression analysis is potentially a very useful statistical technique for establishing the significance of sets of predictor variables. However, when a hierarchical analysis which is based on theory is performed, some estimation procedures for the regression coefficients and their associated standard errors are potentially inappropriate. Specifically, the hierarchical regression equations, the incremental or hierarchical tests, and the parameter estimation of this procedure may not correspond. This problem is investigated by the development of four approaches (simultaneous, stagewise, orthogonal, and hierarchical) of estimation to the analysis. For each method, regression, coefficient estimators and their standard errors are determined. By comparison of these approaches, the use of orthogonalized sets of predictor variates or a modification to a series of simultaneous analyses are recommended as the most sensible technique for a theory driven hierarchical analysis.     

Polytomous multilevel testlet models for testlet‐based assessments with complex sampling designs

A pplications of standard item response theory models assume local independence of items and persons. This paper presents polytomous multilevel testlet models for dual dependence due to item and person clustering in testlet‐based assessments with clustered samples. Simulation and survey data were analysed with a multilevel partial credit testlet model. This model was compared with three alternative models – a testlet partial credit model (PCM), multilevel PCM, and PCM – in terms of model parameter estimation. The results indicated that the deviance information criterion was the fit index that always correctly identified the true multilevel testlet model based on the quantified evidence in model selection, while the Akaike and Bayesian information criteria could not identify the true model. In general, the estimation model and the magnitude of item and person clustering impacted the estimation accuracy of ability parameters, while only the estimation model and the magnitude of item clustering affected the item parameter estimation accuracy. Furthermore, ignoring item clustering effects produced higher total errors in item parameter estimates but did not have much impact on the accuracy of ability parameter estimates, while ignoring person clustering effects yielded higher total errors in ability parameter estimates but did not have much effect on the accuracy of item parameter estimates. When both clustering effects were ignored in the PCM, item and ability parameter estimation accuracy was reduced.     

Effects of Modeling the Heterogeneity on Inferences Drawn from Multilevel Designs

This article uses Monte Carlo techniques to examine the effect of heterogeneity of variance in multilevel analyses in terms of relative bias, coverage probability, and root mean square error (RMSE). For all simulated data sets, the parameters were estimated using the restricted maximum-likelihood (REML) method both assuming homogeneity and incorporating heterogeneity into multilevel models. We find that (a) the estimates for the fixed parameters are unbiased, but the associated standard errors are frequently biased when heterogeneity is ignored; by contrast, the standard errors of the fixed effects are almost always accurate when heterogeneity is considered; (b) the estimates for the random parameters are slightly overestimated; (c) both the homogeneous and heterogeneous models produce standard errors of the variance component estimates that are underestimated; however, taking heterogeneity into account, the REML-estimations give correct estimates of the standard errors at the lowest level and lead to less underestimated standard errors at the highest level; and (d) from the RMSE point of view, REML accounting for heterogeneity outperforms REML assuming homogeneity; a considerable improvement has been particularly detected for the fixed parameters. Based on this, we conclude that the solution presented can be uniformly adopted. We illustrate the process using a real dataset.     

Robustness of parameter and standard error estimates against ignoring a contextual effect of a subject-level covariate in cluster-randomized trials

In experimental research, it is not uncommon to assign clusters to conditions. When analysing the data of such cluster-randomized trials, a multilevel analysis should be applied in order to take into account the dependency of first-level units (i.e., subjects) within a second-level unit (i.e., a cluster). Moreover, the multilevel analysis can handle covariates on both levels. If a first-level covariate is involved, usually the within-cluster effect of this covariate will be estimated, implicitly assuming the contextual effect to be equal. However, this assumption may be violated. The focus of the present simulation study is the effects of ignoring the inequality of the within-cluster and contextual covariate effects on parameter and standard error estimates of the treatment effect, which is the parameter of main interest in experimental research. We found that ignoring the inequality of the within-cluster and contextual effects does not affect the estimation of the treatment effect or its standard errors. However, estimates of the variance components, as well as standard errors of the constant, were found to be biased.     

Robust inference with binary data

In this paper robustness properties of the maximum likelihood estimator (MLE) and several robust estimators for the logistic regression model when the responses are binary are analysed. It is found that the MLE and the classical Rao's score test can be misleading in the presence of model misspecification which in the context of logistic regression means either misclassification's errors in the responses, or extreme data points in the design space. A general framework for robust estimation and testing is presented and a robust estimator as well as a robust testing procedure are presented. It is shown that they are less influenced by model misspecifications than their classical counterparts. They are finally applied to the analysis of binary data from a study on breastfeeding.The author is partially supported by the Swiss National Science Foundation. She would like to thank Rand Wilcox, Eva Cantoni and Elvezio Ronchetti for their helpful comments on earlier versions of the paper, as well as Stephane Heritier for providing the routine to compute the OBRE.     

Exact tests for the rasch model via sequential importance sampling

Rasch proposed an exact conditional inference approach to testing his model but never implemented it because it involves the calculation of a complicated probability. This paper furthers Rasch’s approach by (1) providing an efficient Monte Carlo methodology for accurately approximating the required probability and (2) illustrating the usefulness of Rasch’s approach for several important testing problems through simulation studies. Our Monte Carlo methodology is shown to compare favorably to other Monte Carlo methods proposed for this problem in two respects: it is considerably faster and it provides more reliable estimates of the Monte Carlo standard error.This Research was supported in part by National Science Foundation grant DMS-0203762 and a University of Pennsylvania Research Foundation grant.The authors are grateful to Don Burdick for helpful comments. In addition, the authors wish to thank the editor, the associate editor, and the referees for their helpful suggestions.This revised article was published online in August 2005 with the PDF paginated correctly.     

On normal theory based inference for multilevel models with distributional violations

Data in social and behavioral sciences are often hierarchically organized though seldom normal, yet normal theory based inference procedures are routinely used for analyzing multilevel models. Based on this observation, simple adjustments to normal theory based results are proposed to minimize the consequences of violating normality assumptions. For characterizing the distribution of parameter estimates, sandwich-type covariance matrices are derived. Standard errors based on these covariance matrices remain consistent under distributional violations. Implications of various covariance estimators are also discussed. For evaluating the quality of a multilevel model, a rescaled statistic is given for both the hierarchical linear model and the hierarchical structural equation model. The rescaled statistic, improving the likelihood ratio statistic by estimating one extra parameter, approaches the same mean as its reference distribution. A simulation study with a 2-level factor model implies that the rescaled statistic is preferable.This research was supported by grants DA01070 and DA00017 from the National Institute on Drug Abuse and a University of North Texas faculty research grant. We would like to thank the Associate Editor and two reviewers for suggestions that helped to improve the paper.     

The Impact of Misspecifying the Within-Subject Covariance Structure in Multiwave Longitudinal Multilevel Models: A Monte Carlo Study

This Monte Carlo study examined the impact of misspecifying the 𝚺 matrix in longitudinal data analysis under both the multilevel model and mixed model frameworks. Under the multilevel model approach, under-specification and general-misspecification of the 𝚺 matrix usually resulted in overestimation of the variances of the random effects (e.g., τ₀₀, τ_τ11 ) and standard errors of the corresponding growth parameter estimates (e.g., SE_{β 0}, SE_{β 1}). Overestimates of the standard errors led to lower statistical power in tests of the growth parameters. An unstructured 𝚺 matrix under the mixed model framework generally led to underestimates of standard errors of the growth parameter estimates. Underestimates of the standard errors led to inflation of the type I error rate in tests of the growth parameters. Implications of the compensatory relationship between the random effects of the growth parameters and the longitudinal error structure for model specification were discussed.     

Trend in correlated proportions

A random effects probit model is developed for the case in which the same units are sampled repeatedly at each level of an independent variable. Because the observed proportions may be correlated under these conditions, estimating their trend with respect to the independent variable is no longer a standard problem for probit, logit or loglinear analysis. Using a qualitative analogue of a random regressions model, we employ instead marginal maximum likelihood to estimate the average latent trend line. Likelihood ratio tests of the hypothesis of no trend in the average line, and the hypothesis of no differences in average trend lines between experimental treatments, are proposed. We illustrate the model both with simulated data and with observed data from a clinical experiment in which psychiatric patients on two drug therapies are rated on five occasions for the presence or absence of symptoms.Supported by a grant from the MacArthur Foundation and National Science Foundation Grant BNS85-11774.The authors are indebted to James Heckman for calling our attention to the Clark algorithm.     

Multilevel SEM with random slopes in discrete data using the pairwise maximum likelihood

Pairwise maximum likelihood (PML) estimation is a promising method for multilevel models with discrete responses. Multilevel models take into account that units within a cluster tend to be more alike than units from different clusters. The pairwise likelihood is then obtained as the product of bivariate likelihoods for all within-cluster pairs of units and items. In this study, we investigate the PML estimation method with computationally intensive multilevel random intercept and random slope structural equation models (SEM) in discrete data. In pursuing this, we first reconsidered the general ‘wide format’ (WF) approach for SEM models and then extend the WF approach with random slopes. In a small simulation study we the determine accuracy and efficiency of the PML estimation method by varying the sample size (250, 500, 1000, 2000), response scales (two-point, four-point), and data-generating model (mediation model with three random slopes, factor model with one and two random slopes). Overall, results show that the PML estimation method is capable of estimating computationally intensive random intercept and random slopes multilevel models in the SEM framework with discrete data and many (six or more) latent variables with satisfactory accuracy and efficiency. However, the condition with 250 clusters combined with a two-point response scale shows more bias.