Effects of Modeling the Heterogeneity on Inferences Drawn from Multilevel Designs

This article uses Monte Carlo techniques to examine the effect of heterogeneity of variance in multilevel analyses in terms of relative bias, coverage probability, and root mean square error (RMSE). For all simulated data sets, the parameters were estimated using the restricted maximum-likelihood (REML) method both assuming homogeneity and incorporating heterogeneity into multilevel models. We find that (a) the estimates for the fixed parameters are unbiased, but the associated standard errors are frequently biased when heterogeneity is ignored; by contrast, the standard errors of the fixed effects are almost always accurate when heterogeneity is considered; (b) the estimates for the random parameters are slightly overestimated; (c) both the homogeneous and heterogeneous models produce standard errors of the variance component estimates that are underestimated; however, taking heterogeneity into account, the REML-estimations give correct estimates of the standard errors at the lowest level and lead to less underestimated standard errors at the highest level; and (d) from the RMSE point of view, REML accounting for heterogeneity outperforms REML assuming homogeneity; a considerable improvement has been particularly detected for the fixed parameters. Based on this, we conclude that the solution presented can be uniformly adopted. We illustrate the process using a real dataset.     

Bias and Efficiency in Structural Equation Modeling: Maximum Likelihood Versus Robust Methods

In the structural equation modeling literature, the normal-distribution-based maximum likelihood (ML) method is most widely used, partly because the resulting estimator is claimed to be asymptotically unbiased and most efficient. However, this may not hold when data deviate from normal distribution. Outlying cases or nonnormally distributed data, in practice, can make the ML estimator (MLE) biased and inefficient. In addition to ML, robust methods have also been developed, which are designed to minimize the effects of outlying cases. But the properties of robust estimates and their standard errors (SEs) have never been systematically studied. This article studies two robust methods and compares them against the ML method with respect to bias and efficiency using a confirmatory factor model. Simulation results show that robust methods lead to results comparable with ML when data are normally distributed. When data have heavy tails or outlying cases, robust methods lead to less biased and more efficient estimators than MLEs. A formula to obtain consistent SEs for one of the robust methods is also developed. The formula-based SEs for both robust estimators match the empirical SEs very well with medium-size samples. A sample of the Cross Racial Identity Scale with a 6-factor model is used for illustration. Results also confirm conclusions of the simulation study.     

Incorporating Mobility in Growth Modeling for Multilevel and Longitudinal Item Response Data

Multilevel data often cannot be represented by the strict form of hierarchy typically assumed in multilevel modeling. A common example is the case in which subjects change their group membership in longitudinal studies (e.g., students transfer schools; employees transition between different departments). In this study, cross-classified and multiple membership models for multilevel and longitudinal item response data (CCMM-MLIRD) are developed to incorporate such mobility, focusing on students' school change in large-scale longitudinal studies. Furthermore, we investigate the effect of incorrectly modeling school membership in the analysis of multilevel and longitudinal item response data. Two types of school mobility are described, and corresponding models are specified. Results of the simulation studies suggested that appropriate modeling of the two types of school mobility using the CCMM-MLIRD yielded good recovery of the parameters and improvement over models that did not incorporate mobility properly. In addition, the consequences of incorrectly modeling the school effects on the variance estimates of the random effects and the standard errors of the fixed effects depended upon mobility patterns and model specifications. Two sets of large-scale longitudinal data are analyzed to illustrate applications of the CCMM-MLIRD for each type of school mobility.     

Information matrix estimation procedures for cognitive diagnostic models

Two new methods to estimate the asymptotic covariance matrix for marginal maximum likelihood estimation of cognitive diagnosis models (CDMs), the inverse of the observed information matrix and the sandwich-type estimator, are introduced. Unlike several previous covariance matrix estimators, the new methods take into account both the item and structural parameters. The relationships between the observed information matrix, the empirical cross-product information matrix, the sandwich-type covariance matrix and the two approaches proposed by de la Torre (2009, J. Educ. Behav. Stat., 34, 115) are discussed. Simulation results show that, for a correctly specified CDM and Q-matrix or with a slightly misspecified probability model, the observed information matrix and the sandwich-type covariance matrix exhibit good performance with respect to providing consistent standard errors of item parameter estimates. However, with substantial model misspecification only the sandwich-type covariance matrix exhibits robust performance.     

Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study

When multisource feedback instruments, for example, 360-degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non-standard multilevel structural equation model that incorporates self-ratings (level-2 variables) and others’ ratings from different additional perspectives (level-1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self-ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self-ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade-off effects between the sample sizes of self-ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ² test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level-specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.     

A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models

Group-level variance estimates of zero often arise when fitting multilevel or hierarchical linear models, especially when the number of groups is small. For situations where zero variances are implausible a priori, we propose a maximum penalized likelihood approach to avoid such boundary estimates. This approach is equivalent to estimating variance parameters by their posterior mode, given a weakly informative prior distribution. By choosing the penalty from the log-gamma family with shape parameter greater than 1, we ensure that the estimated variance will be positive. We suggest a default log-gamma(2,λ) penalty with λ→0, which ensures that the maximum penalized likelihood estimate is approximately one standard error from zero when the maximum likelihood estimate is zero, thus remaining consistent with the data while being nondegenerate. We also show that the maximum penalized likelihood estimator with this default penalty is a good approximation to the posterior median obtained under a noninformative prior. Our default method provides better estimates of model parameters and standard errors than the maximum likelihood or the restricted maximum likelihood estimators. The log-gamma family can also be used to convey substantive prior information. In either case—pure penalization or prior information—our recommended procedure gives nondegenerate estimates and in the limit coincides with maximum likelihood as the number of groups increases.     

Polytomous multilevel testlet models for testlet‐based assessments with complex sampling designs

A pplications of standard item response theory models assume local independence of items and persons. This paper presents polytomous multilevel testlet models for dual dependence due to item and person clustering in testlet‐based assessments with clustered samples. Simulation and survey data were analysed with a multilevel partial credit testlet model. This model was compared with three alternative models – a testlet partial credit model (PCM), multilevel PCM, and PCM – in terms of model parameter estimation. The results indicated that the deviance information criterion was the fit index that always correctly identified the true multilevel testlet model based on the quantified evidence in model selection, while the Akaike and Bayesian information criteria could not identify the true model. In general, the estimation model and the magnitude of item and person clustering impacted the estimation accuracy of ability parameters, while only the estimation model and the magnitude of item clustering affected the item parameter estimation accuracy. Furthermore, ignoring item clustering effects produced higher total errors in item parameter estimates but did not have much impact on the accuracy of ability parameter estimates, while ignoring person clustering effects yielded higher total errors in ability parameter estimates but did not have much effect on the accuracy of item parameter estimates. When both clustering effects were ignored in the PCM, item and ability parameter estimation accuracy was reduced.     

Investigating Ceiling Effects in Longitudinal Data Analysis

Score limitation at the top of a scale is commonly termed “ceiling effect.” Ceiling effects can lead to serious artifactual parameter estimates in most data analysis. This study examines the consequences of ceiling effects in longitudinal data analysis and investigates several methods of dealing with ceiling effects through Monte Carlo simulations and empirical data analyses. Data were simulated based on a latent growth curve model with T = 5 occasions. The proportion of the ceiling data [10%–40%] was manipulated by using different thresholds, and estimated parameters were examined for R = 500 replications. The results showed that ceiling effects led to incorrect model selection and biased parameter estimation (shape of the curve and magnitude of the changes) when regular growth curve models were applied. The Tobit growth curve model, instead, performed very well in dealing with ceiling effects in longitudinal data analysis. The Tobit growth curve model was then applied in an empirical cognitive aging study and the results were discussed.     

On the Usefulness of a Multilevel Logistic Regression Approach to Person-Fit Analysis

The logistic person response function (PRF) models the probability of a correct response as a function of the item locations. Reise (2000) Reise, S. P. 2000. Using multilevel logistic regression to evaluate person-fit in IRT models. Multivariate Behavioral Research, 35: 543–568. [Taylor &; Francis Online], [Web of Science ®] , [Google Scholar] proposed to use the slope parameter of the logistic PRF as a person-fit measure. He reformulated the logistic PRF model as a multilevel logistic regression model and estimated the PRF parameters from this multilevel framework. An advantage of the multilevel framework is that it allows relating person fit to explanatory variables for person misfit/fit. We critically discuss Reise's approach. First, we argue that often the interpretation of the PRF slope as an indicator of person misfit is incorrect. Second, we show that the multilevel logistic regression model and the logistic PRF model are incompatible, resulting in a multilevel person-fit framework, which grossly violates the bivariate normality assumption for residuals in the multilevel model. Third, we use a Monte Carlo study to show that in the multilevel logistic regression framework estimates of distribution parameters of PRF intercepts and slopes are biased. Finally, we discuss the implications of these results and suggest an alternative multilevel regression approach to explanatory person-fit analysis. We illustrate the alternative approach using empirical data on repeated anxiety measurements of cardiac arrhythmia patients who had a cardioverter-defibrillator implanted.     

The accuracy of four approximations to noncentralF

The accuracy of the two-moment, three-moment, square root, and cube root approximations to the noncentralF distribution was assessed using 7,920 entries from Tiku’s (1967) power tables. Tiku’s tables list exact values of β for α= .005?.05, ν₁ = 1?12, ν₂ = 2?120, and ?= 0.5?3.0. Analysis of the errors showed generally satisfactory performance for all four approximations. The three-moment approximation was most accurate, registering a maximum error of only .009. The other three approximations had maximum errors of ±.02, except for the square root approximation at ν₂ = 2, where maximum errors of .05 occur. Approximation error increased with decreases in ν₁ and, less consistently, with increases in ν2. Error was nonmonotonically related to ?. A second investigation explored the accuracy of the approximations at values of αranging from .10 to .90. All four approximations degraded substantially in this situation, with maximum errors ranging from ?.09 to .05. If the analysis is restricted to cases where ν₁ τ; 1 and ν₂ τ; 2, maximum errors drop to roughly ±.03. We conclude that the approximations perform reasonably well for small αand moderately well for larger values, if certain restrictions are imposed. From a computational standpoint, however, there is little advantage to using approximate as opposed to exact methods unless exact values ofF _α are known in advance.     

Stopping Rules for Computer Adaptive Testing When Item Banks Have Nonuniform Information

The standard error (SE) stopping rule, which terminates a computer adaptive test (CAT) when the SE is less than a threshold, is effective when there are informative questions for all trait levels. However, in domains such as patient-reported outcomes, the items in a bank might all target one end of the trait continuum (e.g., negative symptoms), and the bank may lack depth for many individuals. In such cases, the predicted standard error reduction (PSER) stopping rule will stop the CAT even if the SE threshold has not been reached and can avoid administering excessive questions that provide little additional information. By tuning the parameters of the PSER algorithm, a practitioner can specify a desired tradeoff between accuracy and efficiency. Using simulated data for the Patient-Reported Outcomes Measurement Information System Anxiety and Physical Function banks, we demonstrate that these parameters can substantially impact CAT performance. When the parameters were optimally tuned, the PSER stopping rule was found to outperform the SE stopping rule overall, particularly for individuals not targeted by the bank, and presented roughly the same number of items across the trait continuum. Therefore, the PSER stopping rule provides an effective method for balancing the precision and efficiency of a CAT.     

Round-robin analysis of social interaction: Exact and estimated standard errors

Kenny has proposed a variance-components model for dyadic social interaction. His Social Relations model estimates variances and covariances from a round-robin of two-person interactions. The current paper presents a matrix formulation of the Social Relations model. It uses the formulation to derive exact and estimated standard errors for round-robin estimates of Social Relations parameters.We are grateful to George Gilbert for comments on an earlier draft.     

Person‐specific versus multilevel autoregressive models: Accuracy in parameter estimates at the population and individual levels

This paper compares the multilevel modelling (MLM) approach and the person‐specific (PS) modelling approach in examining autoregressive (AR) relations with intensive longitudinal data. Two simulation studies are conducted to examine the influences of sample heterogeneity, time series length, sample size, and distribution of individual level AR coefficients on the accuracy of AR estimates, both at the population level and at the individual level. It is found that MLM generally outperforms the PS approach under two conditions: when the sample has a homogeneous AR pattern, namely, when all individuals in the sample are characterized by AR processes with the same order; and when the sample has heterogeneous AR patterns, but a multilevel model with a sufficiently high order (i.e., an order equal to or higher than the maximum order of individual AR patterns in the sample) is fitted and successfully converges. If a lower‐order multilevel model is chosen for heterogeneous samples, the higher‐order lagged effects are misrepresented, resulting in bias at the population level and larger prediction errors at the individual level. In these cases, the PS approach is preferable, given sufficient measurement occasions (T ≥ 50). In addition, sample size and distribution of individual level AR coefficients do not have a large impact on the results. Implications of these findings on model selection and research design are discussed.     

Ignoring Clustering in Confirmatory Factor Analysis: Some Consequences for Model Fit and Standardized Parameter Estimates

In many situations, researchers collect multilevel (clustered or nested) data yet analyze the data either ignoring the clustering (disaggregation) or averaging the micro-level units within each cluster and analyzing the aggregated data at the macro level (aggregation). In this study we investigate the effects of ignoring the nested nature of data in confirmatory factor analysis (CFA). The bias incurred by ignoring clustering is examined in terms of model fit and standardized parameter estimates, which are usually of interest to researchers who use CFA. We find that the disaggregation approach increases model misfit, especially when the intraclass correlation (ICC) is high, whereas the aggregation approach results in accurate detection of model misfit in the macro level. Standardized parameter estimates from the disaggregation and aggregation approaches are deviated toward the values of the macro- and micro-level standardized parameter estimates, respectively. The degree of deviation depends on ICC and cluster size, particularly for the aggregation method. The standard errors of standardized parameter estimates from the disaggregation approach depend on the macro-level item communalities. Those from the aggregation approach underestimate the standard errors in multilevel CFA (MCFA), especially when ICC is low. Thus, we conclude that MCFA or an alternative approach should be used if possible.     

Using data augmentation to obtain standard errors and conduct hypothesis tests in latent class and latent transition analysis

Latent class analysis (LCA) provides a means of identifying a mixture of subgroups in a population measured by multiple categorical indicators. Latent transition analysis (LTA) is a type of LCA that facilitates addressing research questions concerning stage-sequential change over time in longitudinal data. Both approaches have been used with increasing frequency in the social sciences. The objective of this article is to illustrate data augmentation (DA), a Markov chain Monte Carlo procedure that can be used to obtain parameter estimates and standard errors for LCA and LTA models. By use of DA it is possible to construct hypothesis tests concerning not only standard model parameters but also combinations of parameters, affording tremendous flexibility. DA is demonstrated with an example involving tests of ethnic differences, gender differences, and an Ethnicity x Gender interaction in the development of adolescent problem behavior.     

Robust Structural Equation Modeling with Missing Data and Auxiliary Variables

The paper develops a two-stage robust procedure for structural equation modeling (SEM) and an R package rsem to facilitate the use of the procedure by applied researchers. In the first stage, M-estimates of the saturated mean vector and covariance matrix of all variables are obtained. Those corresponding to the substantive variables are then fitted to the structural model in the second stage. A sandwich-type covariance matrix is used to obtain consistent standard errors (SE) of the structural parameter estimates. Rescaled, adjusted as well as corrected and F-statistics are proposed for overall model evaluation. Using R and EQS, the R package rsem combines the two stages and generates all the test statistics and consistent SEs. Following the robust analysis, multiple model fit indices and standardized solutions are provided in the corresponding output of EQS. An example with open/closed book examination data illustrates the proper use of the package. The method is further applied to the analysis of a data set from the National Longitudinal Survey of Youth 1997 cohort, and results show that the developed procedure not only gives a better endorsement of the substantive models but also yields estimates with uniformly smaller standard errors than the normal-distribution-based maximum likelihood.     

Examiner differences on actual WISC protocols

Interrater differences in scoring actual WISC protocols were determined for three different IQ levels. In general, differences among the 94 examiners tended to be within an acceptable range as established by the SE_m; varince on two Verbal subtests occasionally exceeded their corresponding SE_m. Also, examiner's demographic characteristics were not a major consistent source of variance in scoring WISC protocols.     

A practical guide to multilevel modeling

Collecting data from students within classrooms or schools, and collecting data from students on multiple occasions over time, are two common sampling methods used in educational research that often require multilevel modeling (MLM) data analysis techniques to avoid Type-1 errors. The purpose of this article is to clarify the seven major steps involved in a multilevel analysis: (1) clarifying the research question, (2) choosing the appropriate parameter estimator, (3) assessing the need for MLM, (4) building the level-1 model, (5) building the level-2 model, (6) multilevel effect size reporting, and (7) likelihood ratio model testing. The seven steps are illustrated with both a cross-sectional and a longitudinal MLM example from the National Educational Longitudinal Study (NELS) dataset. The goal of this article is to assist applied researchers in conducting and interpreting multilevel analyses and to offer recommendations to guide the reporting of MLM analysis results.     

Family of Origin Predictors of Marital Outcomes Among Coptic Orthodox Christian Egyptian-American Couples

Despite the importance of family within the Coptic community, the extant literature on Coptic family relationships is remarkably limited. The purpose of this study was to explore parental marital quality and family of origin (FOO) quality as predictors of marital satisfaction and marital stability among Coptic Orthodox Christian Egyptian-American couples. Self-report survey data from 255 married Coptic couples living in the United States were analyzed using an actor-partner interdependence model. Results indicated husband FOO quality predicted both marital satisfaction (β =.51, SE?=?.18, p?=?.005) and marital stability (β?=?.51, SE?=?.22, p?=?.022) among Coptic husbands but not among Coptic wives. Results further indicated wife parental marital quality, wife FOO quality, and husband parental marital quality were not predictive of marital satisfaction or marital stability among Coptic husbands or wives. Clinical implications, limitations, and recommendations for future research are discussed.     

Analytical Problems and Suggestions in the Analysis of Behavioral Economic Demand Curves

Behavioral economic demand curves (Hursh, Raslear, Shurtleff, Bauman, & Simmons, 1988) are innovative approaches to characterize the relationships between consumption of a substance and its price. In this article, we investigate common analytical issues in the use of behavioral economic demand curves, which can cause inconsistent interpretations of demand curves, and then we provide methodological suggestions to address those analytical issues.

We first demonstrate that log transformation with different added values for handling zeros changes model parameter estimates dramatically. Second, demand curves are often analyzed using an overparameterized model that results in an inefficient use of the available data and a lack of assessment of the variability among individuals.

To address these issues, we apply a nonlinear mixed effects model based on multivariate error structures that has not been used previously to analyze behavioral economic demand curves in the literature. We also propose analytical formulas for the relevant standard errors of derived values such as P _max, O _max, and elasticity. The proposed model stabilizes the derived values regardless of using different added increments and provides substantially smaller standard errors. We illustrate the data analysis procedure using data from a relative reinforcement efficacy study of simulated marijuana purchasing.