Omitted Variables in Multilevel Models

Statistical methodology for handling omitted variables is presented in a multilevel modeling framework. In many nonexperimental studies, the analyst may not have access to all requisite variables, and this omission may lead to biased estimates of model parameters. By exploiting the hierarchical nature of multilevel data, a battery of statistical tools are developed to test various forms of model misspecification as well as to obtain estimators that are robust to the presence of omitted variables. The methodology allows for tests of omitted effects at single and multiple levels. The paper also introduces intermediate-level tests; these are tests for omitted effects at a single level, regardless of the presence of omitted effects at a higher level. A simulation study shows, not surprisingly, that the omission of variables yields bias in both regression coefficients and variance components; it also suggests that omitted effects at lower levels may cause more severe bias than at higher levels. Important factors resulting in bias were found to be the level of an omitted variable, its effect size, and sample size. A real data study illustrates that an omitted variable at one level may yield biased estimators at any level and, in this study, one cannot obtain reliable estimates for school-level variables when omitted child effects exist. However, robust estimators may provide unbiased estimates for effects of interest even when the efficient estimators fail, and the one-degree-of-freedom test helps one to understand where the problem is located. It is argued that multilevel data typically contain rich information to deal with omitted variables, offering yet another appealing reason for the use of multilevel models in the social sciences. This research was supported by the National Academy of Education/Spencer Foundation and the National Science Foundation, Grant Number SES-0436274.     

Asymptotic bias of normal-distribution-based maximum likelihood estimates of moderation effects with data missing at random

Moderation analysis is useful for addressing interesting research questions in social sciences and behavioural research. In practice, moderated multiple regression (MMR) models have been most widely used. However, missing data pose a challenge, mainly because the interaction term is a product of two or more variables and thus is a non-linear function of the involved variables. Normal-distribution-based maximum likelihood (NML) has been proposed and applied for estimating MMR models with incomplete data. When data are missing completely at random, moderation effect estimates are consistent. However, simulation results have found that when data in the predictor are missing at random (MAR), NML can yield inaccurate estimates of moderation effects when the moderation effects are non-null. Simulation studies are subject to the limitation of confounding systematic bias with sampling errors. Thus, the purpose of this paper is to analytically derive asymptotic bias of NML estimates of moderation effects with MAR data. Results show that when the moderation effect is zero, there is no asymptotic bias in moderation effect estimates with either normal or non-normal data. When the moderation effect is non-zero, however, asymptotic bias may exist and is determined by factors such as the moderation effect size, missing-data proportion, and type of missingness dependence. Our analytical results suggest that researchers should apply NML to MMR models with caution when missing data exist. Suggestions are given regarding moderation analysis with missing data.     

Predicting group-level outcome variables: An empirical comparison of analysis strategies

This study provides a review of two methods for analyzing multilevel data with group-level outcome variables and compares them in a simulation study. The analytical methods included an unadjusted ordinary least squares (OLS) analysis of group means and a two-step adjustment of the group means suggested by Croon and van Veldhoven (2007). The Type I error control, power, bias, standard errors, and RMSE in parameter estimates were compared across design conditions that included manipulations of number of predictor variables, level of correlation between predictors, level of intraclass correlation, predictor reliability, effect size, and sample size. The results suggested that an OLS analysis of the group means, with White’s heteroscedasticity adjustment, provided more power for tests of group-level predictors, but less power for tests of individual-level predictors. Furthermore, this simple analysis avoided the extreme bias in parameter estimates and inadmissible solutions that were encountered with other strategies. These results were interpreted in terms of recommended analytical methods for applied researchers.     

Mixture Factor Analysis for Approximating a Nonnormally Distributed Continuous Latent Factor With Continuous and Dichotomous Observed Variables

Mixture factor analysis is examined as a means of flexibly estimating nonnormally distributed continuous latent factors in the presence of both continuous and dichotomous observed variables. A simulation study compares mixture factor analysis with normal maximum likelihood (ML) latent factor modeling. Different results emerge for continuous versus dichotomous outcomes. For dichotomous outcomes, normal ML path estimates have bias that worsens as latent factor skew/kurtosis increases and does not diminish as sample size increases, whereas the mixture factor analysis model produces nearly unbiased estimators as sample sizes increase (500 and greater) and offers near nominal coverage probability. For continuous outcome variables, both methods produce factor loading estimates with minimal bias regardless of latent factor skew, but the mixture factor analysis is more efficient. The method is demonstrated using data motivated by a study on youth with cystic fibrosis examining predictors of treatment adherence. In summary, mixture factor analysis provides improvements over normal ML estimation in the presence of skewed/kurtotic latent factors, but due to variability in the estimator relating the latent factor to dichotomous outcomes and computational issues, the improvements were only fully realized, in this study, at larger sample sizes (500 and greater).     

The impact of ignoring random features of predictor and moderator variables on sample size for precise interval estimation of interaction effects

The influence of the joint distribution of predictor and moderator variables on the identification of interactions has been well described, but the impact on sample size determinations has received rather limited attention within the framework of moderated multiple regression (MMR). This article investigates the deficiency in sample size determinations for precise interval estimation of interaction effects that can result from ignoring the stochastic nature of continuous predictor and moderator variables in MMR. The primary finding of our examinations is that failure to accommodate the distributional properties of regressors can lead to underestimation of the necessary sample size and distortion of the desired interval precision. In order to take account of the randomness of regressor variables, two general and effective procedures for computing sample size estimates are presented. Moreover, corresponding programs are provided to facilitate use of the suggested approaches. This exposition helps to correct drawbacks in the existing techniques and to advance the practice of reporting confidence intervals in MMR analyses.     

Demonstration and evaluation of a method for assessing mediated moderation

Mediated moderation occurs when the interaction between two variables affects a mediator, which then affects a dependent variable. In this article, we describe the mediated moderation model and evaluate it with a statistical simulation using an adaptation of product-of-coefficients methods to assess mediation. We also demonstrate the use of this method with a substantive example from the adolescent tobacco literature. In the simulation, relative bias (RB) in point estimates and standard errors did not exceed problematic levels of ±10%, although systematic variability in RB was accounted for by parameter size, sample size, and nonzero direct effects. Power to detect mediated moderation effects appears to be severely compromised under one particular combination of conditions: when the component variables that make up the interaction terms are correlated and partial mediated moderation exists. Implications for the estimation of mediated moderation effects in experimental and nonexperimental research are discussed.     

Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data

A 2-step approach for obtaining internal consistency reliability estimates with item-level missing data is outlined. In the 1st step, a covariance matrix and mean vector are obtained using the expectation maximization (EM) algorithm. In the 2nd step, reliability analyses are carried out in the usual fashion using the EM covariance matrix as input. A Monte Carlo simulation examined the impact of 6 variables (scale length, response categories, item correlations, sample size, missing data, and missing data technique) on 3 different outcomes: estimation bias, mean errors, and confidence interval coverage. The 2-step approach using EM consistently yielded the most accurate reliability estimates and produced coverage rates close to the advertised 95% rate. An easy method of implementing the procedure is outlined.     

Sample size and multiple regression analysis

Despite the development of procedures for calculating sample size as a function of relevant effect size parameters, rules of thumb tend to persist in designs of multiple regression studies. One explanation for their persistence may be the difficulty in formulating a reasonable a priori value of an effect size to be detected. This article presents methods for calculating effect sizes in multiple regression from a variety of perspectives and also introduces a new method based on an exchangeability structure among predictor variables. No single method is deemed superior, but rather examples show that a combination of methods is likely to be most valuable in many situations. A simulation provides a 2nd explanation for why rules of thumb for choosing sample size have persisted but also shows that the outcome of such underpowered studies will be a literature consisting of seemingly contradictory results.     

Effect Size Strengths in Subjective Well-Being Research

Subjective well-being (SWB) research is characterized by many large samples, which often results in virtually all variables being significantly related to well-being, even if the associations are small. In this article we explore the strengths of associations between various predictors and SWB outcomes. In addition to standard effect-size statistics, we also examined the range of the SWB scale covered in the distribution of the predictor, allowing us to estimate the strength of influence of each variable, independent of variability in the sample. We analyzed just a few variables to illustrate what our approach reveals. Our analyses included a representative sample of both the world and the United States, and our data included three types of SWB (life satisfaction (LS), positive affect (PA), and negative affect (NA)). The largest effect sizes emerged for societal characteristics, such as between-nations differences, as well as personal characteristics, such as perceived social support. Small or very small effect sizes were consistently found for demographic characteristics, such as sex, age, and marital status. Other effect sizes varied by the type of SWB being considered. For example, income resulted in a large effect size for LS, but small to medium effect sizes for PA and NA. We suggest that when scholars report and interpret the associations of predictor variables with SWB, they consider the strengths of their significant associations.

     

Two classes of models for magnitude estimation

One class of models assumes that presentation of a signal results in an internal representation as a random variable. Depending on whether the signal is close to or far from the preceding signal, the variance of the representation is smaller or larger. Responses are determined largely by this random variable; however, when the signal is close to the preceding one, the response is generated by modifying the representation multiplicatively by some function of the ratio of the previous response to its representation. Power and linear functions are explored. The form of the random variable is assumed to be that arising from either the timing or the counting model operating on a Poisson process. Detailed analyses are carried out successfully only for the timing model with neural sample sizes independent of intensity; however, the data require the sample to increase with intensity. The linear response function coupled with the constant sample size counting model appears somewhat viable, but detailed calculations are very difficult to carry out. The second class of models postulates a power function relation between magnitude estimates and signals intensity for which the exponent is a Gaussian distributed random variable and the unit is the product of two log normal random variables. Again we assume an attention band such that succesive stimuli that are widely separated in intensity lead to independent samples of the random variables while a variety of assumptions is explored for successive stimuli that are near each other in intensity. Although they each give rise to the qualitative features of the data, estimates of parameters are sufficiently inconsistent that we are led to reject all of the submodels studied.     

On a test of dimensionality in redundancy analysis

Lazraq and Cléroux (Psychometrika, 2002, 411–419) proposed a test for identifying the number of significant components in redundancy analysis. This test, however, is ill-conceived. A major problem is that it regards each redundancy component as if it were a single observed predictor variable, which cannot be justified except for the rare situations in which there is only one predictor variable. Consequently, the proposed test leads to drastically biased results, particularly when the number of predictor variables is large, and it cannot be recommended for use. This is shown both theoretically and by Monte Carlo studies.The work reported in this paper was supported by Grant A6394 to the first author from the Natural Sciences and Engineering Research Council of Canada.     

Abstract: Genome-Wide Complex Trait Analysis of Shared Genetic Effects Between Brain Electrophysiology Measures and Alcohol Dependence

Multilevel analysis is an appropriate tool for the analysis of hierarchically structured data. There may, however, be reasons to ignore one of the levels of nesting in the data analysis. In this article a three level model with one predictor variable is used as a reference model and the top or intermediate level is ignored in the data analysis. Analytical results show that this has an effect on the estimated variance components and that standard errors of regression coefficients estimators may be overestimated, leading to a lower power of the test of the effect of the predictor variable. The magnitude of these results depends on the ignored level and the level at which the predictor variable varies, and on the values of the variance components and the sample sizes.     

The biasing effects of selection and attrition on estimating the mean

Organizational and validation researchers often work with data that has been subjected to selection on the predictor and attrition on the criterion. These researchers often use the data observed under these conditions to estimate either the predictor or criterion's restricted population means. We show that the restricted means due to direct or indirect selection are a function of the population means plus the selection ratios. Thus, any difference between selected mean groups reflects the population difference plus the selection ratio difference. When there is also attrition on the criterion, the estimation of group differences becomes even more complicated. The effect of selection and attrition induces measurement bias when estimating the restricted population mean of either the predictor or criterion. A sample mean observed under selection and attrition does not estimate either the population mean or the restricted population mean. We propose several procedures under normality that yield unbiased estimates of the mean. The procedures focus on correcting the effects of selection and attrition. Each procedure was evaluated with a Monte Carlo simulation to ascertain its strengths and weaknesses. Given appropriate sample size and conditions, we show that these procedures yield unbiased estimators of the restricted and unrestricted population means for both predictor and criterion. We also show how our findings have implications for replicating selected group differences.     

有调节的多层中介效应分析

多层中介和多层调节效应分析在社科领域已常有应用,但如果将多层中介和调节整合在一起,可以产生2（多层中介类型）×2（调节变量的层次）×3（调节的中介路径）共12种有调节的多层中介模型。面对有调节的多层中介效应分析,研究者往往束手无策。详述了基于多层线性模型的12种有调节的多层中介的分析方法和基于多层结构方程模型的4类有调节的多层中介分析方法,包括正交分割法,随机系数预测法,潜调节结构方程法和贝叶斯合理值法。这四类方法的核心议题在于如何处理潜调节项。当样本量足够大时,建议选择潜调节结构方程法;当样本量不足时,建议选择贝叶斯合理值法。随后用一个实际例子演示如何进行有调节的多层中介效应分析并有相应的Mplus程序。最后展望了有调节的多层中介效应分析研究的拓展方向。     

A Meta-Analysis of Research on Hindsight Bias

We conducted a meta-analysis of research on hindsight bias to gain an up-to-date summary of the overall strength of hindsight effects and to test hypotheses about potential moderators of hindsight distortion. A total of 95 studies (83 published and 12 unpublished) were included, and 252 independent effect sizes were coded for moderator variables in 3 broad categories involving characteristics of the study, of measurement, and of the experimental manipulation. When excluding missing effect sizes, the overall mean effect size was Md = .39 with a 95% confidence interval of .36 to .42. Five main findings emerged: (a) effect sizes calculated from objective probability estimates were larger than effect sizes calculated from subjective probability estimates; (b) effect sizes of studies that used almanac questions were larger than effect sizes of studies that used real-world events or case histories; (c) studies that included neutral outcomes resulted in larger effect sizes than studies that used positive or negative outcomes; (d) studies that included manipulations to increase hindsight bias resulted in significantly larger effect sizes than studies in which there were no manipulations to reduce or increase hindsight bias; and (e) studies that included manipulations to reduce hindsight bias did not produce lower effect sizes. These findings contribute to our understanding of hindsight bias by updating the state of knowledge, widening the range of known moderator variables, identifying factors that may activate different mediating processes, and highlighting critical gaps in the research literature.     

The Impact of Item Misspecification and Dichotomization on Class and Parameter Recovery in LCA of Count Data

Mixture analysis of count data has become increasingly popular among researchers of substance use, behavioral analysis, and program evaluation. However, this increase in popularity seems to have occurred along with adoption of some conventions in model specification based on arbitrary heuristics that may impact the validity of results. Findings from a systematic review of recent drug and alcohol publications suggested count variables are often dichotomized or misspecified as continuous normal indicators in mixture analysis. Prior research suggests that misspecifying skewed distributions of continuous indicators in mixture analysis introduces bias, though the consequences of this practice when applied to count indicators has not been studied. The present work describes results from a simulation study examining bias in mixture recovery when count indicators are dichotomized (median split; presence vs. absence), ordinalized, or the distribution is misspecified (continuous normal; incorrect count distribution). All distributional misspecifications and methods of categorizing resulted in greater bias in parameter estimates and recovery of class membership relative to specifying the true distribution, though dichotomization appeared to improve class enumeration accuracy relative to all other specifications. Overall, results demonstrate the importance of accurately modeling count indicators in mixture analysis, as misspecification and categorizing data can distort study outcomes.     

Likability and Similarity as Potential Sources of Predictor-Related Criterion Bias in Validation Research

Previous research on the effects of bias in criterion-related validation research has typically involved the use of statistical corrections for halo, leniency, and/or central tendency. We present arguments that likability and similarity of raters to ratees may constitute a form of predictor-related criterion bias. One cannot investigate this form of bias without clear understanding of method, predictor, and criterion constructs and careful direct measurement of each. Measurement and theorizing of method constructs is rarely, if ever, undertaken in criterion-related validation work. The results of a criterion-related validation of the use of quantitative and verbal ability and interview and role-play simulations to predict the performance of 372 federal investigative agents are reported. Using the all-Y LISREL model (Williams & Anderson, 1994), we found that likability and similarity factors were related to interview and role play measures. However, none of these potential "biases" affected both predictor and criterion constructs, hence there was no effect on the estimates of the relationships between the predictors and criteria. Limitations with respect to the generalizability of these results to criterion-related research in which performance data are not as carefully collected as well as advantages and disadvantages of more traditional regression and correlational analyses are discussed.     

Extending the Debate Between Spearman and Wilson 1929: When do Single Variables Optimally Reproduce the Common Part of the Observed Covariances?

The covariances of observed variables reproduced from conventional factor score predictors are generally not the same as the covariances reproduced from the common factors. We sought to find a factor score predictor that optimally reproduces the common part of the observed covariances. It was found algebraically that—under some conditions—the single observed variable with highest loading on a factor reproduces the non-diagonal elements of the observed covariance matrix more exactly than the conventional factor score predictors. This finding is linked to Spearman's and Wilson's 1929 debate on the use of single variables as factor score predictors. A population-based and a sample-based simulation study confirmed the algebraic result that taking a single variable can outperform conventional factor score predictors in reproducing the non-diagonal covariances when the nonzero loading size and the number of nonzero loadings per factor are small. The results indicated that a weighted aggregation of variables does not necessarily lead to an improvement of the score over the variable with the highest loading.