A comparison of two‐stage procedures for testing least‐squares coefficients under heteroscedasticity

This study explores the performance of several two‐stage procedures for testing ordinary least‐squares (OLS) coefficients under heteroscedasticity. A test of the usual homoscedasticity assumption is carried out in the first stage of the procedure. Subsequently, a test of the regression coefficients is chosen and performed in the second stage. Three recently developed methods for detecting heteroscedasticity are examined. In addition, three heteroscedastic robust tests of OLS coefficients are considered. A major finding is that performing a test of heteroscedasticity prior to applying a heteroscedastic robust test can lead to poor control over Type I errors.     

Simplified Estimation and Testing in Unbalanced Repeated Measures Designs

In this paper we propose a simple estimator for unbalanced repeated measures design models where each unit is observed at least once in each cell of the experimental design. The estimator does not require a model of the error covariance structure. Thus, circularity of the error covariance matrix and estimation of correlation parameters and variances are not necessary. Together with a weak assumption about the reason for the varying number of observations, the proposed estimator and its variance estimator are unbiased. As an alternative to confidence intervals based on the normality assumption, a bias-corrected and accelerated bootstrap technique is considered. We also propose the naive percentile bootstrap for Wald-type tests where the standard Wald test may break down when the number of observations is small relative to the number of parameters to be estimated. In a simulation study we illustrate the properties of the estimator and the bootstrap techniques to calculate confidence intervals and conduct hypothesis tests in small and large samples under normality and non-normality of the errors. The results imply that the simple estimator is only slightly less efficient than an estimator that correctly assumes a block structure of the error correlation matrix, a special case of which is an equi-correlation matrix. Application of the estimator and the bootstrap technique is illustrated using data from a task switch experiment based on an experimental within design with 32 cells and 33 participants.

     

The WIG (weighted individual and group) shrinkage estimator

Regularization, or shrinkage estimation, refers to a class of statistical methods that constrain the variability of parameter estimates when fitting models to data. These constraints move parameters toward a group mean or toward a fixed point (e.g., 0). Regularization has gained popularity across many fields for its ability to increase predictive power over classical techniques. However, articles published in JEAB and other behavioral journals have yet to adopt these methods. This paper reviews some common regularization schemes and speculates as to why articles published in JEAB do not use them. In response, we propose our own shrinkage estimator that avoids some of the possible objections associated with the reviewed regularization methods. Our estimator works by mixing weighted individual and group (WIG) data rather than by constraining parameters. We test this method on a problem of model selection. Specifically, we conduct a simulation study on the selection of matching‐law‐based punishment models, comparing WIG with ordinary least squares (OLS) regression, and find that, on average, WIG outperforms OLS in this context.     

Some results on extensions and modifications of the Theil — Sen regression estimator

Many robust regression estimators have been proposed that have a high, finite‐sample breakdown point, roughly meaning that a large porportion of points must be altered to drive the value of an estimator to infinity. But despite this, many of them can be inordinately influenced by two properly placed outliers. With one predictor, an estimator that appears to correct this problem to a fair degree, and simultaneously maintain good efficiency when standard assumptions are met, consists of checking for outliers using a projection‐type method, removing any that are found, and applying the Theil — Sen estimator to the data that remain. When dealing with multiple predictors, there are two generalizations of the Theil — Sen estimator that might be used, but nothing is known about how their small‐sample properties compare. Also, there are no results on testing the hypothesis of zero slopes, and there is no information about the effect on efficiency when outliers are removed. In terms of hypothesis testing, using the more obvious percentile bootstrap method in conjunction with a slight modification of Mahalanobis distance was found to avoid Type I error probabilities above the nominal level, but in some situations the actual Type I error probabilities can be substantially smaller than intended when the sample size is small. An alternative method is found to be more satisfactory.     

Interval estimation of gamma for an<Emphasis Type="Italic">R</Emphasis>×<Emphasis Type="Italic">S</Emphasis> table

When the underlying responses are on an ordinal scale, gamma is one of the most frequently used indices to measure the strength of association between two ordered variables. However, except for a brief mention on the use of the traditional interval estimator based on Wald's statistic, discussion of interval estimation of the gamma is limited. Because it is well known that an interval estimator using Wald's statistic is generally not likely to perform well especially when the sample size is small, the goal of this paper is to find ways to improve the finite-sample performance of this estimator. This paper develops five asymptotic interval estimators of the gamma by employing various methods that are commonly used to improve the normal approximation of the maximum likelihood estimator (MLE). Using Monte Carlo simulation, this paper notes that the coverage probability of the interval estimator using Wald's statistic can be much less than the desired confidence level, especially when the underlying gamma is large. Further, except for the extreme case, in which the underlying gamma is large and the sample size is small, the interval estimator using a logarithmic transformation together with a monotonic function proposed here not only performs well with respect to the coverage probability, but is also more efficient than all the other estimators considered here. Finally, this paper notes that applying an ad hoc adjustment procedure—whenever any observed frequency equals 0, we add 0.5 to all cells in calculation of the cell proportions—can substantially improve the traditional interval estimator. This paper includes two examples to illustrate the practical use of interval estimators considered here.The authors wish to thank the Associate Editor and the two referees for many valuable comments and suggestions to improve the contents and clarity of this paper. The authors also want to thank Dr. C. D. Lin for his graphic assistance.     

Model Implied Instrumental Variables (MIIVs): An Alternative Orientation to Structural Equation Modeling

Few dispute that our models are approximations to reality. Yet when it comes to structural equation models (SEMs), we use estimators that assume true models (e.g. maximum likelihood) and that can create biased estimates when the model is inexact. This article presents an overview of the Model Implied Instrumental Variable (MIIV) approach to SEMs from Bollen (1996). The MIIV estimator using Two Stage Least Squares (2SLS), MIIV-2SLS, has greater robustness to structural misspecifications than system wide estimators. In addition, the MIIV-2SLS estimator is asymptotically distribution free. Furthermore, MIIV-2SLS has equation-based overidentification tests that can help pinpoint misspecifications. Beyond these features, the MIIV approach has other desirable qualities. MIIV methods apply to higher order factor analyses, categorical measures, growth curve models, dynamic factor analysis, and nonlinear latent variables. Finally, MIIV-2SLS permits researchers to estimate and test only the latent variable model or any other subset of equations. In addition, other MIIV estimators beyond 2SLS are available. Despite these promising features, research is needed to better understand its performance under a variety of conditions that represent empirical applications. Empirical and simulation examples in the article illustrate the MIIV orientation to SEMs and highlight an R package MIIVsem that implements MIIV-2SLS.     

Robust inference with binary data

In this paper robustness properties of the maximum likelihood estimator (MLE) and several robust estimators for the logistic regression model when the responses are binary are analysed. It is found that the MLE and the classical Rao's score test can be misleading in the presence of model misspecification which in the context of logistic regression means either misclassification's errors in the responses, or extreme data points in the design space. A general framework for robust estimation and testing is presented and a robust estimator as well as a robust testing procedure are presented. It is shown that they are less influenced by model misspecifications than their classical counterparts. They are finally applied to the analysis of binary data from a study on breastfeeding.The author is partially supported by the Swiss National Science Foundation. She would like to thank Rand Wilcox, Eva Cantoni and Elvezio Ronchetti for their helpful comments on earlier versions of the paper, as well as Stephane Heritier for providing the routine to compute the OBRE.     

Joint Maximum Likelihood Estimation for Diagnostic Classification Models

Joint maximum likelihood estimation (JMLE) is developed for diagnostic classification models (DCMs). JMLE has been barely used in Psychometrics because JMLE parameter estimators typically lack statistical consistency. The JMLE procedure presented here resolves the consistency issue by incorporating an external, statistically consistent estimator of examinees’ proficiency class membership into the joint likelihood function, which subsequently allows for the construction of item parameter estimators that also have the consistency property. Consistency of the JMLE parameter estimators is established within the framework of general DCMs: The JMLE parameter estimators are derived for the Loglinear Cognitive Diagnosis Model (LCDM). Two consistency theorems are proven for the LCDM. Using the framework of general DCMs makes the results and proofs also applicable to DCMs that can be expressed as submodels of the LCDM. Simulation studies are reported for evaluating the performance of JMLE when used with tests of varying length and different numbers of attributes. As a practical application, JMLE is also used with “real world” educational data collected with a language proficiency test.     

Robust regression: Testing global hypotheses about the slopes when there is multicollinearity or heteroscedasticity

A well-known concern regarding the usual linear regression model is multicollinearity. As the strength of the association among the independent variables increases, the squared standard error of regression estimators tends to increase, which can seriously impact power. This paper examines heteroscedastic methods for dealing with this issue when testing the hypothesis that all of the slope parameters are equal to zero via a robust ridge estimator that guards against outliers among the dependent variable. Included are results related to leverage points, meaning outliers among the independent variables. In various situations, the proposed method increases power substantially.     

A new quantile estimator with weights based on a subsampling approach

Quantiles are widely used in both theoretical and applied statistics, and it is important to be able to deploy appropriate quantile estimators. To improve performance in the lower and upper quantiles, especially with small sample sizes, a new quantile estimator is introduced which is a weighted average of all order statistics. The new estimator, denoted NO, has desirable asymptotic properties. Moreover, it offers practical advantages over four estimators in terms of efficiency in most experimental settings. The Harrell–Davis quantile estimator, the default quantile estimator of the R programming language, the Sfakianakis–Verginis SV2 quantile estimator and a kernel quantile estimator. The NO quantile estimator is also utilized in comparing two independent groups with a percentile bootstrap method and, as expected, it is more successful than other estimators in controlling Type I error rates.     

Multilevel Modeling with Correlated Effects

When there exist omitted effects, measurement error, and/or simultaneity in multilevel models, explanatory variables may be correlated with random components, and standard estimation methods do not provide consistent estimates of model parameters. This paper introduces estimators that are consistent under such conditions. By employing generalized method of moments (GMM) estimation techniques in multilevel modeling, the authors present a series of estimators along a robust to efficient continuum. This continuum depends on the assumptions that the analyst makes regarding the extent of the correlated effects. It is shown that the GMM approach provides an overarching framework that encompasses well-known estimators such as fixed and random effects estimators and also provides more options. These GMM estimators can be expressed as instrumental variable (IV) estimators which enhances their interpretability. Moreover, by exploiting the hierarchical structure of the data, the current technique does not require additional variables unlike traditional IV methods. Further, statistical tests are developed to compare the different estimators. A simulation study examines the finite sample properties of the estimators and tests and confirms the theoretical order of the estimators with respect to their robustness and efficiency. It further shows that not only are regression coefficients biased, but variance components may be severely underestimated in the presence of correlated effects. Empirical standard errors are employed as they are less sensitive to correlated effects when compared to model-based standard errors. An example using student achievement data shows that GMM estimators can be effectively used in a search for the most efficient among unbiased estimators. This research was supported by the National Academy of Education/Spencer Foundation and the National Science Foundation, grant number SES-0436274. We thank the editor, associate editor, and referees for detailed feedback that helped improve the paper.     

Checking the Assumptions of Rasch's Model for Speed Tests

Two new tests for a model for the response times on pure speed tests by Rasch (1960) are proposed. The model is based on the assumption that the test response times are approximately gamma distributed, with known index parameters and unknown rate parameters. The rate parameters are decomposed in a subject ability parameter and a test difficulty parameter. By treating the ability as a gamma distributed random variable, maximum marginal likelihood (MML) estimators for the test difficulty parameters and the parameters of the ability distribution are easily derived. Also the model tests proposed here pertain to the framework of MML. Two tests or modification indices are proposed. The first one is focused on the assumption of local stochastic independence, the second one on the assumption of the test characteristic functions. The tests are based on Lagrange multiplier statistics, and can therefore be computed using the parameter estimates under the null model. Therefore, model violations for all items and pairs of items can be assessed as a by-product of one single estimation run. Power studies and applications to real data are included as numerical examples.     

Estimating Success in Predicting a Variable with Nominal Measurement Using Other Variables with Nominal Measurements

In this article we are concerned with the situation where one is estimating the outcome of a variable Y, with nominal measurement, on the basis of the outcomes of several predictor variables, X 1, X 2, ..., X r, each with nominal measurement. We assume that we have a random sample from the population. Here we are interested in estimating p, the probability of successfully predicting a new Y from the population, given the X measurements for this new observation. We begin by proposing an estimator, p_a, which is the success rate in predicting Y from the current sample. We show that this estimator is always biased upwards. We then propose a second estimator, p_b, which divides the original sample into two groups, a holdout group and a training group, in order to estimate p. We show that procedures such as these are always biased downwards, no matter how we divide the original sample into the two groups. Because one of these estimators tends to overestimate p while the other tends to underestimate p, we propose as a heuristic solution to use the mean of these two estimators, p_c, as an estimator for p. We then perform several simulation studies to compare the three estimators with respect to both bias and MSE. These simulations seem to confirm that $ p c is a better estimator than either of the other two.     

Comparing the accuracy of experimental estimates to guessing: a new perspective on replication and the “Crisis of Confidence” in psychology

We develop a general measure of estimation accuracy for fundamental research designs, called v. The v measure compares the estimation accuracy of the ubiquitous ordinary least squares (OLS) estimator, which includes sample means as a special case, with a benchmark estimator that randomizes the direction of treatment effects. For sample and effect sizes common to experimental psychology, v suggests that OLS produces estimates that are insufficiently accurate for the type of hypotheses being tested. We demonstrate how v can be used to determine sample sizes to obtain minimum acceptable estimation accuracy. Software for calculating v is included as online supplemental material (R Core Team, 2012).     

Asymptotically distribution-free (ADF) interval estimation of coefficient alpha

The point estimate of sample coefficient alpha may provide a misleading impression of the reliability of the test score. Because sample coefficient alpha is consistently biased downward, it is more likely to yield a misleading impression of poor reliability. The magnitude of the bias is greatest precisely when the variability of sample alpha is greatest (small population reliability and small sample size). Taking into account the variability of sample alpha with an interval estimator may lead to retaining reliable tests that would be otherwise rejected. Here, the authors performed simulation studies to investigate the behavior of asymptotically distribution-free (ADF) versus normal-theory interval estimators of coefficient alpha under varied conditions. Normal-theory intervals were found to be less accurate when item skewness >1 or excess kurtosis >1. For sample sizes over 100 observations, ADF intervals are preferable, regardless of item skewness and kurtosis. A formula for computing ADF confidence intervals for coefficient alpha for tests of any size is provided, along with its implementation as an SAS macro.     

Some contributions to efficient statistics in structural models: Specification and estimation of moment structures

Current practice in structural modeling of observed continuous random variables is limited to representation systems for first and second moments (e.g., means and covariances), and to distribution theory based on multivariate normality. In psychometrics the multinormality assumption is often incorrect, so that statistical tests on parameters, or model goodness of fit, will frequently be incorrect as well. It is shown that higher order product moments yield important structural information when the distribution of variables is arbitrary. Structural representations are developed for generalizations of the Bentler-Weeks, Jöreskog-Keesling-Wiley, and factor analytic models. Some asymptotically distribution-free efficient estimators for such arbitrary structural models are developed. Limited information estimators are obtained as well. The special case of elliptical distributions that allow nonzero but equal kurtoses for variables is discussed in some detail. The argument is made that multivariate normal theory for covariance structure models should be abandoned in favor of elliptical theory, which is only slightly more difficult to apply in practice but specializes to the traditional case when normality holds. Many open research areas are described.     

On accuracy in reliability estimation

This study in parametric test theory deals with the statistics of reliability estimation when scores on two parts of a test follow a binormal distribution with equal (Case 1) or unequal (Case 2) expectations. In each case biased maximum-likelihood estimators of reliability are obtained and converted into unbiased estimators. Sampling distributions are derived. Second moments are obtained and utilized in calculating mean square errors of estimation as a measure of accuracy. A rank order of four estimators is established. There is a uniformly best estimator. Tables of absolute and relative accuracies are provided for various reliability parameters and sample sizes.     

A latent trait model for response times on tests employing the proportional hazards model

For computer-administered tests, response times can be recorded conjointly with the corresponding responses. This broadens the scope of potential modelling approaches because response times can be analysed in addition to analysing the responses themselves. For this purpose, we present a new latent trait model for response times on tests. This model is based on the Cox proportional hazards model. According to this model, latent variables alter a baseline hazard function. Two different approaches to item parameter estimation are described: the first approach uses a variant of the Cox model for discrete time, whereas the second approach is based on a profile likelihood function. Properties of each estimator will be compared in a simulation study. Compared to the estimator for discrete time, the profile likelihood estimator is more efficient, that is, has smaller variance. Additionally, we show how the fit of the model can be evaluated and how the latent traits can be estimated. Finally, the applicability of the model to an empirical data set is demonstrated.     

On robust estimation of effect size under semiparametric models

Estimation of effect size is of interest in many applied fields such as Psychology, Sociology and Education. However there are few nonparametric estimators of effect size proposed in the existing literature, and little is known about the distributional characteristics of these estimators. In this article, two estimators based on the sample quantiles are proposed and studied. The first one is the estimator suggested by Hedges and Olkin (see page 93 of Hedges & Olkin, 1985) for the situation where a treatment effect is evaluated against a control group (Case A). A modified version of the robust estimator by Hedges and Olkin is also proposed for the situation where two parallel treatments are compared (Case B). Large sample distributions of both estimators are derived. Their asymptotic relative efficiencies with respect to the normal maximum likelihood estimators under several common distributions are evaluated. The robust properties of the proposed estimators are discussed with respect to the sample-wise breakdown points proposed by Akritas (1991). Simulation studies are provided in which the performing characteristics of the proposed estimator are compared to that of the nonparametric estimators by Kraemer and Andrews (1982). Interval estimation of the effect sizes is also discussed. In an example, interval estimates for the data set in Kraemer and Andrews (1982) are calculated for both cases A and B.