SIMSTAT: Bootstrap computer simulation and statistical program for IBM personal computers

SIMSTAT is a computer program that provides bootstrap confidence intervals and estimates distribution parameters for 27 univariate and bivariate statistics. Because it permits variation of the bootstrap sample size, it is a convenient tool for research planning, allowing comparison of power estimates and estimator precision for various estimators and/or sample sizes. SIMSTAT also is a useful tool for studying the robustness of various statistics on real-world data and may be helpful in teaching statistical principles.     

On accuracy in reliability estimation

This study in parametric test theory deals with the statistics of reliability estimation when scores on two parts of a test follow a binormal distribution with equal (Case 1) or unequal (Case 2) expectations. In each case biased maximum-likelihood estimators of reliability are obtained and converted into unbiased estimators. Sampling distributions are derived. Second moments are obtained and utilized in calculating mean square errors of estimation as a measure of accuracy. A rank order of four estimators is established. There is a uniformly best estimator. Tables of absolute and relative accuracies are provided for various reliability parameters and sample sizes.     

An alternative two stage least squares (2SLS) estimator for latent variable equations

The Maximum-likelihood estimator dominates the estimation of general structural equation models. Noniterative, equation-by-equation estimators for factor analysis have received some attention, but little has been done on such estimators for latent variable equations. I propose an alternative 2SLS estimator of the parameters in LISREL type models and contrast it with the existing ones. The new 2SLS estimator allows observed and latent variables to originate from nonnormal distributions, is consistent, has a known asymptotic covariance matrix, and is estimable with standard statistical software. Diagnostics for evaluating instrumental variables are described. An empirical example illustrates the estimator. I gratefully acknowledge support for this research from the Sociology Program of the National Science Foundation (SES-9121564) and the Center for Advanced Study in the Behavioral Sciences, Stanford, California. This paper was presented at the Interdisciplinary Consortium for Statistical Applications at Indiana University at Bloomington (March 2, 1994) and at the RMD Conference on Causal Modeling at Purdue University, West Lafayette, Indiana (March 3-5, 1994).     

Estimating Success in Predicting a Variable with Nominal Measurement Using Other Variables with Nominal Measurements

In this article we are concerned with the situation where one is estimating the outcome of a variable Y, with nominal measurement, on the basis of the outcomes of several predictor variables, X 1, X 2, ..., X r, each with nominal measurement. We assume that we have a random sample from the population. Here we are interested in estimating p, the probability of successfully predicting a new Y from the population, given the X measurements for this new observation. We begin by proposing an estimator, p_a, which is the success rate in predicting Y from the current sample. We show that this estimator is always biased upwards. We then propose a second estimator, p_b, which divides the original sample into two groups, a holdout group and a training group, in order to estimate p. We show that procedures such as these are always biased downwards, no matter how we divide the original sample into the two groups. Because one of these estimators tends to overestimate p while the other tends to underestimate p, we propose as a heuristic solution to use the mean of these two estimators, p_c, as an estimator for p. We then perform several simulation studies to compare the three estimators with respect to both bias and MSE. These simulations seem to confirm that $ p c is a better estimator than either of the other two.     

A Polychoric Instrumental Variable (PIV) Estimator for Structural Equation Models with Categorical Variables

This paper presents a new polychoric instrumental variable (PIV) estimator to use in structural equation models (SEMs) with categorical observed variables. The PIV estimator is a generalization of Bollen’s (Psychometrika 61:109–121, 1996) 2SLS/IV estimator for continuous variables to categorical endogenous variables. We derive the PIV estimator and its asymptotic standard errors for the regression coefficients in the latent variable and measurement models. We also provide an estimator of the variance and covariance parameters of the model, asymptotic standard errors for these, and test statistics of overall model fit. We examine this estimator via an empirical study and also via a small simulation study. Our results illustrate the greater robustness of the PIV estimator to structural misspecifications than the system-wide estimators that are commonly applied in SEMs. Kenneth Bollen gratefully acknowledges support from NSF SES 0617276, NIDA 1-RO1-DA13148-01, and DA013148-05A2. Albert Maydeu-Olivares was supported by the Department of Universities, Research and Information Society (DURSI) of the Catalan Government, and by grant BSO2003-08507 from the Spanish Ministry of Science and Technology. We thank Sharon Christ, John Hipp, and Shawn Bauldry for research assistance. The comments of the members of the Carolina Structural Equation Modeling (CSEM) group are greatly appreciated. An earlier version of this paper under a different title was presented by K. Bollen at the Psychometric Society Meetings, June, 2002, Chapel Hill, North Carolina.     

Threshold estimation in two-alternative forced-choice (2AFC) tasks: the Spearman-Kärber method

The Spearman-K?rber method can be used to estimate the threshold value or difference limen in two-alternative forced-choice tasks. This method yields a simple estimator for the difference limen and its standard error, so that both can be calculated with a pocket calculator. In contrast to previous estimators, the present approach does not require any assumptions about the shape of the true underlying psychometric function. The performance of this new nonparametric estimator is compared with the standard technique of probit analysis. The Spearman-K?rber method appears to be a valuable addition to the toolbox of psychophysical methods, because it is most accurate for estimating the mean (i.e., absolute and difference thresholds) and dispersion of the psychometric function, although it is not optimal for estimating percentile-based parameters of this function.     

Threshold estimation in two-alternative forced-choice (2AFC) tasks: The Spearman-Kärber method

The Spearman-Kärber method can be used to estimate the threshold value or difference limen in two-alternative forced-choice tasks. This method yields a simple estimator for the difference limen and its standard error, so that both can be calculated with a pocket calculator. In contrast to previous estimators, the present approach does not require any assumptions about the shape of the true underlying psychometric function. The performance of this new nonparametric estimator is compared with the standard technique of probit analysis. The Spearman-Kärber method appears to be a valuable addition to the toolbox of psychophysical methods, because it is most accurate for estimating the mean (i.e., absolute and difference thresholds) and dispersion of the psychometric function, although it is not optimal for estimating percentile-based parameters of this function.     

Interval estimation of gamma for an<Emphasis Type="Italic">R</Emphasis>×<Emphasis Type="Italic">S</Emphasis> table

When the underlying responses are on an ordinal scale, gamma is one of the most frequently used indices to measure the strength of association between two ordered variables. However, except for a brief mention on the use of the traditional interval estimator based on Wald's statistic, discussion of interval estimation of the gamma is limited. Because it is well known that an interval estimator using Wald's statistic is generally not likely to perform well especially when the sample size is small, the goal of this paper is to find ways to improve the finite-sample performance of this estimator. This paper develops five asymptotic interval estimators of the gamma by employing various methods that are commonly used to improve the normal approximation of the maximum likelihood estimator (MLE). Using Monte Carlo simulation, this paper notes that the coverage probability of the interval estimator using Wald's statistic can be much less than the desired confidence level, especially when the underlying gamma is large. Further, except for the extreme case, in which the underlying gamma is large and the sample size is small, the interval estimator using a logarithmic transformation together with a monotonic function proposed here not only performs well with respect to the coverage probability, but is also more efficient than all the other estimators considered here. Finally, this paper notes that applying an ad hoc adjustment procedure—whenever any observed frequency equals 0, we add 0.5 to all cells in calculation of the cell proportions—can substantially improve the traditional interval estimator. This paper includes two examples to illustrate the practical use of interval estimators considered here.The authors wish to thank the Associate Editor and the two referees for many valuable comments and suggestions to improve the contents and clarity of this paper. The authors also want to thank Dr. C. D. Lin for his graphic assistance.     

On robust estimation of effect size under semiparametric models

Estimation of effect size is of interest in many applied fields such as Psychology, Sociology and Education. However there are few nonparametric estimators of effect size proposed in the existing literature, and little is known about the distributional characteristics of these estimators. In this article, two estimators based on the sample quantiles are proposed and studied. The first one is the estimator suggested by Hedges and Olkin (see page 93 of Hedges & Olkin, 1985) for the situation where a treatment effect is evaluated against a control group (Case A). A modified version of the robust estimator by Hedges and Olkin is also proposed for the situation where two parallel treatments are compared (Case B). Large sample distributions of both estimators are derived. Their asymptotic relative efficiencies with respect to the normal maximum likelihood estimators under several common distributions are evaluated. The robust properties of the proposed estimators are discussed with respect to the sample-wise breakdown points proposed by Akritas (1991). Simulation studies are provided in which the performing characteristics of the proposed estimator are compared to that of the nonparametric estimators by Kraemer and Andrews (1982). Interval estimation of the effect sizes is also discussed. In an example, interval estimates for the data set in Kraemer and Andrews (1982) are calculated for both cases A and B.     

Sampling behaviour in estimating predictive validity in the context of selection and latent variable modelling: A Monte Carlo study

While the effect of selection in predictive validity studies has long been recognized and discussed in psychometric studies, little consideration has been given to this problem in the context of latent variable models. In a recent paper, Muthén & Hsu (1993) proposed and compared estimators of predictive validity of a multifactorial test. Both selectivity and measurement error were considered in the estimation of predictive validity. The purpose of the present paper is to expand on Muthén & Hsu (1993) by examining and comparing the sampling behaviour of three estimators for predictive validity, LQL (listwise, quasi-likelihood estimator), FQL (full, quasi-likelihood estimator) and FS (factor score estimator), using a Monte Carlo approach. Effects of selection procedures, selection ratios and sample sizes on the sampling behaviours of the estimators are also investigated. The results show that FQL and FS are the two preferred estimators and each has different strengths and weaknesses. A real data application is presented to illustrate the practical implementation of the estimators.     

Confidence intervals for the overall effect size in random-effects meta-analysis

One of the main objectives in meta-analysis is to estimate the overall effect size by calculating a confidence interval (CI). The usual procedure consists of assuming a standard normal distribution and a sampling variance defined as the inverse of the sum of the estimated weights of the effect sizes. But this procedure does not take into account the uncertainty due to the fact that the heterogeneity variance (tau2) and the within-study variances have to be estimated, leading to CIs that are too narrow with the consequence that the actual coverage probability is smaller than the nominal confidence level. In this article, the performances of 3 alternatives to the standard CI procedure are examined under a random-effects model and 8 different tau2 estimators to estimate the weights: the t distribution CI, the weighted variance CI (with an improved variance), and the quantile approximation method (recently proposed). The results of a Monte Carlo simulation showed that the weighted variance CI outperformed the other methods regardless of the tau2 estimator, the value of tau2, the number of studies, and the sample size.     

A note on the unbiased estimation of the intraclass correlation

The intraclass correlation,, is a parameter featured in much psychological research. Two commonly used estimators of, the maximum likelihood and least squares estimators, are known to be negatively biased. Olkin and Pratt (1958) derived the minimum variance unbiased estimator of the intraclass correlation, but use of this estimator has apparently been impeded by the lack of a closed form solution. This note briefly reviews the unbiased estimator and gives a FORTRAN 77 subroutine to calculate it.The first author was supported by an All-University Fellowship from the University of Southern California.     

Investigating the Impact of Uncertainty About Item Parameters on Ability Estimation

Asymptotic expansions of the maximum likelihood estimator (MLE) and weighted likelihood estimator (WLE) of an examinee’s ability are derived while item parameter estimators are treated as covariates measured with error. The asymptotic formulae present the amount of bias of the ability estimators due to the uncertainty of item parameter estimators. A numerical example is presented to illustrate how to apply the formulae to evaluate the impact of uncertainty about item parameters on ability estimation and the appropriateness of estimating ability using the regular MLE or WLE method.     

A Limited Information Estimator for Dynamic Factor Models

Structural equation modeling (SEM) is an increasingly popular method for examining multivariate time series data. As in cross-sectional data analysis, structural misspecification of time series models is inevitable, and further complicated by the fact that errors occur in both the time series and measurement components of the model. In this article, we introduce a new limited information estimator and local fit diagnostic for dynamic factor models within the SEM framework. We demonstrate the implementation of this estimator and examine its performance under both correct and incorrect model specifications via a small simulation study. The estimates from this estimator are compared to those from the most common system-wide estimators and are found to be more robust to the structural misspecifications considered.     

On the Relationships Between Jeffreys Modal and Weighted Likelihood Estimation of Ability Under Logistic IRT Models

This paper focuses on two estimators of ability with logistic item response theory models: the Bayesian modal (BM) estimator and the weighted likelihood (WL) estimator. For the BM estimator, Jeffreys’ prior distribution is considered, and the corresponding estimator is referred to as the Jeffreys modal (JM) estimator. It is established that under the three-parameter logistic model, the JM estimator returns larger estimates than the WL estimator. Several implications of this result are outlined.     

Quantile maximum likelihood estimation of response time distributions

Queen’s University, Kingston, Ontario, Canada We introduce and evaluate via a Monte Carlo study a robust new estimation technique that fits distribution functions to grouped response time (RT) data, where the grouping is determined by sample quantiles. The new estimator, quantile maximum likelihood (QML), is more efficient and less biased than the best alternative estimation technique when fitting the commonly used ex-Gaussian distribution. Limitations of the Monte Carlo results are discussed and guidance provided for the practical application of the new technique. Because QML estimation can be computationally costly, we make fast open source code for fitting available that can be easily modified     

Estimating the polyserial correlation coefficient

We develop simple noniterative estimators of the polyserial correlation coefficient. A general relationship between the polyserial correlation and the point polyserial correlation is exploited to give extensions of Pearson's, Brogden's, and Lord's biserial estimators to the multicategory setting. The small sample and asmptotic properties of these estimators are studied in some detail. A comparison with maximum likelihood estimates shows that Lord's polyserial estimator is fairly efficient across three probability models.The authors would like to thank the referees for suggestions that improved the presentation of the paper.     

The polyserial correlation coefficient

The polyserial and point polyserial correlations are discussed as generalizations of the biserial and point biserial correlations. The relationship between the polyserial and point polyserial correlation is derived. The maximum likelihood estimator of the polyserial correlation is compared with a two-step estimator and with a computationally convenient ad hoc estimator. All three estimators perform reasonably well in a Monte Carlo simulation. Some practical applications of the polyserial correlation are described.By coincidence, the first author and the second and third authors learned that they were working independently on closely related problems and, consequently, decided to write a jointly authored paper.     

ESTIMATION OF SAMPLING VARIANCE OF CORRELATIONS IN META-ANALYSIS

Monte Carlo simulations were conducted to compare the performance of the traditional (Fisher, 1954) and mean (Hunter & Schmidt, 1990) estimators of the sampling variance of correlations in meta-analysis. The mean estimator differs from the traditional estimator in that it uses the mean observed correlation, averaged across studies, in the sampling variance formula. The simulations investigated the homogeneous (i.e., no true correlation variance across studies) and heterogeneous case (i.e., true correlation variance across studies). Results reveal that, compared to the traditional estimator, the mean estimator provides less negatively biased estimates of sampling variance in the homogeneous and heterogeneous cases and more positively biased estimates in the heterogenous case. Thus, results support the use of the mean estimator unless strong, theory-based hypotheses regarding moderating effects exist.