A Monte Carlo study comparing PIV,ULS and DWLS in the estimation of dichotomous confirmatory factor analysis

We conducted a Monte Carlo study to investigate the performance of the polychoric instrumental variable estimator (PIV) in comparison to unweighted least squares (ULS) and diagonally weighted least squares (DWLS) in the estimation of a confirmatory factor analysis model with dichotomous indicators. The simulation involved 144 conditions (1,000 replications per condition) that were defined by a combination of (a) two types of latent factor models, (b) four sample sizes (100, 250, 500, 1,000), (c) three factor loadings (low, moderate, strong), (d) three levels of non‐normality (normal, moderately, and extremely non‐normal), and (e) whether the factor model was correctly specified or misspecified. The results showed that when the model was correctly specified, PIV produced estimates that were as accurate as ULS and DWLS. Furthermore, the simulation showed that PIV was more robust to structural misspecifications than ULS and DWLS.     

More efficient parameter estimates for factor analysis of ordinal variables by ridge generalized least squares

Data in psychology are often collected using Likert‐type scales, and it has been shown that factor analysis of Likert‐type data is better performed on the polychoric correlation matrix than on the product‐moment covariance matrix, especially when the distributions of the observed variables are skewed. In theory, factor analysis of the polychoric correlation matrix is best conducted using generalized least squares with an asymptotically correct weight matrix (AGLS). However, simulation studies showed that both least squares (LS) and diagonally weighted least squares (DWLS) perform better than AGLS, and thus LS or DWLS is routinely used in practice. In either LS or DWLS, the associations among the polychoric correlation coefficients are completely ignored. To mend such a gap between statistical theory and empirical work, this paper proposes new methods, called ridge GLS, for factor analysis of ordinal data. Monte Carlo results show that, for a wide range of sample sizes, ridge GLS methods yield uniformly more accurate parameter estimates than existing methods (LS, DWLS, AGLS). A real‐data example indicates that estimates by ridge GLS are 9–20% more efficient than those by existing methods. Rescaled and adjusted test statistics as well as sandwich‐type standard errors following the ridge GLS methods also perform reasonably well.     

Limited information estimation and testing of Thurstonian models for paired comparison data under multiple judgment sampling

We relate Thurstonian models for paired comparisons data to Thurstonian models for ranking data, which assign zero probabilities to all intransitive patterns. We also propose an intermediate model for paired comparisons data that assigns nonzero probabilities to all transitive patterns and to some but not all intransitive patterns.There is a close correspondence between the multidimensional normal ogive model employed in educational testing and Thurstone's model for paired comparisons data under multiple judgment sampling with minimal identification restrictions. Alike the normal ogive model, Thurstonian models have two formulations, a factor analytic and an IRT formulation. We use the factor analytic formulation to estimate this model from the first and second order marginals of the contingency table using estimators proposed by Muthén. We also propose a statistic to assess the fit of these models to the first and second order marginals of the contingency table. This is important, as a model may reproduce well the estimated thresholds and tetrachoric correlations, yet fail to reproduce the marginals of the contingency table if the assumption of multivariate normality is incorrect.A simulation study is performed to investigate the performance of three alternative limited information estimators which differ in the procedure used in their final stage: unweighted least squares (ULS), diagonally weighted least squares (DWLS), and full weighted least squares (WLS). Both the ULS and DWLS show a good performance with medium size problems and small samples, with a slight better performance of the ULS estimator.This paper is based on the author's doctoral dissertation; Ulf Böckenholt, advisor. The final stages of this research took place while the author was at the Department of Statistics and Econometrics, Universidad Carlos III de Madrid. The author is indebted to Adolfo Hernández for stimulating discussions that helped improve this paper, and to Ulf Böckenholt and the Associate Editor for a number of helpfulsuggestions to a previous draft.     

The Influence of Number of Categories and Threshold Values on Fit Indices in Structural Equation Modeling with Ordered Categorical Data

This study examines the unscaled and scaled root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker–Lewis index (TLI) of diagonally weighted least squares (DWLS) and unweighted least squares (ULS) estimators in structural equation modeling with ordered categorical data. We show that the number of categories and threshold values for categorization can unappealingly impact the DWLS unscaled and scaled fit indices, as well as the ULS scaled fit indices in the population, given that analysis models are misspecified and that the threshold structure is saturated. Consequently, a severely misspecified model may be considered acceptable, depending on how the underlying continuous variables are categorized. The corresponding CFI and TLI are less dependent on the categorization than RMSEA but are less sensitive to model misspecification in general. In contrast, the number of categories and threshold values do not impact the ULS unscaled fit indices in the population.     

Effect of non-normality on test statistics for one-way independent groups designs

The data obtained from one‐way independent groups designs is typically non‐normal in form and rarely equally variable across treatment populations (i.e. population variances are heterogeneous). Consequently, the classical test statistic that is used to assess statistical significance (i.e. the analysis of variance F test) typically provides invalid results (e.g. too many Type I errors, reduced power). For this reason, there has been considerable interest in finding a test statistic that is appropriate under conditions of non‐normality and variance heterogeneity. Previously recommended procedures for analysing such data include the James test, the Welch test applied either to the usual least squares estimators of central tendency and variability, or the Welch test with robust estimators (i.e. trimmed means and Winsorized variances). A new statistic proposed by Krishnamoorthy, Lu, and Mathew, intended to deal with heterogeneous variances, though not non‐normality, uses a parametric bootstrap procedure. In their investigation of the parametric bootstrap test, the authors examined its operating characteristics under limited conditions and did not compare it to the Welch test based on robust estimators. Thus, we investigated how the parametric bootstrap procedure and a modified parametric bootstrap procedure based on trimmed means perform relative to previously recommended procedures when data are non‐normal and heterogeneous. The results indicated that the tests based on trimmed means offer the best Type I error control and power when variances are unequal and at least some of the distribution shapes are non‐normal.     

Analysing multitrait–multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: What sample size is needed for valid results?

Convergent and discriminant validity of psychological constructs can best be examined in the framework of multitrait–multimethod (MTMM) analysis. To gain information at the level of single items, MTMM models for categorical variables have to be applied. The CTC(M?1) model is presented as an example of an MTMM model for ordinal variables. Based on an empirical application of the CTC(M?1) model, a complex simulation study was conducted to examine the sample size requirements of the robust weighted least squares mean‐ and variance‐adjusted χ² test of model fit (WLSMV estimator) implemented in Mplus. In particular, the simulation study analysed the χ² approximation, the parameter estimation bias, the standard error bias, and the reliability of the WLSMV estimator depending on the varying number of items per trait–method unit (ranging from 2 to 8) and varying sample sizes (250, 500, 750, and 1000 observations). The results showed that the WLSMV estimator provided a good – albeit slightly liberal – χ² approximation and stable and reliable parameter estimates for models of reasonable complexity (2–4 items) and small sample sizes (at least 250 observations). When more complex models with 5 or more items were analysed, larger sample sizes of at least 500 observations were needed. The most complex model with 9 trait–method units and 8 items (72 observed variables) requires sample sizes of at least 1000 observations.     

Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study

When multisource feedback instruments, for example, 360-degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non-standard multilevel structural equation model that incorporates self-ratings (level-2 variables) and others’ ratings from different additional perspectives (level-1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self-ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self-ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade-off effects between the sample sizes of self-ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ² test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level-specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.     

Comparing the accuracy of experimental estimates to guessing: a new perspective on replication and the “Crisis of Confidence” in psychology

We develop a general measure of estimation accuracy for fundamental research designs, called v. The v measure compares the estimation accuracy of the ubiquitous ordinary least squares (OLS) estimator, which includes sample means as a special case, with a benchmark estimator that randomizes the direction of treatment effects. For sample and effect sizes common to experimental psychology, v suggests that OLS produces estimates that are insufficiently accurate for the type of hypotheses being tested. We demonstrate how v can be used to determine sample sizes to obtain minimum acceptable estimation accuracy. Software for calculating v is included as online supplemental material (R Core Team, 2012).     

A study of algorithms for covariance structure analysis with specific comparisons using factor analysis

Several algorithms for covariance structure analysis are considered in addition to the Fletcher-Powell algorithm. These include the Gauss-Newton, Newton-Raphson, Fisher Scoring, and Fletcher-Reeves algorithms. Two methods of estimation are considered, maximum likelihood and weighted least squares. It is shown that the Gauss-Newton algorithm which in standard form produces weighted least squares estimates can, in iteratively reweighted form, produce maximum likelihood estimates as well. Previously unavailable standard error estimates to be used in conjunction with the Fletcher-Reeves algorithm are derived. Finally all the algorithms are applied to a number of maximum likelihood and weighted least squares factor analysis problems to compare the estimates and the standard errors produced. The algorithms appear to give satisfactory estimates but there are serious discrepancies in the standard errors. Because it is robust to poor starting values, converges rapidly and conveniently produces consistent standard errors for both maximum likelihood and weighted least squares problems, the Gauss-Newton algorithm represents an attractive alternative for at least some covariance structure analyses.Work by the first author has been supported in part by Grant No. Da01070 from the U. S. Public Health Service. Work by the second author has been supported in part by Grant No. MCS 77-02121 from the National Science Foundation.     

Two-step weighted least squares factor analysis of dichotomized variables

A two-step weighted least squares estimator for multiple factor analysis of dichotomized variables is discussed. The estimator is based on the first and second order joint probabilities. Asymptotic standard errors and a model test are obtained by applying the Jackknife procedure.     

More on least squares estimation of the transition matrix in a stationary first-order markov process from sample proportions data

Miller suggested ordinary least squares estimation of a constant transition matrix; Madansky proposed a relatively more efficient weighted least squares estimator which corrects for heteroscedasticity. In this paper an efficient generalized least squares estimator is derived which utilizes the entire covariance matrix of the distrubances. This estimator satisfies the condition that each row of the transition matrix must sum to unity. Madansky noted that estimates of the variances could be negative; a method for obtaining consistent non-negative estimates of the variances is suggested in this paper. The technique is applied to the hypothetical sample data used by Miller and Madansky.I am indebted to a referee for his thoughtful suggestions on content and style.     

A note on permutation tests of significance for multiple regression coefficients

In the vast majority of psychological research utilizing multiple regression analysis, asymptotic probability values are reported. This paper demonstrates that asymptotic estimates of standard errors provided by multiple regression are not always accurate. A resampling permutation procedure is used to estimate the standard errors. In some cases the results differ substantially from the traditional least squares regression estimates.     

Exploratory second-order analyses for components and factors

Abstract: Exploratory methods using second‐order components and second‐order common factors were proposed. The second‐order components were obtained from the resolution of the correlation matrix of obliquely rotated first‐order principal components. The standard errors of the estimates of the second‐order component loadings were derived from an augmented information matrix with restrictions for the loadings and associated parameters. The second‐order factor analysis proposed was similar to the classical method in that the factor correlations among the first‐order factors were further resolved by the exploratory method of factor analysis. However, in this paper the second‐order factor loadings were estimated by the generalized least squares using the asymptotic variance‐covariance matrix for the first‐order factor correlations. The asymptotic standard errors for the estimates of the second‐order factor loadings were also derived. A numerical example was presented with simulated results.     

A note on the sampling properties of the Vincentizing (quantile averaging) procedure

To assess the effect of a manipulation on a response time distribution, psychologists often use Vincentizing or quantile averaging to construct group or “average” distributions. We provide a theorem characterizing the large sample properties of the averaged quantiles when the individual RT distributions all belong to the same location-scale family. We then apply the theorem to estimating parameters for the quantile-averaged distributions. From the theorem, it is shown that parameters of the group distribution can be estimated by generalized least squares. This method provides accurate estimates of standard errors of parameters and can therefore be used in formal inference. The method is benchmarked in a small simulation study against both a maximum likelihood method and an ordinary least-squares method. Generalized least squares essentially is the only method based on the averaged quantiles that is both unbiased and provides accurate estimates of parameter standard errors. It is also proved that for location-scale families, performing generalized least squares on quantile averages is formally equivalent to averaging parameter estimates from generalized least squares performed on individuals. A limitation on the method is that individual RT distributions must be members of the same location-scale family.     

The Infinitesimal Jackknife with Exploratory Factor Analysis

The infinitesimal jackknife, a nonparametric method for estimating standard errors, has been used to obtain standard error estimates in covariance structure analysis. In this article, we adapt it for obtaining standard errors for rotated factor loadings and factor correlations in exploratory factor analysis with sample correlation matrices. Both maximum likelihood estimation and ordinary least squares estimation are considered.     

Standard Errors of Kernel Equating: Accounting for Bandwidth Estimation

In standardized testing, equating is used to ensure comparability of test scores across multiple test administrations. One equipercentile observed-score equating method is kernel equating, where an essential step is to obtain continuous approximations to the discrete score distributions by applying a kernel with a smoothing bandwidth parameter. When estimating the bandwidth, additional variability is introduced which is currently not accounted for when calculating the standard errors of equating. This poses a threat to the accuracy of the standard errors of equating. In this study, the asymptotic variance of the bandwidth parameter estimator is derived and a modified method for calculating the standard error of equating that accounts for the bandwidth estimation variability is introduced for the equivalent groups design. A simulation study is used to verify the derivations and confirm the accuracy of the modified method across several sample sizes and test lengths as compared to the existing method and the Monte Carlo standard error of equating estimates. The results show that the modified standard errors of equating are accurate under the considered conditions. Furthermore, the modified and the existing methods produce similar results which suggest that the bandwidth variability impact on the standard error of equating is minimal.     

A comparison of two‐stage procedures for testing least‐squares coefficients under heteroscedasticity

This study explores the performance of several two‐stage procedures for testing ordinary least‐squares (OLS) coefficients under heteroscedasticity. A test of the usual homoscedasticity assumption is carried out in the first stage of the procedure. Subsequently, a test of the regression coefficients is chosen and performed in the second stage. Three recently developed methods for detecting heteroscedasticity are examined. In addition, three heteroscedastic robust tests of OLS coefficients are considered. A major finding is that performing a test of heteroscedasticity prior to applying a heteroscedastic robust test can lead to poor control over Type I errors.     

On the asymptotic standard error of a class of robust estimators of ability in dichotomous item response models

In item response theory, the classical estimators of ability are highly sensitive to response disturbances and can return strongly biased estimates of the true underlying ability level. Robust methods were introduced to lessen the impact of such aberrant responses on the estimation process. The computation of asymptotic (i.e., large‐sample) standard errors (ASE) for these robust estimators, however, has not yet been fully considered. This paper focuses on a broad class of robust ability estimators, defined by an appropriate selection of the weight function and the residual measure, for which the ASE is derived from the theory of estimating equations. The maximum likelihood (ML) and the robust estimators, together with their estimated ASEs, are then compared in a simulation study by generating random guessing disturbances. It is concluded that both the estimators and their ASE perform similarly in the absence of random guessing, while the robust estimator and its estimated ASE are less biased and outperform their ML counterparts in the presence of random guessing with large impact on the item response process.     

The analysis of structural equation models by means of derivative free nonlinear least squares

It is shown that the PAR Derivative-Free Nonlinear Regression program in BMDP can be used to fit structural equation models, producing generalized least squares estimates, standard errors, and goodness-of-fit test statistics. Covariance structure models more general than LISREL can be analyzed. The approach is particularly useful for dealing with new non-standard models and experimenting with alternate methods of estimation. The research of the second author was supported by the NSF grant MCS 83-01587. We wish to thank our referees for some very valuable suggestions.