Constructing Confidence Intervals for Effect Size Measures of an Indirect Effect

Confidence intervals for an effect size can provide the information about the magnitude of an effect and its precision as well as the binary decision about the existence of an effect. In this study, the performances of five different methods for constructing confidence intervals for ratio effect size measures of an indirect effect were compared in terms of power, coverage rates, Type I error rates, and widths of confidence intervals. The five methods include the percentile bootstrap method, the bias-corrected and accelerated (BCa) bootstrap method, the delta method, the Fieller method, and the Monte Carlo method. The results were discussed with respect to the adequacy of the distributional assumptions and the nature of ratio quantity. The confidence intervals from the five methods showed similar results for samples of more than 500, whereas, for samples of less than 500, the confidence intervals were sufficiently narrow to convey the information about the population effect sizes only when the effect sizes of regression coefficients defining the indirect effect are large.     

Sample Sizes for Two-Group Second-Order Latent Growth Curve Models

Second-order latent growth curve models (S. C. Duncan &; Duncan, 1996 Duncan, S. C. and Duncan, T. E. 1996. A multivariate growth curve analysis of adolescent substance use.. Structural Equation Modeling, 3: 323–347. [Taylor &; Francis Online], [Web of Science ®] , [Google Scholar]; McArdle, 1988 McArdle, J. J. 1988. “Dynamic but structural equation modeling of repeated measures data.”. In Handbook of multivariate experimental psychology, , 2nd ed. Edited by: Cattell, R. B. and Nesselroade, J. 564–614. New York: Plenum.. [Crossref] , [Google Scholar]) can be used to study group differences in change in latent constructs. We give exact formulas for the covariance matrix of the parameter estimates and an algebraic expression for the estimation of slope differences. Formulas for calculations of the required sample size are presented, illustrated, and discussed. They are checked by Monte Carlo simulations in Mplus and also by Satorra and Saris's (1985) Satorra, A. and Saris, W. E. 1985. The power of the likelihood ratio test in covariance structure analysis.. Psychometrika, 50: 83–90. [Crossref], [Web of Science ®] , [Google Scholar] power approximation techniques for small and medium effect sizes (Cohen, 1988 Cohen, J. 1988. Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum..  [Google Scholar]). Results are similar across methods. Not surprisingly, sample sizes decrease with effect sizes, indicator reliabilities, number of indicators, frequency of observation, and duration of study. The relative importance of these factors is also discussed, alone and in combination. The use of the sample size formula is illustrated using a modification of empirical results from Stoel, Peetsma, and Roeleveld (2003) Stoel, R. D., Peetsma, T. T. and Roeleveld, J. 2003. Relations between the development of school investment, self-confidence, and language achievement in elementary education: A multivariate latent growth curve approach.. Learning and Individual Differences, 13: 313–333. [Crossref], [Web of Science ®] , [Google Scholar].     

Confidence Intervals for the Probability of Superiority Effect Size Measure and the Area Under a Receiver Operating Characteristic Curve

It is good scientific practice to the report an appropriate estimate of effect size and a confidence interval (CI) to indicate the precision with which a population effect was estimated. For comparisons of 2 independent groups, a probability-based effect size estimator (A) that is equal to the area under a receiver operating characteristic curve and closely related to the popular Wilcoxon-Mann-Whitney nonparametric statistical tests has many appealing properties (e.g., easy to understand, robust to violations of parametric assumptions, insensitive to outliers). We performed a simulation study to compare 9 analytic and 3 empirical (bootstrap) methods for constructing a CI for A that can yield very different CIs for the same data. The experimental design crossed 6 factors to yield a total of 324 cells representing challenging but realistic data conditions. Results were examined using several criteria, with emphasis placed on the extent to which observed CI coverage probabilities approximated nominal levels. Based on the simulation study results, the bias-corrected and accelerated bootstrap method is recommended for constructing a CI for the A statistic; bootstrap methods also provided the least biased and most accurate standard error of A. An empirical illustration examining score differences on a citation-based index of scholarly impact across faculty at low-ranked versus high-ranked research universities underscores the importance of choosing an appropriate CI method.     

A Comparative Investigation of Confidence Intervals for IndependentVariables in Linear Regression

In linear regression, the most appropriate standardized effect size for individual independent variables having an arbitrary metric remains open to debate, despite researchers typically reporting a standardized regression coefficient. Alternative standardized measures include the semipartial correlation, the improvement in the squared multiple correlation, and the squared partial correlation. No arguments based on either theoretical or statistical grounds for preferring one of these standardized measures have been mounted in the literature. Using a Monte Carlo simulation, the performance of interval estimators for these effect-size measures was compared in a 5-way factorial design. Formal statistical design methods assessed both the accuracy and robustness of the four interval estimators. The coverage probability of a large-sample confidence interval for the semipartial correlation coefficient derived from Aloe and Becker was highly accurate and robust in 98% of instances. It was better in small samples than the Yuan-Chan large-sample confidence interval for a standardized regression coefficient. It was also consistently better than both a bootstrap confidence interval for the improvement in the squared multiple correlation and a noncentral interval for the squared partial correlation.     

Sample Size Planning for the Squared Multiple Correlation Coefficient: Accuracy in Parameter Estimation via Narrow Confidence Intervals

Methods of sample size planning are developed from the accuracy in parameter approach in the multiple regression context in order to obtain a sufficiently narrow confidence interval for the population squared multiple correlation coefficient when regressors are random. Approximate and exact methods are developed that provide necessary sample size so that the expected width of the confidence interval will be sufficiently narrow. Modifications of these methods are then developed so that necessary sample size will lead to sufficiently narrow confidence intervals with no less than some desired degree of assurance. Computer routines have been developed and are included within the MBESS R package so that the methods discussed in the article can be implemented. The methods and computer routines are demonstrated using an empirical example linking innovation in the health services industry with previous innovation, personality factors, and group climate characteristics.     

单维测验合成信度三种区间估计的比较

已有许多研究建议使用合成信度来估计测验信度, 并报告其置信区间。有三种方法或途径可以计算单维测验合成信度的置信区间, 包括Bootstrap法、Delta法和直接用统计软件(如LISREL)输出的标准误进行计算。本文通过模拟研究进行比较, 发现Delta法与Bootstrap法得到的置信区间相当接近, 但用LISREL输出的标准误计算的与Bootstrap法得到的结果相差很大。推荐用Delta法估计合成信度的置信区间(使用Mplus容易实现), 但不能直接用LISREL输出的标准误来计算。举例说明了如何计算单维测验的合成信度以及用Delta法计算其置信区间。     

Confidence in Religious Leaders in Korea: A Research Note

Korea may provide an important testing ground for assessing religious growth as a correlate of religious authority. In Korea from 1985 to 1995, all religious groups experienced growth, but from 1995 to 2005 only the Catholic population did so. Favorable images of Korean Catholicism compared to other Korean religions point to one factor that may account for this trend, namely, confidence in religious leaders. Up to now there has been no empirical test measuring confidence in religious leaders among different religious groups in Korea. Using the 2003–2007 Korean General Social Surveys cumulative data, we found a hierarchy of confidence in religious leaders ranging from highest to lowest as follows: Catholics, Protestants, Buddhists, no religion. Our finding may suggest the continued vitality of Catholicism in Korea.     

Comparing Latent Means Without Mean Structure Models: A Projection-Based Approach

The conventional setup for multi-group structural equation modeling requires a stringent condition of cross-group equality of intercepts before mean comparison with latent variables can be conducted. This article proposes a new setup that allows mean comparison without the need to estimate any mean structural model. By projecting the observed sample means onto the space of the common scores and the space orthogonal to that of the common scores, the new setup allows identifying and estimating the means of the common and specific factors, although, without replicate measures, variances of specific factors cannot be distinguished from those of measurement errors. Under the new setup, testing cross-group mean differences of the common scores is done independently from that of the specific factors. Such independent testing eliminates the requirement for cross-group equality of intercepts by the conventional setup in order to test cross-group equality of means of latent variables using chi-square-difference statistics. The most appealing piece of the new setup is a validity index for mean differences, defined as the percentage of the sum of the squared observed mean differences that is due to that of the mean differences of the common scores. By analyzing real data with two groups, the new setup is shown to offer more information than what is obtained under the conventional setup.     

Abstract: Sample Size Planning for Latent Curve Models

When designing a study that uses structural equation modeling (SEM), an important task is to decide an appropriate sample size. Historically, this task is approached from the power analytic perspective, where the goal is to obtain sufficient power to reject a false null hypothesis. However, hypothesis testing only tells if a population effect is zero and fails to address the question about the population effect size. Moreover, significance tests in the SEM context often reject the null hypothesis too easily, and therefore the problem in practice is having too much power instead of not enough power.

An alternative means to infer the population effect is forming confidence intervals (CIs). A CI is more informative than hypothesis testing because a CI provides a range of plausible values for the population effect size of interest. Given the close relationship between CI and sample size, the sample size for an SEM study can be planned with the goal to obtain sufficiently narrow CIs for the population model parameters of interest.

Latent curve models (LCMs) is an application of SEM with mean structure to studying change over time. The sample size planning method for LCM from the CI perspective is based on maximum likelihood and expected information matrix. Given a sample, to form a CI for the model parameter of interest in LCM, it requires the sample covariance matrix S, sample mean vector , and sample size N. Therefore, the width (w) of the resulting CI can be considered a function of S, , and N. Inverting the CI formation process gives the sample size planning process. The inverted process requires a proxy for the population covariance matrix Σ, population mean vector μ, and the desired width ω as input, and it returns N as output. The specification of the input information for sample size planning needs to be performed based on a systematic literature review. In the context of covariance structure analysis, Lai and Kelley (2011) discussed several practical methods to facilitate specifying Σ and ω for the sample size planning procedure.     

Optimal and Most Exact Confidence Intervals for Person Parameters in Item Response Theory Models

The common way to calculate confidence intervals for item response theory models is to assume that the standardized maximum likelihood estimator for the person parameter θ is normally distributed. However, this approximation is often inadequate for short and medium test lengths. As a result, the coverage probabilities fall below the given level of significance in many cases; and, therefore, the corresponding intervals are no longer confidence intervals in terms of the actual definition. In the present work, confidence intervals are defined more precisely by utilizing the relationship between confidence intervals and hypothesis testing. Two approaches to confidence interval construction are explored that are optimal with respect to criteria of smallness and consistency with the standard approach.     

Bootstrap Confidence Intervals for Ordinary Least Squares Factor Loadings and Correlations in Exploratory Factor Analysis

This article is concerned with using the bootstrap to assign confidence intervals for rotated factor loadings and factor correlations in ordinary least squares exploratory factor analysis. Coverage performances of SE-based intervals, percentile intervals, bias-corrected percentile intervals, bias-corrected accelerated percentile intervals, and hybrid intervals are explored using simulation studies involving different sample sizes, perfect and imperfect models, and normal and elliptical data. The bootstrap confidence intervals are also illustrated using a personality data set of 537 Chinese men. The results suggest that the bootstrap is an effective method for assigning confidence intervals at moderately large sample sizes.     

A Direct Method for Obtaining Approximate Standard Error and Confidence Interval of Maximal Reliability for Composites With Congeneric Measures

Unlike a substantial part of reliability literature in the past, this article is concerned with weighted combinations of a given set of congeneric measures with uncorrelated errors. The relationship between maximal coefficient alpha and maximal reliability for such composites is initially dealt with, and it is shown that the former is a lower bound of the latter. A direct method for obtaining approximate standard error and confidence interval for maximal reliability is then outlined. The procedure is based on a second-order Taylor series approximation and is readily and widely applicable in empirical research via use of covariance structure modeling. The described method is illustrated with a numerical example.     

A Note on the Age of Marriage and Size of Family of two Comparable Vocational Groups

Summary

In a 4 × 2 design the effects of utility and attitudinal-supportiveness of information on message selection were tested. From data obtained in the pretest session, subjects were assigned to one of four Prior Attitude levels (Strongly Anti, Anti, Pro, Strongly Pro), and required to make either a Proabortion or an Antiabortion speech. Statements for and against legalized abortion were reproduced on slides which subjects selected and viewed to prepare their speeches. Unobtrusive measures of the proportion of Proabortion slides viewed and the proportion of time spent viewing Proabortion slides constituted the major dependent variables of the study. A main effect of information utility was hypothesized, and this expectation was strongly supported (p < .001). Within each speech condition subjects were expected to prefer attitudinally supportive information. This hypothesized additivity of supportive and useful information was evident (p < .002) for subjects assigned to make Proabortion speeches, but insignificant differences were obtained for subjects assigned Antiabortion speeches.     

Self-Report Measures of Individual Differences in Regulatory Focus: A Cautionary Note

Regulatory focus theory distinguishes between two independent structures of strategic inclination, promotion versus prevention. However, the theory implies two potentially independent definitions of these inclinations, the self-guide versus the reference-point definitions. Two scales (the Regulatory Focus Questionnaire, Higgins al., 2001, and the General Regulatory Focus Measure, Lockwood, Jordan, & Kunda, 2002) have been widely used to measure dispositional regulatory focus. We suggest that these two scales align respectively with the two definitions, and find that the two scales are largely uncorrelated. Both conceptual and methodological implications are discussed.     

Confidence Bounds and Power for the Reliability of Observational Measures on the Quality of a Social Setting

Social scientists are frequently interested in assessing the qualities of social settings such as classrooms, schools, neighborhoods, or day care centers. The most common procedure requires observers to rate social interactions within these settings on multiple items and then to combine the item responses to obtain a summary measure of setting quality. A key aspect of the quality of such a summary measure is its reliability. In this paper we derive a confidence interval for reliability, a test for the hypothesis that the reliability meets a minimum standard, and the power of this test against alternative hypotheses. Next, we consider the problem of using data from a preliminary field study of the measurement procedure to inform the design of a later study that will test substantive hypotheses about the correlates of setting quality. The preliminary study is typically called the ??generalizability study?? or ??G study?? while the later, substantive study is called the ??decision study?? or ??D study.?? We show how to use data from the G study to estimate reliability, a confidence interval for the reliability, and the power of tests for the reliability of measurement produced under alternative designs for the D study. We conclude with a discussion of sample size requirements for G studies.