Note on two generalizations of coefficient alpha

An important relationship is given for two generalizations of coefficient alpha, Rajaratnam, Cronbach and Gleser's generalizability formula for stratified-parallel tests and Raju's coefficient beta.The author gratefully acknowledges the generous assistance given by reviewers and the editor.     

Asymptotically distribution-free (ADF) interval estimation of coefficient alpha

The point estimate of sample coefficient alpha may provide a misleading impression of the reliability of the test score. Because sample coefficient alpha is consistently biased downward, it is more likely to yield a misleading impression of poor reliability. The magnitude of the bias is greatest precisely when the variability of sample alpha is greatest (small population reliability and small sample size). Taking into account the variability of sample alpha with an interval estimator may lead to retaining reliable tests that would be otherwise rejected. Here, the authors performed simulation studies to investigate the behavior of asymptotically distribution-free (ADF) versus normal-theory interval estimators of coefficient alpha under varied conditions. Normal-theory intervals were found to be less accurate when item skewness >1 or excess kurtosis >1. For sample sizes over 100 observations, ADF intervals are preferable, regardless of item skewness and kurtosis. A formula for computing ADF confidence intervals for coefficient alpha for tests of any size is provided, along with its implementation as an SAS macro.     

On robusiness of the normal-theory based asymptotic distributions of three reliability coefficient estimates

This paper studies the asymptotic distributions of three reliability coefficient estimates: Sample coefficient alpha, the reliability estimate of a composite score following a factor analysis, and the estimate of the maximal reliability of a linear combination of item scores following a factor analysis. Results indicate that the asymptotic distribution for each of the coefficient estimates, obtained based on a normal sampling distribution, is still valid within a large class of nonnormal distributions. Therefore, a formula for calculating the standard error of the sample coefficient alpha, recently obtained by van Zyl, Neudecker and Nel, applies to other reliability coefficients and can still be used even with skewed and kurtotic data such as are typical in the social and behavioral sciences.This research was supported by grants DA01070 and DA00017 from the National Institute on Drug Abuse and a University of North Texas faculty research grant. We would like to thank the Associate Editor and two reviewers for suggestions that helped to improve the paper.     

Statistical approaches to achieving sufficiently high test score reliabilities for research purposes

The author provides statistical approaches to aid investigators in assuring that sufficiently high test score reliabilities are achieved for specific research purposes. The statistical approaches use tests of statistical significance between the obtained reliability and lowest population reliability that an investigator will tolerate. The statistical approaches work for coefficient alpha and related coefficients and for alternate-forms, split-half (2-part alpha), and retest reliabilities. The author shows that, in some instances, a formula can help to estimate the sample size necessary for the statistical test.     

A note on the relation between coefficient alpha and guttman's “split-half” lower bounds

Use of the same term split-half for division of ann-item test into two subtests containing equal [Cronbach], and possibly unequal [Guttman], numbers of items sometimes leads to a misunderstanding about the relation between Guttman's maximum split-half bound and Cronbach's coefficient alpha.Coefficient alpha is the average of split-half bounds in the Cronbach sense and so is not larger than the maximum split-half bound in either sense whenn is even. Whenn is odd, however, splithalf bounds exist only in the Guttman sense and the largest of these may be smaller than coefficient alpha.     

Estimating the reliability of a test split into two parts of equal or unequal length

When the reliability of test scores must be estimated by an internal consistency method, partition of the test into just 2 parts may be the only way to maintain content equivalence of the parts. If the parts are classically parallel, the Spearman-Brown formula may be validly used to estimate the reliability of total scores. If the parts differ in their standard deviations but are tau equivalent, Cronbach's alpha is appropriate. However, if the 2 parts are congeneric, that is, they are unequal in functional length or they comprise heterogeneous item types, a less well-known estimate, the Angoff-Feldt coefficient, is appropriate. Guidelines in terms of the ratio of standard deviations are proposed for choosing among Spearman-Brown, alpha, and Angoff-Feldt coefficients.     

Reliability if deleted,not ‘alpha if deleted’: Evaluation of scale reliability following component deletion

This paper demonstrates that the widely available and routinely used index ‘coefficient alpha if item deleted’ can be misleading in the process of construction and revision of multiple‐component instruments with congeneric measures. An alternative approach to evaluation of scale reliability following deletion of each component in a given composite is outlined that can be recommended in general for scale development purposes. The method provides ranges of plausible values for instrument reliability when dispensing with single components in a tentative composite, and permits testing hypotheses about reliability of resulting scale versions. The proposed procedure is illustrated with an example.     

A Meta-Analysis of the Reliability of Free and For-Pay Big Five Scales

The present study meta-analytically compared coefficient alpha reliabilities reported for free and for-pay Big Five scales. We collected 288 studies from five previous meta-analyses of Big Five traits and harvested 1,317 alphas from these studies. We found that free and for-pay scales measuring Big Five traits possessed comparable reliabilities. However, after we controlled for the numbers of items in the scales with the Spearman-Brown formula, we found that free scales possessed significantly higher alpha coefficients than for-pay scales for each of the Big Five traits. Thus, the study offers initial evidence that Big Five scales that are free more efficiently measure these traits for research purposes than do for-pay scales.     

Internal consistency of the object relations and social cognition scales for the Thematic Apperception Test.

We examined 8 data sets to determine whether it is possible to attain acceptable levels of internal consistency (coefficient alpha) reliability for the 4 Object Relations and Social Cognition scales (ORSC; Westen, Lohr, Silk, Kerber, & Goodrich, 1989) for the Thematic Apperception Test (TAT; Murray, 1943) when cards are considered as items in a scale. Number of cards used in the data sets ranged from 4 to 10, and the Spearman-Brown prophecy formula was applied to estimate the number of cards that would be required to attain alpha levels of different magnitudes. The two more structural subscales (Complexity of Representations and Understanding of Social Causality) have somewhat higher consistencies than the two more affective ones (Affect Tone and Capacity for Emotional Investment and Moral Standards). The results suggest that the use of 10 to 12 cards provides internal consistencies of alpha > or = .70 across each of the 4 ORSC scales.     

Alpha if item deleted: A note on loss of criterion validity in scale development if maximizing coefficient alpha

This note is concerned with a validity‐related limitation of the widely available and routinely used index ‘alpha if item deleted’ in the process of construction and development of multiple‐component measuring instruments. Attention is drawn to the fact that this statistic can suggest dispensing with such scale components, whose removal leads to loss of criterion validity while maximizing the popular coefficient alpha. As an alternative, a latent variable modelling approach is discussed that can be used for point and interval estimation of composite criterion validity (as well as reliability) after deletion of single components. The method can also be utilized to test conventional or minimum level hypotheses about associated population change in measurement quality indices.     

Questionnaire of executive function for dancers: an ecological approach

There is a current debate about the ecological validity of executive function (EF) tests. Consistent with the verisimilitude approach, this research proposes the ballet executive scale (BES), a self-rating questionnaire that assimilates idiosyncratic executive behaviors of classical dance community. The BES was administrated to 149 adolescents, students of the Cuban Ballet School. Results present a Cronbach's alpha coefficient of .80 and a split-half Spearman-Brown coefficient r (SB) = .81. An exploratory factor analysis describes a bifactorial pattern of EF dimensions, with a self-regulation component, which explains more than 40% of variance, and a Developmental component, which accounts for more than 20% of variance. The questionnaire's total scores fit linear regression models with two external criteria of academic records, confirming concurrent validity. These findings support the hypothesis that the internalization of specific contextual cultural meanings has a mediating influence in the development of EF.     

Meta-analysis of coefficient alpha

The meta-analysis of coefficient alpha across many studies is becoming more common in psychology by a methodology labeled reliability generalization. Existing reliability generalization studies have not used the sampling distribution of coefficient alpha for precision weighting and other common meta-analytic procedures. A framework is provided for a statistically grounded meta-analysis of coefficient alpha using its sampling distribution. Two empirical examples are offered to illustrate these methods, and limitations of reliability generalization are described.     

A coefficient alpha for test-retest data

Transient errors are caused by variations in feelings, moods, and mental states over time. If these errors are present, coefficient alpha is an inflated estimate of reliability. A true-score model is presented that incorporates transient errors for test-retest data, and a reliability estimate is derived. This estimate, referred to as the test-retest alpha, is less than coefficient alpha if transient error is present and is less susceptible to effects due to item recall than a test-retest correlation. An assumption underlying the test-retest alpha is essential tau equivalency of items. A test-retest split-half coefficient is presented as an alternative to the test-retest alpha when this assumption is violated. The test-retest alpha is the mean of all possible test-retest split-half coefficients.     

Resampling-Based Inference Methods for Comparing Two Coefficients Alpha

The two-sample problem for Cronbach’s coefficient \(\alpha _C\), as an estimate of test or composite score reliability, has attracted little attention compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistical procedures of comparing two coefficients \(\alpha _{C,1}\) and \(\alpha _{C,2}\). The null hypothesis of interest is \(H_0 : \alpha _{C,1} = \alpha _{C,2}\), which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed for two-group multivariate non-normal models under the general asymptotically distribution-free (ADF) setting. These statistical tests ensure a better control of the type-I error, in finite or very small sample sizes, when the state-of-affairs ADF large-sample test may fail to properly attain the nominal significance level. By proper choice of a studentized test statistic, the resampling tests are modified in order to be valid asymptotically even in non-exchangeable data frameworks. Moreover, extensions of this approach to other designs and reliability measures are discussed as well. Finally, the usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.     

Generalization of Scott's formula for retractions from generalized Alexandroff's cube

In the paper [2] the following theorem is shown: Theorem (Th. 3,5, [2]), If α=0 or δ=∞ or α?δ, then a closure space X is an absolute extensor for the category of 〈α, δ〉 -closure spaces iff a contraction of X is the closure space of all 〈α, δ〉-filters in an 〈α, δ〉-semidistributive lattice. In the case when α=ω and δ=∞, this theorem becomes Scott's theorem: Theorem ([7]). A topological space X is an absolute extensor for the category of all topological spaces iff a contraction of X is a topological space of “Scott's open sets” in a continuous lattice. On the other hand, when α=0 and δ=ω, this theorem becomes Jankowski's theorem: Theorem ([4]). A closure space X is an absolute extensor for the category of all closure spaces satisfying the compactness theorem iff a contraction of X is a closure space of all filters in a complete Heyting lattice. But for separate cases of α and δ, the Theorem 3.5 from [2] is proved using essentialy different methods. In this paper it is shown that this theorem can be proved using, for retraction, one uniform formula. Namely it is proved that if α= 0 or δ= ∞ or α ? δ and \(F_{\alpha ,\delta } \left( L \right) \subseteq B_{\alpha ,\delta }^\mathfrak{n} \) and if L is an 〈α, δ〉-semidistributive lattice, then the function $$r:{\text{ }}B_{\alpha ,\delta }^\mathfrak{n} \to F_{\alpha ,\delta } \left( L \right)$$ such that for x ε ? ( \(\mathfrak{n}\) ): (*) $$r\left( x \right) = inf_L \left\{ {l \in L|\left( {\forall A \subseteq L} \right)x \in C\left( A \right) \Rightarrow l \in C\left( A \right)} \right\}$$ defines retraction, where C is a proper closure operator for \(B_{\alpha ,\delta }^\mathfrak{n} \) . It is also proved that the formula (*) defines retraction for all 〈α, δ〉, whenever L is an 〈α, δ〉 -pseudodistributive lattice. Moreover it is proved that when α=ω and δ=∞, the formula (*) defines identical retraction to the formula given in [7], and when α = 0 and δ=ω, the formula (*) defines identical retraction to the formula given in [4].     

From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation

Coefficient alpha is the most popular measure of reliability (and certainly of internal consistency reliability) reported in psychological research. This is noteworthy given the numerous deficiencies of coefficient alpha documented in the psychometric literature. This mismatch between theory and practice appears to arise partly because users of psychological scales are unfamiliar with the psychometric literature on coefficient alpha and partly because alternatives to alpha are not widely known. We present a brief review of the psychometric literature on coefficient alpha, followed by a practical alternative in the form of coefficient omega. To facilitate the shift from alpha to omega, we also present a brief guide to the calculation of point and interval estimates of omega using a free, open source software environment.     

Some cautionary notes regarding the use of aggregated scores as a measure of behavioral stability

The utility of the method of aggregation as a measure of behavioral consistency was examined in 26 studies involving computer-generated, repeated-measurement data. The first series of studies involved rectangular distributions in which score constant. A second series of studies used normally distributed z scores, and score consistency was manipulated by inducing a desired correlation between the scores in adjacent trials. In both sets of studies, the aggregate stability coefficient was a strictly increasing function of the number of aggregated trials, and even trivial amounts of score stability resulted in large stability coefficients. In a third series of studies, high stability coefficients occurred when computed on combined unstable subsamples which differed from each other only in central tendency. Terminal aggregate coefficients were compared with Spearman-Brown prophecy and Cronbach's alpha reliability coefficients computed on the experimental data. It was concluded that the method of aggregation produces spuriously high estimates of behavioral consistency. It was further shown that the Spearman-Brown prophecy formula and coefficient alpha accurately predict the results of the aggregation method, suggesting that aggregation is an internal consistency reliability procedure. The equating of stability with traditional notions of reliability was questioned.     

Internal consistency of tests: Analyses old and new

A coefficient derived from communalities of test parts has been proposed as greatest lower bound to Guttman's immediate retest reliability. The communalities have at times been calculated from covariances between itemsets, which tends to underestimate appreciably. When items are experimentally independent, a consistent estimate of the greatest defensible internal-consistency coefficient is obtained by factoring item covariances. In samples of modest size, this analysis capitalizes on chance; an estimate subject to less upward bias is suggested. For estimating alternate-forms reliability, communality-based coefficients are less appropriate than stratified alpha.I thank Edward Haertel for comments and suggestions, and Andrew Comrey for data.     

The reliability of the Thematic Apperception Test

Controversy over the TAT's reliability may stem largely from the mis-application of traditional psychometric measures, which are inappropriate to this test. The TAT is implicitly based on a multiple regression model, for which coefficient alpha is not appropriate. Also, test-retest correlations may be adversely affected by the standard instructions to write a "creative" story. In a test-retest study, 47 high school students retook the TAT after a year with instructions designed to break the implicit set to produce a new and different story from that previously written. The test-retest correlations were r = .48 (need for affiliation) and .56 (need for intimacy), or approximately the same as those for, e.g., the MMPI, 16PF, and CPI, It was demonstrated that this high stability over time was not due to subjects' recalling and repeating previous responses. Finally, it was shown that alpha considerably underestimated the test-retest reliability, contrary to assumptions of classical psychometrics.