A note on using alpha and stratified alpha to estimate the reliability of a test composed of item parcels

Several authors have suggested that prior to conducting a confirmatory factor analysis it may be useful to group items into a smaller number of item ‘parcels’ or ‘testlets’. The present paper mathematically shows that coefficient alpha based on these parcel scores will only exceed alpha based on the entire set of items if W, the ratio of the average covariance of items between parcels to the average covariance of items within parcels, is greater than unity. If W is less than unity, however, and errors of measurement are uncorrelated, then stratified alpha will be a better lower bound to the reliability of a measure than the other two coefficients. Stratified alpha are also equal to the true reliability of a test when items within parcels are essentially tau‐equivalent if one assumes that errors of measurement are not correlated.     

Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds

Let Σ _x be the (population) dispersion matrix, assumed well-estimated, of a set of non-homogeneous item scores. Finding the greatest lower bound for the reliability of the total of these scores is shown to be equivalent to minimizing the trace of Σ _x by reducing the diagonal elements while keeping the matrix non-negative definite. Using this approach, Guttman's bounds are reviewed, a method is established to determine whether his λ₄ (maximum split-half coefficient alpha) is the greatest lower bound in any instance, and three new bounds are discussed. A geometric representation, which sheds light on many of the bounds, is described. Present affiliation of the second author: Department of Statistics, University of Nigeria (Nsukka Campus). Work on this paper was carried out while on study leave in Aberystwyth.     

Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: II: A search procedure to locate the greatest lower bound

Finding the greatest lower bound for the reliability of the total score on a test comprisingn non-homogenous items with dispersion matrix Σ _x is equivalent to maximizing the trace of a diagonal matrix Σ _E with elements θ _I , subject to Σ _E and Σ _T =Σ _x − Σ _E being non-negative definite. The casesn=2 andn=3 are solved explicity. A computer search in the space of the θ _i is developed for the general case. When Guttman's λ₄ (maximum split-half coefficient alpha) is not the g.l.b., the maximizing set of θ _i makes the rank of Σ _T less thann − 1. Numerical examples of various bounds are given. Present affiliation of the first author: St. Hild's College of Education, Durham City, England.     

A note on reliability estimation for a test with components of unknown functional lengths

This research note proposes two reliability coefficients for tests with components of unknown functional lengths. The derived coefficients are extensions of the techniques devised by Kristof and Feldt and do not require a reduction of test components into parts. Simulation study indicates that the new coefficients yield reasonably stable reliability estimates when the number of test components is small.     

Coefficient alpha and the reliability of composite measurements

Following a general approach due to Guttman, coefficientα is rederived as a lower bound on the reliability of a test. A necessary and sufficient condition under which equality is attained in this inequality and hence thatα is equal to the reliability of the test is derived and shown to be closely related to the recent redefinition of the concept of parallel measurements due to Novick. This condition is then also shown to be closely related to the unit rank assumption originally adopted by Kuder and Richardson in the derivation of their formula 20. The assumption later adopted by Jackson and Ferguson and the one adopted by Gulliksen are shown to be related to the necessary and sufficient condition derived here. It is then pointed out that the statement that “coefficientα is equal to the mean of the split-half reliabilities” is true only under the restricted condition assumed by Cronbach in the body of his derivation of this result. Finally some limitations on the uses of any function ofα as a measure of internal consistency are noted.     

A note on the standard length of a test

This paper describes a relationship between the variance-covariance matrix of test items and Woodbury's concept of the standard length of a test. An index of item-test relationship is described in standard length terms. The sum of these indices for the items in a test is equal to the square of Jackson's coefficient of sensitivity.     

A series of lower bounds to the reliability of a test

Two well-known lower bounds to the reliability in classical test theory, Guttman's ₂ and Cronbach's coefficient alpha, are shown to be terms of an infinite series of lower bounds. All terms of this series are equal to the reliability if and only if the test is composed of items which are essentially tau-equivalent. Some practical examples, comparing the first 7 terms of the series, are offered. It appears that the second term (₂) is generally worth-while computing as an improvement of the first term (alpha) whereas going beyond the second term is not worth the computational effort. Possibly an exception should be made for very short tests having widely spread absolute values of covariances between items. The relationship of the series and previous work on lower bound estimates for the reliability is briefly discussed.The authors are obliged to Henk Camstra for providing a computer program that was used in this study.     

A note on Tryon's measure of reliability

Two alternative formulas, based upon the analysis of variance, are given for Tryon's general form for the reliability coefficient.     

A note on ROC analysis and non-parametric estimate of sensitivity

In the signal detection paradigm, the non-parametric index of sensitivity A′, as first introduced by Pollack and Norman (1964), is a popular alternative to the more traditional d′ measure of sensitivity. Smith (1995) clarified a confusion about the interpretation of A′ in relation to the area beneath proper receiver operating characteristic (ROC) curves, and provided a formula (which he called A′′) for this commonly held interpretation. However, he made an error in his calculations. Here, we rectify this error by providing the correct formula (which we call A) and compare the discrepancy that would have resulted. The corresponding measure for bias b is also provided. Since all such calculations apply to “proper” ROC curves with non-decreasing slopes, we also prove, as a separate result, the slope-monotonicity of ROC curves generated by likelihood-ratio criterion.This revised article was published online in August 2005 with the PDF paginated correctly.     

On the reliability of a weighted composite

A general formula for the reliability of a weighted composite has been derived by which that reliability can be estimated from a knowledge of the weights whatever their source, reliabilities, dispersions, and intercorrelations of the components. The Spearman-Brown formula has been shown to be a special case of the general statement. The effect of the internal consistency or intercorrelation of the components has been investigated and the conditions defining the set of weights yielding maximum reliability shown to be that the weight of a component is proportional to the sum of its intercorrelations with the remaining components and inversely proportional to its error variance.     

A subset selection technique for scoring items on a multiple choice test

On a multiple-choice test in which each item hask alternative responses, the test taker is permitted to choose any subset which he believes contains the one correct answer. A scoring system is devised that depends on the size of the subset and on whether or not the correct answer is eliminated. The mean and variance of the score per item are obtained. Methods are derived for determining the total number of items that should be included on the test so that the average score on all items can be regarded as a good measure of the subject's knowledge. Efficiency comparisons between conventional and the subset selection scoring procedures are made. The analogous problem ofr > 1 correct answers for each item (withr fixed and known) is also considered.The authors are grateful to M. Aitkin, C. Coombs, F. Lord, and the reviewers for their comments and suggestions.     

A note on the sequential probability ratio test

Link and Heath [1975] have analysed a random walk model for two-choice reaction time on the assumption that the two probability density functions (p.d.f.s) of the step-size, each p.d.f. corresponding to one stimulus, are mirror reflections of each other; and they have demonstrated the critical role played by the symmetry of the moment generating function (m.g.f.) of the step size in the determination of whether or not error and correct reaction times are equal. It is shown here that, given reflection symmetry, m.g.f. symmetry is necessary and sufficient for the random walk model to be equivalent to a sequential probability ratio test.     

A note on the calculation of weights for maximum battery reliability

Weights may be determined for combining tests so that the composite has maximum reliability, but the calculation of these weights by means of the original equations is cumbersome. It is shown that the desired weighted composite is the first principal axis of a matrix closely related to the intercorrelation matrix. Thus, simple and straightforward procedures are available for calculating the weights.     

A unifying expression for the maximal reliability of a linear composite

A formally simple expression for the maximal reliability of a linear composite is provided, its theoretical implications and its relation to existing results for reliability are discussed.     

Sensitivity of a linear composite of predictor items to differential item weighting

Many authors have demonstrated for idealized item configurations that equal item weights are often virtually as good for a particular predictive purpose as the item weights that are theoretically optimal. What has not been heretofore clear, however, is what happens to the similarity between weighted and unweighted composites of the same items when the item configuration's variance structure is complex.