Coefficients alpha and reliabilities of unrotated and rotated components

It has been shown by Kaiser that the sum of coefficients alpha of a set of principal components does not change when the components are transformed by an orthogonal rotation. In this paper, Kaiser's result is generalized. First, the invariance property is shown to hold for any set of orthogonal components. Next, a similar invariance property is derived for the reliability of any set of components. Both generalizations are established by considering simultaneously optimal weights for components with maximum alpha and with maximum reliability, respectively. A short-cut formula is offered to evaluate the coefficients alpha for orthogonally rotated principal components from rotation weights and eigenvalues of the correlation matrix. Finally, the greatest lower bound to reliability and a weighted version are discussed.Comments by Henk A.L. Kiers and by anonymous referees are gratefully acknowledged.     

Coefficients of feasibility for ordinal response scales

Cronbach's alpha is systematically used in the social sciences to estimate internal consistency. However, this coefficient assumes the continuity of the variables and this assumption is not met by ordinal response items or Likert scales. This work shows two alternatives to the alpha coefficient, the ordinal alpha and beta coefficients. Both of them take into account the ordinal nature of the data. The new coefficients were applied to several scales and compared with the traditional estimation of alpha. This paper also shows how to estimate those coefficients.     

Resampling-Based Inference Methods for Comparing Two Coefficients Alpha

The two-sample problem for Cronbach’s coefficient \(\alpha _C\), as an estimate of test or composite score reliability, has attracted little attention compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistical procedures of comparing two coefficients \(\alpha _{C,1}\) and \(\alpha _{C,2}\). The null hypothesis of interest is \(H_0 : \alpha _{C,1} = \alpha _{C,2}\), which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed for two-group multivariate non-normal models under the general asymptotically distribution-free (ADF) setting. These statistical tests ensure a better control of the type-I error, in finite or very small sample sizes, when the state-of-affairs ADF large-sample test may fail to properly attain the nominal significance level. By proper choice of a studentized test statistic, the resampling tests are modified in order to be valid asymptotically even in non-exchangeable data frameworks. Moreover, extensions of this approach to other designs and reliability measures are discussed as well. Finally, the usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.     

A Note on the Reliability Coefficients for Item Response Model-Based Ability Estimates

Assuming item parameters on a test are known constants, the reliability coefficient for item response theory (IRT) ability estimates is defined for a population of examinees in two different ways: as (a) the product-moment correlation between ability estimates on two parallel forms of a test and (b) the squared correlation between the true abilities and estimates. Due to the bias of IRT ability estimates, the parallel-forms reliability coefficient is not generally equal to the squared-correlation reliability coefficient. It is shown algebraically that the parallel-forms reliability coefficient is expected to be greater than the squared-correlation reliability coefficient, but the difference would be negligible in a practical sense.     

An Alternative Interpretation of the Linearly Weighted Kappa Coefficients for Ordinal Data

When two (or more) observers are independently categorizing a set of observations, Cohen’s kappa has become the most notable measure of interobserver agreement. When the categories are ordinal, a weighted form of kappa becomes desirable. The two most popular weighting schemes are the quadratic weights and linear weights. Quadratic weights have been justified by the fact that the corresponding weighted kappa is asymptotically equivalent to an intraclass correlation coefficient. This paper deals with linear weights and shows that the corresponding weighted kappa is equivalent to the unweighted kappa when cumulative probabilities are substituted for probabilities. A numerical example is provided.     

A coefficient alpha for test-retest data

Transient errors are caused by variations in feelings, moods, and mental states over time. If these errors are present, coefficient alpha is an inflated estimate of reliability. A true-score model is presented that incorporates transient errors for test-retest data, and a reliability estimate is derived. This estimate, referred to as the test-retest alpha, is less than coefficient alpha if transient error is present and is less susceptible to effects due to item recall than a test-retest correlation. An assumption underlying the test-retest alpha is essential tau equivalency of items. A test-retest split-half coefficient is presented as an alternative to the test-retest alpha when this assumption is violated. The test-retest alpha is the mean of all possible test-retest split-half coefficients.     

An on-line integrator for alpha quantification

A system for the on-line graphic quantification of alpha frequencies (or other frequency bands extracted from EEG signals) is described. Alpha-frequency components (8 to 13 Hz) are actively filtered from complex EEG waveforms and presented to a laboratory-developed solid state integrator. The high-level output of the integrator, which is compatible with all popular EEG and polygraph recorders, is in the form of an ascending analog voltage ramp, reset at a predetermined level, to provide a rapid quantitative determination of alpha activity and variability. The design of the integrator permits an alternate mode of operation where time-synchronized resets, in lieu of voltage-dependent resets, produce variable height voltage ramps, whose amplitude is a direct function of the accumulated alpha activity per unit time. This method of preprocessing and signal conditioning the alpha components of the EEG waveform provides a simple analog registry of on-line real-time correlation of alpha activity with various physiological and behavioral activity.     

Evidence for two components of object-based selection

A wealth of research has shown that observers can bias visual processing toward specific locations, but the role of object-based selection is less clear. In support of object-based selection, previous research has shown that when two objects are presented simultaneously, observers are better at reporting two attributes from one of the objects than one attribute from each object. However, there has been controversy over whether this effect is best explained by object-based selection or spatial selection. Our work suggests that there are two separate components of selection in this task: (a) a spatial component that is observed when the relevant targets are cued for observers before the onset of the stimulus display and (b) an object-based component that can still be observed when the first component has been eliminated. The latter effect replicates the initial evidence in favor of object-based selection, and can be demonstrated even when the relevant targets are cued after the offset of the target stimuli.     

Hemisphere differences for components of mental rotation

The hemispheric functional lateralization of components of mental rotation performance was investigated. Twenty right-handed males were presented with rotated alphanumerics and unfamiliar characters in the left or right visual field. Subjects decided if the laterally presented stimulus was identical to or a mirror image of a center standard stimulus. Reaction time and errors were measured. Previous mental rotation findings were replicated and the visual field variable produced significant effects for both dependent measures. An overall right visual field advantage was observed in the latency data, suggesting a left hemisphere superiority for at least one component process of the task. A significant interaction in the error data showed that alphanumerics produced less errors in the right visual field than in the left visual field, consistent with a left hemisphere superiority for processing verbal symbolic material. No such hemispheric difference in accuracy was found for unfamiliar characters.     

Evidence for several components of word familiarity

Familiarity with a word can be divided into two main components: familiarity with the form of the word (due to both its lexicality and its specific form) and familiarity with its meaning. In this study, ratings of familiarity were compared for words whose meaning was unknown to participants (UM words), for words of known meaning (KM words), and for unknown words (U words). Linguistic and experiential frequencies were equivalent. Rated familiarity was lower for UM than KM words and even lower for U words. Next, we built pseudowords from these stimuli by changing one letter and submitted them to two familiarity rating tasks that differed in the nature of the additional stimuli: either only nonwords or nonwords plus words. It was assumed that familiarity ratings would be lower for pseudowords built from UM words than for pseudowords built from KM words. The data were consistent with this assumption, and ratings depended on the initial categories of stimuli. These results support the view that usual word familiarity has two components, familiarity with form and familiarity with meaning, and a double source, processing of word form and processing of word meaning. The full set of these materials and norms may be downloaded from www.psychonomic.org/archive.     

Exploratory second-order analyses for components and factors

Abstract: Exploratory methods using second‐order components and second‐order common factors were proposed. The second‐order components were obtained from the resolution of the correlation matrix of obliquely rotated first‐order principal components. The standard errors of the estimates of the second‐order component loadings were derived from an augmented information matrix with restrictions for the loadings and associated parameters. The second‐order factor analysis proposed was similar to the classical method in that the factor correlations among the first‐order factors were further resolved by the exploratory method of factor analysis. However, in this paper the second‐order factor loadings were estimated by the generalized least squares using the asymptotic variance‐covariance matrix for the first‐order factor correlations. The asymptotic standard errors for the estimates of the second‐order factor loadings were also derived. A numerical example was presented with simulated results.     

A New Interpretation of the Weighted Kappa Coefficients

Reliability and agreement studies are of paramount importance. They do contribute to the quality of studies by providing information about the amount of error inherent to any diagnosis, score or measurement. Guidelines for reporting reliability and agreement studies were recently provided. While the use of the kappa-like family is advised for categorical and ordinal scales, no further guideline in the choice of a weighting scheme is given. In the present paper, a new simple and practical interpretation of the linear- and quadratic-weighted kappa coefficients is given. This will help researchers in motivating their choice of a weighting scheme.     

A loss function for alpha factor analysis

Most of the factor solutions can be got by minimizing a corresponding loss function. However, up to now, a loss function for the alpha factor analysis (AFA) has not been known. The present paper establishes such a loss function for the AFA. Some analogies to the maximum likelihood factor analysis are discussed.The author is greatly indebted to Prof. Henry F. Kaiser (University of California, Berkeley) for his kind encouragement. He is also indebted to an anonymous referee ofPsychometrika for having confronted him with the problem in 1977. Financial support by the Wiener Hochschuljubiläumsstiftung is gratefully acknowledged.     

Biases and Standard Errors of Standardized Regression Coefficients

The paper obtains consistent standard errors (SE) and biases of order O(1/n) for the sample standardized regression coefficients with both random and given predictors. Analytical results indicate that the formulas for SEs given in popular text books are consistent only when the population value of the regression coefficient is zero. The sample standardized regression coefficients are also biased in general, although it should not be a concern in practice when the sample size is not too small. Monte Carlo results imply that, for both standardized and unstandardized sample regression coefficients, SE estimates based on asymptotics tend to under-predict the empirical ones at smaller sample sizes.     

Estimating Group-Level Relationships: General Recommendations and Considerations for the Use of Intraclass Correlation Coefficients

Purpose

In this investigation, we argue for why and how available intraclass correlation coefficients and other types of reliability estimates can be employed as sample-based reliability estimates within primary and meta-analytic studies when relationships between group-level phenomena are of interest.

Design/Methodology/Approach

Group-level correlations and reliability estimates were obtained from 46 studies examining organizational climate–performance relationships. We illustrate how the group-level reliability estimates can be used to correct correlations for predictor and criterion unreliability. Procedures are presented for computing the sampling variances of individually corrected correlations that account for sampling error in the group-level reliability estimates.

Findings

Support was found for the conservative nature of meta-analytic parameter estimates when group-level reliability information is sample-based as opposed to assumed population values. In addition, our analyses indicated that conclusions about substantive relationships between group-level variables can change based on availability of sample-based reliabilities within both primary and meta-analytic studies.

Implications

Results from this study suggest that researchers should rely on sample-based meta-analytic procedures when examining the generalizability of group-level relationships. This study also demonstrates the importance of using all available reliability information and accounting for sampling error in the reliability estimates when conducting meta-analyses at the group level of analysis.

Originality/Value

This study breaks ground by systematically examining the use of intraclass correlation coefficients as reliability estimates within group-level meta-analytic studies. Furthermore, illustrative analyses provide guidance to primary and meta-analytic researchers in regard to how to correct group-level correlations for unreliability in the predictor, criterion, or both whenever and in whatever proportions the artifact information is available.

     

Further evidence for sublexical components in implicit memory for novel words

This study provides further evidence for the notion, proposed by Dorfman (1994), that activation of sublexical components plays an important role in implicit memory for novel words. Priming in an identification task was observed consistently when nonwords were formed out of morphemes; weaker, more variable effects were observed when nonwords were composed of syllables or pseudosyllables. Priming was insensitive to changes in the surface features (type case) of the stimuli and was disrupted by rearrangement of the stimulus components. Results are interpreted in terms of the activation and integration of preexisting sublexical representations and as evidence against the role of newly established episodic or perceptual representations. It is concluded that findings of priming for novel information should not be taken as evidence against an activation view.     

An Investigation of the Accuracy of Standardized Path Coefficients

The present study investigated potential sex differences in preferences for spicy, hot or unusual foods and in food aversions or neophobia. This study also reexamined the issue of a gender difference in preference for sweet foods. Questionnaires concerning past and current food use and preferences as well as food and condiment use in one actual meal were completed by 148 people between 17 and 32 years of age. Their responses clearly support the prediction that men tend to have a stronger preference than women for spicy, hot foods. The results also support the prediction that men are more likely than women to seek unusual and new foods. Both sexes showed the same degree of preference for condiment use.     

A k-sample significance test for independent alpha coefficients

The earlier two-sample procedure of Feldt [1969] for comparing independent alpha reliability coefficients is extended to the case ofK 2 independent samples. Details of a normalization of the statistic under consideration are presented, leading to computational procedures for the overallK-group significance test and accompanying multiple comparisons. Results based on computer simulation methods are presented, demonstrating that the procedures control Type I error adequately. The results of a power comparison of the case ofK=2 with Feldt's [1969]F test are also presented. The differences in power were negligible. Some final observations, along with suggestions for further research, are noted.The authors gratefully acknowledge the assistance of Michael E. Masson, in the computations performed, and of Leonard S. Feldt, in suggesting the data generation procedures used in the study. In addition, the authors thank James Zidek and the Institute of Applied Mathematics and Statistics, University of British Columbia, for advice concerning some of the theoretical development.