Alternate methods and software for calculating interobserver agreement for continuous observation data

Portable electronic data collection devices permit investigators to collect large amounts of observational data in a form ready for computer analysis. These devices are particularly efficient for gathering continuous data on multiple behavior categories. We expect that the increasing availability of these devices will lead to greater use of continuous data collection methods in observational research. This paper addresses the difficulties encountered when calculating traditional interobserver agreement statistics for continuous, multiple-code scoring. Two alternative strategies are described that yield interobserver agreement values based on the exact time of behavior code entries by the primary and secondary observers.Work on this paper was supported in part by NICHD Grants P01HD15051 and R01HD17650 and Office of Special Education and Rehabilitation Services Grant G008302980.     

On the reliability of the extreme score

Under the assumption of normality, a formula is derived for the reliability of the maximum score. It is shown that the maximum score is more reliable than each of the single observations, but less reliable than their composite score.     

Tests for equality of several alpha coefficients when their sample estimates are dependent

In a variety of measurement situations, the researcher may wish to compare the reliabilities of several instruments administered to the same sample of subjects. This paper presents eleven statistical procedures which test the equality ofm coefficient alphas when the sample alpha coefficients are dependent. Several of the procedures are derived in detail, and numerical examples are given for two. Since all of the procedures depend on approximate asymptotic results, Monte Carlo methods are used to assess the accuracy of the procedures for sample sizes of 50, 100, and 200. Both control of Type I error and power are evaluated by computer simulation. Two of the procedures are unable to control Type I errors satisfactorily. The remaining nine procedures perform properly, but three are somewhat superior in power and Type I error control.A more detailed version of this paper is also available.     

Stability of the Millon Clinical Multiaxial Inventory among psychiatric inpatients

This study investigated the stability of the Millon Clinical Multiaxial Inventory (MCMI) in a sample of psychiatric inpatients over a longer time period and from different perspectives than previous reliability studies. Test-retest reliability was assessed from three perspectives: correlation/regression, equivalence of dimensional structures across testings upon different admissions, and stability of personality style across testings. Some stability from a correlational perspective was found, with higher stability estimates indicated for the basic personality scales in comparison with the clinical symptom scales which is consistent with Millon's theory. The internal structure of the MCMI was essentially identical across testings. Although source of reinforcement and instrumental or coping style were stable across testings beyond chance, the MCMI profiles of a considerable number of subjects were inconsistent across testings. Correspondingly, clinicians should be cautious in using the MCMI to make categorical decisions regarding personality style with inpatients who are tested at admission and who have had repeated hospitalizations.     

Graphic comparisons of interobserver agreement indices

How changes in the interobserver agreement and disagreement cells in the reliability matrix are reflected differently in eight commonly used reliability indices is shown graphically. Indices which take into account expected chance difference are compared to those indices which do not. Differences between indices which do and do not treat the agreement and the disagreement cells equally are also illustrated.     

Internal consistency of tests: Analyses old and new

A coefficient derived from communalities of test parts has been proposed as greatest lower bound to Guttman's immediate retest reliability. The communalities have at times been calculated from covariances between itemsets, which tends to underestimate appreciably. When items are experimentally independent, a consistent estimate of the greatest defensible internal-consistency coefficient is obtained by factoring item covariances. In samples of modest size, this analysis capitalizes on chance; an estimate subject to less upward bias is suggested. For estimating alternate-forms reliability, communality-based coefficients are less appropriate than stratified alpha.I thank Edward Haertel for comments and suggestions, and Andrew Comrey for data.     

Psychometric evaluation of the dating anxiety survey: A self-report questionnaire for the assessment of dating anxiety in males and females

The Dating Anxiety Survey (DAS) was constructed to assess dating anxiety in males and females. Factor analysis of the survey revealed three factors: passive contact, active intentions for dating, and dating interactions. The reliabilities of the three subscales, as determined by coefficient alpha, were .87, .91, and .93 for males and .90, .90, and .92 for females, respectively. Correlations with dating history and a measure of social anxiety were generally of a low but significant magnitude, providing some support for concurrent validity. The results of the factor analysis lend support to the construct validity of the DAS. These findings suggest that the DAS is a potentially useful instrument in the self-report of dating anxiety.Portions of this paper were presented at the meeting of the Southeastern Psychological Association, Atlanta, GA, March 1983.     

Experience and desire: A new format for Sensation Seeking Scales

A new form (VI) of the Sensation Seeking Scale (SSS) was developed which separates reports of past experiences from desired or intended future experiences on both Disinhibition (Dis) and Thrill and Adventure Seeking (TAS) factors. Factor analyses were used to select items for the scales. High internal reliabilities were found for the Experience-Dis, Intention-TAS, and Intention-Dis scales, but only moderate reliability was found for the Experience-TAS scale. Retest reliabilities were high for all scales. The Experience-TAS and -Dis scales were highly correlated for males but not for females. The Experience- and Intention-TAS scales were moderately correlated, and the Experience- and Intention-Dis scales were highly correlated for both sexes. Both the TAS and the Dis scales on form V were highly correlated with the corresponding Intention scales on form VI. Uses for the new SS scales in individual assessment are suggested.A copy of SSS form VI may be obtained from the author.     

Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds

Let Σ _x be the (population) dispersion matrix, assumed well-estimated, of a set of non-homogeneous item scores. Finding the greatest lower bound for the reliability of the total of these scores is shown to be equivalent to minimizing the trace of Σ _x by reducing the diagonal elements while keeping the matrix non-negative definite. Using this approach, Guttman's bounds are reviewed, a method is established to determine whether his λ₄ (maximum split-half coefficient alpha) is the greatest lower bound in any instance, and three new bounds are discussed. A geometric representation, which sheds light on many of the bounds, is described. Present affiliation of the second author: Department of Statistics, University of Nigeria (Nsukka Campus). Work on this paper was carried out while on study leave in Aberystwyth.