Interdisciplinary agreement in the diagnosis of headache types

The degree of agreement in independent diagnosis of headache by a neurologist and a clinical psychologist was determined to be 86.4% in a series of 66 consecutive patients. Fifty-five percent of the disagreements were due to the patients' giving different information to the two assessors. This source of disagreement was associated with the patients' having had previous psychiatric treatment.This research was supported in part by Grant NS-15235 from the National Institute of Neurological and Communicative Disorders and Stroke.     

Measures of interobserver agreement: Calculation formulas and distribution effects

Seventeen measures of association for observer reliability (interobserver agreement) are reviewed and computational formulas are given in a common notational system. An empirical comparison of 10 of these measures is made over a range of potential reliability check results. The effects on percentage and correlational measures of occurrence frequency, error frequency, and error distribution are examined. The question of which is the best measure of interobserver agreement is discussed in terms of critical issues to be considered     

Simplifying continuous monitoring of multiple-response/multiple-subject classroom interactions

In order to facilitate the field monitoring of three subjects interacting according to one or more of 18 response categories, a modified version of several available, but oftentimes mechanically incompatible, observational procedures was designed. Its continuous recording strategy, sectioned into one-minute observational units, enabled researchers to derive highly representative behavior samples, and when accompanied by the specially tailored coding form and recording apparatus, observers achieved over 90% agreement across all reliability sessions. This procedure provides applied researchers with a simple, highly reliable, and adaptable observation tool for continuously and simultaneously monitoring the behaviors of one or more subjects.     

Accuracy of observational data with whole interval,partial interval,and momentary time-sampling recording techniques

The purpose of the study was to compare the accuracies obtained from whole interval, partial interval, and momentary time-sample recording procedures. Two types of accuracies were defined. Between-methods accuracy is the degree to which an observer using a particular recording method agrees with a standard using a continuous recording method. Within-methods accuracy is the degree to which an observer and a standard agree when both use the same recording method. Fifty-four undergraduate students viewed a videotape of a woman twisting her hair and recorded the occurrence or nonoccurrence of the behavior using one of the three recording methods. Tapes were divided into segments with low, intermediate, and high rates of behavior. Partial and whole interval recording obtained poorer between-methods accuracies than momentary time sampling. Results showed that whole interval and momentary time-sampling procedures yielded higher within-method accuracies than partial interval recording. Thus, momentary time-sampling recording provided the advantages of both greater representativeness and as few or fewer observer errors than whole or partial interval recording procedures. Use of the interval recording methods by researchers must be justifiable within this framework.     

Reliability and expected loss: A unifying principle

We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the proportional reduction in loss (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.Bruce Cooil is an Associate Professor of Statistics, and Roland T. Rust is a Professor and area head for Marketing. The authors thank three anonymous reviewers and an Associate Editor for their helpful comments and suggestions. This work was supported in part by the Dean's Fund for Faculty Research of the Owen Graduate School of Management, Vanderbilt University.     

ANOVA and ANCOVA of pre- and post-test,ordinal data

With random assignment to treatments and standard assumptions, either a one-way ANOVA of post-test scores or a two-way, repeated measures ANOVA of pre- and post-test scores provides a legitimate test of the equal treatment effect null hypothesis for latent variable . In an ANCOVA for pre- and post-test variablesX andY which are ordinal measures of and , respectively, random assignment and standard assumptions ensure the legitimacy of inferences about the equality of treatment effects on latent variable . Sample estimates of adjustedY treatment means are ordinal estimators of adjusted post-test means on latent variable .     

On the estimation of polychoric correlations and their asymptotic covariance matrix

A general theory for parametric inference in contingency tables is outlined. Estimation of polychoric correlations is seen as a special case of this theory. The asymptotic covariance matrix of the estimated polychoric correlations is derived for the case when the thresholds are estimated from the univariate marginals and the polychoric correlations are estimated from the bivariate marginals for given thresholds. Computational aspects are also discussed.The research was supported by the Swedish Council for Research in the Humanities and Social Sciences (HSFR) under the programMultivariate Statistical Analysis. The author thanks a reviewer for pointing out an error in the original version of the paper.     

Some remarks on Scheiblechner's treatment of isop models

Scheiblechner (1995) proposes a probabilistic axiomatization of measurement called ISOP (isotonic ordinal probabilistic models) that replaces Rasch's (1980) specific objectivity assumptions with two interesting ordinal assumptions. Special cases of Scheiblechner's model include standard unidimensional factor analysis models in which the loadings are held constant, and the Rasch model for binary item responses. Closely related are the doubly-monotone item response models of Mokken (1971; see also Mokken & Lewis, 1982; Sijtsma, 1988; Molenaar, 1991; Sijtsma & Junker, 1996; and Sijtsma & Hemker, in press). More generally, strictly unidimensional latent variable models have been considered in some detail by Holland and Rosenbaum (1986), Ellis and van den Wollenberg (1993), and Junker (1990, 1993). The purpose of this note is to provide connections with current research in foundations and nonparametric latent variable and item response modeling that are missing from Scheiblechner's (1995) paper, and to point out important related work by Hemker, Sijtsma, Molenaar, & Junker (1996, 1997), Ellis and Junker (in press) and Junker and Ellis (1997). We also discuss counterexamples to three major theorems in the paper. By carrying out these three tasks, we hope to provide researchers interested in the foundations of measurement and item response modeling the opportunity to give the ISOP approach the careful attention it deserves. This research was supported by the National Science Foundation, Grant DMS-94.04438. I thank the Editor and three anonymous referees for careful readings that improved the completeness of this note.     

模糊偏序关系在心理学研究中的应用

本文首次给出建立模糊偏序关系矩阵的可行途径。以大学生的颜色爱好实验为例,论证了采用本文所建立的模糊偏序关系矩阵进行排序的方法是制作顺序量表的一种较佳方法。     

Rorschach interrater agreement estimates: An empirical evaluation

A standardized estimation of Rorschach interrater agreement is needed. Percentage agreement, although widely used, is found to be unsuitable. Forty-one protocols from adults in both a normal and a psychiatric sample were scored by two or three scorers, making 85 scoring pairs. Percentage agreement, correlations (phi and Pearson's r ), and kappa were computed on single response, total score, and category level. Percentage agreement shows minimal variation. Even when exceeding 0.80, it can obscure major disagreements. Kappa and correlations both vary in a similar way with level of disagreement. Total score level does not give additional information compared to single score and category levels. Kappa proved to be conservative and reliable and is therefore suggested as a standard estimate.