Structural analysis of subjective categorical data

A general approach to the analysis of subjective categorical data is considered, in which agreement matrices of two or more raters are directly expressed in terms of error and agreement parameters. The method provides focused analyses of ratings from several raters for whom ratings have measurement error distributions that may induce bias in the evaluation of substantive questions of interest. Each rater's judgment process is modeled as a mixture of two components: an error variable that is unique for the rater in question as well as an agreement variable that operationalizes the true values of the units of observation. The statistical problems of identification, estimation, and testing of such measurement models are discussed.The general model is applied in several special cases. The most simple situation is that underlying Cohen's Kappa, where two raters place units into unordered categories. The model provides a generalization and systematization of the Kappa-idea to correct for agreement by chance. In applications with typical research designs, including a between-subjects design and a mixed within-subjects, between-subjects design, the model is shown to disentangle structural and measurement components of the observations, thereby controlling for possible confounding effects of systematic rater bias. Situations considered include the case of more than two raters as well as the case of ordered categories. The different analyses are illustrated by means of real data sets.The authors wish to thank Lawrence Hubert and Ivo Molenaar for helpful and detailed comments on a previous draft of this paper. Thanks are also due to Jens Möller und Bernd Strauß for the data from the 1992 Olympic Games. We thank the editor and three anonymous reviewers for valuable comments on an earlier draft.     

Probability matrix decomposition models

In this paper, we consider a class of models for two-way matrices with binary entries of 0 and 1. First, we considerBoolean matrix decomposition, conceptualize it as alatent response model (LRM) and, by making use of this conceptualization, generalize it to a larger class of matrix decomposition models. Second,probability matrix decomposition (PMD) models are introduced as a probabilistic version of this larger class of deterministic matrix decomposition models. Third, an algorithm for the computation of the maximum likelihood (ML) and the maximum a posteriori (MAP) estimates of the parameters of PMD models is presented. This algorithm is an EM-algorithm, and is a special case of a more general algorithm that can be used for the whole class of LRMs. And fourth, as an example, a PMD model is applied to data on decision making in psychiatric diagnosis. This paper is based on a chapter of the first author's doctoral dissertation, written at the University of Leuven and supervised by Paul De Boeck.     

Use of normative peer data as a standard for evaluating classroom treatment effects

This study illustrated the use of normative behavioral observation data as a standard for evaluating the practicality of treatment effects produced in other settings. Three groups of eight subjects each, displaying relatively low proportions of appropriate classroom behavior when compared with regular classroom peers, were selected for treatment within an experimental classroom setting. The three groups were exposed to intervention procedures designed to reinforce either direct academic performance and/or facilitative nonacademic classroom responses. The treatment was effective in changing levels of appropriate behavior (1) above baseline levels in the experimental classroom, and (2) to within normal peer-defined limits when reintegrated into the regular classroom. Further, the data reflect successful maintenance of these effects for a seven- to 12-week follow-up period. Several applications of a normative model for evaluating treatment, generalization, and maintenance effects were presented and discussed.     

Police and prosecution systems: An evaluation of a police criminal case preparation program

Program evaluation can provide objective information relevant to decisions on program maintenance. A program to address problems in the preparation of criminal investigation reports in a metropolitan police department was evaluated. The program permanently altered environmental conditions under which reports were prepared to facilitate performance. Police officers, who had previously prepared reports without assistance, visited the Case Preparation Room to prepare reports with assistance from office personnel. Compared to reports prepared without assistance, reports prepared in the Case Preparation Room documented more case elements required by the state legal code for criminal prosecution, were completed in fewer days following arrests, and received higher ratings from Assistant District Attorneys. Operation of a permanent program available to approximately 945 officers proved a practical solution to improving the preparation of criminal investigation reports.     

Social interest,extreme response style,and implications for adjustment

As part of a series of studies examining Adler's proposition that social interest, or interest and concern for others, is a major factor facilitating personal adjustment, the research investigated the relation between social interest and Extreme Response Style (ERS). ERS was chosen as an indirect measure of maladjustment since research has confirmed Adler's contention that it is especially prevalent among neurotic individuals. In three studies involving a total of 221 subjects, inverse relations were found between a measure of social interest and ERS scores derived from four questionnaires concerning attitudes toward self, others, and a wide variety of attitudinal issues. The data also indicated that the results were not due to either apathy or repression being associated with social interest. The results provided further evidence that positive relations between social interest and adjustment are not limited to self-reports of the latter.     

Intervention effects and relative variation as dimensions in experts' use of visual inference

Recent research indicates that when analyzing graphically presented single-subject data, subjects trained in visual inference appear to attend to large changes between phases regardless of relative variation and do not differentiate among common intervention effect patterns. In this follow-up study, experts in applied behavior analysis completed a free-sort task designed to assess the effects of these dimensions on their use of visual inference. The results indicate that they tended to differentiate among common intervention effect patterns but did not attend to relative variation in the data.     

The principal components of mixed measurement level multivariate data: An alternating least squares method with optimal scaling features

A method is discussed which extends principal components analysis to the situation where the variables may be measured at a variety of scale levels (nominal, ordinal or interval), and where they may be either continuous or discrete. There are no restrictions on the mix of measurement characteristics and there may be any pattern of missing observations. The method scales the observations on each variable within the restrictions imposed by the variable's measurement characteristics, so that the deviation from the principal components model for a specified number of components is minimized in the least squares sense. An alternating least squares algorithm is discussed. An illustrative example is given.Copies of this paper and of the associated PRINCIPALS program may be obtained by writing to Forrest W. Young, Psychometric Laboratory, Davie Hall 013-A, Chapel Hill, NC 27514.     

Evaluating interobserver reliability of interval data

Previous recommendations to employ occurrence, nonoccurrence, and overall estimates of interobserver reliability for interval data are reviewed. A rationale for comparing obtained reliability to reliability that would result from a random-chance model is explained. Formulae and graphic functions are presented to allow for the determination of chance agreement for each of the three indices, given any obtained per cent of intervals in which a response is recorded to occur. All indices are interpretable throughout the range of possible obtained values for the per cent of intervals in which a response is recorded. The level of chance agreement simply changes with changing values. Statistical procedures that could be used to determine whether obtained reliability is significantly superior to chance reliability are reviewed. These procedures are rejected because they yield significance levels that are partly a function of sample sizes and because there are no general rules to govern acceptable significance levels depending on the sizes of samples employed.     

Social skills training for juvenile delinquents

Twenty-seven incarcerated juvenile delinquents matched on the number of previous offenses, age, WISC-R IQ, and locus of control were assigned to a Social Skills. Discussion or Control group. Social skills training consisted of instruction, feedback, modeling, behavior rehearsal, social reinforcement and graduated homework assignments. All subjects were assessed before and after treatment on a variety of self-report, role-play and behavioral measures. Analyses of variance for difference scores indicated that the Social Skills group improved significantly more than the Discussion and Control groups, which did not differ. Appropriate interpersonal skills were learned, state anxiety was reduced, internal locus of control was increased, and significant shifts in adjustment to the institutional program were evidenced for the Social Skills group.     

Estimation for the multiple factor model when data are missing

A maximum likelihood method of estimating the parameters of the multiple factor model when data are missing from the sample is presented. A Monte Carlo study compares the method with 5 heuristic methods of dealing with the problem. The present method shows some advantage in accuracy of estimation over the heuristic methods but is considerably more costly computationally.This paper is based on the author's doctoral dissertation at the Department of Psychology, University of Illinois at Urbana-Champaign. The author gratefully acknowledges the aid of Drs. Robert Bohrer, Charles Lewis, Robert Linn, Maurice Tatsuoka, and Ledyard Tucker.