A Measure for the Reliability of a Rating Scale Based on Longitudinal Clinical Trial Data

A new measure for reliability of a rating scale is introduced, based on the classical definition of reliability, as the ratio of the true score variance and the total variance. Clinical trial data can be employed to estimate the reliability of the scale in use, whenever repeated measurements are taken. The reliability is estimated from the covariance parameters obtained from a linear mixed model. The method provides a single number to express the reliability of the scale, but allows for the study of the reliability’s time evolution. The method is illustrated using a case study in schizophrenia. The authors are grateful to J&J PRD for kind permission to use their data. We gratefully acknowledge support from the Belgian IUAP/PAI network “Statistical Techniques and Modeling for Complex Substantive Questions with Complex Data.”     

Hierarchical Classes Modeling of Rating Data

Hierarchical classes (HICLAS) models constitute a distinct family of structural models for N-way N-mode data. All members of the family include N simultaneous and linked classifications of the elements of the N modes implied by the data; those classifications are organized in terms of hierarchical, if–then-type relations. Moreover, the models are accompanied by comprehensive, insightful graphical representations. Up to now, the hierarchical classes family has been limited to dichotomous or dichotomized data. In the present paper we propose a novel extension of the family to two-way two-mode rating data (HICLAS-R). The HICLAS-R model preserves the representation of simultaneous and linked classifications as well as of generalized if–then-type relations, and keeps being accompanied by a comprehensive graphical representation. It is shown to bear interesting relationships with classical real-valued two-way component analysis and with methods of optimal scaling. The research reported in this paper was supported by the Research Fund of the University of Leuven (GOA/00/02 and GOA/05/04) and by the Fund for Scientific Research-Flanders (project G.0146.06). Eva Ceulemans is a Post-doctoral Researcher supported by the Fund for Scientific Research, Flanders. The authors gratefully acknowledge the help of Gert Quintiens and Kaatje Bollaerts in collecting the data used in Section 4 and of Jan Schepers in additional analyses of these data.     

Measuring the ability of transitive reasoning,using product and strategy information

Cognitive theories disagree about the processes and the number of abilities involved in transitive reasoning. This led to controversies about the influence of task characteristics on individuals' performance and the development of transitive reasoning. In this study, a computer test was constructed containing 16 transitive reasoning tasks having different characteristics with respect to presentation form, task format, and task content. Both product and strategy information were analyzed to measure the performance of 6- to 13-year-old children. Three methods (MSP, DETECT, and Improved DIMTEST) were used to determine the number of abilities involved and to test the assumptions imposed on the data by item response models. Nonparametric IRT models were used to construct a scale for transitive reasoning. Multiple regression was used to determine the influence of task characteristics on the difficulty level of the tasks. It was concluded that: (1) the qualitatively distinct abilities predicted by Piaget's theory could not be distinguished by means of different dimensions in the data structure; (2) transitive reasoning could be described by one ability, and some task characteristics influenced the difficulty of a task; and (3) strategy information provided a stronger scale than product information.     

Paradoxes in Double Extension Set Theories

Three systems of double extension set theory have been proposed by Andrzej Kisielewicz in two papers. In this paper, it is shown that the two stronger systems are inconsistent, and that the third, weakest system does not admit extensionality for general sets or the use of general sets as parameters in its comprehension scheme. The parameter-free version of the comprehension principle of double extension set theory is also shown to be inconsistent with extensionality. The definitions of the systems and a self-contained exposition of their properties is given, sufficient to develop the inconsistency proofs.     

Enhancing frequency recording by developmental disabilities treatment staff

We evaluated a staff training and management package for increasing accuracy of recording frequency of problem behavior in a residential care facility. A multiple baseline design across the first and second work shifts showed that 2 of 8 participants increased their accuracy following in-service training, and all 8 improved during a condition with supervisor presence and feedback. Improvements were maintained when feedback was removed and generalized to activity periods when neither supervisor presence nor feedback was provided. Other staff behavior was not adversely affected by the intervention package.     

Consumer brand choice: individual and group analyses of demand elasticity

Following the behavior-analytic tradition of analyzing individual behavior, the present research investigated demand elasticity of individual consumers purchasing supermarket products, and compared individual and group analyses of elasticity. Panel data from 80 UK consumers purchasing 9 product categories (i.e., baked beans, biscuits, breakfast cereals, butter, cheese, fruit juice, instant coffee, margarine and tea) during a 16-week period were used. Elasticity coefficients were calculated for individual consumers with data from all or only 1 product category (intra-consumer elasticities), and for each product category using all data points from all consumers (overall product elasticity) or 1 average data point per consumer (interconsumer elasticity). In addition to this, split-sample elasticity coefficients were obtained for each individual with data from all product categories purchased during weeks 1 to 8 and 9 to 16. The results suggest that: 1) demand elasticity coefficients calculated for individual consumers purchasing supermarket food products are compatible with predictions from economic theory and behavioral economics; 2) overall product elasticities, typically employed in marketing and econometric research, include effects of interconsumer and intraconsumer elasticities; 3) when comparing demand elasticities of different product categories, group and individual analyses yield similar trends; and 4) individual differences in demand elasticity are relatively consistent across time, but do not seem to be consistent across products. These results demonstrate the theoretical, methodological, and managerial relevance of investigating the behavior of individual consumers.     

The Short Mood and Feelings Questionnaire (SMFQ): A Unidimensional Item Response Theory and Categorical Data Factor Analysis of Self-Report Ratings from a Community Sample of 7-through 11-Year-Old Children

Item response theory (IRT) and categorical data factor analysis (CDFA) are complementary methods for the analysis of the psychometric properties of psychiatric measures that purport to measure latent constructs. These methods have been applied to relatively few child and adolescent measures. We provide the first combined IRT and CDFA analysis of a clinical measure (the Short Mood and Feelings Questionnaire—SMFQ) in a community sample of 7-through 11-year-old children. Both latent variable models supported the internal construct validity of a single underlying continuum of severity of depressive symptoms. SMFQ items discriminated well at the more severe end of the depressive latent trait. Item performance was not affected by age, although age correlated significantly with latent SMFQ scores suggesting that symptom severity increased within the age period of 7–11. These results extend existing psychometric studies of the SMFQ and confirm its scaling properties as a potential dimensional measure of symptom severity of childhood depression in community samples.     

A two-stage logistic regression model for analyzing inter-rater agreement

Studies of agreement commonly occur in psychiatric research. For example, researchers are often interested in the agreement among radiologists in their review of brain scans of elderly patients with dementia or in the agreement among multiple informant reports of psychopathology in children. In this paper, we consider the agreement between two raters when rating a dichotomous outcome (e.g., presence or absence of psychopathology). In particular, we consider logistic regression models that allow agreement to depend on both rater- and subject-level covariates. Logistic regression has been proposed as a simple method for identifying covariates that are predictive of agreement (Coughlin et al., 1992). However, this approach is problematic since it does not take account of agreement due to chance alone. As a result, a spurious association between the probability (or odds) of agreement and a covariate could arise due entirely to chance agreement. That is, if the prevalence of the dichotomous outcome varies among subgroups of the population, then covariates that identify the subgroups may appear to be predictive of agreement. In this paper we propose a modification to the standard logistic regression model in order to take proper account of chance agreement. An attractive feature of the proposed method is that it can be easily implemented using existing statistical software for logistic regression. The proposed method is motivated by data from the Connecticut Child Study (Zahner et al., 1992) on the agreement among parent and teacher reports of psychopathology in children. In this study, parents and teachers provide dichotomous assessments of a child's psychopathology and it is of interest to examine whether agreement among the parent and teacher reports is related to the age and gender of the child and to the time elapsed between parent and teacher assessments of the child.The authors thank the Associate Editor and the referees for their helpful comments and suggestions. We also thank Gwen Zahner for use of data from the Connecticut Child Study, which was conducted under contract to the Connecticut Department of Children and Youth Services. This research was supported by grants HL 69800, AHRQ 10871, HL52329, HL61769, GM 29745, MH 54693 and MH 17119 from the National Institutes of Health.     

1. On note taking

In this paper the author explores the theoretical and technical issues relating to taking notes of analytic sessions, using an introspective approach. The paper discusses the lack of a consistent approach to note taking amongst analysts and sets out to demonstrate that systematic note taking can be helpful to the analyst. The author describes his discovery that an initial phase where as much data was recorded as possible did not prove to be reliably helpful in clinical work and initially actively interfered with recall in subsequent sessions. The impact of the nature of the analytic session itself and the focus of the analyst's interest on recall is discussed. The author then describes how he modified his note taking technique to classify information from sessions into four categories which enabled the analyst to select which information to record in notes. The characteristics of memory and its constructive nature are discussed in relation to the problems that arise in making accurate notes of analytic sessions.