Millon Clinical Multiaxial Inventory (MCMI) scores with a collegiate sample: Long-term stability and self-other agreement

This study examined (a) the stability of the personality and symptom scales of the Millon Clinical Multiaxial Inventory (MCMI) for a sample of university undergraduates and (b) the correspondence between MCMI scores from self-report versus scores obtained from knowledgeable others who responded by describing the subject rather than themselves. The MCMI was administered to subjects during their freshman year and then again during their senior year. As in clinical populations, stability estimates in this collegiate sample were greater for the basic personality scales than for the symptom scales. Comparison of the results with those from previous studies further showed that the stability coefficients for the collegiate sample were somewhat lower overall than those from treatment follow-up studies with clinical populations. The results also showed that reasonable agreement between self- and others' ratings was obtained on most MCMI scales. The greatest agreement was found for scales that reflect more observable behaviors and relate to an introversion-extroversion dimension, whereas the least agreement was found for scales related to a psychoticism dimension. Lower levels of agreement were also found for scales having a lower mean proportion of items endorsed.This study was sponsored in part by PHS Grant R01 MH31750-01-6, by PHS Grant 5R01 AA06754-01-03, and by funds from the University of Connecticut Research Foundation and Computer Center.     

Reliability and validity of two widely-used worry questionnaires: self-report and self-peer convergence

The reliability and validity of the Penn State Worry Questionnaire (PSWQ) and the Worry Domains Questionnaire (WDQ) were examined with self-ratings from a non-clinical sample of 148 students in a test-retest design across four weeks. Ratings from three well-acquainted peers were also obtained. With internal consistencies and test-retest correlations of at least 0.85, the present study confirmed the high reliability of the questionnaires. Moreover, both measures demonstrated substantial convergent validity: Average agreement among peers was 0.42 (PSWQ) and 0.47 (WDQ) and aggregated self-peer agreement was 0.55 (PSWQ) and 0.49 (WDQ). Self-peer agreement was not biased by social desirability. These findings challenge views that worry is an unreliable and unobservable phenomenon.     

Referees, editors, and publication practices: Improving the reliability and usefulness of the peer review system

The documented low levels of reliability of the peer review process present a serious challenge to editors who must often base their publication decisions on conflicting referee recommendations. The purpose of this article is to discuss this process and examine ways to produce a more reliable and useful peer review system. The author is a Senior Research Scientist & Biostatistician, Child Study Center and Department of Psychiatry in the Yale University School of Medicine. Note: The main points of this report were presented at the Annual Meeting of the American Association for the Advancement of Science (AAAS) in Baltimore, Maryland, on February 9, 1996.     

Some developments in multivariate generalizability

This article is concerned with estimation of components of maximum generalizability in multifacet experimental designs involving multiple dependent measures. Within a Type II multivariate analysis of variance framework, components of maximum generalizability are defined as those composites of the dependent measures that maximize universe score variance for persons relative to observed score variance. The coefficient of maximum generalizability, expressed as a function of variance component matrices, is shown to equal the squared canonical correlation between true and observed scores. Emphasis is placed on estimation of variance component matrices, on the distinction between generalizability- and decision-studies, and on extension to multifacet designs involving crossed and nested facets. An example of a two-facet partially nested design is provided.Appreciation is expressed to the Office of Research in Medical Education, University of Texas Medical Branch, for permitting use of their data.     

Effect of a reliability observer on the frequency of positive parent behavior recorded during naturalistic parent-child interactions

The purpose of the present study was to examine the effect of the presence of a reliability observer on the number of positive parent behaviors recorded by a primary observer during naturalistic parent-child interactions. Thirty parents and their young clinic-referred children served as subjects. After two initial home observations, a reliability observer was present to record data in observation session 3, but not session 4, for one-half the subjects. For the remaining subjects the reliability observer was present in session 4 but not session 3. The results of a 2 × 2 analysis of variance indicated a group by session interaction which resulted from an increase in maternal attention to the child in the presence of a reliability observer. Hypotheses to explain the finding are presented and implications of the results are discussed.This research was supported in part by NIMH Grant MH28859-01.     

The effect of witnessing consequences on the behavioral recordings of experimental observers

The cueing effects of interviewer praise contingent on a target behavior and expectation of behavior change were examined with six observers. Experiment I investigated the effect of cues in conjunction with expectation. Experiment II assessed the relative contributions of cues and expectation, and Experiment III examined the effect of cues in the absence of expectation. The frequencies of two behaviors, client eye contact and face touching, were held constant throughout a series of videotaped interviews between an "interviewer" and a "client". A within-subjects design was used in each experiment. During baseline conditions, praise did not follow eye contact by the client on the videotape. In all experimental conditions, praise statements from the interviewer followed each occurrence of eye contact with an equal number of praises delivered at random times when there was no eye contact. Three of the six observers dramatically increased their recordings of eye contact during the first experimental phase, but these increases were not replicated in a second praise condition. There were no systematic changes in recorded face touching. Witnessing the delivery of consequences, rather than expectation seemed to be responsible for the effect. This potential threat to the internal validity of studies using observational data may go undetected by interobserver agreement checks.     

Measures of interobserver agreement: Calculation formulas and distribution effects

Seventeen measures of association for observer reliability (interobserver agreement) are reviewed and computational formulas are given in a common notational system. An empirical comparison of 10 of these measures is made over a range of potential reliability check results. The effects on percentage and correlational measures of occurrence frequency, error frequency, and error distribution are examined. The question of which is the best measure of interobserver agreement is discussed in terms of critical issues to be considered     

Evaluating interobserver reliability of interval data

Previous recommendations to employ occurrence, nonoccurrence, and overall estimates of interobserver reliability for interval data are reviewed. A rationale for comparing obtained reliability to reliability that would result from a random-chance model is explained. Formulae and graphic functions are presented to allow for the determination of chance agreement for each of the three indices, given any obtained per cent of intervals in which a response is recorded to occur. All indices are interpretable throughout the range of possible obtained values for the per cent of intervals in which a response is recorded. The level of chance agreement simply changes with changing values. Statistical procedures that could be used to determine whether obtained reliability is significantly superior to chance reliability are reviewed. These procedures are rejected because they yield significance levels that are partly a function of sample sizes and because there are no general rules to govern acceptable significance levels depending on the sizes of samples employed.     

Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds

Let Σ _x be the (population) dispersion matrix, assumed well-estimated, of a set of non-homogeneous item scores. Finding the greatest lower bound for the reliability of the total of these scores is shown to be equivalent to minimizing the trace of Σ _x by reducing the diagonal elements while keeping the matrix non-negative definite. Using this approach, Guttman's bounds are reviewed, a method is established to determine whether his λ₄ (maximum split-half coefficient alpha) is the greatest lower bound in any instance, and three new bounds are discussed. A geometric representation, which sheds light on many of the bounds, is described. Present affiliation of the second author: Department of Statistics, University of Nigeria (Nsukka Campus). Work on this paper was carried out while on study leave in Aberystwyth.