A simple computational alternative to analysis of variance formulas for estimating the k-judge reliability

Formulas requiring the computation of only three standard deviations are presented for computing the interjudge reliability coefficient for any number of judges. These formulas yield coefficients identical to those obtained from a one-way repeated-measures analysis of variance. Even researchers with small handheld calculators can use this simple approach.     

Comparing judgments of stuttering made by students, clinicians, and highly experienced judges

Purpose: The purpose of this study was to compare judgments of stuttering made by students and clinicians with previously available judgments made by highly experienced judges in stuttering. Method: On two occasions, 41 university students and 31 speech-language pathologists judged the presence or absence of stuttering in each of 216 audiovisually recorded 5-s intervals of the speech of adults who stutter. Intrajudge and interjudge agreement were calculated, and comparisons were made to judgments previously made about the same recordings by 10 highly experienced judges of stuttering. Results: Students and clinicians showed similar and relatively high levels of intrajudge and interjudge agreement, but both students and clinicians identified less than half as much stuttering as the highly experienced judges had identified. Conclusions: These results replicate previous findings of high agreement coexisting with low accuracy in students’ judgments of stuttering, extending those findings to show that similar problems are evident in judgments made by practicing clinicians. Implications include the need for explicit stuttering judgment training programs for both students and practicing clinicians.

Educational objectives: After reading this article, the reader will be able to: (1) describe different methods for identifying stuttering and possible problems associated with each method; (2) describe two different methods for reporting interjudge reliability; (3) describe how the identification of stuttering differs for student, clinician, and highly experienced judges.     

The Adult Attachment Projective Picture System: integrating attachment into clinical assessment

This article summarizes the development and validation of the Adult Attachment Projective System (AAP), a measure we developed from the Bowlby-Ainsworth developmental tradition to assess adult attachment status. The AAP has demonstrated excellent concurrent validity with the Adult Attachment Interview (George, Kaplan, & Main, 1984/1985/1996; Main & Goldwyn, 1985-1994; Main, Goldwyn, & Hesse, 2003), interjudge reliability, and test-retest reliability, with no effects of verbal intelligence or social desirability. The AAP coding and classification system and application in clinical and community samples are summarized. Finally, we introduce the 3 other articles that are part of this Special Section and discuss the use of the AAP in therapeutic assessment and treatment.     

Study samples are too small to produce sufficiently precise reliability coefficients

In a survey of journal articles, test manuals, and test critique books, the author found that a mean sample size (N) of 260 participants had been used for reliability studies on 742 tests. The distribution was skewed because the median sample size for the total sample was only 90. The median sample sizes for the internal consistency, retest, and interjudge reliabilities were 182, 64, and 36, respectively. The author presented sample size statistics for the various internal consistency methods and types of tests. In general, the author found that the sample sizes that were used in the internal consistency studies were too small to produce sufficiently precise reliability coefficients, which in turn could cause imprecise estimates of examinee true-score confidence intervals. The results also suggest that larger sample sizes have been used in the last decade compared with those that were used in earlier decades.     

Cognitive anxiety: a method of content analysis for verbal samples

The work of such psychologists as Kelly, McReynolds, Epstein, and Lazarus suggested the need for a measure of cognitive anxiety and provided a definition of that construct. A method of content analysis of verbal samples was devised and found to have adequate interjudge reliability. Normative data for five groups of subjects were provided. The validity of the measure as representative of a reaction to being unable to anticipate and integrate experience meaningfully was demonstrated in (a) the higher scores of groups of subjects who were currently coping with new experiences than those who were not, (b) the significant correlation of its scores with a state rather than trait anxiety measures, (c) the variability of its scores over time as observed in a generalizeability study, and (d) the higher scores of subjects when they were dealing with experiences for which meaningful anticipation was relatively difficult.     

The effects of self-monitoring and self-reported consistency on the consistency of personality judgments made by strangers and intimates

How consistent are strangers' and intimates' judgments of stimulus people's personalities, and how is this interjudge consistency affected by stimulus persons' assessed self-monitoring and self-reported behavioral consistency? To answer these questions, 38 stimulus subjects rated themselves on the personality dimensions of extraversion and anxiety and also rated their cross-situational consistency on these dimensions. Strangers, friends, mothers, and fathers of stimulus persons also rated them on extraversion and anxiety. The results indicated that: For judgments of anxiety there was lower interjudge consistency for high than for low self-monitoring stimulus subjects. For judgments of extraversion there was no difference in interjudge consistency for low and high self-monitoring stimulus subjects. The results also showed that anxiety was a more “private” trait in that intimates' but not strangers' judgments correlated with stimulus subjects' self-reported anxiety (r = .50 and r = .11, respectively), while extraversion was a more “public” trait in that both intimates' and strangers' judgments correlated with stimulus subjects' self-reported extraversion (r = .42 and r = .51, respectively). These results suggest that self-monitoring of stimulus persons affects interjudge consistency of peroonality judgments particularly for “private” traits such as anxiety, which are most subject to expressive control and inhibition.     

The Development of a Scale for Evaluating Creative Culinary Products

The purpose of this study was to develop a “Scale for Evaluating Creative Culinary Products” based on the nature of these products. We adopted the Consensual Assessment Technique (CAT) to establish the credibility of the scale. The scale comprises 8 categories: professional technique; aroma, taste, and texture; color; modeling and arrangement; garnish; dishware; handling of ingredients; and overall assessment. There are 34 assessment criteria for these 8 categories. Nine culinary experts were invited to evaluate, by means of these criteria, the products of 28 college students studying Chinese cuisine. The degree of interjudge reliability was .98, and the correlation between each criterion and the total scale was between .578 and .856. Through confirmatory factor analysis, the scale was demonstrated to possess a high degree of validity.     

Interjudge Agreement of Draw-A-Person Diagnostic Impressions

Summary

The interjudge agreement of Draw-A-Person (DAP)-based diagnostic impressions was assessed. Twenty-four psychologists were asked to categorize 48 DAP protocols as being the productions of organics, paranoid schizophrenics, non-paranoid schizophrenics or normal controls. The mean between-judges proportion of agreement was only .41 and the mean proportion correct was a mere .28. (A proportion of .25 could be anticipated in each case by chance.) Strength of interjudge agreement did not vary with the extent to which judges utilized the test but did seem dependent upon the subjects' diagnoses. The results indicated that diagnostic impressions based on the DAP alone seem to be neither usefully valid nor impressively consistent across psychologists. It is suggested that the profession is due for a re-evaluation of its DAP training techniques and its use of the test.     

Low accuracy in judgments of others' psychological well-being as seen from a phenomenological perspective

The generally low degree of agreement between self-ratings on personality traits and ratings by others may be interpreted from the viewpoint that self-reports reflect people's experience of themselves but not necessarily their behaviors. A detailed analysis of self and other ratings on subjective well-being as a central dimension of human experience is consistent with this phenomenological view. Ratings of well-being were not significantly correlated with rated behaviors either in self-ratings or in ratings by others. Screening subjects in terms of avowed consistency and observability on a trait did not improve self-other agreement for well-being, nor did it replicate the individual trait effects reported by Kenrick and Stringfield (1980). Judgments by others were found to have poor interjudge reliability and to reflect biases associated with projection of own well-being and a halo effect organized around the subject's perceived friendliness or likability. It was demonstrated that pooling the judgments of several observers should not and does not lead to accurate prediction of the phenomenal personality, and that accuracy may generally depend on the level of self-disclosure.     

Some Arguments Against Analyzing Client Change Graphically

Although single-subject research techniques are relevant and valuable to counselors, analyzing change by graphs alone is open to major sources of error. The author discusses these sources and explores the issues of serial dependency, unreliability of graphs, interjudge disagreement on graphed data, and the use of a more reliable statistical procedure (the time-series statistic) with relevance to the counselor in the field.     

Determinants of Interjudge Agreement on Personality Traits: The Big Five Domains, Observability, Evaluativeness, and the Unique Perspective of the Self

ABSTRACT We examined several determinants of interjudge agreement on personality traits. The findings, which were cross-validated in two samples, suggest that agreement is a function of four factors: which Big Five content domain the trait represents, how observable relevant behaviors are, how evaluative the trait is, and whether the self is one of the judges. Agreement was highest for traits related to Extraversion and lowest for traits related to Agreeableness. More observable and less evaluative traits elicited higher interjudge agreement. On average, self-peer agreement was lower than peer-peer agreement. However, this effect was limited to evaluative traits; for neutral traits, self-peer agreement was as high as peer-peer agreement. These findings suggest that self- and peer perception proceed through similar processes for neutral traits but not for highly evaluative traits, raising the possibility that self-perceptions become distorted when the trait is affectively charged.     

Psychometric Properties and Clinical Utility of the Scale for Suicide Ideation with Inpatient Children

The purpose of the current study was to examine the psychometric properties and clinical utility of the Scale for Suicide Ideation (SSI) for children. The SSI was administered to 100 children who were hospitalized. Data were not used from 13 subjects due to the presence of a psychotic disorder or IQ less than 70, leaving 87 participants. The SSI was examined regarding its internal consistency, concurrent validity, construct validity, and factor structure when used with children. Based on the findings, the SSI appears to have adequate psychometric properties and can be used by clinicians and researchers examining children in an inpatient setting. Recommendations for clinicians are included to help make the SSI more functional in use for children.     

A pilot study for the analysis of dream reports using Maslow's need categories: an extension to the emotional selection hypothesis

The emotional selection hypothesis describes a cyclical process that uses dreams to modify and test select mental schemas. An extension is proposed that further characterizes these schemas as facilitators of human need satisfaction. A pilot study was conducted in which this hypothesis was tested by assigning 100 dream reports (10 randomly selected from 10 dream logs at an online web site) to one or more categories within Maslow's hierarchy of needs. A "match" was declared when at least two of three judges agreed both for category and for whether the identified need was satisfied or thwarted in the dream narrative. The interjudge reliability of the judged needs was good (92% of the reports contained at least one match). The number of needs judged as thwarted did not differ significantly from the number judged as satisfied (48 vs. 52%, respectively). The six "higher" needs (belongingness, esteem, cognitive, aesthetic, self-actualization, and transcendence) were scored significantly more frequently (81%) than were the two lowest or "basic" needs (physiological and safety, 19%). Basic needs were also more likely to be judged as thwarted, while higher needs were more likely to be judged as satisfied. These findings are discussed in the context of Maslow's hierarchy of needs as a framework for investigating theories of dream function, including the emotional selection hypothesis and other contemporary dream theories.     

Strange-but-true: a (quick) new argument for contextualism about ‘know’

A powerful objection to subject-sensitive invariantism (SSI) concerns various ‘strange-but-true’ (or “embarrassing”) conditionals. One popular response to this objection is to argue that strange-but-true conditionals pose a problem for non-sceptical epistemological theories in general. In the present paper, it is argued that strange-but-true conditionals are not a problem for contextualism about ‘know’. This observation undercuts the proposed defence of SSI, and supplies a surprising new argument for contextualism.     

Differences between traits: properties associated with interjudge agreement

The present study concerns the relation between properties of personality traits and the agreement with which they are applied to real individuals. Subjects rated the 100 personality items of the California Q-Set on nine subjective dimensions, six of which loaded highly on a first principal component. This factor was interpreted as reflecting each trait's "easy visibility" to an outside observer. Actual interjudge agreement in applying each trait to real individuals was assessed in two ways: Self-other agreement was assessed in two independent samples, and interpeer agreement was assessed in three samples. Impressive and stable agreement was found for most Q items. The traits that were applied to individuals with the greatest interjudge agreement were the same ones that seemed most easily visible and tended to be positively relevant to extraversion and negatively relevant to neuroticism (identified through a factor analysis by McCrae, Costa, & Busch, 1986). The results suggest that traits defining extraversion are revealed relatively directly in social behavior and, therefore, are easy to judge, that traits defining neuroticism are less visible and, so, are judged less accurately, and that lay perceivers of personality are generally sensitive to this difference between traits.