Reliability of parental reports using the Katz Adjustment Scales: before and after hospital admission for schizophrenia

We examined the interrater and test-retest reliability of the KATZ Adjustment Scale (Relative rated or R form) longitudinally in a sample of schizophrenic patients, assessing their function before hospital admission, as well as at 1 and 9 mo. after discharge. Changes in mean scores over those assessments suggested sensitivity to change while mothers and fathers both completed the measure with moderate consistency over time. Interrater reliability was moderate at best and quite poor at initial testing, when the subjects were most disturbed clinically, suggesting that the scales may be acceptable when the individual is stable but that ratings may be unreliable when there is an exacerbation in clinical state.     

The Tourette's Disorder Scale (TODS): development, reliability, and validity

To address the lack of a simple and standardized instrument to assess overall illness severity of Tourette's disorder (TD), the authors developed and tested a 15-item scale to measure a broad range of common symptoms including tics, inattention, hyperactivity, obsessions, compulsions, aggression, and emotional symptoms. Independent investigators used the 15-item Tourette's Disorder Scale (TODS) to assess 60 TD patients who were taking part in a double-blind placebo-controlled multicenter 8-week treatment study. Interrater reliability, internal consistency, convergent and discriminant validity, and sensitivity to change were examined. The TODS was associated with good interrater reliability, excellent internal consistency, and favorable levels of validity and sensitivity to change. Individual TODS items showed good convergent and discriminant validity against other measures. The TODS is a simple, efficient way for clinicians and parents to rate the severity of multiple symptoms commonly found in patients with Tourette's disorder.     

Interrater reliability and agreement of performance ratings: A methodological comparison

This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, theT index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearsonr, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.This paper is based, in part, on a thesis submitted to East Carolina University by the second author. Portions of this study were presented at the American Psychological Association meeting in New Orleans, LA, August, 1989. The authors would like to thank Michael Campion and two anonymous reviewers for their comments on earlier drafts of this paper.     

The Overvalued Ideas Scale: development, reliability and validity in obsessive-compulsive disorder.

The presence of overvalued ideas in obsessive-compulsive disorder (OCD) has been theoretically linked to poorer treatment outcome [Kozak, M. J. & Foa, E. B. (1994). Obsessions, overvalued ideas and delusions in obsessive-compulsive disorder. Behaviour Research and Therapy, 32, 343-353]. To date, no measures have been developed which quantitatively assess levels of overvalued ideas in obsessive-compulsives. The present studies examined the psychometric properties of a scale developed to measure this form of psychopathology, the Overvalued Ideas Scale (OVIS). In study 1, 102 patients diagnosed with OCD were administered a battery of instruments including the OVIS at baseline and two weeks later, prior to initiating treatment. Results indicate that the OVIS has adequate internal consistency reliability (coefficient alpha = 0.88 at baseline), test-retest reliability (r = 0.86) and interrater reliability (r = 0.88). Moderate to high levels of convergent validity was found with measures of obsessive-compulsive symptoms, a single item assessment of overvalued ideas and psychotic symptoms. Medium levels of discriminant validity with measures of anxiety and depression was obtained in this study. Individuals determined to have high OVI showed greater stability of this pathology than those with lower OVI, suggesting that overvalued ideas are stable for extreme scorers. In study 2 a total of 40 patients participated who were diagnosed with OCD. The same battery of instruments was administered as in study 1, as well as the Beck Depression Inventory and Beck Anxiety Inventories. Results were similar to that obtained in study 1, including a relative lack of discriminant validity with self-report measures of depression and anxiety. It is suggested that further research with the OVIS may show predictive value in treatment outcome studies of OCD.     

Therapist interventions in short-term dynamic psychotherapy: six expert treatments

This paper describes the profile of verbal response modes utilised in the expert application of Short-Term Dynamic Psychotherapy (STDP). One hundred and fifteen randomly selected segments from six treatments of STDP were analysed. Trained raters used a verbal response mode coding system to examine the individual speaking turns of an expert therapist. Based on the profile of therapist interventions reported, it was concluded that the actual conduct of this treatment in routine practice illustrates the empirically informed modifications to STDP technique integrated alongside the common characteristics of STDP based on the therapist (i) adopting an active stance, (ii) maintaining treatment focus using frequent confrontations and the ‘Triangle of Conflict’, and (iii) tailoring treatment to participant functioning using a combination of supportive and expressive interventions. Furthermore, specific differences in therapist activity were observed across treatment phases as well as between participants.     

Identification of Nearly Fatal Suicide Attempts: Self-Inflicted Injury Severity Form

The Self-inflicted Injury Severity Form (SIISF) was developed as an epidemiological research tool for identifying individuals in hospital emergency departments who have life-threatening self-inflicted injuries. Data were collected from 715 patients with self-inflicted injuries in two large hospitals. In 295 of these cases, a second set of data was independently collected for assessment of interrater reliability. Validity was assessed by comparing the SIISF results with simultaneously collected Risk—Rescue Ratings. Assessment of interrater reliability found that only 2.4% of physicians disagreed on the suicide method used. The kappa statistic for method used was .94, indicating excellent agreement. The SIISF was found to distinguish between severe and less severe injuries. Thus, it appears to provide a simple method to distinguish patients who have life-threatening self-inflicted injuries.     

The developmental profile: preliminary results on interrater reliability and construct validity

This study presents the preliminary results of research into the interrater reliability and construct validity of the Developmental Profile (DP). In the DP a number of developmental lines, such as Object-Relations, Self-Images, and Problem-Solving Capacities, are assessed and classified according to the level of functioning. A total of 108 profiles were assessed, drawn from three different categories of patients. The weighted kappa values for interrater reliability were sufficient. On the adaptive level, but also on the maladaptive levels Symbiosis and Resistance, significant differences were found between psychiatric patients, "normal controls" (dental patients) and somatic patients. No differences were recorded between the latter two groups. The conclusion is that the DP is a promising instrument, of which the reliability and validity has to be further investigated in order to contribute to scientific support for psychodynamic theory formation.     

Estimating Coefficients of Equivalence and Stability for Job Performance Ratings: The importance of controlling for transient error on criterion measurement

Previous research on measurement error in job performance ratings estimated reliability using coefficients: alpha, test–retest, and interrater correlation. None of these three coefficients control for the four main sources of error in performance ratings. For this reason, coefficient of equivalence and stability (CES) has been suggested as the ideal estimate of reliability. This article presents the estimates of CES for a time interval of 1, 2, and 3 years. The values obtained for a single rater were .51, .48, and .44, respectively. For two raters, the values were .59, .55, and .51. The findings suggest that previous reliability estimates based on alpha, test–retest, and interrater coefficients overestimated the reliability of job performance ratings. In the present study, the interrater coefficient overestimates reliability by 13.6–25.4% for an interval time of 1–3 years, as it does not control for transient error. Results also showed that the importance of transient error increases as the length of the interval between the measures increases. Based on the results, it is suggested that corrected validities based on interrater reliability underestimate the magnitude of the validity. The implications of these findings for future efforts to estimate criterion reliability and predictor validity are discussed.     

Taking “the boss” into the real world: Field interrater reliability of the Short‐Term Assessment of Risk and Treatability: Adolescent Version

There is emerging evidence that the performance of risk assessment instruments is weaker when used for clinical decision‐making than for research purposes. For instance, research has found lower agreement between evaluators when the risk assessments are conducted during routine practice. We examined the field interrater reliability of the Short‐Term Assessment of Risk and Treatability: Adolescent Version (START:AV). Clinicians in a Dutch secure youth care facility completed START:AV assessments as part of the treatment routine. Consistent with previous literature, interrater reliability of the items and total scores was lower than previously reported in non‐field studies. Nevertheless, moderate to good interrater reliability was found for final risk judgments on most adverse outcomes. Field studies provide insights into the actual performance of structured risk assessment in real‐world settings, exposing factors that affect reliability. This information is relevant for those who wish to implement structured risk assessment with a level of reliability that is defensible considering the high stakes.     

The Collaborative Longitudinal Personality Disorders Study: reliability of axis I and II diagnoses

Both the interrater and test-retest-retest reliability of axis I and axis II disorders were assessed using the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I) and the Diagnostic Interview for DSM-IV Personality Disorders (DIPD-IV). Fair-good median interrater kappa (.40-.75) were found for all axis II disorders diagnosed five times or more, except antisocial personality disorder (1.0). All of the test-retest kappa for axis II disorders, except for narcissistic personality disorder (1.0) and paranoid personality disorder (.39), were also found to be fair-good. Interrater and test-retest dimensional reliability figures for axis II were generally higher than those for their categorical counterparts; most were in the excellent range (> .75). In terms of axis I, excellent median interrater kappa were found for six of the 10 disorders diagnosed five times or more, whereas fair-good median interrater kappa were found for the other four axis I disorders. In general, test-retest reliability figures for axis I disorders were somewhat lower than the interrater reliability figures. Three test-retest kappa were in the excellent range, six were in the fair-good range, and one (for dysthymia) was in the poor range (.35). Taken together, the results of this study suggest that both axis I and axis II disorders can be diagnosed reliably when using appropriate semistructured interviews. They also suggest that the reliability of axis II disorders is roughly equivalent to that reliability found for most axis I disorders.     

Reliability and Validity of the Dyadic Observed Communication Scale (DOCS)

We evaluated the reliability and validity of the Dyadic Observed Communication Scale (DOCS) coding scheme, which was developed to capture a range of communication components between parents and adolescents. Adolescents and their caregivers were recruited from mental health facilities for participation in a large, multi-site family-based HIV prevention intervention study. Seventy-one dyads were randomly selected from the larger study sample and coded using the DOCS at baseline. Preliminary validity and reliability of the DOCS was examined using various methods, such as comparing results to self-report measures and examining interrater reliability. Results suggest that the DOCS is a reliable and valid measure of observed communication among parent-adolescent dyads that captures both verbal and nonverbal communication behaviors that are typical intervention targets. The DOCS is a viable coding scheme for use by researchers and clinicians examining parent-adolescent communication. Coders can be trained to reliably capture individual and dyadic components of communication for parents and adolescents and this complex information can be obtained relatively quickly.     

Employment Interview Reliability: New meta‐analytic estimates by structure and format

This study sought to provide an update on evidence regarding the interrater reliability of employment interviews. Using a final dataset of 125 coefficients with a total sample size of 32,428, our results highlight the importance of taking all three sources of measurement error (random response, transient, and conspect) into account. For instance, the mean interrater reliability was considerably higher for panel interviews than for separate interviews conducted by different interviewers (.74 vs. .44). A strong implication of our findings is that interview professionals should not base perceptions of the psychometric properties of their interview process on interrater estimates that do not include all three sources. A number of directions for future research were identified, including the influence of cues in medium structure panel interviews (e.g., changes in tone or pitch) and the lower than expected reliability for highly structured interviews conducted separately by different interviewers.     

An Analytic Creativity Assessment Scale for Digital Game Story Design: Construct Validity,Internal Consistency and Interrater Reliability

Mobile technology has rapidly made digital games a popular entertainment to this digital generation, and thus digital game design received considerable attention in both the game industry and design education. Digital game design involves diverse dimensions in which digital game story design (DGSD) particularly attracts our interest, as the literature needs more information, especially the creativity assessment of DGSD. Existing measuring tools do not adequately address the characteristics of game-story duality. Thus, an analytic creativity assessment scale of DGSD (CAS-DGSD) based on literature and original ideas was developed in our previous work. This study aims to statistically examine its construct validity, internal consistency reliability, and interrater reliability to verify its effectiveness. Three commercial games of 3 different game genres (action, puzzle, and role-play) were rated by 32 student raters and 4 expert raters. Statistical results show acceptable construct validity, internal consistency reliability, and interrater reliability of the CAS-DGSD. The CAS-DGSD not only helps evaluators like teachers identify which aspects of DGSD are short of creativity, but also serves as a guideline for digital game story designers like design students and product developers to tailor creative and entertaining game stories.     

INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS

Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.     

Development of a Patient Adherence Scale for Exposure and Response Prevention Therapy

Exposure and response prevention (EX/RP) is an evidence-based treatment for obsessive-compulsive disorder (OCD). For EX/RP to be maximally effective, it is believed that patients must adhere outside of sessions to the procedures they learn in therapy. To date, there is no standard measure of patient EX/RP adherence, despite the importance of accurately assessing EX/RP adherence in both clinical research and practice. This paper describes the development of the Patient EX/RP Adherence Scale (PEAS), which assesses the patient's between-session adherence to the therapist's EX/RP instructions, and presents initial data on the scale's reliability and validity. The scale was designed to focus on the key procedures of EX/RP and to be brief enough to be used at each treatment session. The scale demonstrates excellent interrater reliability and good face and content validity. The usefulness of the scale is considered in the context of being an important tool to researchers trying to understand and improve outcomes of EX/RP for OCD as well as to EX/RP therapists in clinical practice. Future research will need to test the scale's reliability and validity in a larger sample of patients over the course of treatment.     

The Revised Memory and Behavior Problems Checklist--Nursing Home: instrument development and measurement of burden among certified nursing assistants

Confirmatory factor analysis of the Revised Memory and Behavior Problems Checklist--Nursing Home (RMBPC) replicated the factor structure of the community-based RMBPC (L. Teri et al., 1992). The reliability of the total score was high as indexed by estimates of internal consistency (alpha = .95), test-retest reliability (r = .86), and interrater reliability between 2 interviewers (r = .88). Notably, the interrater reliability between 2 independent certified nursing assistants (CNAs) regarding residents' behavior problem frequency was more modest (r = .46), possibly reflecting the degree to which resident behaviors capture individual CNA's attention. This may have implications for the interpretation of data from the Minimum Data Set. CNAs reported moderately severe burden associated with behavior problems in 47% of residents under their care.     

A thermal vascular test for distinguishing between patients with Raynaud's phenomenon and healthy controls. Raynaud's Treatment Study Investigators.

This study tested the reliability and validity of a diagnostic thermal vascular test (TVT) for patients with Raynaud's Phenomenon (RP). The TVT examined digital blood pressure responses to combined cooling and occlusion and was developed as part of the Raynaud's Treatment Study, a multicenter clinical trial comparing the efficacy of biofeedback and pharmacological treatment. A computerized system permitted efficient, accurate, and uniform testing at different geographical sites. A comparison of 199 patients with RP and 52 healthy controls is reported. The TVT showed a sensitivity of 79% and a specificity of 88%. Test-retest reliability was acceptable (r = .80). Addition of a psychological challenge failed to improve the discrimination between patients with RP and controls. The TVT separated patients with RP and controls as well as or better than existing tests and did so with enhanced ease of operation.     

Client process in short term dynamic psychotherapy1

In this study, a single case design was used to examine significant client events in a process of short-term dynamic psychotherapy. The Category System of Client Good Moments (Mahrer, 1988) was used as a measure of therapeutic process. Client speaking turns were rated for three sessions (early, middle, and late) of the complete sixteen session treatment conducted according to davanloo's (1978) model of short-term dynamic psychotherapy (STDP), and categories of good moments identified. The results showed that the salient client change-events were related to the client's (a) provision of significant information, (b) exploration of feelings, and (c) insight and understanding across the three sessions. Additionally, the results also showed a statistically significant increase in client-reported behavior change and sense of well-being for the late session. The results suggest that in-session report of behavior change is an important component of therapeutic process. The implications of these results for an atheoretical, data-driven understanding of therapeutic change processes are presented.