CONTINUOUS RECORDING AND INTEROBSERVER AGREEMENT ALGORITHMS REPORTED IN THE JOURNAL OF APPLIED BEHAVIOR ANALYSIS (1995–2005)

We reviewed all research articles in 10 recent volumes of the Journal of Applied Behavior Analysis (JABA): Vol. 28(3), 1995, through Vol. 38(2), 2005. Continuous recording was used in the majority (55%) of the 168 articles reporting data on free‐operant human behaviors. Three methods for reporting interobserver agreement (exact agreement, block‐by‐block agreement, and time‐window analysis) were employed in more than 10 of the articles that reported continuous recording. Having identified these currently popular agreement computation algorithms, we explain them to assist researchers, software writers, and other consumers of JABA articles.     

Treatment integrity of school-based interventions with children in the journal of applied behavior analysis 1991-2005

We reviewed all school-based experimental studies with individuals 0 to 18 years published in the Journal of Applied Behavior Analysis (JABA) between 1991 and 2005. A total of 142 articles (152 studies) that met review criteria were included. Nearly all (95%) of these experiments provided an operational definition of the independent variable, but only 30% of the studies provided treatment integrity data. Nearly half of studies (45%) were judged to be at high risk for treatment inaccuracies. Treatment integrity data were more likely to be included in studies that used teachers, multiple treatment agents, or both. Although there was a substantial increase in reporting operational definitions of independent variables, results suggest that there was only a modest improvement in reported integrity over the past 30 years of JABA studies. Recommendations for research and practice are discussed.     

CHARACTERISTICS AND TRENDS IN APPLIED BEHAVIOR ANALYSIS1

Characteristics and trends in applied behavior analysis were examined by evaluating the contents of the Journal of Applied Behavior Analysis (JABA) from 1968 to 1974. Investigations were categorized according to the settings, populations, target behaviors, and research methodology employed. The analyses, completed separately for each dimension, revealed a relatively high proportion of investigations conducted in classrooms, with children and adolescents, and for problematic-disruptive behaviors. Although research in each of these areas did not constitute the majority of JABA investigations, only two categories across each content dimension were required to constitute the majority. The majority of JABA research has been conducted in classrooms and institutional settings, with children and adolescents and retardates of all ages, and for problematic-disruptive and academic behaviors. Despite the relatively restricted focus, the proportion of investigations conducted in naturalistic or open-field settings (e.g., movie theaters, parks, zoos, busses) and, to a lesser extent, day-care facilities has increased in recent years. Also, the focus upon academic responses, training individuals to employ behavior modification, and community relevant behaviors (e.g., employment, littering) has increased. As expected, examination of research design and evaluation techniques revealed extensive reliance upon intrasubject replication designs. Indeed, the majority of investigations have employed reversal (ABAB) designs. Interestingly, the use of multiple-baseline and between-group designs has increased in recent years. Finally, the use of statistical tests, not at all a rarity in JABA investigations, has not systematically increased since the inception of JABA.     

Gender-role conflict scale: College men's fear of femininity

Gender-role conflict exists when gender roles have negative consequences for people. This research reports initial validity and reliability data on measures of gender-role conflict for men. Two measures, Gender Role Conflict Scale I and II (GRCS-I and GRCS-II) were constructed to assess patterns of gender-role conflict described in the literature. GRCS-I assesses men's personal gender-role attitudes, behaviors, and conflicts. GRCS-II assesses men's gender-role conflicts in specific gender-role conflict situations. Both GRCS measures and the Personal Attributes Questionnaire (PAQ) were administered to male college students (N=527). Initial factor-analytic data for GRCS-I and GRCS-II demonstrated eight meaningful factors. Acceptable test-retest and internal consistency reliabilities were found for both measures. MANOVA, ANOVA, and Tukey procedures indicated differences for subjects across the four PAQ categories. Significant gender-role conflict differences across the factors were found for men who were instrumental, expressive, or both instrumental and expressive. Results of these differences are reported, as well as implications for future development of both scales.The authors are indebted to Dr. Nancy Betz (Ohio State University) who generously gathered data for us during the Spring Semester, 1982. This research was supported by a grant from the General Research Fund of the University of Kansas, Lawrence, Kansas. This paper was presented at the American Psychological Association Annual Convention, Washington, DC, August 26, 1982.     

Publication trends in 25 years of the Journal of Applied Behavior Analysis

All articles published in the first 25 years of JABA (1968 to 1992) were reviewed to classify the percentage of articles published in the following categories: (a) type of article, (b) subjects, (c) setting, (d) behavior-change agent, (e) target behavior, (f) use of basic principles, and (g) miscellaneous procedures. Overall percentages and trends are reported in each category. Results indicate an increase in the percentage of articles with participants and target behaviors in developmental disabilities and a decrease in the percentage of studies targeting academic behavior, verbal behavior, and other child behavioral excesses. The most frequent setting continues to be a school; however, there is a clear trend towards community and other naturalistic settings. Results also highlight the increasing complexity and multicomponent nature of JABA interventions. Potential implications for future applications are discussed.     

Self- and cross-citations in the Journal of Applied Behavior Analysis and the Journal of the Experimental Analysis of Behavior: 1993-2003

Self- and cross-citations in JABA and JEAB from 1993 through 2003 were examined. Yearly mean levels of self-citation for JABA and JEAB were 34.9% and 33.2%, respectively. Overall, 7.8% of JABA citations were JEAB articles, and 0.6% of JEAB citations were JABA articles. The former value, but not the latter, is substantially higher than the cross-citation level reported for earlier years. The two JEAB articles most often cited in JABA were published over 20 years ago and are concerned with establishing operations and the matching law.     

A review of the observational data-collection and reliability procedures reported in The Journal of Applied Behavior Analysis

The research published in the Journal of Applied Behavior Analysis (1968 to 1975) was surveyed for three basic elements: data-collection methods, reliability procedures, and reliability scores. Three-quarters of the studies reported observational data. Most of these studies' observational methods were variations of event recording, trial scoring, interval recording, or time-sample recording. Almost all studies reported assessment of observer reliability, usually total or point-by-point percentage agreement scores. About half the agreement scores were consistently above 90%. Less than one-quarter of the studies reported that reliability was assessed at least once per condition.     

Interrater reliability and agreement of performance ratings: A methodological comparison

This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, theT index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearsonr, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.This paper is based, in part, on a thesis submitted to East Carolina University by the second author. Portions of this study were presented at the American Psychological Association meeting in New Orleans, LA, August, 1989. The authors would like to thank Michael Campion and two anonymous reviewers for their comments on earlier drafts of this paper.     

Gender differences in motivations for course selection: Academically talented students in an intensive summer program

This study examines the motivational factors contributing to course selection in a sample of students enrolled in an intensive three-week academic experience for exceptionally talented adolescents. Our sample included 947 gifted adolescents, of whom 3.7% were Black, 73.5% were White, 2.2% Hispanic, 0.1% Native American, 19.8% Asian, and 0.6% categorized themselves as other. Girls and boys selected different types of classes, with course participation falling along traditional gender-stereotyped lines. Boys and girls both performed exceptionally well, and said they chose their course because the subject was interesting, useful for future goals, and challenging. Females more often than males enrolled in classes because they perceived them as challenging, different from usual, not offered at school, and as making them more well-rounded. Males more often than females tended to select classes because they thought they would do well and/or because they viewed these classes as being useful for future schooling or career. However, these gender differences varied by the type of course the students took. The resulting complex picture of students' motivations is discussed in terms of the need to specify both the contexts in which males and females make their academic choices, and the interactive determinants of their motivations.An earlier version of this paper was presented at the Third Annual Esther Katz Rosen Symposium on the Psychological Development of Gifted Children, University of Kansas, Lawrence, Kansas, February 19–20, 1993.     

Effects of verbal imitation and comprehension training on verbal production

This study sought to determine whether some combination of imitation training and comprehension training was necessary to achieve verbal production or whether comprehension training alone was sufficient to result in verbal production. Fifty-one first- and second-grade subjects were trained to imitate eight words (Spanish nouns) for which there were no referents. An additional set of eight words was trained in a comprehension task where overt verbal rehearsal was prohibited. Once criterion of two successive sessions of 100% performance was reached in training for both imitation and comprehension, training modes were reversed so that items initially trained in imitation were then trained in comprehension andvice versa. Results showed that initial comprehension training did result in some verbal production, whereas initial imitation training, as expected, did not. A marked improvement in verbal production performance was observed when initial comprehension training was followed by imitation training. When initial imitation training was followed by comprehension training, verbal production also resulted but not to the degree or with the consistency of performance which marked the comprehension-then-imitation sequence. Furthermore, comprehension training required more trials to reach a similar level of proficiency when it followed imitation training than when it preceded imitation. These data indicate that initial imitation training interferes with acquisition of subsequent comprehension and production responses and that the preferred training sequence is one which initially focuses on comprehension and follows this with verbal imitation.This study was supported by Grant HD 00870 from the National Institute of Child Health and Human Development and Grant NS 10468 from the National Institute of Neurological Diseases and Stroke awarded to the Bureau of Child Research, University of Kansas, Lawrence, Kansas.     

Estimating the Religious Composition of All Nations: An Empirical Assessment of the World Christian Database

The international religious data in the World Christian Database (WCD), and its print predecessor, the World Christian Encyclopedia (WCE) have been used frequently in academic studies and the popular press. Scholars have raised questions about the WCD's estimates categories, and potential bias, but the data have not yet been systematically assessed. We test the reliability of the WCD by comparing its religious composition estimates to four other data sources (World Values Survey, Pew Global Assessment Project,CIA World Factbook, and the U.S. Department of State), finding that estimates are highly correlated. In comparing the WCD estimates for Islamic countries and American Christian adherents with local data sources, we identify specific groups for which estimates differ. In addition, we discuss countries where the data sets provide inconsistent religious estimates. Religious composition estimates in the WCD are generally plausible and consistent with other data sets. The WCD also includes comprehensive nonreligious data. Recommendations regarding the use of the WCD are given.     

Organizational, Nonorganizational, and Intrinsic religiosity and academic dishonesty

The present study was a preliminary examination of the relations among the Organizational, Nonorganizational, and Intrinsic dimensions of religiosity and academic dishonesty. 244 college students completed the Duke Religion Index and nine questions assessing academic dishonesty. Analysis indicated that (1) regardless of sex, High Nonorganizational and Intrinsic religiosity was associated with lower reported rates of academic dishonesty, and (2) there was an interaction between Organizational religiosity and sex, with High Organizational women and men reporting similar rates of academic dishonesty. Furthermore, the frequency of academic dishonesty reported by High Organizational women was higher than the rates reported by Moderate and Minimal Organizational women.     

The impact of the transition to high school on the self-system and perceived social context of poor urban youth

Examined the effects of the normative school transition to senior high school (n=330) on the self-system and perceived school and peer social contexts of poor, black (n=83), European American (n=115), Latino (n=105), and Asian American (n=27) youth in the public school systems of three Eastern urban cities. The only negative effect of the school transition on the self-system was a decline in grade point average (GPA). Concurrently, the school transition was perceived to be associated with changes in the school and peer contexts. Across the transition, students reported increased disengagement from school (i.e., increased social support and extracurricular involvement) and increased engagement with peers (i.e., decreased daily hassles and increased involvement). These changes in the school and peer microsystems, like the changes in the self-system, were also common across race/ethnicity and gender. In addition, transition-associated school changes, and in particular changes in daily academic demands/hassles and involvement in school activities, were associated with changes in the academic dimensions of the self-system (i.e., academic efficacy expectations and GPA). Results and implications for preventive intervention are discussed within a developmental mismatch framework. This research was supported in part by grants from the National Institute of Mental Health (MH43084) and the Carnegie Corporation (B4850) awarded to Edward Seidman, J. Lawrence Aber, LaRue Allen, and Christina Mitchell. We express our appreciation to the children and schools whose cooperation made this study possible.     

Further evidence for score reliability and validity of the Anxiety Depression Distress Inventory-27 scale

We conducted several analyses with data from undergraduate students (N = 309) and active duty US Air Force Security Forces personnel (N = 273) to examine the utility of scores on the Anxiety Depression Distress Inventory-27 (ADDI-27). The three specific dimensions of the ADDI-27 include Positive Affect, Somatic Anxiety, and General Distress, each composed of nine relevant and representative items. Internal consistency reliability estimates for scores on the scales were strong across the study groups (coefficient-α values ≥ .80). Results of analyses using differential item functioning showed that the groups interpreted the contents of the ADDI-27 items similarly. At the scale-level analyses, the undergraduate student sample reported higher somatic and general distress symptoms compared to the Air Force sample. Evidence for concurrent validity was adequate. Examples of the concurrent measures were the meaning in life, mental health functioning, and sources of social support self-report instruments. Taken together, results support the use of the ADDI-27 for assessing clusters of somatic anxiety, depression, and general distress in the current study samples.     

Depressive Symptoms in Mothers and Daughters: Attachment Style Moderates Reporter Agreement

Parents and adolescents show only modest agreement when reporting on depressive symptoms. Drawing from attachment theory and previous research on informant discrepancies, we tested hypotheses about how adolescent attachment style may impact reporting agreement in a sample of 184 low-income mother-adolescent daughter dyads (adolescent mean age = 15.4 (SD = 1.05), maternal mean age = 41.4 (SD = 7.60); 58 % Latina, 26 % African-American/Black, 16 % as non-Hispanic, White). Mothers and adolescents reported on their own and each others’ depressive symptoms and adolescents reported on attachment style. Using a moderated Actor Partner Interdependence Model (APIM) to calculate reporter bias and accuracy estimates, we tested whether attachment style moderated maternal and adolescent accuracy in theoretically consistent ways. Mothers and adolescents showed similar levels of accuracy and bias when reporting on each other. Consistent with hypotheses, we found that adolescents who reported high levels of preoccupation were less accurate when reporting on their mothers because they tended to observe symptoms that their mothers did not endorse. Conversely, mothers were the most accurate in these dyads, potentially because preoccupied adolescents tend to elevate displays of emotional distress. Reporting accuracy was not affected by a dismissive style. These results add to literature indicating that parent-child reporting discrepancies often reflect meaningful information about relationships, and highlight the need to consider different sources of reporting bias and accuracy in assessment and treatment.     

Beyond statistical significance: Five principles for the new era of data analysis and reporting

A crisis of confidence in research findings in consumer psychology and other academic disciplines has led to various proposals to abandon, replace, strengthen, or supplement the null hypothesis significance testing paradigm. The proliferation of such proposals, and their often-conflicting recommendations, can increase confusion among researchers. We aim to bring some clarity by proposing five simple principles for the new era of data analysis and reporting of research in consumer psychology. We avoid adding to researchers' confusion and proposing more onerous or rigid standards. Our goal is to offer straightforward practical principles that are easy for researchers to keep in mind while analyzing their data and reporting their findings. These principles involve (1) interpreting p-values as continuous measures of the strength of evidence, (2) being aware of assumptions that determine whether one can rely on p-values, (3) using theory to establish the applicability of findings to new settings, (4) employing multiple measures of evidence and various processes to obtain them, but assigning special privilege to none, and (5) reporting procedures and findings transparently and completely. We hope that these principles provide researchers with some guidance and help to strengthen the reliability of the conclusions derived from their data, analyses, and findings.     

Discrepancies among mother,child, and teacher reports: Examining the contributions of maternal depression and anxiety

The relationship between maternal affective symptomatology and discrepancies in maternal reports of child symptoms, relative to teacher and child reports, was evaluated in a community sample of 188 children ages 9–12 years. Mothers, teachers, and children were administered a structured interview about child psychopathology. In general, mothers reported more child behavior problems than children and teachers, regardless of maternal symptomatology. However, maternal affective symptoms were associated with discrepancies between mothers' and daughters' reports and between mothers' and teachers' reports of girls' externalizing symptoms. Furthermore, mothers who reported high levels of both anxiety and depressive symptomatology tended to report a large number of symptoms that were not confirmed by either their daughters or teachers. Findings are discussed as possible evidence of the role of maternal affective symptomatology in both actual increases in child symptomatology and maternal reporting distortions. Although maternal depressive and anxiety symptoms shared variance in reporting discrepancies, only anxiety explained unique variance. Consistent with previous studies, cross-informant agreement was modest to moderate (r= .16 to .50) and all informants reported more behavior problems in boys than in girls. This work was undertaken with the support of National Institute of Mental Health Grants MH43909 and MH46717.     

Women in applied behavior analysis

The status of women in applied behavior analysis was examined by comparing the participation of women in the Journal of Applied Behavior Analysis (JABA) to their participation in three similar journals. For all journals, the percentage of articles with at least one female author, the percentage of authors who are female, and the percentage of articles with a female first author increased from 1978 to 1997. Participation by women in JABA was equal to or greater than participation by women in the comparison journals. However, women appeared as authors on papers in special sections of Behavior Modification substantially more often when the editor was female than when the editor was male. In addition, female membership on the editorial boards of JABA, Behavior Modification, and Behaviour Research and Therapy failed to increase from 1978 to 1997. We conclude that a "glass ceiling" reduces the participation of women at the highest levels of applied behavior analysis and related fields.     

Self- and cross-citations in the Journal of Applied Behavior Analysis and the Journal of the Experimental Analysis of Behavior: 1983-1992

We examined self- and cross-citation practices in JABA and JEAB from 1983 through 1992. Mean levels of self-citation for JABA and for JEAB were 22.6% and 36.1%, respectively. Overall, 2.4% of JABA citations were JEAB articles, and 0.6% of JEAB citations were JABA articles, which suggests limited integration of basic and applied research.     

Health Care Reform and Psychologists in Medical Schools: Stress, Challenge, and Change

Psychologists from 45 academic health science centers throughout the United States and Canada gathered at Georgetown University for a national conference November 2–5, 1995 organized by the Association of Medical School Psychologists. This paper introduces the proceedings of that conference described in the next four articles in this issue of the Journal of Clinical Psychology in Medical Settings[Volume 4, number 1]. Papers prepared by the working groups focus upon clinical services, education and training, research, and governance and administration within the academic medical setting.