质性研究中编码者信度的多种方法考察

质性研究中检验编码者信度的方法有归类一致性指数、编码信度系数、相关系数、中位数检验、概化系数等。基于教师胜任力访谈数据集,对编码者信度考察结果表明,归类一致性指数和编码信度系数受相同编码数影响而不稳定,相关系数受数据类型制约,中位数检验受研究设计影响,概化系数则受编码者和编码项目的数量影响。研究中须合理选用。     

Rorschach intercoder reliability for protocol-level comprehensive system variables in an international sample

This study examines the intercoder reliability of Rorschach Comprehensive System (CS; Exner, 2001) protocol-level variables. A large international sample was combined to obtain intercoder agreement for 489 Rorschach protocols coded using the CS. Intercoder agreement was calculated using an Iota coefficient, a statistical coefficient similar to kappa that is corrected for chance. Iota values for the variables analyzed ranged from .31 to 1.00, with 2 in the poor range of agreement, 4 in the fair range, 25 in the good range, and 116 in the excellent range of agreement. Discrepancies between variables are discussed.     

Rorschach Intercoder Reliability for Protocol-Level Comprehensive System Variables in an International Sample

This study examines the intercoder reliability of Rorschach Comprehensive System (CS; Exner, 2001) protocol-level variables. A large international sample was combined to obtain intercoder agreement for 489 Rorschach protocols coded using the CS. Intercoder agreement was calculated using an Iota coefficient, a statistical coefficient similar to kappa that is corrected for chance. Iota values for the variables analyzed ranged from .31 to 1.00, with 2 in the poor range of agreement, 4 in the fair range, 25 in the good range, and 116 in the excellent range of agreement. Discrepancies between variables are discussed.     

Why Not Open the Black Box of Journal Editing in Philosophy? Make Peer Reviews of Published Papers Available

Despite general agreement within philosophy that peer review is indispensable, its fairness and reliability is often questioned. This article suggests that such worries can to a large extent be met by adopting the practice that reviews as well as earlier versions of papers are made publicly available when the final version of a paper is published. This suggestion combines the advantages of transparency with the merits of anonymity of reviewers. While there are obstacles to this suggestion, the article argues that it would be worthwhile to implement it because it can help map patterns of conduct and secure confidence in the fairness and reliability of review procedures and journal editing within philosophy.     

Two Kinds of Learning

Episodic analysis is the separation of a novel into its component episodes —i.e., into short stories each of which has a unitary theme—in order to provide equivalent units for content analysis. Independent coders show two types of disagreement. Neither is serious for content analysis, and both may be minimized by some suggested ground rules. In a study of 13 stylistically different books, intercoder reliability ranged from 76% to 94%. Variations in accuracy were clearly related to literary style. Relevant information on the coding process and reliability, as well as data on the number and length of episodes in the 13 books, are given in tabular form. Problems connected with reliable coding are discussed and some solutions are suggested.     

Content analysis in negotiation research: A review and guide

Content analysis, a technique in which inferences are derived from the communication of interacting parties, is not utilized enough in mainstream negotiation research. The purpose of the present article is to outline the benefits and limitations inherent in the content analysis technique, to provide a comprehensive review of content analysis systems that have been used in the past, and to evaluate the existing systems in an effort to guide readers in the decision process. This article aids the potential consumer—any person considering the use of content analysis in negotiation research—in making informed choices regarding this technique. Choice among existing systems should be based upon one’s research question and upon accepted standards of quality. A system that is relatively comprehensive, well grounded in theory, and demonstrates acceptable levels of reliability is advocated. Specific recommendations regarding quality systems are provided.     

An investigation of the theoretical content of physical activity brochures

New evidence-based physical activity (PA) guidelines and recommendations for constructing messages supplementing the guidelines have been put forth. As well, recent reviews have identified theoretical constructs that hold promise as targets for intervention: self-regulation, outcome expectancies and self-efficacy. The purpose of this study was to examine the integration of messages targeting self-regulation, self-efficacy and outcome expectancies in existing physical activity brochures. Twenty-two PA brochures from Canadian and American National Health Organizations were assessed for their use self-efficacy, self-regulatory processes and outcome expectancies. Brochures were analyzed line-by-line using a modified version of the validated Content Analysis Approach to Theory-Specified Persuasive Educational Communication (CAATSPEC; Abraham, Southby, Quandte, Krahé, & van der Sluijs, 2007). Two independent raters coded a third of the brochures (n = 7). Inter-rater reliability was acceptable for 17 of the 20 categories (rs > .79). Discrepancies in all categories were discussed and agreement was reached. The remaining brochures were coded by one of the two raters. Usage of the three key theoretical constructs accounted for only 36.43% of brochure content (20.23% self-efficacy, 10.40% outcome expectancies, 5.80% self-regulation). Brochures lacked the use of a variety of theoretical strategies, specifically goal-setting, planning and verbal persuasion and rarely highlighted the affective benefits of physical activity. In the future brochures should aim to place increased emphasis on self-regulation, self-efficacy, and affective outcome expectancies.     

Opinion Climates in Social Media: Blending Mass and Interpersonal Communication

Social media's capacity for users to generate, comment on, and forward content (including mass media messages) to other users has created new forms of mass interpersonal communication. These systems render observable processes underlying the formation of opinion climates. Five attributes of contemporary electronic opinion environments can alter the way users gauge, form, and express opinions on topics of public interest: the juxtaposition of mass media and user‐generated content, ideological homogeneity and heterogeneity of online networks, technical ease with which to express opinions, the reach of messages, and networked audiences. These attributes facilitate analysis of theoretical and empirical works from different scholarly traditions, suggesting lines of inquiry that can enrich the analysis of (public) opinion formation via current communication technologies.     

The Challenges of Qualitatively Coding Ancient Texts

We respond to several important and valid concerns about our study (“The Prevalence of Folk Dualism in Early China,”Cognitive Science 35: 997–1007) by Klein and Klein, defending our interpretation of our data. We also argue that, despite the undeniable challenges involved in qualitatively coding texts from ancient cultures, the standard tools used throughout the cognitive sciences—large quantities of data, coders as blind to the hypothesis as possible, intercoder reliability measures, and statistical analysis—allow the noise of randomly distributed interpretative differences to be distinguished from the signal of genuine historical patterns.     

Reliability and convergent validity of the Job Components Inventory

Further evidence is presented on the reliability and validity of the Job Components Inventory (JCI), a recently developed job analysis technique. The JCI covers the use of tools and equipment, physical and perceptual requirements, mathematical requirements, communication requirements, decision-making and responsibility requirements. Results of a study of 75 postmen and postwomen and seven supervisors are reported. Inter-rater reliability, supervisor-job incumbent agreement and inter-office agreement were all found to be high. This study indicates that the JCI can be used to analyse jobs other than low skill youth jobs for which it was originally developed     

Evidence for clinical smoking cessation for adolescents.

OBJECTIVE: This report reviews the evidence that informs the role of health and mental health care providers in addressing youth smoking cessation. DESIGN: Qualitative literature review. RESULTS: Physicians do not consistently screen adolescents for tobacco use and fail to provide recommended cessation advice. Challenges to addressing smoking cessation include the need for procedures to ensure confidentiality and the existence of competing demands to provide other services. Few published studies have specifically addressed the effectiveness of clinical interventions. Interventions that require return visits or follow-up phone contacts are technically difficult to implement in this population. Successful interventions may require resources not available in nonresearch settings. Most studies have used brief clinical intervention as a control condition, making it impossible to evaluate its effectiveness. CONCLUSION: There is little evidence that supports current clinical smoking cessation guidelines for adolescents. More research is needed to develop inexpensive, efficient clinical interventions that can provide youths access to smoking cessation help. Future challenges include reorganizing clinical systems to offer greater counseling by support staff or in electronic formats and to provide effective booster messages and follow-up care in a population that is difficult to track.     

Advances in juvenile adjudicative competence: A 10-year update

This article provides a comprehensive review of juvenile adjudicative competence (AC) literature published between 2010 and 2019. Publications included in this article are peer-reviewed and disseminate original research or provide new commentary on forensic evaluation, policy, or theory. The review is organized in the following sequence: (i) factors associated with juvenile AC, (ii) evaluating juvenile AC (assessment tools and techniques, quality of evaluations, evaluation recommendations), (iii) remediation (remediation recommendations), (iv) systemic issues (inconsistency in statutes and court processes, defense attorneys' concerns about AC, age-related issues, developmental immaturity), and (v) special topics (special populations, international research). Systemic processes and variability in statutes have contributed to current concerns regarding reliability of juvenile AC evaluation and remediation. Clear and consistent standards must be developed to address these problems. Continued research is necessary to clarify how to accurately assess juvenile AC and appropriately remediate those adjudicated incompetent. Practice and policy implications as well as future directions for research are discussed.     

An Inductively Derived Typology of Verbal Aggression and Its Association to Distress

Threats elicit distress in the form of negative affect. To examine how communication functions as a threat, this study developed a typology of verbal aggression and then examined its association to distress. To develop the typology, similarity data for 100 written aggressive messages were submitted to a hierarchical cluster analysis. The results revealed messages clustered into three broad domains of attack: group membership, personal failings, and relational failings. The typology demonstrated reliability, predictive validity, and content validity. In addition, a multidimensional scaling was conducted on an audiotaped subset of the 100 messages. The results revealed aggressive messages can be characterized along three dimensions (disposition, target, and force) that define the nature of an attack. These three dimensions elicited different emotional reactions in recipients and suggest nonverbals allow recipients to discriminate aggressive messages in terms of the tone and the force of an attack.     

Naturalistic observation in clinical assessment

Summary The development of the Behavioral Coding System (BCS) used by the Social Learning Project has encompassed approximately 8 years of clinical and research experience with naturalistic observation as a clinical assessment tool. The BCS, while originally designed to accomplish certain broad purposes, illustrates a solution to an assessment task that should be applicable to other research and clinical settings in which naturalistic observation of family interactions are needed.A variety of reliability analyses, ranging from traditional interobserver agreement among coders to generalizability analyses, have supported the measurement precision of the BCS scores, for their intended purposes. In conducting this series of investigations, certain problems in psychometric analysis of observation data have arisen and been documented. Most notably, the tradition of estimating reliability via interobserver agreement has been questioned, mainly on the grounds that behavioral complexity intrudes into such analyses in ways that suggest that current observer reliability estimates may be substantially biased. The usefulness of generalizability theory is argued, particularly for observational data collected under varying assessment conditions which may influence behavioral scores.Three types of validity have been reported for BCS scores: content, concurrent, and construct validity. The BCS has favorably withstood these psychometric investigations, showing that the behavioral measures are justified on content grounds, that outside reports of behavior coincide satisfactorily with the BCS scores, and that expected behavioral changes following treatment are readily indexed by the BCS scores.Excerpts and abstracts from chapters by Jones, Reid, and Patterson (1975, pp. 42–95), and from Reid (1977).The assessment procedures were developed as part of an extended series of grants from the Section on Crime and Delinquency, National Institute of Mental Health.J. B. Reid (1977) has recently edited a manual which presents a much fuller report of the topics covered in this report, including: a more extensive literature review, operational definitions of code categories, normative data, video training tapes, and procedures for training observers.     

Outpatient standards of care and the suicidal patient.

This article is a review of the legal and clinical literature on standards of care for nonhospitalized adult suicidal patients. The authors discuss effective assessment, management and treatment procedures that balance the need for high-quality care by a reasonable and prudent practitioner with the requirements of court-determined and statutory standards. Through a review of malpractice claims data and through an examination of the clinical literature we detail the essential guidelines for sound assessment, intervention, and management procedures. The authors specifically discuss common failure situations in outpatient care (e.g., problems in pharmacotherapy, the decision to hospitalize, inappropriate and dual relationships, the assessment of imminence and lethality, and so on). Details regarding practical considerations in developing an outpatient standard of care are provided, as are suggestions that such a standard of care must include an ongoing assessment of the therapeutic alliance.     

Exploratory factor analysis in transportation research: Current practices and recommendations

Exploratory factor analysis (EFA) is a widely used statistical method in traffic and transportation research, particularly for the development and validation of measurement instruments. This article critically examines current practices in conducting and reporting EFA in published transportation studies. One hundred and eighty papers published between 2016 and 2018 were examined, of which eighty-two were included in the present study after applying eligibility criteria. The review suggests that the quality of EFA reported in the field is routinely poor: (a) researchers fail to provide sufficient information to be able to adequately assess the appropriateness and quality of both the input data and the reported output; and (b) the decisions underlying the choices of EFA methods are not justified and rely mostly on procedures advised against, particularly the Little-Jiffy approach. In summary, a significant gap between current practice and experts' recommendations exists. We provide some guidelines that may help in conducting, reporting and reviewing EFA in transportation research.     

Review of State Medication Policies/Guidelines Regarding Psychotropic Medications in Public Schools

It is currently estimated that up to 6 million children take psychotropic medications for the treatment of mental health problems. The highest prevalence rates (50–76 %) are typically found among students with special needs, especially among those with ADHD and emotional disturbance. The Individuals with Disabilities Education Act (IDEA) and Section 504 of the Rehabilitation Act of 1973 require medications be administered by schools whenever it is deemed necessary for the child to have access to educational services. However, these requirements do not extend to all students, nor do they provide guidance regarding the safest and most efficacious manner in which psychotropic medications should be administered. The authors reviewed existing state medication policies and guidelines to assess the level of guidance currently provided to school staff. Results showed the vast majority of states (48) provided guidance related to the administration of medications to students, with slightly fewer (44) states discussing required documentation procedures. Surprisingly, only 15 states addressed monitoring students for adverse side effects of medications, and even fewer (11) states specifically discussed psychotropic medication in their policies/guidelines. The vast majority (42) of states also addressed requirements for the safe and proper storage of medications, while slightly more than half of all states (31) provided any guidance regarding training of unlicensed personnel (e.g., secretaries) who frequently administer medications to students. The authors highlight several model guidelines/policies and review recommendations for best practice.     

Reliability in Content Analysis

In a recent article in this journal, Lombard, Snyder‐Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests, compared the virtues and drawbacks of five popular reliability measures, and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data. This article proposes three conditions for statistical measures to serve as indices of the reliability of data and examines the mathematical structure and the behavior of the five coefficients discussed by the authors, as well as two others. It compares common beliefs about these coefficients with what they actually do and concludes with alternative recommendations for testing reliability in content analysis and similar data‐making efforts.     

Critical elements of the crisis intervention team model of jail diversion: An expert survey

The Crisis Intervention Team (CIT) model of jail diversion is a promising approach to addressing the over-involvement of people with mental illness with the criminal justice system. Despite its popularity and promising empirical support, the literature has yet to clarify CIT's critical elements. The aim of this study was to assess the degree to which experts agreement on the importance and perceived implementation of the critical elements of CIT. Study 1 used a literature review to cull potential elements. Three experts familiar with the CIT model reviewed these elements. Study 2 utilized an online survey of 55 CIT co-ordinators/officers and published experts to assess agreement regarding importance and degree the element is implemented. A preliminary list of 36 elements was identified with a high level of expert agreement on their importance regarding importance and implementation. This study developed a preliminary list of elements of the CIT's program. Further research should develop consensus, explore elements' association with outcomes, and differentiate elements from those common to all jail diversion programs. Copyright © 2010 John Wiley & Sons, Ltd.     

A review of behavioral interventions to enhance sports performance

Research on applied behavior analysis and sports performance has become an increasingly popular subject. It involves using behavioral measures to assess the utility of numerous interventions to enhance athletic performance. We identified 101 studies published since the late 1960s focusing on behavioral interventions to enhance sports performance. To determine the breadth of the current research, we identified each sport and the different interventions used to improve performance. Of the studies included in this review, research has investigated 23 interventions with 21 sports. The purpose of this paper is to review the current literature regarding what procedures have been evaluated, the results of this research, some of the limitations, and future directions of behavioral research on sports performance enhancement. In doing so, this review will allow readers to easily browse the current literature by the sport or intervention of interest.