Interobserver agreement and procedural fidelity: An odd asymmetry

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior-analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.     

Socially Situated Transmission: The Bias to Transmit Negative Information is Moderated by the Social Context

Cultural evolutionary theory has identified a range of cognitive biases that guide human social learning. Naturalistic and experimental studies indicate transmission biases favoring negative and positive information. To address these conflicting findings, the present study takes a socially situated view of information transmission, which predicts that bias expression will depend on the social context. We report a large-scale experiment (N = 425) that manipulated the social context and examined its effect on the transmission of the positive and negative information contained in a narrative text. In each social context, information was progressively lost as it was transmitted from person to person, but negative information survived better than positive information, supporting a negative transmission bias. Importantly, the negative transmission bias was moderated by the social context: Higher social connectivity weakened the bias to transmit negative information, supporting a socially situated account of information transmission. Our findings indicate that our evolved cognitive preferences can be moderated by our social goals.     

Pre‐service teacher‐delivered behavioral skills training: A pyramidal training approach

This study evaluated the effectiveness of using a university‐based pyramidal approach (i.e., train the trainer) to teach four pre‐service teachers to train another person. These pre‐service teachers were taught to use behavioral skills training (BST) techniques to train other professionals to use an evidence‐based practice. Transfer to a generalization trainee was probed and three participants maintenance of BST fidelity was probed one year later, after the participants had begun teaching. Results indicate that the participants quickly acquired and maintained a high degree of fidelity with BST. Implications for research and practice are discussed.     

Examiner error in curriculum-based measurement of oral reading

Although curriculum based measures of oral reading (CBM-R) have strong technical adequacy, there is still a reason to believe that student performance may be influenced by factors of the testing situation, such as errors examiners make in administering and scoring the test. This study examined the construct-irrelevant variance introduced by examiners using a cross-classified multilevel model. We sought to determine the extent of variance in student CBM-R scores attributable to examiners and, if present, the extent to which it was moderated by students' grade level and English learner (EL) status. Fit indices indicated that a cross-classified random effects model (CCREM) best fits the data with measures nested within students, students nested within schools, and examiners crossing schools. Intraclass correlations of the CCREM revealed that roughly 16% of the variance in student CBM-R scores was associated between examiners. The remaining variance was associated with the measurement level, 3.59%; between students, 75.23%; and between schools, 5.21%. Results were moderated by grade level but not by EL status. The discussion addresses the implications of this error for low-stakes and high-stakes decisions about students, teacher evaluation systems, and hypothesis testing in reading intervention research.     

Coding Counsellor Behaviour in Motivational Interviewing Sessions: Inter‐Rater Reliability for the Swedish Motivational Interviewing Treatment Integrity Code (MITI)

The aim of this study was to evaluate inter‐rater reliability when using the Swedish version of the Motivational Interviewing Treatment Code (MITI) as an adjunct to MI training, clinical practice and research. Coders were trained to use the MITI for scoring taped sessions. The 4‐month basic training had a duration of 39 hours. Following training, 60 audio‐taped live interviews were randomly assigned for MITI coding. Mean intra‐class correlation (ICC) coefficients were calculated for 7 coders across all pairs of coders. Cronbach's alpha was calculated to estimate the covariance between each pair across their common interviews. Six months later, a second inter‐rater reliability test was performed, when 5 coders coded the same 15 randomly selected tapes. At the second reliability testing the mean ICC was 0.81 and the mean Cronbach's alpha was 0.96. However, the ICC varied for different sub‐variables of the MITI, ranging from 0.42 empathy to 0.79 for number of Closed questions. In conclusion, MITI shows promising potential to be a reliable tool to confirm and enhance MI training as well as practice in clinical settings and in evaluating MI integrity in clinical MI research. However, coder assessment of empathy and MI‐spirit, “global” variables, requires further refinement.     

Validation of a reflective thinking instrument for third-year undergraduate nursing students participating in high-fidelity simulation

Background: nursing students are required to think reflectively in both real and simulated clinical practice. Although the Reflective Thinking instrument is reliable in its measurement of reflective thinking, its validity is unknown. Method: confirmatory factor analysis was undertaken in an iterative manner within a non-equivalent control-group study to measure nursing students’ reflective thinking and satisfaction with high-fidelity simulation. The validity and reliability of the Reflective Thinking instrument was tested. Results: the resulting instrument consisted of 15 items across four factors. The final model was plausible as it demonstrated ‘goodness of fit’; however, it had neither convergent or discriminatory validity, nor reliability. Internal consistency was fair to reasonable for the four subscales. Conclusion: the testing of the Reflective Thinking instrument using confirmatory factor analysis is a vital initial step in the modification of this instrument. Further development using exploratory factor analysis will enhance the instrument’s relevance within high-fidelity simulation.     

Training of practitioners and beliefs about family skills in family-based prevention programmes

To ensure close adherence to the contents of a family-based cognitive-behavioural programme, and maintain a high standard of quality in its implementation, it is important that practitioners’ attitudes and beliefs regarding family skills coincide with those of the programme. This paper describes and analyses (observed) potential changes in these beliefs, after training courses held prior to the application of a prevention programme for families: the Family Competence Programme. A questionnaire was devised to examine practitioners’ beliefs regarding family skills; and completed both before and after the training course. The results point to significant changes in family dynamics, realistic expectations and active listening. Thus, it can be demonstrated that preliminary training courses promote practitioner attitudes and beliefs which are more coherent with a subsequent programme’s approach.     

论曾国藩的忠孝观

在中国传统伦理文化中 ,忠孝是两个重要的范畴。曾国藩的“先于孝而后于忠”忠者观是值得研究的。     

A Randomized Trial to Identify Accurate Measurement Methods for Adherence to Cognitive-Behavioral Therapy

Clinician fidelity to cognitive behavioral therapy (CBT) is an important mechanism by which desired clinical outcomes are achieved and is an indicator of care quality. Despite its importance, there are few fidelity measurement methods that are efficient and have demonstrated reliability and validity. Using a randomized trial design, we compared three methods of assessing CBT adherence—a core component of fidelity—to direct observation, the gold standard. Clinicians recruited from 27 community mental health agencies (n = 126; M age = 37.69 years, SD = 12.84; 75.7% female) were randomized 1:1:1 to one of three fidelity conditions: self-report (n = 41), chart-stimulated recall (semistructured interviews with the chart available; n = 42), or behavioral rehearsal (simulated role-plays; n = 43). All participating clinicians completed fidelity assessments for up to three sessions with three different clients that were recruited from clinicians’ caseloads (n = 288; M age = 13.39 years SD = 3.89; 41.7% female); sessions were also audio-recorded and coded for comparison to determine the most accurate method. All fidelity measures had parallel scales that yielded an adherence maximum score (i.e., the highest-rated intervention in a session), a mean of techniques observed, and a count total of observed techniques. Results of three-level mixed effects regression models indicated that behavioral rehearsal produced comparable scores to observation for all adherence scores (all ps > .01), indicating no difference between behavioral rehearsal and observation. Self-report and chart-stimulated recall overestimated adherence compared to observation (ps < .01). Overall, findings suggested that behavioral rehearsal indexed CBT adherence comparably to direct observation, the gold-standard, in pediatric populations. Behavioral rehearsal may at times be able to replace the need for resource-intensive direct observation in implementation research and practice.