Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Considerations in the choice of interobserver reliability estimates

Two types of interobserver reliability values may be needed in treatment studies in which observers constitute the primary data-acquisition system: trial reliability and the reliability of the composite unit or score which is subsequently analyzed, e.g., daily or weekly session totals. Two approaches to determining interobserver reliability are described: percentage agreement and "correlational" measures of reliability. The interpretation of these estimates, factors affecting their magnitude, and the advantages and limitations of each approach are presented.     

A descriptive study of e-counsellor attitudes,ethics, and practice

Introduction: The purpose of this study was to examine questions related to the process, perceived outcomes, and ethics of therapists who provide counselling and therapy online. Method: An online survey was conducted (N=93) with identified e-counsellors with at least a Master's Degree. Questions included demographic information and Likert-type scales related to extent of practice, theoretical orientation, training and supervision, attitudes about appropriate practice, referral, legal and ethical issues, practice difficulties, and therapist satisfaction. Results: Overall, e-counsellors are satisfied with their practice and believe it is effective. They generally do not have formal training or supervision in online practice. Current e-counselling appears to be a part-time, supplemental practice with little overlap to FTF practice. There was little agreement among respondents on attitudes, practice, ethical issues, and knowledge of regulations related to e-counselling. Discussion: Lack of consensus about ethical obligations and practice suggest the need for formal training in e-counselling by professional programmes and international cooperation in formulating practice ethics. Areas for further research are discussed.     

Comparing Traditional and Ebook-Augmented Parent-Child Interaction Therapy (PCIT): A Randomized Control Trial of Pocket PCIT

Disruptive behavior in young children is one of the most common referrals to behavioral health providers. While numerous effective parenting programs, such as parent–child interaction therapy (PCIT), exist for improving children’s behaviors, challenges with treatment engagement and retention limit the intended positive impact on child and caregiver outcomes, particularly for racial and ethnic minority families. In an effort to address barriers contributing to poor engagement and retention in traditional PCIT service delivery and among ethnic and racial minority families, a multimedia PCIT ebook was developed and evaluated. In a sample of the general public that utilized the ebook, users were found to be more engaged in viewing embedded videos within the ebook that were related to expert skill explanations and skill demonstrations than caregiver testimonies. A randomized controlled trial was also conducted to evaluate the extent that the ebook + PCIT improved treatment engagement, retention, parenting skills, skill acquisition efficiency, and child behavior above and beyond traditional PCIT. Participating families were randomly assigned to either the traditional PCIT (n = 71) or ebook + PCIT (n = 107) group using an online random number generator. Forty-nine caregivers (traditional PCIT n = 24, ebook + PCIT n = 25) were excluded from analyses because they were lost to follow-up during the intervention. Families in both the traditional PCIT and ebook + PCIT groups demonstrated generally equivalent positive outcomes in treatment engagement (i.e., attendance, treatment length, completion rate) and caregiver skill acquisition efficiency at midtreatment, posttreatment, and 3-month follow-up. The addition of the ebook to PCIT also reduced child disruptive behavior at midtreatment, above and beyond traditional PCIT, but not at posttreatment or follow-up. Clinical implications and future directions are discussed.     

Maintenance of Relationship and Individual Functioning Gains Following Online Relationship Programs for Low‐Income Couples

Initial evidence suggests that gains in relationship functioning from brief, web‐based programs are maintained through one year following the intervention; however, whether these results generalize to a low‐income sample is unclear. Furthermore, previous research from in‐person couple therapy suggests there may be different shapes of maintenance slopes for behavioral versus acceptance‐based techniques. This study contacted 668 individuals who enrolled in online behavioral (ePREP) or acceptance‐based (OurRelationship) programs one year following completion of the program. Multilevel modeling was used to examine linear and quadratic rates of change in the year following the online intervention as well as total amount of change from pretreatment to 12‐month follow‐up for both relationship and individual functioning. The majority of couples who responded continued to be in a relationship with the same partner (68.3%). Examinations of relationship functioning indicated couples in both programs maintained their gains over follow‐up (i.e., no significant linear or quadratic changes), with medium‐to‐large within‐group effect sizes from pre‐ to one‐year follow‐up. There were no significant differences in relationship outcomes between OurRelationship and ePREP. Similarly, examinations of individual functioning outcomes indicated couples maintained their gains over follow‐up or continued to improve. In total, couples experienced small‐to‐medium within‐group effect sizes from pretreatment to one‐year follow‐up, with larger effects for individuals who were initially distressed. These results suggest that online programs create lasting change for low‐income couples in relationship and individual functioning, with minimal differences between behavioral and acceptance‐based orientations.     

Parent-child interactions during joint engagement with touchscreen technology: A comparison of younger versus older toddlers

With a plethora of touchscreen apps aimed at young children, parents are receiving mixed messages about the appropriateness of such technology for their toddlers. The American Academy of Pediatrics (2016) advises limited engagement with digital media for this age group and encourages parents to co-engage with children when they are using screens. However, very little is known about parent-child interaction in the context of joint engagement with digital screen media in the toddler years. This study observed 56 toddlers (M = 32.5 months old; 53 % female) and a parent (52 mothers; 4 fathers) performing a 3-minute drawing task on a touchscreen tablet (digital condition), and on an Etch-A-Sketch (non-digital condition) using a repeated measures design. Observations were analysed using global ratings of dyadic interaction, comparing warmth, cooperation and conflict between digital and non-digital conditions. A mixed MANCOVA analysis, controlling for levels of daily usage of touchscreens, revealed lower levels of parent-child cooperation and warmth in the digital condition compared to the non-digital condition. In addition, there was a main effect of age with younger dyads displaying less cooperation overall, particularly in the digital condition where interactions were also less warm. Results suggest that co-engaging with digital technology can be a challenging and potentially emotionally charged context for both parents and young children. Younger toddlers, especially, may be more likely to experience less cooperative interactions when co-engaging with digital technology with a parent. Results are discussed in relation to developmental differences between 2- and 3-year olds, and the need for more nuanced guidance for parents supporting young children’s interaction with digital media.     

Choosing what to choose from: Preference for inclusion over exclusion when constructing consideration sets from large choice sets

Decision making is a two‐stage process, consisting of, first, consideration set construction and then final choice. Decision makers can form a consideration set from a choice set using one of two strategies: including the options they wish to further consider or excluding those they do not wish to further consider. The authors propose that decision makers have a relative preference for an inclusion (vs. exclusion) strategy when choosing from large choice sets and that this preference is driven primarily by a lay belief that inclusion requires less effort than exclusion, particularly in large choice sets. Study 1 demonstrates that decision makers prefer using an inclusion (vs. exclusion) strategy when faced with large choice sets. Study 2 replicates the effect of choice set size on preference for consideration set construction strategy and demonstrates that the belief that exclusion is more effortful mediates the relative preference for inclusion in large choice sets. Studies 3 and 4 further support the importance of perceived effort, demonstrating a greater preference for inclusion in large choice sets when decision makers are primed to think about effort (vs. accuracy; Study 3) and when the choice set is perceived as requiring more effort because of more information being presented about each alternative (vs. more alternatives in the choice set; Study 4). Finally, Study 5 manipulates consideration set construction strategy, showing that using inclusion (vs. exclusion) in large choice sets leads to smaller consideration sets, greater confidence in the decision process, and a higher quality consideration set.     

mMotiv8: A smartphone‐based contingency management intervention to promote smoking cessation

Cigarette smoking is the leading preventable cause of death and illness in the United States. We tested the usability, acceptability, and efficacy of a smartphone‐based contingency management treatment to promote cessation. We used a nonconcurrent multiple‐baseline design. Participants (N = 14) provided breath carbon monoxide (CO) samples by using a CO meter that was connected to the user's smartphone. An app (mMotiv8) housed on participants' smartphones automatically captured pictures of the CO sampling procedure to validate the end user's identity, and it prompted submissions via a push message delivered to participants' smartphones. Participants earned a $10 incentive for daily abstinence, which was added to a reloadable debit card. Overall, 4% of the CO samples were negative during baseline, and 89% were negative during treatment. Self‐reported usability and acceptability were high, and 85% of the prompted samples were submitted. A smartphone intervention could be scalable and reduce the health consequences and costs associated with cigarette smoking, particularly in rural and low‐income populations.     

Computer‐based training to teach observers to accurately score problem behavior using fast forwarding at 5x normal speed

The current study evaluated whether a computer‐based training program could improve observers' accuracy in scoring discrete instances of problem behavior at 5x normal speed using a multiple‐baseline design across subjects. During pretraining and posttraining, observers attempted to score multiple examples of problem behavior at 5.0x without feedback. During training, participants scored multiple examples of problem behavior at 5.0x with automated feedback. Researchers measured omission (missing problem behavior) and commission (scoring other behavior as problem behavior) errors and the total duration of scoring time to determine the observers' accuracy and efficiency, respectively. After training, all participants scored instances of problem behavior with less than 11% error using 5.0x. The time required to score the videos across 90‐min observations was reduced by 66%. Results extend previous evaluations of fast forwarding by demonstrating that the training program could be used to teach observers to accurately score problem behavior using a speed faster than 3.5x.