Conceptual,methodological, and clinical issues in cognitive-behavioral assessment

This article reviews cognitive-behavioral assessment principles and procedures. The functions of such assessments are noted, and issues and advances in cognitive assessment are discussed. The need for integrated cognitive, behavioral, and affective assessments is then explored. Finally, conceptual methodological, and clinical issues in the integration of assessments and study of reciprocal determinants are presented.Preparation of this paper was supported in part by a General Research Board Faculty Research Award from the University of Maryland. The author would like to thank Steve Graham, Donald Meichenbaum, Barbara Keogh, and Bernice Wong for their feedback during the preparation of this paper.     

Response competition and stimulus preference in the treatment of automatically reinforced behavior: a comparison

Clinicians are particularly challenged by the development of interventions for behavior maintained by automatic reinforcement because reinforcers that maintain the responses often cannot be directly observed or manipulated. Researchers have conducted either preference assessments or competing items assessments when developing effective treatments for behavior maintained by automatic reinforcement. However, interventions based on these assessments have not been directly compared. The current study evaluated procedures to make such a comparison. High-competition items resulted in greater reductions in vocal stereotypy than did high-preference items for a preschool boy with autism.     

Increasing accurate preference assessment implementation through pyramidal training

Preference assessments directly evaluate items that may serve as reinforcers, and their implementation is an important skill for individuals who work with children. This study examined the effectiveness of pyramidal training on teachers' implementation of preference assessments. During Experiment 1, 3 special education teachers taught 6 trainees to conduct paired-choice, multiple-stimulus without replacement, and free-operant preference assessments. All trainees acquired skills necessary to implement preference assessments with 90% or greater accuracy during the training sessions and demonstrated generalization of skills to their classrooms or clinic. During Experiment 2, 5 teachers who served as trainees in Experiment 1 trained 18 preschool teachers. All preschool teachers met the mastery criterion following training. Training teachers to implement preference assessments may increase teachers' acceptance and use of behavior-analytic procedures in school settings.     

Perceived Risks and Benefits in a Text Message Study of Substance Abuse and Sexual Behavior

To inform ethical procedures for human subjects research using mobile health (mHealth), we examined perceived risks and benefits of study participation among emerging adults (N = 54) with drug use who completed text message assessments of substance use and sexual behaviors. Most participants reported comfort with participation and some reported perceived benefits, such as improved relationships. Perceived risks were infrequently reported, including negative emotions and legal or financial concerns. In conclusion, participants from a vulnerable population reported few perceived harms of participation in longitudinal mHealth assessments of sensitive behaviors. Researchers should continue characterizing participants’ perspectives on ethical aspects of mHealth research.     

A Comparative Review of Current Practices in Personality Assessment Norming

Because consequential personnel decisions are often made by comparing an individual against a norm group, having accurate and representative norms is as important as having sound selection procedures. Existing standards provide specific guidance on selection procedures, but are less clear on norms. To fill this gap and define best practices, we (a) review existing literature and guidelines; (b) review current applied practices using 30 personality assessments offered in the United States, United Kingdom, and Canada; and (c) provide recommendations for strengthening existing guidelines and helping publishers develop, present, and apply norms for personality assessments.     

Toward Increasing Fairness in Score Scale Calibrations Employed in International Large-Scale Assessments

In this article, we investigate the creation of comparable score scales across countries in international assessments. We examine potential improvements to current score scale calibration procedures used in international large-scale assessments. Our approach seeks to improve fairness in scoring international large-scale assessments, which often ignore item misfit in score scale calibrations. We also seek to obtain improved model-data fit estimates when calibrating international score scales. To this end, we examine the use of two alternative score scale calibration procedures: (a) a language-based score scale and (b) a more parsimonious international scale wherein a large proportion of international parameters are used with a subset of country-based parameters for items that misfit in the international scale. In our analyses, we used data from all 40 countries participating in the Progress in International Reading Literacy Study. Our findings revealed that current score scale calibration procedures yield large numbers of misfitting items (higher than 25% for some countries). Our proposed approach diminished the effects of proportion of item misfit on score scale calibrations and also yielded enhanced model-data fit estimates. These results lead to enhancing confidence in measurements obtained from international large-scale assessments.     

Using an abbreviated assessment to identify effective error‐correction procedures for individual learners during discrete‐trial instruction

Previous research comparing the effectiveness of error‐correction procedures has involved lengthy assessments that may not be practical in applied settings. We used an abbreviated assessment to compare the effectiveness of five error‐correction procedures for four children with autism spectrum disorder or a developmental delay. During the abbreviated assessment, we sampled participants' responding with each procedure and completed the assessment before participants reached our mastery criterion. Then, we used the results of the abbreviated assessment to predict the most efficient procedure for each participant. Next, we conducted validation assessments, comparing the number of sessions, trials, and time required for participants to master targets with each procedure. Results showed correspondence between the abbreviated assessment and validation assessments for two of four participants and partial correspondence for the other two participants. Findings suggest that a brief assessment may be a useful tool for identifying the most efficient error‐correction procedure for individual learners.     

Issues associated with repeated neuropsychological assessments

Distinguishing practice effects from other factors in repeated neuropsychological assessments are discussed in the context of research studies and clinical/forensic assessments. Potential methodological procedures for reducing the impact of practice effects in research settings are outlined. In contrast, the potential clinical utility and interpretation of practice effects in clinical assessments and forensic evaluations are highlighted.     

IS COGNITIVE COMPLEXITY LOQUACITY? A REPLY TO POWERS,JORDAN, AND STREET

Powers, Jordan, and Street recently presented evidence suggesting that assessments of cognitive complexity obtained from a widely used procedure may be significantly influenced by general loquacity and verbal complexity. This report argues that Powers et al.'s results are an artifact of the procedures employed, and presents additional evidence bearing on the relationship between cognitive complexity and general verbal abilities. In one study, conceptually independent assessments of cognitive complexity, loquacity, and writing speed were obtained from a group of college students. Cognitive complexity was found generally unrelated to both writing speed and loquacity. In a second set of studies, independent assessments of interpersonal cognitive complexity and two indices of general verbal abilities were obtained from three diverse populations (college students, mothers of young children, and grade-school children). Only low and generally nonsignificant relationships were found between cognitive complexity scores and the measures of verbal facility. These results are discussed within the framework of a construct! vist theory of cognitive development.     

Further evaluation of the use of preference categories to identify novel reinforcers: A systematic replication

Preference assessments are frequently used to identify reinforcers for individuals with autism spectrum disorder and intellectual and developmental disabilities. Although well‐established procedures have been shown to identify preferred stimuli for individual items that can be used in skill acquisition and behavior reduction programs, little research has been conducted on identifying categories of preferred items. In this study, paired‐stimulus preference assessments were conducted with 4 individuals with autism spectrum disorder. Researchers classified the edible stimuli as belonging to 1 of 4 categories: chocolate (e.g., chocolate chips and M&M's^®), salty/crunchy (e.g., chips and crackers), gummy (e.g., Swedish Fish^® and Starburst^®), or fruit/vegetable (e.g., grape and apple). Preference hierarchies were identified for individual stimuli and for categories of stimuli. For all participants, at least 3 of the 4 most preferred items came from the same category. Novel items (i.e., items not included in the preference assessments) identified by the researchers as belonging to the high‐preference category functioned as effective reinforcers during subsequent reinforcer assessments for all participants. This finding suggests that clinicians could identify likely effective edible reinforcers based on an individual's categorical preference without explicit testing.     

A Comparison of Function-Based Differential Reinforcement Interventions for Children Engaging in Disruptive Classroom Behavior

This study provides a direct comparison of differential reinforcement of other behavior (DRO) and differential reinforcement of alternative behavior (DRA). Participants included three children in center-based classrooms referred for functional assessments due to disruptive classroom behavior. Functional assessments included interviews and brief functional analyses. An alternating treatments design was used to evaluate the relative effectiveness of function-based DRO and DRA interventions. Results indicated that both intervention procedures effectively reduced disruptive behavior, but the DRA procedure consistently resulted in greater reductions in disruptive behavior across all participants. Results are discussed in terms of directions for future functional assessment and intervention research as well as implications for applied practice.     

Neonatal stress and coping in intensive care

Examples are given of research on (a) stressful procedures and assessments in the neonatal intensive care nursery and (b) natural caregiving stimulation and self-comforting behaviors that alleviate these stressful experiences. The data from the studies on stressful procedures and assessments revealed that (a) weaning from mechanical ventilation was associated with increased plasma cortisol levels; (b) heelsticks and tube insertions for gavage feedings were accompanied by decreases in oxygen tension (TcPO2); and (c) Brazelton neonatal behavior assessments were followed by increased cortisol levels and decreased growth hormone levels. To alleviate stress, various interventions were tried including caregiver comforting techniques and giving the neonates opportunities to comfort themselves by sucking on pacifiers. The caregiver intervention studies revealed that (a) a simple placement of hands on the head and abdomen of the NICU neonate resulted in a lesser need for oxygen and (b) massaging the NICU neonate resulted in greater weight gain and earlier discharge. In those studies that explored the infant's self-comforting, nonnutritive sucking behavior, the following results emerged: (a) sucking on pacifiers during heelsticks resulted in less crying and lower heart rate and respiration and (b) sucking on pacifiers during tube feedings resulted in smoother bottle feedings later and in greater weight gain. Thus, it appears that several forms of cost-effective interventions can alleviate these stressful procedures and facilitate infant growth and development.     

A Field Experiment Comparing Anchored and Unanchored Criteria Weights in the Analytic Hierarchy Process

This study reports the results of a field experiment comparing the predictive validity of two approaches to multicriteria assessments: the absolute measurement mode of AHP and the absolute measurement mode of linking pin AHP. The questioning procedures of the two differ only in that the former employs unanchored criteria weight assessments and the latter anchored criteria weight assessments. The decision task required insurance agents to respond to a mailed questionnaire in which they evaluated non-monetary incentives (contests) according to (1) the public recognition received from winning, (2) the criteria for winning and (3) the nature of the reward. There were four levels for each dimension. A between subjects design was used, with each subject receiving one of the two methods. In addition, all subjects divided 100 points among four contests and these hold-out assignments were employed as a validity check. Linking pin AHP was found to be superior to conventional AHP in this experiment, lending weight to the argument that the use of unanchored criteria weights is problematic. © 1997 by John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 6 : 140–149 (1997) No. of Figures: 3. No. of Tables: 5. No. of References: 24.     

Response-restriction analysis: II. Alteration of activity preferences

We used response-restriction (RR) assessments to identify the preferences of 7 individuals with mental retardation for a variety of vocational and leisure activities. We subsequently increased their engagement in nonpreferred activities using several procedures: response restriction per se versus a Premack-type contingency (Study 1), supplemental reinforcement for engagement in target activities (Study 2), and noncontingent pairing of reinforcers with nonpreferred activities (Study 3). Results indicated that preferences are not immutable and can be altered through a variety of relatively benign interventions and that the results of RR assessments may be helpful in determining which types of procedures may be most effective on an individual basis.     

The Effect of Method Characteristics on Retest Score Gains and Criterion-Related Validity

Purpose

We sought to empirically assess the effect of predictor method characteristics (test form, item-type, and test-type) on retest score change associated with an invariant construct—general mental ability (GMA)—and to evaluate the effect of retesting on the criterion-related validity of assessments that vary in their susceptibility to retest effects.

Design

Three hundred seven individuals completed a battery of GMA assessments. After a 6-week interval, participants returned to the testing site to retest using both alternate and identical forms of the initial assessments.

Findings

Greater score gains were observed on assessments comprising heterogeneous item-types than homogeneous item-types, and on performance-based assessments than self-report assessments. However, despite variations in score gains, the relationships between the initial test scores and criterion scores were no different than the relationships between retest scores and criterion scores for all assessments.

Implications

Tests and procedures that reduce reliance on test- or item-specific knowledge and skill may help minimize score changes due to retesting across multiple administrations. Moreover, under the boundary conditions present in this study, the criterion-related validity of ability assessments may not be affected by increases in test-specific knowledge and skills.

Originality/Value

Despite the prevalence and industry support of retesting, a comprehensive understanding of retest score change still eludes researchers and practitioners. This ambiguity may be due in part to neglecting the method-construct distinctions in the retest literature. This is the first report to explicitly utilize the method-construct distinction in an effort to examine the causes and consequences of retest effects.

     

Adaptive spatial working memory assessments for aging pet dogs

Assessments for spatial working memory (SWM) in pet dogs that can detect age-related cognitive deficits in a single session may aid in diagnosing canine dementia and may facilitate translational research on Alzheimer’s disease in humans. Adaptive testing procedures are widely used in single-session assessments for humans with diverse cognitive abilities. In this study, we designed and deployed two up-down staircase assessments for SWM in which 26 pet dogs were required to recall the location of a treat hidden behind one of two identical boxes following delays of variable length. In the first experiment, performance tended to decline with age but few dogs completed the test (n?=?10). However, all of the dogs that participated in the second experiment (n?=?24) completed the assessment and provided reliable evidence of learning and retaining the task. Delay length and age significantly predicted performance supporting the validity of this assessment. The relationships between age and performance were described by inverted U-shaped functions as both old and young dogs displayed deficits in weighted cumulative-scores and trial-by-trial performance. Thus, SWM in pet dogs may develop until midlife and decline thereafter. Exploratory analyses of non-mnemonic fixation strategies, sustained engagement, inhibitory control, and potential improvements for future SWM assessments which adopt this paradigm are also discussed.

     

An Upgrading Procedure for Adaptive Assessment of Knowledge

In knowledge space theory, existing adaptive assessment procedures can only be applied when suitable estimates of their parameters are available. In this paper, an iterative procedure is proposed, which upgrades its parameters with the increasing number of assessments. The first assessments are run using parameter values that favor accuracy over efficiency. Subsequent assessments are run using new parameter values estimated on the incomplete response patterns from previous assessments. Parameter estimation is carried out through a new probabilistic model for missing-at-random data. Two simulation studies show that, with the increasing number of assessments, the performance of the proposed procedure approaches that of gold standards.     

A diagnostic tree model for polytomous responses with multiple strategies

Constructed-response items have been shown to be appropriate for cognitively diagnostic assessments because students’ problem-solving procedures can be observed, providing direct evidence for making inferences about their proficiency. However, multiple strategies used by students make item scoring and psychometric analyses challenging. This study introduces the so-called two-digit scoring scheme into diagnostic assessments to record both students’ partial credits and their strategies. This study also proposes a diagnostic tree model (DTM) by integrating the cognitive diagnosis models with the tree model to analyse the items scored using the two-digit rubrics. Both convergent and divergent tree structures are considered to accommodate various scoring rules. The MMLE/EM algorithm is used for item parameter estimation of the DTM, and has been shown to provide good parameter recovery under varied conditions in a simulation study. A set of data from TIMSS 2007 mathematics assessment is analysed to illustrate the use of the two-digit scoring scheme and the DTM.     

Using Brief Academic Assessments to Determine Generalization Strategies

This study examined the utility of brief academic assessments to identify effective generalization procedures for individual students. Specifically, the study built on the proposal that brief assessments of antecedent and consequence manipulations can identify the most effective generalization strategy for individual students. The design was an alternating treatments design nested within a multiple baseline across six students. Students learned how to solve a set of multiplication facts using a common strategy while spontaneous generalization to other sets of facts was measured. Next, researchers determined whether an antecedent- or consequent-based generalization strategy would be more effective for increasing generalization across multiplication skills and conducted an extended analysis with an alternating treatment phase to confirm results of the brief assessment. Results indicated that the assessment correctly identified the most effective generalization strategy for five of the six students.     

Conflict management: A premarital training program in mutual problem solving

Abstract

The purpose of this study was to evaluate the effectiveness of a structured educational program designed to train premarital couples in communication and mutual problem-solving skills. Twenty-six couples participated in an eight-week, 24-hour problem-solving training program, while 28 similar couples participated in an eight-week, 24-hour relationship discussion group. Results indicated that the problem-solving group, as compared to the relationship-discussion group, showed a significant increase in communication and mutual problem-solving skills. Discussion focused on the benefits for premarital couples of learning effective problemsolving procedures as well as the need for follow-up assessments of both behavioral skill level and relationship satisfaction and adjustment.