Use of Brief Experimental Analyses in Outpatient Clinic and Home Settings

We describe the historic and current use of brief experimental analysis procedures in outpatient clinic and home settings. We discuss some applications of the designs and suggest design modifications for improving internal validity. We describe our application of the designs to longitudinal, in-home programs for children with severe behavior disorders and demonstrate how related versions of brief experimental analysis procedures can be linked to form a more comprehensive evaluation.     

Methodological developments in personality research: An introduction

This article first provides a brief overview of some of the empirical and conceptual work in personality following Mischel's (1968) critique Strengths and weaknesses of traditional personality research designs including laboratory experiments, experimental personality designs, and preexperimental correlational studies are presented Some more recent approaches to designing laboratory research that attempt to address issues of external validity and approaches to designing field Investigations that attempt to address issues of internal validity are discussed Issues associated with the measurement of constructs, the broadening of measurement methods, and newer data analytic techniques are noted Finally, a brief orientation to the other papers in the special issue is presented     

The internal validity of efficacy studies: design and statistical power in studies of language therapy for aphasics.

In this study the internal validity of efficacy studies of language therapy for aphasic patients is discussed. The lack of sufficient internal validity is demonstrated with respect to research designs used in these studies and the statistical power of their statistical significance tests. The internal validity problems are viewed as the major cause of the conflicting conclusions in the efficacy studies in the past three decades.     

A NOTE ON EVALUATING BEHAVIOR MODIFICATION AND BEHAVIOR MODELING AS INDUSTRIAL TRAINING TECHNIQUES

The training literature (1967-1976) was searched for reports of scientific evaluations of behavior modification and behavior modeling used in industrial training. No reported scientific evaluations of behavior modification were found. Four reports (1976) of scientific evaluation of behavior modeling used in training managers were found. The research designs used in these studies were analyzed for possible threats to internal validity. Enough threats to internal validity were discovered in the designs used to question the reported results of behavior modeling training of managers.     

Individual differences methods for randomized experiments

Experiments allow researchers to randomly vary the key manipulation, the instruments of measurement, and the sequences of the measurements and manipulations across participants. To date, however, the advantages of randomized experiments to manipulate both the aspects of interest and the aspects that threaten internal validity have been primarily used to make inferences about the average causal effect of the experimental manipulation. This article introduces a general framework for analyzing experimental data to make inferences about individual differences in causal effects. Approaches to analyzing the data produced by a number of classical designs and 2 more novel designs are discussed. Simulations highlight the strengths and weaknesses of the data produced by each design with respect to internal validity. Results indicate that, although the data produced by standard designs can be used to produce accurate estimates of average causal effects of experimental manipulations, more elaborate designs are often necessary for accurate inferences with respect to individual differences in causal effects. The methods described here can be diversely applied by researchers interested in determining the extent to which individuals respond differentially to an experimental manipulation or treatment and how differential responsiveness relates to individual participant characteristics.     

Computer-Assisted Career Guidance and Single-Subject Research: A Scientist-Practitioner Approach to Accountability

In this article the author discusses how computer-assisted career counselors can become more accountable for their services by using intensive research designs. An overview of computer-assisted career guidance (CACG) is presented, and the relevance of the scientist-practitioner model is discussed. Applicable single-subject research, including simple and complex phase changes, alternating treatment designs, and replication are presented with examples for accountable CACG counseling. If your client has changed, how do you know it? —Jerry Schmidt To understand individuals, it is necessary to study individuals. —Leo Goldman     

Methodological and assessment considerations in evaluating reinforcement programs in applied settings

The extensive use of reinforcement programs in applied settings has led to experimentation that often fails to consider potential problems in design. The logic of the within-subject design is reviewed and specific designs employed in reinforcement programs are discussed. For each design (ABAB, or multiple-baseline design across behaviors, individuals, or situations), effects are discussed that make that design less powerful with respect to demonstrating the effect of the experimental variable. Problems in interpreting results of experiments in this area of inquiry are evaluated from the standpoint of internal and external validity. The issue of control groups is presented with considerations as to situations that require their use. Finally, the assessment strategy for evaluating operant programs is discussed and recommendations are made for measurement of behaviors in addition to the target response.     

The intervention selection bias: an underrecognized confound in intervention research

Selection bias can be the most important threat to internal validity in intervention research, but is often insufficiently recognized and controlled. The bias is illustrated in research on parental interventions (punishment, homework assistance); medical interventions (hospitalization); and psychological interventions for suicide risk, sex offending, and juvenile delinquency. The intervention selection bias is most adequately controlled in randomized studies or strong quasi-experimental designs, although recent statistical innovations can enhance weaker designs. The most important points are to increase awareness of the intervention selection bias and to systematically evaluate plausible alternative explanations of data before making causal conclusions.     

Experimental and Quasi-Experimental Designs in Visitor Studies: A Critical Reflection on Three Projects

Identifying causal relationships is an important aspect of research and evaluation in visitor studies, such as making claims about the learning outcomes of a program or exhibit. Experimental and quasi-experimental approaches are powerful tools for addressing these causal questions. However, these designs are arguably underused in visitor studies. In this article, we offer examples of the use of experimental and quasi-experimental designs in science museums to aide investigators interested in expanding their methods toolkit and increasing their ability to make strong causal claims about programmatic experiences or relationships among variables. Using three designs from recent research (fully randomized experiment, posttest only quasi-experimental design with comparison condition, and posttest with independent pretest design), we discuss challenges and tradeoffs related to feasibility, participant experience, alignment with research questions, and internal and external validity. We end the article with broader reflections on the role of experimental and quasi-experimental designs in visitor studies.     

D-optimal design for the Rasch counts model with multiple binary predictors

In this paper we derive optimal designs for the Rasch Poisson counts model and its extended version of the (generalized) negative binomial counts model incorporating several binary predictors for the difficulty parameter. To efficiently estimate the regression coefficients of the predictors, locally D-optimal designs are developed. After an introduction to the Rasch Poisson counts model and its extension, we will specify these models as particular generalized linear models. Based on this embedding, optimal designs for both models including several binary explanatory variables will be presented. Therefore, we will derive conditions on the effect sizes for certain designs to be locally D-optimal. Finally, it is pointed out that the results derived for the Rasch Poisson models can be applied for more general Poisson regression models which should receive more attention in future psychological research.     

Single‐Case Experimental Designs for the Evaluation of Treatments for Self‐Injurious and Suicidal Behaviors

Single‐case experimental designs (SCEDs) provide a time‐ and cost‐effective alternative to randomized clinical trials and offer significant advantages in terms of internal and external validity. A brief history and primer on SCEDs is provided, specifically for use in suicide intervention research. Various SCED methodologies, such as AB, ABAB, multiple baseline, and changing criterion designs, are described. Advantages and disadvantages of their use specifically for intervention research for suicidal and self‐injurious behaviors are detailed.     

Different randomized multiple-baseline models for different situations: A practical guide for single-case intervention researchers

Single-case researchers often implement multiple-baseline designs as their preferred methodology for intervention evaluations. Recent writings and empirical investigations have argued in favor of incorporating various forms of randomization into such designs for the purpose of elevating the intervention study's internal validity and scientific credibility. In this article, we consider a variety of randomized multiple-baseline designs and associated randomization statistical tests, along with their potential strengths and limitations. In what amounts to a practical guide, we refer school psychology researchers to these versatile randomization procedures for planning and executing their intervention studies.     

INTERACTION EFFECTS IN MULTIELEMENT DESIGNS: INEVITABLE,DESIRABLE, AND IGNORABLE

A single-subject design often used to compare the effectiveness of two or more independent variables (like treatment programs) is the multielement (alternating treatments or simultaneous treatments) design. Variants of this design approximate the concurrent comparison of the effects of two or more variables (or levels of variables) by programming the variables (or levels) in rapid alternation, typically across or within daily sessions. Properly combined with conventional reversal designs, these designs can also display a variety of interaction effects, some of them worrisome, others highly desirable for the future development of the field. A worrisome model is the possibility that when Treatment B alternates rapidly with Treatment C, the effects of each will not be the same as when each is the only treatment used. A desirable model is the use of the multielement design as a fast-paced component of an otherwise conventional reversal design examining contextual control of some relationship: the possibility that some behavior responds differently to Controlling Variables A and B in Context X than in Context Y. This second possibility opens single-subject designs to the more efficient examination of all interactive effects and is highly desirable, considering the prevalence and importance of interactions in determining the limits and the generality of currently understood behavioral phenomena.     

Bayesian analysis of longitudinal multitrait–multimethod data with ordinal response variables

A new multilevel latent state graded response model for longitudinal multitrait–multimethod (MTMM) measurement designs combining structurally different and interchangeable methods is proposed. The model allows researchers to examine construct validity over time and to study the change and stability of constructs and method effects based on ordinal response variables. We show how Bayesian estimation techniques can address a number of important issues that typically arise in longitudinal multilevel MTMM studies and facilitates the estimation of the model presented. Estimation accuracy and the impact of between‐ and within‐level sample sizes as well as different prior specifications on parameter recovery were investigated in a Monte Carlo simulation study. Findings indicate that the parameters of the model presented can be accurately estimated with Bayesian estimation methods in the case of low convergent validity with as few as 250 clusters and more than two observations within each cluster. The model was applied to well‐being data from a longitudinal MTMM study, assessing the change and stability of life satisfaction and subjective happiness in young adults after high‐school graduation. Guidelines for empirical applications are provided and advantages and limitations of a Bayesian approach to estimating longitudinal multilevel MTMM models are discussed.     

Meta-Analysis of Single-Case Design Research: Introduction to the Special Issue

Single-case design (SCD) research focuses on finding powerful effects, but the influence of this methodology on the evidence-based practice (EBP) movement is questionable. Meta-analytic procedures may help facilitate the role of SCD research in the EBP movement, but meta-analyses of SCDs are controversial. The current article provides an introduction to the special issue on meta-analyses of SCD research by discussing concerns regarding the internal and external validity of these designs. Specific considerations for increasing the validity of SCD meta-analyses are provided, as are brief overviews of the articles included in the special issue.     

Development of a short form of the Treatment Evaluation Inventory for acceptability of psychological interventions

The Treatment Evaluation Inventory of Kazdin, French, and Sherick is a 19-item measure of the perceived acceptability of behavioural treatments. Development of two brief forms was based on data from two sources. For Study 1, data from 218 completed questionnaires were used to develop internally consistent brief scales. In Study 2 internal consistency and the validity of the brief forms were estimated for a set of 131 questionnaires. Item reduction was achieved by analysis of item-total minus item correlations. Brief forms with 3, 6, 9, and 12 items were proposed. Their internal consistency (Cronbach alpha) and construct validity were based on correlations of scores on each short form with the full scale scores and on comparing means of different forms. Discriminant validity was based on the difference between two groups (estimated effect size 0.7). Scores for all forms showed high internal consistency and correlated highly with total scale scores. Only the 12-item brief scale yielded mean scores similar to the full scale. The 3-item form could be used as a quick screen, and the 12-item form for more intensive purposes as it is most similar to the full-scale.     

Rehabilitation of executive disorders after brain injury: Are interventions effective?

In this paper, the effectiveness of interventions for executive disorders was reviewed. The objective was to evaluate the internal and external validity of intervention studies. A total of 46 papers, describing 54 studies, conducted in the last two decades meeting several preset inclusion criteria, was included in this review. The studies were categorized into three treatment approaches in order to enhance comparability. The overall results show that many interventions yield positive outcomes and seem to be effective in reducing executive problems in brain‐injured subjects. However, several studies have only an explorative intent and are based on less sophisticated experimental designs. The verification of their results is generally more tenuous. The internal validity, or the set‐up of experimental conditions necessary to draw valid conclusions about treatment effectiveness, including the choice of well‐matched control groups, or the randomization of patients over treatment and control conditions, is not always strong. The same conclusion can be drawn for the external validity of a number of the intervention studies; often evidence of generalization to real‐life situations, long‐term follow‐up, and transfer to non‐trained situations, were (partially) lacking in the studies under review. The authors are aware that the design of proper randomized controlled trials for the investigation of the treatment effectiveness of executive disorders is cumbersome and time consuming. Nonetheless, the provisional results of several well‐designed studies described in this review make the effort worthwhile.     

Application of multiple baseline designs in behavior analytic research: Evidence for the influence of new guidelines

The multiple baseline (MBL) design is a single‐case experimental design that has both research and applied utility. Although the concurrent and nonconcurrent MBL variants are valid designs, each rules out different threats to internal validity. To help clarify these differences, studies have provided guidelines for graphically depicting and distinguishing between concurrent and nonconcurrent MBLs. This study assessed the extent to which data‐presentation guidelines have been adopted by examining single‐case experimental design studies published in 3 behavior‐analytic journals from 2000 to 2015. Results suggest that data‐depiction guidelines have increased correct identification of concurrent multiple baseline designs but also indicate that these guidelines have not been universally adopted.     

The measurement of participant-reported memory across diverse populations and settings: a systematic review and meta-analysis of the Multifactorial Memory Questionnaire

ABSTRACT

The Multifactorial Memory Questionnaire (MMQ) is a participant-reported measure of memory satisfaction, ability, and strategy use. Initially validated with healthy older adults, it has since been used in many different populations and settings for a variety of purposes. We conducted a systematic review and meta-analysis of the measurement properties of the MMQ across multiple, diverse studies. Methods: The study was designed using the Consensus-Based Standards for the Selection of Health Measurement Instruments and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. We calculated aggregate statistics and evaluated the methodological quality of 29 studies retrieved from PsycINFO, MEDLINE, EMBASE, and Web of Science. Results: Analyses revealed high-quality evidence for internal consistency, stability, measurement error, convergent validity, and known-groups validity of the three MMQ scales. There was moderate-quality evidence for responsiveness and structural validity, with some studies identifying separate factors for internal and external memory strategy use. Measurement properties were similar across languages, participant samples, and study designs. Conclusions: The MMQ is a valid, reliable, and responsive measure across diverse settings and populations. Future research is needed to determine whether more detailed information can be obtained from the scales, specifically, internal versus external strategy use.