Evaluation of publication bias in response interruption and redirection: A meta-analysis

Publication bias is the disproportionate representation of studies with large effects and statistically significant findings in the published research literature. If publication bias occurs in single-case research design studies on applied behavior-analytic (ABA) interventions, it can result in inflated estimates of ABA intervention effects. We conducted an empirical evaluation of publication bias on an evidence-based ABA intervention for children diagnosed with autism spectrum disorder, response interruption and redirection (RIRD). We determined effect size estimates for published and unpublished studies using 3 metrics, percentage of nonoverlapping data (PND), Hedges' g, and log response ratios (LRR). Omnibus effect size estimates across all 3 metrics were positive, supporting that RIRD is an effective treatment for reducing problem behavior maintained by nonsocial consequences. We observed larger PND for published compared to unpublished studies, small and nonsignificant differences in LRR for published compared to unpublished studies, and significant differences in Hedges' g for published compared to unpublished studies, with published studies showing slightly larger effect. We found little, if any, difference in methodological quality between published and unpublished studies. While RIRD appears to be an effective intervention for challenging behavior maintained by nonsocial consequences, our results reflect some degree of publication bias present in the RIRD research literature.     

A primer for using multilevel models to meta-analyze single case design data with AB phases

Meta-analytic methods provide a way to synthesize data across treatment evaluation studies. However, these well-accepted methods are infrequent with behavior analytic studies. Multilevel models may be a promising method to meta-analyze single-case data. This technical article provides a primer for how to conduct a multilevel model with single-case designs with AB phases using data from the differential-reinforcement-of-low-rate behavior literature. We provide details, recommendations, and considerations for searching for appropriate studies, organizing the data, and conducting the analyses. All data sets are available to allow the reader to follow along with this primer. The purpose of this technical article is to minimally equip behavior analysts to complete a meta-analysis that will summarize a current state of affairs as it relates to the science of behavior analysis and its practice. Moreover, we aim to demonstrate the value of analyses of this sort for behavior analysis.     

The generalized Logit-Linear Item Response Model for Binary-Designed Items

This paper introduces the generalized logit-linear item response model (GLLIRM), which represents the item-solving process as a series of dichotomous operations or steps. The GLLIRM assumes that the probability function of the item response is a logistic function of a linear composite of basic parameters which describe the operations, and the coefficients depend on three design matrices X, Y and Z. The GLLIRM provides a tool for testing hypotheses on the item-solving process and generalizes existing models. An empirical application is included, in which the model is applied to evaluate sources of difficulty and pairwise item interactions in a logical analysis test. This research was supported by the Comunidad de Madrid grant CCG06-UAM/ESP-0043.     

A Speeded Item Response Model with Gradual Process Change

An item response theory model for dealing with test speededness is proposed. The model consists of two random processes, a problem solving process and a random guessing process, with the random guessing gradually taking over from the problem solving process. The involved change point and change rate are considered random parameters in order to model examinee differences in both respects. The proposed model is evaluated on simulated data and in a case study. The research reported in this paper was supported by IAP P5/24 and GOA/2005/04, both awarded to Paul De Boeck and Iven Van Mechelen, and by IAP P6/03, awarded to Iven Van Mechelen. Yuri Goegebeur’s research was supported by a grant of the Danish Natural Science Research Council.     

干预-应答模式鉴别学习障碍的有效性及其调节因素:20年研究的元分析

本研究采用元分析方法探讨在学习障碍的鉴别中兴起的干预-应答（Response to Intervention,RTI）模式鉴别学习障碍内部亚组的有效性及其调节因素。通过系统检索1996-2015年的文献,获得了34项研究,包括6127名学生的45个样本、261个效应值。元分析结果表明,RTI模式可有效区分学习障碍风险儿童的内部亚组,对干预无应答和有应答的学生在学业成就、认知技能、行为等多方面存在系统、显著和持久的差异,但仍存在个体应答状态的进一步分化。RTI模式对于学习障碍风险儿童内部变异的区分效果受到干预对象、干预层次、干预时间、应答指标选择、测量方法和切分点等因素的影响。本元分析结果不仅为认识RTI模式鉴别学习障碍的有效性提供了进一步证据,更重要的是通过系列调节效应分析,为合理实施RTI模式鉴别学习障碍提供了直接依据。     

Evaluating single-case research data for systematic review: A commentary for the special issue

The purpose of this commentary is to provide observation on the statistical procedures described throughout this special section from the perspective of researchers with experience in conducting systematic reviews and meta-analyses of single-case research to address issues of evidence-based practice. It is our position that both visual and statistical analyses are complimentary methods for evaluating single-case research data for these purposes. Given the recent developments regarding the use of single-case research to inform evidence-based practice and policy, the developments described in the present issue will be contextualized within the need for a widely accepted process for data evaluation to assist with extending the impact of single-case research. The commentary will, therefore, begin with providing an overview of the conceptual underpinnings of a systematic review of single-case research and will be followed by a discussion of several features that are essential to the development of a conceptually sound and widely used statistical procedure for single-case research. The commentary will conclude with recommendations and guidelines for the use of both visual and statistical analyses within primary research reports and recommendations for future research.     

A generalized least squares regression approach for computing effect sizes in single-case research: application examples

A new method for deriving effect sizes from single-case designs is proposed. The strategy is applicable to small-sample time-series data with autoregressive errors. The method uses Generalized Least Squares (GLS) to model the autocorrelation of the data and estimate regression parameters to produce an effect size that represents the magnitude of treatment effect from baseline to treatment phases in standard deviation units. In this paper, the method is applied to two published examples using common single case designs (i.e., withdrawal and multiple-baseline). The results from these studies are described, and the method is compared to ten desirable criteria for single-case effect sizes. Based on the results of this application, we conclude with observations about the use of GLS as a support to visual analysis, provide recommendations for future research, and describe implications for practice.     

A generalized linear factor model approach to the hierarchical framework for responses and response times

We show how the hierarchical model for responses and response times as developed by van der Linden (2007), Fox, Klein Entink, and van der Linden (2007), Klein Entink, Fox, and van der Linden (2009), and Glas and van der Linden (2010) can be simplified to a generalized linear factor model with only the mild restriction that there is no hierarchical model at the item side. This result is valuable as it enables all well‐developed modelling tools and extensions that come with these methods. We show that the restriction we impose on the hierarchical model does not influence parameter recovery under realistic circumstances. In addition, we present two illustrative real data analyses to demonstrate the practical benefits of our approach.     

因子分析元分析的三阶段整合模型

因子分析元分析(meta-analysis of factor analyses, MAFA)整合了因子分析和元分析两种方法学传统。开展MAFA方法学研究可结束过去几十年MAFA研究缺乏根据的状况, 服务心理学等领域内的知识生产。三阶段整合模型认为MAFA包括数据准备、数据合成和数据分析三个主要阶段。模型及数据合成技术的有效性将用心理测量学方法, 包括蒙特卡洛模拟研究进行检验。研究结果将阐明正确运用MAFA的一般规则、操作步骤和注意事项, 进而增进我们对MAFA的认识, 丰富应用心理等领域的方法学体系, 为MAFA方法的推广应用奠定基础。     

基于等级反应模型的属性层级方法

给出基于等级反应模型的属性层级方法（Attribute Hierarchy Method, AHM）,并简记为GRM-AHM,提出了相应的确定GRM-AHM的期望项目反应模式全集的方法和一种新的归类法LL。用蒙特卡洛模拟实验比较GRM-AHM的几种归类法的归准率（属性模式归准率和单个属性的平均判准率）。结果发现,新归类法的归准率与AHM中的方法A差不多,但比方法B高很多;随着被试作答失误率的提高,它们的归准率都有所下降。在归类精度和简单性方面,GRM-AHM都比Bolt等(2004)提出的多级评分融合模型（Fusion Model）好     

A Polytomous Model of Cognitive Diagnostic Assessment for Graded Data

Pursuing the line of the difference models in IRT (Thissen &; Steinberg, 1986 Thissen, D., &; Steinberg, L. (1986). A taxonomy of item response models. Psychometrika, 51:567–577. doi:10.1007/BF02295596.[Crossref], [Web of Science ®] , [Google Scholar]), this article proposed a new cognitive diagnostic model for graded/polytomous data based on the deterministic input, noisy, and gate (Haertel, 1989 Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333–352. doi:10.1111/j.1745-3984.1989.tb00336.x.[Crossref], [Web of Science ®] , [Google Scholar]; Junker &; Sijtsma, 2001 Junker, B. W., &; Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. doi:10.1177/01466210122032064.[Crossref], [Web of Science ®] , [Google Scholar]), which is named the DINA model for graded data (DINA-GD). We investigated the performance of a full Bayesian estimation of the proposed model. In the simulation, the classification accuracy and item recovery for the DINA-GD model were investigated. The results indicated that the proposed model had acceptable examinees' correct attribute classification rate and item parameter recovery. In addition, a real-data example was used to illustrate the application of this new model with the graded data or polytomously scored items.     

卫生行为干预的认知-情感网络模型

行为的影响因素错综复杂,有效的干预方案的设计离不开合理的理论模型指导。提出的模型认为,个体的行为是由其大脑中的认知-情感神经网络决定的,它包括自己、他人、环境和行为动机四个层次化的子系统。该理论模型不仅为系统理解卫生行为提供了有用的思维框架,还对行为干预提出了许多有益的启示。     

A general linear model for estimating effect size in the presence of publication bias

When the process of publication favors studies with smallp-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailedp-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.Authors' note: The contributions of the authors are considered equal, and the order of authorship was chosen to be reverse-alphabetical.     

The Graded Response Differential Discrimination Model Accounting for Extreme Response Style

Extreme response style or, more generally, individual differences in response spacing have been shown to be an influential bias when analyzing questionnaire data. Recently a promising model adjusting for this bias — the differential discrimination model — has been proposed. An advantage to other related approaches is that the model can be fitted using standard structural equation modeling software. However, the model is designed for analyzing continuous item responses, whereas graded response formats are certainly more prominent in behavioral sciences. To resolve this limitation, the present article extends the differential discrimination model to analyzing graded responses. Empirical examples as well as a small simulation study are presented.     

A Signal Detection Model for Multiple-Choice Exams

A model for multiple-choice exams is developed from a signal-detection perspective. A correct alternative in a multiple-choice exam can be viewed as being a signal embedded in noise (incorrect alternatives). Examinees are assumed to have perceptions of the plausibility of each alternative, and the decision process is to choose the most plausible alternative. It is also assumed that each examinee either knows or does not know each item. These assumptions together lead to a signal detection choice model for multiple-choice exams. The model can be viewed, statistically, as a mixture extension, with random mixing, of the traditional choice model, or similarly, as a grade-of-membership extension. A version of the model with extreme value distributions is developed, in which case the model simplifies to a mixture multinomial logit model with random mixing. The approach is shown to offer measures of item discrimination and difficulty, along with information about the relative plausibility of each of the alternatives. The model, parameters, and measures derived from the parameters are compared to those obtained with several commonly used item response theory models. An application of the model to an educational data set is presented.     

A Protective Factors Model for Alcohol Abuse and Suicide Prevention Among Alaska Native Youth

This study provides an empirical test of a culturally grounded theoretical model for prevention of alcohol abuse and suicide risk with Alaska Native youth, using a promising set of culturally appropriate measures for the study of the process of change and outcome. This model is derived from qualitative work that generated an heuristic model of protective factors from alcohol (Allen et al. in J Prev Interv Commun 32:41–59, 2006; Mohatt et al. in Am J Commun Psychol 33:263–273, 2004a; Harm Reduct 1, 2004b). Participants included 413 rural Alaska Native youth ages 12–18 who assisted in testing a predictive model of Reasons for Life and Reflective Processes about alcohol abuse consequences as co-occurring outcomes. Specific individual, family, peer, and community level protective factor variables predicted these outcomes. Results suggest prominent roles for these predictor variables as intermediate prevention strategy target variables in a theoretical model for a multilevel intervention. The model guides understanding of underlying change processes in an intervention to increase the ultimate outcome variables of Reasons for Life and Reflective Processes regarding the consequences of alcohol abuse.     

Sample Size Planning for Detecting Mediation Effects: A Power Analysis Procedure Considering Uncertainty in Effect Size Estimates

Abstract

When planning mediation studies, researchers are often interested in the sample size needed to achieve adequate power for testing mediation. Power depends on population effect sizes, which are unknown in practice. In conventional power analysis, effect size estimates, however, are often used as population values, which could result in underpowered studies. Uncertainty in effect size estimates has been considered in other sample size planning contexts (e.g., t-test, ANOVA), but has not been handled properly for planning mediation studies. In the current study, we proposed an easy-to-use sample size planning method for testing mediation with uncertainty in effect size estimates considered. We conducted simulation studies to demonstrate the impact of uncertainty in effect size estimates on power of testing mediation, and to provide sample size suggestions under different levels of uncertainty. Empirical examples were provided to illustrate the application of our method. R functions and a web application were developed to facilitate implementation.     

项目反应时间的对数偏正态模型

近年来,项目反应时间数据的建模是心理和教育测量领域的热门方向之一。针对反应时间的对数正态模型和Box-Cox正态模型的不足,本文在van der Linden的分层模型框架下基于偏正态分布建立一个反应时间的对数线性模型,并成功给出模型参数估计的马尔科夫链蒙特卡罗（Markov Chain Monte Carlo, MCMC）算法。模拟研究和实例分析的结果均表明,与对数正态模型和Box-Cox正态模型相比,对数偏正态模型表现出更加优良的拟合效果,具有更强的灵活性和适用性。     

A Hierarchical Rater Model for Longitudinal Data

Research studies in psychology and education often seek to detect changes or growth in an outcome over a duration of time. This research provides a solution to those interested in estimating latent traits from psychological measures that rely on human raters. Rater effects potentially degrade the quality of scores in constructed response and performance assessments. We develop an extension of the hierarchical rater model (HRM), which yields estimates of latent traits that have been corrected for individual rater bias and variability, for ratings that come from longitudinal designs. The parameterization, called the longitudinal HRM (L-HRM), includes an autoregressive time series process to permit serial dependence between latent traits at adjacent timepoints, as well as a parameter for overall growth. We evaluate and demonstrate the feasibility and performance of the L-HRM using simulation studies. Parameter recovery results reveal predictable amounts and patterns of bias and error for most parameters across conditions. An application to ratings from a study of character strength demonstrates the model. We discuss limitations and future research directions to improve the L-HRM.     

A sequential cognitive diagnosis model for polytomous responses

This paper proposes a general polytomous cognitive diagnosis model for a special type of graded responses, where item categories are attained in a sequential manner, and associated with some attributes explicitly. To relate categories to attributes, a category‐level Q‐matrix is used. When the attribute and category association is specified a priori, the proposed model has the flexibility to allow different cognitive processes (e.g., conjunctive, disjunctive) to be modelled at different categories within a single item. This model can be extended for items where categories cannot be explicitly linked to attributes, and for items with unordered categories. The feasibility of the proposed model is examined using simulated data. The proposed model is illustrated using the data from the Trends in International Mathematics and Science Study 2007 assessment.