Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables

This paper uses log-linear models with latent variables (Hagenaars, in Loglinear Models with Latent Variables, 1993) to define a family of cognitive diagnosis models. In doing so, the relationship between many common models is explicitly defined and discussed. In addition, because the log-linear model with latent variables is a general model for cognitive diagnosis, new alternatives to modeling the functional relationship between attribute mastery and the probability of a correct response are discussed.     

Detecting Differential Item Functioning Using Multiple-Group Cognitive Diagnosis Models

This study proposes a multiple-group cognitive diagnosis model to account for the fact that students in different groups may use distinct attributes or use the same attributes but in different manners (e.g., conjunctive, disjunctive, and compensatory) to solve problems. Based on the proposed model, this study systematically investigates the performance of the likelihood ratio (LR) test and Wald test in detecting differential item functioning (DIF). A forward anchor item search procedure was also proposed to identify a set of anchor items with invariant item parameters across groups. Results showed that the LR and Wald tests with the forward anchor item search algorithm produced better calibrated Type I error rates than the ordinary LR and Wald tests, especially when items were of low quality. A set of real data were also analyzed to illustrate the use of these DIF detection procedures.     

计算机化多维测验中作答时间和作答精度数据的联合分析

随着心理与教育测量研究的发展和科技的进步,计算机化(大规模)测验逐渐受到人们的关注。为探究在计算机化多维测验中如何利用作答时间数据来辅助评估多维潜在能力,以及为我国义务教育阶段教育质量监测提供数据分析方法上的理论支持。本研究以2012年和2015年国际学生能力评估(PISA)计算机化数学测验数据为例,提出了一种可同时利用作答时间和作答精度数据的联合作答与时间的多维Rasch模型。根据新模型对PISA数据的分析结果,表明引入作答时间数据,不仅有助于提高模型参数的估计精度,还有助于数据分析者利用被试的作答时间信息来做进一步的决策和干预(e.g., 对异常作答行为或预备知识的诊断)。     

Hidden Markov Item Response Theory Models for Responses and Response Times

Current approaches to model responses and response times to psychometric tests solely focus on between-subject differences in speed and ability. Within subjects, speed and ability are assumed to be constants. Violations of this assumption are generally absorbed in the residual of the model. As a result, within-subject departures from the between-subject speed and ability level remain undetected. These departures may be of interest to the researcher as they reflect differences in the response processes adopted on the items of a test. In this article, we propose a dynamic approach for responses and response times based on hidden Markov modeling to account for within-subject differences in responses and response times. A simulation study is conducted to demonstrate acceptable parameter recovery and acceptable performance of various fit indices in distinguishing between different models. In addition, both a confirmatory and an exploratory application are presented to demonstrate the practical value of the modeling approach.     

Accuracy of the DIBELS Oral Reading Fluency Measure for Predicting Third Grade Reading Comprehension Outcomes

We evaluated the validity of DIBELS (Dynamic Indicators of Basic Early Literacy Skills) ORF (Oral Reading Fluency) for predicting performance on the Florida Comprehensive Assessment Test (FCAT-SSS) and Stanford Achievement Test (SAT-10) reading comprehension measures. The usefulness of previously established ORF risk-level cutoffs [Good, R.H., Simmons, D.C., and Kame’enui, E.J. (2001). The importance and decision-making utility of a continuum of fluency-based indicators of foundational reading skills for third-grade high-stakes outcomes. Scientific Studies of Reading, 5, 257–288.] for third grade students were evaluated on calibration (n_S1 = 16,539) and cross-validation (n_S2 = 16,908) samples representative of Florida's Reading First population. The strongest correlations were the third (February/March) administration of ORF with both FCAT-SSS and SAT-10 (r_S = .70–.71), when the three tests were administered concurrently. Recalibrated ORF risk-level cut scores derived from ROC (receiver-operating characteristic) curve analyses produced more accurate identification of true positives than previously established benchmarks. The recalibrated risk-level cut scores predict performance on the FCAT-SSS equally well for students from different socio-economic, language, and race/ethnicity categories.     

Using Response Times to Model Not-Reached Items due to Time Limits

Missing values at the end of a test typically are the result of test takers running out of time and can as such be understood by studying test takers’ working speed. As testing moves to computer-based assessment, response times become available allowing to simulatenously model speed and ability. Integrating research on response time modeling with research on modeling missing responses, we propose using response times to model missing values due to time limits. We identify similarities between approaches used to account for not-reached items (Rose et al. in ETS Res Rep Ser 2010:i–53, 2010) and the speed-accuracy (SA) model for joint modeling of effective speed and effective ability as proposed by van der Linden (Psychometrika 72(3):287–308, 2007). In a simulation, we show (a) that the SA model can recover parameters in the presence of missing values due to time limits and (b) that the response time model, using item-level timing information rather than a count of not-reached items, results in person parameter estimates that differ from missing data IRT models applied to not-reached items. We propose using the SA model to model the missing data process and to use both, ability and speed, to describe the performance of test takers. We illustrate the application of the model in an empirical analysis.

     

Speed-Accuracy Response Models: Scoring Rules based on Response Time and Accuracy

Starting from an explicit scoring rule for time limit tasks incorporating both response time and accuracy, and a definite trade-off between speed and accuracy, a response model is derived. Since the scoring rule is interpreted as a sufficient statistic, the model belongs to the exponential family. The various marginal and conditional distributions for response accuracy and response time are derived, and it is shown how the model parameters can be estimated. The model for response accuracy is found to be the two-parameter logistic model. It is found that the time limit determines the item discrimination, and this effect is illustrated with the Amsterdam Chess Test II.     

Using Response Times for Joint Modeling of Response and Omission Behavior

Abstract

For adequate modeling of missing responses, a thorough understanding of the nonresponse mechanisms is vital. As a large number of major testing programs are in the process or already have been moving to computer-based assessment, a rich body of additional data on examinee behavior becomes easily accessible. These additional data may contain valuable information on the processes associated with nonresponse. Bringing together research on item omissions with approaches for modeling response time data, we propose a framework for simultaneously modeling response behavior and omission behavior utilizing timing information for both. As such, the proposed model allows (a) to gain a deeper understanding of response and nonresponse behavior in general and, in particular, of the processes underlying item omissions in LSAs, (b) to model the processes determining the time examinees require to generate a response or to omit an item, and (c) to account for nonignorable item omissions. Parameter recovery of the proposed model is studied within a simulation study. An illustration of the model by means of an application to real data is provided.     

认知诊断模型发展及其应用方法述评

认知心理学和心理测量学结合派生出的认知诊断理论, 利用现代统计方法和计算机技术作为工具, 诊断被试的认知结构和认知过程。认知诊断有多种模型, 不同的模型有不同的特点及应用条件。模型的选择和认知诊断方法的应用对认知诊断的结果有重要的影响, 因此在选择模型之时需要了解各种认知诊断模型的发展过程及优缺点。     

Using Percentile Schedules to Increase Academic Fluency

After a learner becomes accurate with a task, fluency must be shaped over time to reach the target goal. However, shaping criteria can be somewhat arbitrary; thus, an objective criteria has the potential to improve implementation consistency. One such method is through the use of percentile schedules. The purpose of the current study was to use a percentile schedule as a means of determining the reinforcement criterion to improve the fluency for three academic tasks. The participant was a 14‐year‐old boy diagnosed with developmental disabilities. The use of the percentile schedule based reinforcement criterion resulted in increased fluency with two of the three academic tasks. This study suggests that percentile schedules may provide an objective criterion for improving fluency. Copyright © 2016 John Wiley & Sons, Ltd.     

Evaluation of Model Fit in Cognitive Diagnosis Models

Cognitive diagnosis models (CDMs) estimate student ability profiles using latent attributes. Model fit to the data needs to be ascertained in order to determine whether inferences from CDMs are valid. This study investigated the usefulness of some popular model fit statistics to detect CDM fit including relative fit indices (AIC, BIC, and CAIC), and absolute fit indices (RMSEA2, ABS(fcor) and MAX(χ²_jj′)). These fit indices were assessed under different CDM settings with respect to Q-matrix misspecification and CDM misspecification. Results showed that relative fit indices selected the correct DINA model most of the times and selected the correct G-DINA model well across most conditions. Absolute fit indices rejected the true DINA model if the Q-matrix was misspecified in any way. Absolute fit indices rejected the true G-DINA model whenever the Q-matrix was under-specified. RMSEA2 could be artificially low when the Q-matrix was over-specified.     

A Study of Reliability Across Multiple Raters When Using the NAEP and MDFS Rubrics to Measure Oral Reading Fluency

Becoming a fluent reader has been established as important to reading comprehension. Prosody (expression) is an indicator of fluent reading that is linked to improved comprehension in students across elementary, middle, and secondary grades. Fluent reading is most often evaluated by classroom teachers through the use of a rubric, with the most common being the Multi-Dimensional Fluency Scale (MDFS) and the National Assessment of Educational Progress (NAEP) scale. This investigation uses a generalizability study (G-study) and a decision study (G-study) to determine reliability and efficiency of the two rubrics across five raters and two rating occasions in 177 first- through third-grade students. The results revealed the MDFS and NAEP to be parallel instruments with variance attributable to raters ranging from nearly 0 to 2.2%. Generalizability coefficients ranging from 0.91 to 0.94, indicating high reliability were found for both instruments. Recommendations for administration efficiency of each rubric are provided and instructional implications are discussed.     

The Stability of Literacy-Related Cognitive Contributions to Chinese Character Naming and Reading Fluency

The present study examined the developmental issue of cognitive factors that explain Chinese literacy. Phonological awareness, rapid automatized naming, short-term memory, orthographic awareness and morphological awareness and two literacy tasks (character naming and reading fluency) were administered to 408 second-graders, 428 fourth-graders and 496 six-graders. Results from linear regression analysis and path analysis model showed that the five reading-related cognitive constructs explained unique variances in character naming. Second, character naming is primary for reading fluency after controlling other cognitive constructs; third, the relation between the cognitive factors and literacy changes significantly as a function of reading skills. Results give a clear direction to understanding Chinese reading development.     

Stochastic Dynamic Models of Response Time and Accuracy: A Foundational Primer

A large class of statistical decision models for performance in simple information processing tasks can be described by linear, first-order, stochastic differential equations (SDEs), whose solutions are diffusion processes. In such models, the first passage time for the diffusion process through a response criterion determines the time at which an observer makes a decision about the identity of a stimulus. Because the assumptions of many cognitive models lead to SDEs that are time inhomogeneous, classical methods for solving such first passage time problems are usually inapplicable. In contrast, recent integral equation methods often yield solutions to both the one-sided and the two-sided first passage time problems, even in the presence of time inhomogeneity. These methods, which are of particular relevance to the cognitive modeler, are described in detail, together with illustrative applications. Copyright 2000 Academic Press.     

Predicting Response to Therapist-Assisted Internet-Delivered Cognitive Behavior Therapy for Depression or Anxiety Within an Open Dissemination Trial

Therapist-assisted Internet-delivered cognitive behavior therapy (ICBT) is efficacious for treating anxiety and depression, but predictors of response to treatment when delivered in clinical practice are not well understood. In this study, we explored demographic, clinical, and program variables that predicted modules started and symptom improvement (i.e., Generalized Anxiety Disorder-7 or Patient Health Questionnaire-9 total scores over pre-, mid-, and posttreatment) within a previously published open dissemination trial (Hadjistavropoulos et al., 2014). The sample consisted of 195 patients offered 12 modules of therapist-assisted ICBT for depression or generalized anxiety; ICBT was delivered by therapists working in six geographically dispersed clinics. Consistent across ICBT for depression or generalized anxiety, starting fewer modules was associated with more phone calls from therapists reflecting that therapists tended to call patients who did not start modules as scheduled. Also consistent for both ICBT programs, greater pretreatment condition severity and completion of more modules was associated with superior ICBT-derived benefit. Other predictors of response to treatment varied across the two programs. Younger age, lower education, taking psychotropic medication, being in receipt of psychiatric care and lower comfort with written communication were associated with either fewer program starts or lower symptom improvement in one of the two programs. It is concluded that monitoring response to ICBT may be particularly important in patients with these characteristics. Research directions for identifying patients who are less likely to benefit from ICBT are discussed.     

面向“为学习而测评”的纵向认知诊断模型

基于“为学习而测评”的理念,以促进学生学习为目的,客观量化学习现状并提供诊断反馈的测评模式日益受到重视。相比于横断认识诊断测评,纵向认知诊断测评更有利于实现促进学生发展的目标。为使国内学者系统性地了解纵向认知诊断模型,首先,依据建模逻辑将已有纵向认知诊断模型划分为基于潜在转换分析的和基于高阶潜在结构模型的两类,并逐一介绍和说明两类模型的理论基础和应用情景;然后,通过模拟研究为读者呈现如何使用纵向认知诊断模型进行数据分析及如何解读相应的诊断结果。最后,提炼出四个可进一步研究的议题。     

Joint Modeling of Ability and Differential Speed Using Responses and Response Times

With computerized testing, it is possible to record both the responses of test takers to test questions (i.e., items) and the amount of time spent by a test taker in responding to each question. Various models have been proposed that take into account both test-taker ability and working speed, with the many models assuming a constant working speed throughout the test. The constant working speed assumption may be inappropriate for various reasons. For example, a test taker may need to adjust the pace due to time mismanagement, or a test taker who started out working too fast may reduce the working speed to improve accuracy. A model is proposed here that allows for variable working speed. An illustration of the model using the Amsterdam Chess Test data is provided.     

Creativity in Physics: Response Fluency and Task Specificity

The purpose of this study was to explore creativity in the domain of physics and, specifically, its relation to fluency of responses (divergent thinking) and type of task. Fifty-four university students were pretested on their knowledge of relevant physics concepts. They then were asked to solve 3 ill-defined problems representing different types of tasks. The appropriate responses given to each problem were evaluated as to their number (fluency) and frequency (originality). Task-specific components were found to influence creativity independently and to moderate the effects of general factors such as fluency of responses. Efforts to predict and facilitate creativity in educational settings, therefore, also must take into account the way creativity is manifested within particular domains and the constraints that different types of tasks may impose.