An Illustration of Diagnostic Classification Modeling in Student Learning Outcomes Assessment

The assessment of higher-education student learning outcomes is an important component in understanding the strengths and weaknesses of academic and general education programs. This study illustrates the application of diagnostic classification models, a burgeoning set of statistical models, in assessing student learning outcomes. To facilitate understanding and future applications of diagnostic modeling, the log-linear cognitive diagnosis model used in this study is presented in a didactic manner. The model is applied in a context where undergraduate students were assessed along four learning outcomes related to psychosocial research across two time points. Results focus on implications and methods to aid stakeholders’ interpretation of the analyses. Contrasts to traditional measurement models and potential future applications are also discussed.     

A semi-parametric within-subject mixture approach to the analyses of responses and response times

In item response theory, modelling the item response times in addition to the item responses may improve the detection of possible between- and within-subject differences in the process that resulted in the responses. For instance, if respondents rely on rapid guessing on some items but not on all, the joint distribution of the responses and response times will be a multivariate within-subject mixture distribution. Suitable parametric methods to detect these within-subject differences have been proposed. In these approaches, a distribution needs to be assumed for the within-class response times. In this paper, it is demonstrated that these parametric within-subject approaches may produce false positives and biased parameter estimates if the assumption concerning the response time distribution is violated. A semi-parametric approach is proposed which resorts to categorized response times. This approach is shown to hardly produce false positives and parameter bias. In addition, the semi-parametric approach results in approximately the same power as the parametric approach.     

A generalized linear factor model approach to the hierarchical framework for responses and response times

We show how the hierarchical model for responses and response times as developed by van der Linden (2007), Fox, Klein Entink, and van der Linden (2007), Klein Entink, Fox, and van der Linden (2009), and Glas and van der Linden (2010) can be simplified to a generalized linear factor model with only the mild restriction that there is no hierarchical model at the item side. This result is valuable as it enables all well‐developed modelling tools and extensions that come with these methods. We show that the restriction we impose on the hierarchical model does not influence parameter recovery under realistic circumstances. In addition, we present two illustrative real data analyses to demonstrate the practical benefits of our approach.     

Human operant learning under concurrent reinforcement of response variability

This study asked whether the concurrent reinforcement of behavioral variability facilitates learning to emit a difficult target response. Sixty students repeatedly pressed sequences of keys, with an originally infrequently occurring target sequence consistently being followed by positive feedback. Three conditions differed in the feedback given to non-target sequences: concurrent positive feedback presented contingent on response variability, positive feedback presented non-contingently, or no reinforcement for any non-target responses (control condition). Contrary to the result of analogous rat studies, if anything, the participants in the control condition more readily learned to emit the target sequence than did the subjects in each of the other two conditions. It is argued that these contradictory findings are primarily caused by procedural differences, such as differences in the density of the reinforcement schedule applied to non-target behavior, rather than reflecting a true species difference.     

A temporal common ground for learning: The moderating effect of shared mental models on the relation between team learning behaviours and performance improvement

In this longitudinal study, we integrated a team process and a learning curve perspective on team learning and empirically analysed whether team learning processes lead to performance improvement. In addition, we tested whether this relation is moderated by the similarity of team members’ task, team, and temporal mental models. We tested our model on a sample of 67 teams (314 individuals) competing in a management simulation over five consecutive time periods, using random coefficient modelling (RCM). Our findings suggest that team learning behaviours do not have a direct effect on the team learning curve, but temporal and task mental models are crucial for the translation of team learning behaviours into performance improvement. We found that when teams have similar task and temporal mental models, engaging in team learning processes is beneficial, whereas, when teams have dissimilar task and temporal mental models, it is detrimental to performance improvement. We did not find a significant effect for the moderating role of team mental model similarity. Our study emphasizes the importance of integrating different perspectives on team learning and provides support for the role of team cognition as a catalyst for team learning.     

A multivariate counting process with Weibull-distributed first-arrival times

In this paper, we study a method to construct a multivariate counting process with positive dependencies between the event occurrences. Conditional on a random effect with a positive stable distribution, the univariate counting processes are independent non-homogeneous Poisson processes with a power intensity function. The applicability of the model is illustrated in three examples: a horse race model with several dependent channels, a dependent parallel-counter model and an interactive coactivation model.     

What response times tell of children's behavior on the balance scale task

Analysis of accuracy of responses to balance scale problems gives a global idea of the cognitive processes that underlie problem-solving behavior on this task. We show that response times (RTs) provide additional detailed information about the kind and duration of these processes. We derive predictions about the RTs from Siegler's (1981) model for the balance scale task, including the counterintuitive prediction that young adults are slower than children in solving particular balance scale problems. The predictions were tested in a study in which 191 6- to 22-year-old participants were presented with a computerized balance scale task. RTs were analyzed with regression models. In addition to qualitative differences between items, we also modeled quantitative differences between items in the regression models. Analyses supported the predictions and provided additional knowledge on the rules. Rule II was reformulated as a rule that always involves the encoding, but not always the correct application of the distance cue. RTs provided evidence for the use of a buggy-rule and not an addition-rule. Finally, a relation between rule inconsistency and increased RT was found.     

Using response times to detect aberrant responses in computerized adaptive testing

A lognormal model for response times is used to check response times for aberrances in examinee behavior on computerized adaptive tests. Both classical procedures and Bayesian posterior predictive checks are presented. For a fixed examinee, responses and response times are independent; checks based on response times offer thus information independent of the results of checks on response patterns. Empirical examples of the use of classical and Bayesian checks for detecting two different types of aberrances in response times are presented. The detection rates for the Bayesian checks outperformed those for the classical checks, but at the cost of higher false-alarm rates. A guideline for the choice between the two types of checks is offered.This study received funding from the Law School Admission Council (LSAC). The opinions and conclusions contained in this paper are those of the authors and do not necessarily reflect the policy and position of LSAC. The authors are most indebted to Wim M. M. Tielen for his computational assistance and to the US Defense Manpower Data Center for the permission to use the ASVAB data set in the empirical examples.     

Fast and accurate calculations for first-passage times in Wiener diffusion models

We propose a new method for quickly calculating the probability density function for first-passage times in simple Wiener diffusion models, extending an earlier method used by [Van Zandt, T., Colonius, H., & Proctor, R. W. (2000). A comparison of two response-time models applied to perceptual matching. Psychonomic Bulletin & Review, 7, 208-256]. The method relies on the observation that there are two distinct infinite series expansions of this probability density, one of which converges quickly for small time values, while the other converges quickly at large time values. By deriving error bounds associated with finite truncation of either expansion, we are able to determine analytically which of the two versions should be applied in any particular context. The bounds indicate that, even for extremely stringent error tolerances, no more than 8 terms are required to calculate the probability density. By making the calculation of this distribution tractable, the goal is to allow more complex extensions of Wiener diffusion models to be developed.     

Ethical and legal issues associated with using response-to-intervention to assess learning disabilities

The Individuals with Disabilities Education Improvement Act of 2004 allows schools to use a child's response to research-based intervention (RTI) as a part of procedures to identify students with learning disabilities. This paper considers whether RTI-based assessment models meet ethical and legal standards for acceptable assessment practices. Based on a review of available research, it was concluded that RTI-based assessment practices, when carefully crafted and implemented, have the potential to be multifaceted, fair, valid, and useful. Threats to acceptable RTI-based assessment practices include: the lack of research-based interventions appropriate for diverse academic domains, ethnic groups, grades K-12, and students with limited English proficiency; uncertainty regarding how to determine when nonresponse to intervention warrants formal referral for evaluation of special education eligibility; difficulty translating scientifically sound RTI practices to the local school level; and inadequate staff training and poor treatment fidelity. Suggested directions for future research are included.     

Comparison of dual discrepancy criteria to assess response to intervention

Much of the research regarding diagnosing children as LD with responsiveness-to-intervention (RTI) has examined methods to identify children as non-responsive. The current study examined the dual discrepancy (DD) definition in which children score low on a post-intervention reading measure and score below their peers in reading growth rates. Four definitions of DD were compared, student growth below the 25th percentile, 33rd percentile, 50th percentile, and less than one standard deviation below the mean. Participants were 151 children in grades 1 through 3 who were identified as experiencing reading difficulties. Results suggested that the three percentile groups significantly differentiated reading scores between DD and non-DD children, but the one standard deviation group did not. No ethnic or gender biases were noted and the 25th and 33rd percentile criteria closely matched estimates of LD prevalence. Implications for future research are included.     

A mixture model for responses and response times with a higher-order ability structure to detect rapid guessing behaviour

Many educational and psychological assessments focus on multidimensional latent traits that often have a hierarchical structure to provide both overall-level information and fine-grained diagnostic information. A test will usually have either separate time limits for each subtest or an overall time limit for administrative convenience and test fairness. In order to complete the items within the allocated time, examinees frequently adopt different test-taking behaviours during the test, such as solution behaviour and rapid guessing behaviour. In this paper we propose a new mixture model for responses and response times with a hierarchical ability structure, which incorporates auxiliary information from other subtests and the correlation structure of the abilities to detect rapid guessing behaviour. A Markov chain Monte Carlo method is proposed for model estimation. Simulation studies reveal that all model parameters could be recovered well, and the parameter estimates had smaller absolute bias and mean squared error than the mixture unidimensional item response theory (UIRT) model. Moreover, the true positive rate of detecting rapid guessing behaviour is also higher than when using the mixture UIRT model separately for each subscale, whereas the false detection rate is much lower than the mixture UIRT model. The deviance information criterion and the logarithm of the pseudo-marginal likelihood are employed to evaluate the model fit. Finally, a real data analysis is presented to demonstrate the practical value of the proposed model.     

Additive and multiplicative models for gamma distributed random variables,and their application as psychometric models for response times

A class of models for gamma distributed random variables is presented. These models are shown to be more flexible than the classical linear models with respect to the structure that can be imposed on the expected value. In particular, both additive, multiplicative, and combined additive-multiplicative models can be formulated. As a special case, a class of psychometric models for reaction times is presented, together with their psychological interpretation. By means of a comparison with existing models, this class of models is shown to offer some possibilities that are not available in existing methods. Parameter estimation by means of maximum likelihood (ML) is shown to have some attractive properties, since the models belong to the exponential family. Then, the results of a simulation study of the bias in the ML estimates are presented. Finally, the application of these models is illustrated by an analysis of the data from a mental rotation experiment. This analysis is preceded by an evaluation of the appropriateness of the gamma distribution for these data.     

On the mean and variance of response times under the diffusion model with an application to parameter estimation

We give closed form expressions for the mean and variance of RTs for Ratcliff’s diffusion model [Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85, 59-108] under the simplifying assumption that there is no variability across trials in the parameters. These expressions are more general than those currently available. As an application, we demonstrate their use in a method-of-moments estimation procedure that addresses some of the weaknesses of the EZ method [Wagenmakers, E.-J., van der Maas, H. L. J., & Grasman, R. P. P. P. (2007). An EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14, 3-22], and illustrate this with lexical decision data. We discuss further possible applications.     

Drawing conclusions from choice response time models: A tutorial using the linear ballistic accumulator

Cognitive models of choice and response times can lead to deeper insights into the processes underlying decisions than standard analyses of accuracy and response time data. The application of these models, however, has historically been reserved for the authors of the models, and their associates. Recently, choice response time models have become more accessible through the release of user-friendly software for estimating their parameters. The aim of this tutorial is to provide guidance about the process of using these parameter estimates and associated model fits to make conclusions about experimental data. We use an application of one response time model, the linear ballistic accumulator, as an example to demonstrate the steps required to select an appropriate parametric characterization of a data set. We also discuss how to evaluate the quality of the agreement between model and data, including guidelines for presenting model predictions for group-level data.     

Robust estimation of the hierarchical model for responses and response times

Van der Linden's (2007, Psychometrika, 72, 287) hierarchical model for responses and response times in tests has numerous applications in psychological assessment. The success of these applications requires the parameters of the model to have been estimated without bias. The data used for model fitting, however, are often contaminated, for example, by rapid guesses or lapses of attention. This distorts the parameter estimates. In the present paper, a novel estimation approach is proposed that is robust against contamination. The approach consists of two steps. In the first step, the response time model is fitted on the basis of a robust estimate of the covariance matrix. In the second step, the item response model is extended to a mixture model, which allows for a proportion of irregular responses in the data. The parameters of the mixture model are then estimated with a modified marginal maximum likelihood estimator. The modified marginal maximum likelihood estimator downweights responses of test-takers with unusual response time patterns. As a result, the estimator is resistant to several forms of data contamination. The robustness of the approach is investigated in a simulation study. An application of the estimator is demonstrated with real data.     

Retrieval of bindings between task-irrelevant stimuli and responses can facilitate behaviour under conditions of high response certainty

Repetition priming can be driven by the encoding and retrieval of stimulus–response (S–R) bindings. When a previously encoded S–R binding is retrieved, and is congruent with the response currently required, it can bias response-selection processes towards selecting the retrieved response, resulting in facilitation. Previous studies have used classification tasks at retrieval. Here, two (or more) response options are competing, and it is likely that any evidence (e.g., an S–R binding) in favour of one option will be utilized to effect a decision. Thus, S–R effects are likely to be seen when using such a task. It is unclear whether such effects can be seen under conditions of higher response certainty, when participants are explicitly cued to make a response. Across two experiments, evidence for a modulating influence of S–R bindings is seen despite using a response cueing method at retrieval to minimize response uncertainty and despite stimuli being task irrelevant. Finally, the results suggest that responses within these S–R bindings are coded at the level of left versus right hand, and not a more fine-grained within-hand thumb versus index finger. The results underline the resilience of S–R effects, suggesting that they are present even under conditions where no explicit object-oriented decision is required.     

Computational modeling of organisational learning by self-modeling networks

Within organisational learning literature, mental models are considered a vehicle for both individual learning and organizational learning. By learning individual mental models (and making them explicit), a basis for formation of shared mental models for the level of the organization is created, which after its formation can then be adopted by individuals. This provides mechanisms for organizational learning. These mechanisms have been used as a basis for an adaptive computational network model. The model is illustrated by a not too complex but realistic case study.     

Improving precision of ability estimation: Getting more from response times

By considering information about response time (RT) in addition to response accuracy (RA), joint models for RA and RT such as the hierarchical model (van der Linden, 2007) can improve the precision with which ability is estimated over models that only consider RA. The hierarchical model, however, assumes that only the person's speed is informative of ability. This assumption of conditional independence between RT and ability given speed may be violated in practice, and ignores collateral information about ability that may be present in the residual RTs. We propose a posterior predictive check for evaluating the assumption of conditional independence between RT and ability given speed. Furthermore, we propose an extension of the hierarchical model that contains cross-loadings between ability and RT, which enables one to take additional collateral information about ability into account beyond what is possible in the standard hierarchical model. A Bayesian estimation procedure is proposed for the model. Using simulation studies, the performance of the model is evaluated in terms of parameter recovery, and the possible gain in precision over the standard hierarchical model and an RA-only model is considered. The model is applied to data from a high-stakes educational test.     

Procedural learning: A developmental study of motor sequence learning and probabilistic classification learning in school-aged children

In this study, we investigated motor and cognitive procedural learning in typically developing children aged 8–12 years with a serial reaction time (SRT) task and a probabilistic classification learning (PCL) task. The aims were to replicate and extend the results of previous SRT studies, to investigate PCL in school-aged children, to explore the contribution of declarative knowledge to SRT and PCL performance, to explore the strategies used by children in the PCL task via a mathematical model, and to see whether performances obtained in motor and cognitive tasks correlated. The results showed similar learning effects in the three age groups in the SRT and in the first half of the PCL tasks. Participants did not develop explicit knowledge in the SRT task whereas declarative knowledge of the cue–outcome associations correlated with the performances in the second half of the PCL task, suggesting a participation of explicit knowledge after some time of exposure in PCL. An increasing proportion of the optimal strategy use with increasing age was observed in the PCL task. Finally, no correlation appeared between cognitive and motor performance. In conclusion, we extended the hypothesis of age invariance from motor to cognitive procedural learning, which had not been done previously. The ability to adopt more efficient learning strategies with age may rely on the maturation of the fronto-striatal loops. The lack of correlation between performance in the SRT task and the first part of the PCL task suggests dissociable developmental trajectories within the procedural memory system.