An automated test development of parallel tests from a seed test

Binary programming models are presented to generate parallel tests from an itembank. The parallel tests are created to match item for item an existing seed test and match user supplied taxonomic specifications. The taxonomic specifications may be either obtained from the seed test or from some other user requirement. An algorithm is presented along with computational results to indicate the overall efficiency of the process. Empirical findings based on an itembank for the Arithmetic Reasoning section of the Armed Services Vocational Aptitude Battery are given.The Office of Naval Research, Program in Cognitive Science, N00014-87-C-0696 partially supported the work of Douglas H. Jones. The Rutgers Research Resource Committee of the Graduate School of Management partially supported the work of Douglas H. Jones and Ing-Long Wu. A Thomas and Betts research fellowship partially supported the work of Ing-Long Wu. The Human Resources Laboratory, United States Air Force, partially supported the work of Ronald Armstrong. The authors benefited from conversations with Dr. Wayne Shore, Operational Technologies, San Antonio, Texas. The order of authors' names is alphabetical and denotes equal authorship.     

Millon Clinical Multiaxial Inventory (MCMI) computer-generated diagnoses: How do they compare to clinician judgment?

During the past several decades, computers have achieved increasing prominence in psychological assessment procedures. This is particularly true for computer-based test interpretation and diagnosis. This study reports on a study designed to compare the accuracy of computer-based diagnoses with clinician-generated diagnoses. The Millon Clinical Multiaxial Inventory (MCMI) was administered to 151 consecutively admitted inpatients at a large private psychiatric hospital. The computer-generated diagnoses were compared with those generated by admitting psychiatrists. The results indicated that the MCMI diagnostic impressions underestimated the severity of depressive disorders when compared with clinician diagnoses on Axis I. Specifically, clinicians diagnosed major depression much more frequently than did the MCMI. In addition, clinicians diagnosed anxiety disorders much less frequently than did the MCMI.     

Stability criteria

Three approaches to the determination of behavioral stability were examined. In the first, a learning curve was fit to acquisition data (from Cumming and Schoenfeld, 1960), and the “experiment” stopped when the data approached sufficiently close to the theoretical asymptote. In the second, the data were analyzed for variability and linear and quadratic trend. In the third, the experiment was stopped when the magnitude of the daily changes in the data fell below a criterion. Accuracy was measured as deviation between the average value of the dependent variable when the experiment was stopped, and the average value over the last 100 sessions. The first approach was most accurate, but at the cost of requiring the most sessions and being the most difficult to apply. Both the second and third approaches provided acceptable criteria with a reasonable cost-accuracy tradeoff. The second approach permits a continuous adjustment of the criteria to accommodate the variability intrinsic in the experimental paradigm. The third, nomothetic, approach also takes into account the decreasing marginal utility of extended training sessions.     

INTERACTIONS BETWEEN TEACHER GUIDANCE AND CONTINGENT ACCESS TO PLAY IN DEVELOPING PREACADEMIC SKILLS OF DEVIANT PRESCHOOL CHILDREN1

Token-mediated access to play and snacks was made contingent on completion of academic tasks in the Baseline Experiment. This contingency produced stable completion rates that were subsequently doubled, and then tripled, for four deviant children in a special preschool. A reversal design demonstrated that the contingency was functional in maintaining the children's rates of task completion. The Guidance Experiment examined the role of a social event, teacher guidance, in the acquisition of task-completion skills, in a multiple-baseline-across-tasks design (with reversals). The analysis demonstrated that teacher guidance was an important supplement to the token-mediated contingency in establishing significant increases in task completions for a second group of three deviant children in the special class. The importance of teacher guidance was related to the difficulty level of the children's tasks.     

A note on computing Louis’ observed information matrix identity for IRT and cognitive diagnostic models

Using Louis’ formula, it is possible to obtain the observed information matrix and the corresponding large-sample standard error estimates after the expectation–maximization (EM) algorithm has converged. However, Louis’ formula is commonly de-emphasized due to its relatively complex integration representation, particularly when studying latent variable models. This paper provides a holistic overview that demonstrates how Louis’ formula can be applied efficiently to item response theory (IRT) models and other popular latent variable models, such as cognitive diagnostic models (CDMs). After presenting the algebraic components required for Louis’ formula, two real data analyses, with accompanying numerical illustrations, are presented. Next, a Monte Carlo simulation is presented to compare the computational efficiency of Louis’ formula with previously existing methods. Results from these presentations suggest that Louis’ formula should be adopted as a standard method when computing the observed information matrix for IRT models and CDMs fitted with the EM algorithm due to its computational efficiency and flexibility.     

多级评分认知诊断模型述评

认知诊断评估旨在探讨个体内部的知识掌握结构,并提供关于学生优缺点的详细诊断信息,以促进个体的全面发展。当前研究者已开发了大量0-1评分的认知诊断模型,但对于多级评分认知诊断模型的研究还比较少。本文对已有的多级评分认知诊断模型进行了归纳,介绍了模型的假设,计量特征以及适用范围,为实际应用者和研究者在多级评分认知诊断模型的比较和选用上提供借鉴和参考。最后,对未来关于多级评分诊断模型的研究方向进行了展望。     

A Theoretical Framework for Analysing the Notion of Relative Importance of Criteria

Multiple-criteria decision aid almost always requires the use of weights, importance coefficients or even a hierarchy of criteria, veto thresholds, etc. These are importance parameters that are used to differentiate the role devoted to each criterion in the construction of comprehensive preferences. Many researchers have studied the problem of how to assign values to such parameters, but few of them have tried to analyse in detail what underlies the notion of importance of criteria and to give a clear formal definition of it. In this paper our purpose is to define a theoretical framework so as to analyse the notion of the importance of criteria under very general conditions. Within this framework it clearly appears that the importance of criteria is taken into account in very different ways in various aggregation procedures. This framework also allows us to shed new light on fundamental questions such as: Under what conditions is it possible to state that one criterion is more important than another? Are importance parameters of the various aggregation procedures dependent on or independent of the encoding of criteria? What are the links between the two concepts of the importance of criteria and the compensatoriness of preferences? This theoretical framework seems to us sufficiently general to ground further research in order to define theoretically valid elicitation methods for importance parameters.     

Selecting a new technology strategy using multiattribute,multilevel decision trees

Research indicates that selecting a strategy to best exploit a new technology is a complex decision-making process. The task involves making a series of decisions with multiple alternatives, each to be evaluated by multiple criteria whose values have high levels of uncertainty. This paper presents a methodology for modelling a new technology decision using decision trees and an optimizing algorithm. A problem of a mining company considering the adoption of new technology is used to illustrate the decision-making task and modelling methodology. A numerical solution to the case demonstrates the potential of the optimizing technique in strategy selection.     

The Performance Diagnostic Checklist-Human Services: A brief review

The Performance Diagnostic Checklist-Human Services (PDC-HS) is an informant-based tool designed to identify the variables contributing to poor employee performance in human service settings, such as clinics, schools, and residential facilities. Upon completion of the tool, an intervention indicated by PDC-HS results is used to improve employee performance. To date, the PDC-HS has been used in a number of studies. This review describes the existing research on the PDC-HS and provides suggestions for future research.