Estimation of error variances of test scores

The representation of test scores asn-dimensional points leads directly to an estimate of error variance at a particular score level in the case of equivalent items. Approximations are suggested for the case of non-equivalent items. These approximations are compared, with satisfactory results, with empirical data prepared by Dr. Mollenkopf.     

The Children's Depression Inventory: error in cutoff scores for screening purposes

The Children's Depression Inventory (CDI) is used to screen for depression in school-age children. Such screening uses the manual's reported information on suggested cutoff scores. These cutoff scores are based on an erroneous base-rate calculation and an inadequate methodology. Data are provided showing that for the suggested cutoff scores of 13 or 20, the CDI has poor receiver-operating characteristics. Indeed, for the cutoff score of 20, suggested as being suitable for screening in the general population (e.g., schools), clinicians will miss 86% of depressed children. In conclusion, it is recommended that the CDI is better suited as a continuous measure of mood and that cutoff scores should not be used to screen for the likely presence or absence of depression.     

A nonrandomized minimax solution for passing scores in the binomial error model

A nonrandomized minimax solution is presented for passing scores in the binomial error model. The computation does not require prior knowledge regarding an individual examinee or group test data for a population of examinees. The optimum passing score minimizes the maximum risk which would be incurred by misclassifications. A closed-form solution is provided for the case of constant losses, and tables are presented for a variety of situations including linear and quadratic losses. A scheme which allows for correction for guessing is also described.This work was performed pursuant to Grant No. NIE-G-78-0087 with the National Institute of Education, Department of Health, Education, and Welfare, Huynh Huynh, Principal Investigator. Points of view or opinions stated do not necessarily reflect NIE position or policy and no official endorsement should be inferred. The editorial assistance and comments of Anthony J. Nitko and of Joseph C. Saunders are gratefully acknowledged.     

基于双因子模型的测验总分和维度分的合成方法

双因子模型可以同时包含一个全局因子和多个局部因子,在描述多维测验结构时有其独特优势,近些年应用越来越广泛。文章基于双因子模型,提出了4种合成总分和维度分的方法,分别是:原始分法,加和法,全局题目加权加和法和局部题目加权加和法,并采用模拟的方法,在样本量、测验长度、维度间相关变化的条件下考察了这些方法与传统多维IRT方法的表现。最后,通过实证研究对结果进行了验证。结果显示:(1)全局加权加和法和局部加权加和法,尤其是局部加权加和法合成的总分和维度分与真值最接近、信度最高。(2)在维度间相关较高,测验长度较长的条件下,局部加权加和法的结果较好,部分条件下甚至优于多维IRT法。(3)仅有局部加权加和法合成的维度分能够反应维度间真实的相关关系。     

Prediction of true test scores from observed item scores and ancillary data

In many educational tests which involve constructed responses, a traditional test score is obtained by adding together item scores obtained through holistic scoring by trained human raters. For example, this practice was used until 2008 in the case of GRE^®General Analytical Writing and until 2009 in the case of TOEFL^® iBT Writing. With use of natural language processing, it is possible to obtain additional information concerning item responses from computer programs such as e‐rater^®. In addition, available information relevant to examinee performance may include scores on related tests. We suggest application of standard results from classical test theory to the available data to obtain best linear predictors of true traditional test scores. In performing such analysis, we require estimation of variances and covariances of measurement errors, a task which can be quite difficult in the case of tests with limited numbers of items and with multiple measurements per item. As a consequence, a new estimation method is suggested based on samples of examinees who have taken an assessment more than once. Such samples are typically not random samples of the general population of examinees, so that we apply statistical adjustment methods to obtain the needed estimated variances and covariances of measurement errors. To examine practical implications of the suggested methods of analysis, applications are made to GRE General Analytical Writing and TOEFL iBT Writing. Results obtained indicate that substantial improvements are possible both in terms of reliability of scoring and in terms of assessment reliability.     

Statistical consideration of mastery scores

The kind of mastery score problem often encountered in educational and psychological testing may be roughly described as follows: given that a certain unit of instruction must be mastered by a subject, a test is administered at the end of this unit and, on the basis of the observed score or an estimate of the true score, a decision is made about the grant of the mastery status. If the subject is declared master of the unit, he will be allowed to proceed to the next unit of instruction. Otherwise, remedial work will be given. Given relevant consequences of misclassification, the problem is to determine an optimum decision rule.Part of this work was completed while the author was Visiting Researcher at the Learning Research and Development Center, University of Pittsburgh, in the summer of 1973. Gratitude is extended to Robert Glaser for the invitation and to Anthony Nitko for helpful comments and suggestions.     

Accepting error to make less error

In this article I argue that the clinical and statistical approaches rest on different assumptions about the nature of random error and the appropriate level of accuracy to be expected in prediction. To examine this, a case is made for each approach. The clinical approach is characterized as being deterministic, causal, and less concerned with prediction than with diagnosis and treatment. The statistical approach accepts error as inevitable and in so doing makes less error in prediction. This is illustrated using examples from probability learning and equal weighting in linear models. Thereafter, a decision analysis of the two approaches is proposed. Of particular importance are the errors that characterize each approach: myths, magic, and illusions of control in the clinical; lost opportunities and illusions of the lack of control in the statistical. Each approach represents a gamble with corresponding risks and benefits.     

The error in the error theory

Moral error theory of the kind defended by J. L. Mackie and Richard Joyce is premised on two claims: (1) that moral judgements essentially presuppose that moral value has absolute authority, and (2) that this presupposition is false, because nothing has absolute authority. This paper accepts (2) but rejects (1). It is argued first that (1) is not the best explanation of the evidence from moral practice, and second that even if it were, the error theory would still be mistaken, because the assumption does not contaminate the meaning or truth-conditions of moral claims. These are determined by the essential application conditions for moral concepts, which are relational rather than absolute. An analogy is drawn between moral judgements and motion judgements.     

Formulas for component scores

General formulas for obtaining scores for individuals on components (factors derived from correlation matrices with unit communality estimates) are given. They are specialized to give Hotelling's formula for principal component scores. Formulas for scores on components rotated from principal components are developed.The final version of this paper was prepared while the author was a U. S. Public Health Service Fellow at the Center for Advanced Study in the Behavioral Sciences.     

Quadratic prediction of factor scores

Factor scores are naturally predicted by means of their conditional expectation given the indicatorsy. Under normality this expectation is linear iny but in general it is an unknown function ofy. It is discussed that under nonnormality factor scores can be more precisely predicted by a quadratic function ofy.The authors would like to thank Edith Nijenhuis, the anonymous referees, and the associate editor for their helpful comments and suggestions.     

The reliability of component scores

A method is given for determining the reliability of each of the components resulting from a factor analysis by the principal axis method.On leave for military service.