Estimating true score in the compound binomial error model

Several procedures have been proposed in the statistical literature for estimating simultaneously the mean of each ofk binomial populations. In terms of mental test theory, however, it is not clear that these procedures should be used when an item sampling model applies since the binomial error model is usually viewed as an oversimplification of the true situation. In this study we compare empirically several of these estimation techniques. Particular attention is given to situations where observations are generated according to a two-term approximation to the compound binomial distribution.The author would like to thank Shelley Niwa for writing the computer programs used in this study.The work upon which this publication is based was performed pursuant to Grant # NIE-G-76-0083 with the National Institute of Education, Department of Health, Education and Welfare. Points of view or opinions stated do not necessarily represent official NIE position or policy.     

Bayesian inference and the classical test theory model: Reliability and true scores

A general one-way analysis of variance components with unequal replication numbers is used to provide unbiased estimates of the true and error score variance of classical test theory. The inadequacy of the ANOVA theory is noted and the foundations for a Bayesian approach are detailed. The choice of prior distribution is discussed and a justification for the Tiao-Tan prior is found in the particular context of the “n-split” technique. The posterior distributions of reliability, error score variance, observed score variance and true score variance are presented with some extensions of the original work of Tiao and Tan. Special attention is given to simple approximations that are available in important cases and also to the problems that arise when the ANOVA estimate of true score variance is negative. Bayesian methods derived by Box and Tiao and by Lindley are studied numerically in relation to the problem of estimating true score. Each is found to be useful and the advantages and disadvantages of each are discussed and related to the classical test-theoretic methods. Finally, some general relationships between Bayesian inference and classical test theory are discussed. Supported in part by the National Institute of Child Health and Human Development under Research Grant 1 PO1 HDO1762. Reproduction, translation, use or disposal by or for the United States Government is permitted.     

Estimation of true score and error variance for tests under various equivalence assumptions

Maximum-likelihood estimators of true score variance and error variance for mental tests are derived for six different models of equivalent measurements. Statistical properties of the estimators are examined. Main emphasis is placed upon essentially-equivalent measurements. A statistical criterion for this type of measurement is given. The solution of the comparatively simple maximum-likelihood equations is effected by means of a rapid Newton-Raphson procedure. Two different initial estimators are considered and their relative merits in terms of second moments evaluated. Four numerical examples are appended by way of illustration.Paper read at the Mathematical Psychology Meeting, August 28, 1968, at Stanford, Calfornia.This research was supported in part by the National Institute of Child Health and Human Development, Research Grant 1 P01 HD01762.     

Estimation of reliability and true score variance from a split of a test into three arbitrary parts

This paper gives a method of estimating the reliability of a test which has been divided into three parts. The parts do not have to satisfy any statistical criteria like parallelism or-equivalence. If the parts are homogeneous in content (congeneric),i.e., if their true scores are linearly related and if sample size is large then the method described in this paper will give the precise value of the reliability parameter. If the homogeneity condition is violated then underestimation will typically result. However, the estimate will always be at least as accurate as coefficient and Guttman's lower bound ₃ when the same data are used. An application to real data is presented by way of illustration. Seven different splits of the same test are analyzed. The new method yields remarkably stable reliability estimates across splits as predicted by the theory. One deviating value can be accounted for by a certain unsuspected peculiarity of the test composition. Both coefficient and ₃ would not have led to the same discovery.Expanded version of a paper given at the Psychometric Society Meeting in Stanford, California, March 1974.     

The theory of the estimation of test reliability

The theoretically best estimate of the reliability coefficient is stated in terms of a precise definition of the equivalence of two forms of a test. Various approximations to this theoretical formula are derived, with reference to several degrees of completeness of information about the test and to special assumptions. The familiar Spearman-Brown Formula is shown to be a special case of the general formulation of the problem of reliability. Reliability coefficients computed in various ways are presented for comparative purposes.     

The error in the error theory

Moral error theory of the kind defended by J. L. Mackie and Richard Joyce is premised on two claims: (1) that moral judgements essentially presuppose that moral value has absolute authority, and (2) that this presupposition is false, because nothing has absolute authority. This paper accepts (2) but rejects (1). It is argued first that (1) is not the best explanation of the evidence from moral practice, and second that even if it were, the error theory would still be mistaken, because the assumption does not contaminate the meaning or truth-conditions of moral claims. These are determined by the essential application conditions for moral concepts, which are relational rather than absolute. An analogy is drawn between moral judgements and motion judgements.     

The variance error of the P50-discriminant

The P₅₀-discriminant has been reported elsewhere in connection with its use in predicting whether selective service registrants if inducted would become normal operative soldiers or would commit offenses causing their imprisonment. The standard error of the P₅₀-discriminant is a good measure to use in determining how far to the side of this statistic a particular case falls. The standard error formula itself has also been published elsewhere; but its derivation, as the variance error, is given here.The author gratefully acknowledges the very extensive assistance kindly given to him by Dr. Truman L. Kelley and Dr. Frederick Mosteller. This assistance was given without reference to the utility of the P₅₀-discriminant, upon which matter the author reports elsewhere and for which he takes full responsibility.     

The effects on the predictive variance of a new subject’s score for a new test

Abstract: It is often required to predict the scores or their variations under interest. Ishii and Watanabe (2001) investigated, in the context of psychological measurement, the Bayesian predictive distribution of a new subject’s scores for tests and subjects’ scores for a new test. In this paper, the Bayesian posterior predictive distribution of a new subject’s scores for a new parallel test were considered. And the effects of the number of subjects, the number of the tests, and the test reliability were investigated. Then, it was found that, under assumptions that (co)variance parameters are known, the predictive variance of a new subject’s score for a new test was equal to the predictive variances of the new subject’s scores for the existent tests. It was also found that the effect of the number of subjects was relatively large and the effect of the number of tests was relatively small, when a new subject’s scores for existent tests were not observed.     

Weakly parallel tests in latent trait theory with some criticisms of classical test theory

A new concept of weakly parallel tests, in contrast to strongly parallel tests in latent trait theory, is proposed. Some criticisms of the fundamental concepts in classical test theory, such as the reliability of a test and the standard error of estimation, are given.     

The variance theory of the mirror effect in recognition memory

The mirror effect refers to a rather general empirical finding showing that, for two classes of stimuli, the class with the higher hit rates also has a lower false alarm rate. In this article, a parsimonious theory is proposed to account for the mirror effect regarding, specifically, high- and low-frequency items and the associated receiver-operating curves. The theory is implemented in a recurrent network in which one layer represents items and the other represents contexts. It is shown that the frequency mirror effect is found in this simple network if the decision is based on counting the number of active nodes in such a way that performance is optimal or near optimal. The optimal performance requires that the number of active nodes is low, only nodes active in the encoded representation are counted, the activation threshold is set between the old and the new distributions, and normalization is based on the variance of the input. Owing to the interference caused by encoding the to-be-recognized item in several preexperimental contexts, the variance of the input to the context layer is greater for highthan for low-frequency items, which yields lower hit rates and higher false alarm rates for high- than for low-frequency items. Although initially the theory was proposed to account for the mirror effect with respect to word frequency, subsequent simulations have shown that the theory also accounts for strength-based mirror effects within a list and between lists. In this case, consistent with experimental data, the variance theory suggests that focusing attention to the more difficult class within a list affects the hit rate, but not the false alarm rate and not the standard deviations of the underlying density, leading to no mirror effect.     

The average error of a learning model,estimation and use in testing the fit of models

A measure of the discrepancy between observed transition frequencies and those predicted by a learning model, an average error of a learning model, is presented. The maximum-likelihood estimator of the average error is derived and its use in a modified test of goodness of fit is demonstrated.     

The mapping model: a cognitive theory of quantitative estimation

How do people make quantitative estimations, such as estimating a car's selling price? Traditionally, linear-regression-type models have been used to answer this question. These models assume that people weight and integrate all information available to estimate a criterion. The authors propose an alternative cognitive theory for quantitative estimation. The mapping model, inspired by the work of N. R. Brown and R. S. Siegler (1993) on metrics and mappings, offers a heuristic approach to decision making. The authors test this model against established alternative models of estimation, namely, linear regression, an exemplar model, and a simple estimation heuristic. With 4 experimental studies the authors compare the models under different environmental conditions. The mapping model proves to be a valid model to predict people's estimates.     

The classical limit of quantum theory

Both physicists and philosophers claim that quantum mechanics reduces to classical mechanics as 0, that classical mechanics is a limiting case of quantum mechanics. If so, several formal and non-formal conditions must be satisfied. These conditions are satisfied in a reduction using the Wigner transformation to map quantum mechanics onto the classical phase plane. This reduction does not, however, assist in providing an adequate metaphysical interpretation of quantum theory.Earlier versions of this essay received helpful criticism from Bruce Knight, Clark Glymour, and Donald Martin.     

Method-specific variance in the implicit association test

The Implicit Association Test (IAT; A. G. Greenwald, D. E. McGhee, & J. L. K. Schwartz, 1998) can be used to assess interindividual differences in the strength of associative links between representational structures such as attitude objects and evaluations. Four experiments are reported that explore the extent of method-specific variance in the IAT. The most important findings are that conventionally scored IAT effects contain reliable interindividual differences that are method specific but independent of the measures' content, and that IAT effects can be obtained in the absence of a preexisting association between the response categories. Several techniques to decrease the impact of method-specific variance are evaluated. The best results were obtained with the D measures recently proposed by A. G. Greenwald, B. A. Nosek, and M. R. Banaji (2003).     

On the mean and variance of response times under the diffusion model with an application to parameter estimation

We give closed form expressions for the mean and variance of RTs for Ratcliff’s diffusion model [Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85, 59-108] under the simplifying assumption that there is no variability across trials in the parameters. These expressions are more general than those currently available. As an application, we demonstrate their use in a method-of-moments estimation procedure that addresses some of the weaknesses of the EZ method [Wagenmakers, E.-J., van der Maas, H. L. J., & Grasman, R. P. P. P. (2007). An EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14, 3-22], and illustrate this with lexical decision data. We discuss further possible applications.