Statistical inference based on latent ability estimates

The quality of approximations to first and second order moments (e.g., statistics like means, variances, regression coefficients) based on latent ability estimates is being discussed. The ability estimates are obtained using either the Rasch, or the two-parameter logistic model. Straightforward use of such statistics to make inferences with respect to true latent ability is not recommended, unless we account for the fact that the basic quantities are estimates. In this paper true score theory is used to account for the latter; the counterpart of observed/true score being estimated/true latent ability. It is shown that statistics based on the true score theory are virtually unbiased if the number of items presented to each examinee is larger than fifteen. Three types of estimators are compared: maximum likelihood, weighted maximum likelihood, and Bayes modal. Furthermore, the (dis)advantages of the true score method and direct modeling of latent ability is discussed.     

Some critical observations of the test information function as a measure of local accuracy in ability estimation

The test information function serves important roles in latent trait models and in their applications. Among others, it has been used as the measure of accuracy in ability estimation. A question arises, however, if the test information function is accurate enough for all meaningful levels of ability relative to the test, especially when the number of test items is relatively small (e.g., less than 50). In the present paper, using the constant information model and constant amounts of test information for a finite interval of ability, simulated data were produced for eight different levels of ability and for twenty different numbers of test items ranging between 10 and 200. Analyses of these data suggest that it is desirable to consider some modification of the test information function when it is used as the measure of accuracy in ability estimation.     

Weighted likelihood estimation of ability in item response theory

Applications of item response theory, which depend upon its parameter invariance property, require that parameter estimates be unbiased. A new method, weighted likelihood estimation (WLE), is derived, and proved to be less biased than maximum likelihood estimation (MLE) with the same asymptotic variance and normal distribution. WLE removes the first order bias term from MLE. Two Monte Carlo studies compare WLE with MLE and Bayesian modal estimation (BME) of ability in conventional tests and tailored tests, assuming the item parameters are known constants. The Monte Carlo studies favor WLE over MLE and BME on several criteria over a wide range of the ability scale.     

Bayesian estimation in the two-parameter logistic model

A Bayesian procedure is developed for the estimation of parameters in the two-parameter logistic item response model. Joint modal estimates of the parameters are obtained and procedures for the specification of prior information are described. Through simulation studies it is shown that Bayesian estimates of the parameters are superior to maximum likelihood estimates in the sense that they are (a) more meaningful since they do not drift out of range, and (b) more accurate in that they result in smaller mean squared differences between estimates and true values.The research reported here was performed pursuant to Grant No. N0014-79-C-0039 with the Office of Naval Research.     

Unbiased estimators of ability parameters,of their variance,and of their parallel-forms reliability

Given known item parameters, unbiased estimators are derived i) for an examinee's ability parameter and for his proportion-correct true score, ii) for the variances of and across examinees in the group tested, and iii) for the parallel-forms reliability of the maximum likelihood estimator .This work was supported in part by contract N00014-80-C-0402, project designation NR 150-453 between the Office of Naval Research and Educational Testing Service. Reproduction in whole or in part is permitted for any purpose of the United States Government.     

The role of collateral information about examinees in item parameter estimation

Standard procedures for estimating item parameters in item response theory (IRT) ignore collateral information that may be available about examinees, such as their standing on demographic and educational variables. This paper describes circumstances under which collateral information about examineesmay be used to make inferences about item parameters more precise, and circumstances under which itmust be used to obtain correct inferences.This work was supported by Contract No. N00014-85-K-0683, project designation NR 150-539, from the Cognitive Science Program, Cognitive and Neural Sciences Division, Office of Naval Research. Reproduction in whole or in part is permitted for any purpose of the United States Government. We are indebted to Tim Davey, Eugene Johnson, and three anonymous referees for their comments on earlier versions of the paper.     

四参数Logistic模型潜在特质参数的Warm加权极大似然估计

本文以四参数Logistic(4-parameter Logistic,4PL)模型为研究对象,根据Warm的加权极大似然估计技巧,提出了4PL模型潜在特质参数的加权极大似然估计方法,并借助模拟研究对加权极大似然估计的性质进行验证。研究结果表明,与通常的极大似然估计和后验期望估计相比,加权极大似然估计的偏差(bias)明显减小,并且具有良好的返真性能。此外,在测试的长度较短和项目的区分度较小的情况下,加权极大似然估计依然保持了良好的统计性质,表现出更加显著的优势。     

Estimating Optimal Weights for Compound Scores: A Multidimensional IRT Approach

A method is proposed for constructing indices as linear functions of variables such that the reliability of the compound score is maximized. Reliability is defined in the framework of latent variable modeling [i.e., item response theory (IRT)] and optimal weights of the components of the index are found by maximizing the posterior variance relative to the total latent variable variance. Three methods for estimating the weights are proposed. The first is a likelihood-based approach, that is, marginal maximum likelihood (MML). The other two are Bayesian approaches based on Markov chain Monte Carlo (MCMC) computational methods. One is based on an augmented Gibbs sampler specifically targeted at IRT, and the other is based on a general purpose Gibbs sampler such as implemented in OpenBugs and Jags. Simulation studies are presented to demonstrate the procedure and to compare the three methods. Results are very similar, so practitioners may be suggested the use of the easily accessible latter method. A real-data set pertaining to the 28-joint Disease Activity Score is used to show how the methods can be applied in a complex measurement situation with multiple time points and mixed data formats.     

Ability estimation for conventional tests

Five different ability estimators—maximum likelihood [MLE ()], weighted likelihood [WLE ()], Bayesian modal [BME ()], expected a posteriori [EAP ()] and the standardized number-right score [Z ()]—were used as scores for conventional, multiple-choice tests. The bias, standard error and reliability of the five ability estimators were evaluated using Monte Carlo estimates of the unknown conditional means and variances of the estimators. The results indicated that ability estimates based on BME (), EAP () or WLE () were reasonably unbiased for the range of abilities corresponding to the difficulty of a test, and that their standard errors were relatively small. Also, they were as reliable as the old standby—the number-right score.     

Nonunique solutions to the likelihood equation for the three-parameter logistic model

Samejima identified the possibility of multiple solutions to the likelihood equation (multiple maxima in the likelihood function) for estimating an examinee's trait value for the three-parameter logistic model. In the practical applications that Lord studied, he found that multiple solutions did not occur when the number of items was 20. In the present paper, fourteen multiple-choice achievement tests with from 20 to 50 items were examined to see if it was possible for them to produce item response vectors with multiple maxima; such vectors were found for all the tests. Examination of response vectors for large groups of real examinees found that from 0 to 3.1% of them had response vectors with multiple maxima. The implications of these results for multiple-choice tests are discussed.     

A semiparametric approach for item response function estimation to detect item misfit

When scaling data using item response theory, valid statements based on the measurement model are only permissible if the model fits the data. Most item fit statistics used to assess the fit between observed item responses and the item responses predicted by the measurement model show significant weaknesses, such as the dependence of fit statistics on sample size and number of items. In order to assess the size of misfit and to thus use the fit statistic as an effect size, dependencies on properties of the data set are undesirable. The present study describes a new approach and empirically tests it for consistency. We developed an estimator of the distance between the predicted item response functions (IRFs) and the true IRFs by semiparametric adaptation of IRFs. For the semiparametric adaptation, the approach of extended basis functions due to Ramsay and Silverman (2005) is used. The IRF is defined as the sum of a linear term and a more flexible term constructed via basis function expansions. The group lasso method is applied as a regularization of the flexible term, and determines whether all parameters of the basis functions are fixed at zero or freely estimated. Thus, the method serves as a selection criterion for items that should be adjusted semiparametrically. The distance between the predicted and semiparametrically adjusted IRF of misfitting items can then be determined by describing the fitting items by the parametric form of the IRF and the misfitting items by the semiparametric approach. In a simulation study, we demonstrated that the proposed method delivers satisfactory results in large samples (i.e., N ≥ 1,000).     

Application of the bootstrap methods in factor analysis

A Monte Carlo experiment is conducted to investigate the performance of the bootstrap methods in normal theory maximum likelihood factor analysis both when the distributional assumption is satisfied and unsatisfied. The parameters and their functions of interest include unrotated loadings, analytically rotated loadings, and unique variances. The results reveal that (a) bootstrap bias estimation performs sometimes poorly for factor loadings and nonstandardized unique variances; (b) bootstrap variance estimation performs well even when the distributional assumption is violated; (c) bootstrap confidence intervals based on the Studentized statistics are recommended; (d) if structural hypothesis about the population covariance matrix is taken into account then the bootstrap distribution of the normal theory likelihood ratio test statistic is close to the corresponding sampling distribution with slightly heavier right tail.This study was carried out in part under the ISM cooperative research program (91-ISM · CRP-85, 92-ISM · CRP-102). The authors would like to thank the editor and three reviewers for their helpful comments and suggestions which improved the quality of this paper considerably.     

Improving precision of ability estimation: Getting more from response times

By considering information about response time (RT) in addition to response accuracy (RA), joint models for RA and RT such as the hierarchical model (van der Linden, 2007) can improve the precision with which ability is estimated over models that only consider RA. The hierarchical model, however, assumes that only the person's speed is informative of ability. This assumption of conditional independence between RT and ability given speed may be violated in practice, and ignores collateral information about ability that may be present in the residual RTs. We propose a posterior predictive check for evaluating the assumption of conditional independence between RT and ability given speed. Furthermore, we propose an extension of the hierarchical model that contains cross-loadings between ability and RT, which enables one to take additional collateral information about ability into account beyond what is possible in the standard hierarchical model. A Bayesian estimation procedure is proposed for the model. Using simulation studies, the performance of the model is evaluated in terms of parameter recovery, and the possible gain in precision over the standard hierarchical model and an RA-only model is considered. The model is applied to data from a high-stakes educational test.     

Exact and best confidence intervals for the ability parameter of the Rasch model

A commonly used method to evaluate the accuracy of a measurement is to provide a confidence interval that contains the parameter of interest with a given high probability. Smallest exact confidence intervals for the ability parameter of the Rasch model are derived and compared to the traditional, asymptotically valid intervals based on the Fisher information. Tables of the exact confidence intervals, termed Clopper-Pearson intervals, can be routinely drawn up by applying a computer program designed by and obtainable from the author. These tables are particularly useful for tests of only moderate lengths where the asymptotic method does not provide valid confidence intervals.     

Bayes modal estimation in item response models

This article describes a Bayesian framework for estimation in item response models, with two-stage prior distributions on both item and examinee populations. Strategies for point and interval estimation are discussed, and a general procedure based on the EM algorithm is presented. Details are given for implementation under one-, two-, and three-parameter binary logistic IRT models. Novel features include minimally restrictive assumptions about examinee distributions and the exploitation of dependence among item parameters in a population of interest. Improved estimation in a moderately small sample is demonstrated with simulated data.This research was supported by a grant from the Spencer Foundation, Chicago, IL. Comments and suggestions on earlier drafts by Charles Lewis, Frederic Lord, Rosenbaum, James Ramsey, Hiroshi Watanabe, the editor, and two anonymous referees are gratefully acknowledged.     

An item response tree model with not-all-distinct end nodes for non-response modelling

The non-response model in Knott et al. (1991, Statistician, 40, 217) can be represented as a tree model with one branch for response/non-response and another branch for correct/incorrect response, and each branch probability is characterized by an item response theory model. In the model, it is assumed that there is only one source of non-responses. However, in questionnaires or educational tests, non-responses might come from different sources, such as test speededness, inability to answer, lack of motivation, and sensitive questions. To better accommodate such more realistic underlying mechanisms, we propose a a tree model with four end nodes, not all distinct, for non-response modelling. The Laplace-approximated maximum likelihood estimation for the proposed model is suggested. The validation of the proposed estimation procedure and the advantage of the proposed model over traditional methods are demonstrated in simulations. For illustration, the methodologies are applied to data from the 2012 Programme for International Student Assessment (PISA). The analysis shows that the proposed tree model has a better fit to PISA data than other existing models, providing a useful tool to distinguish the sources of non-responses.     

Optimum examinee samples for item parameter estimation in item response theory: A multi-objective programming approach

This paper proposes a multi-objective programming method for determining samples of examinees needed for estimating the parameters of a group of items. In the numerical experiments, optimum samples are compared to uniformly and normally distributed samples. The results show that the samples usually recommended in the literature are well suited for estimating the difficulty parameters. Furthermore, they are also adequate for estimating the discrimination parameters in the three-parameter model, butnot for the guessing parameters.     

On the estimation of parameters in latent structure analysis

In this note, we describe the iterative procedure introduced earlier by Goodman to calculate the maximum likelihood estimates of the parameters in latent structure analysis, and we provide here a simple and direct proof of the fact that the parameter estimates obtained with the iterative procedure cannot lie outside the allowed interval. Formann recently stated that Goodman's algorithm can yield parameter estimates that lie outside the allowed interval, and we prove in the present note that Formann's contention is incorrect.This research was supported in part by Research Contract No. NSF SOC 76-80389 from the Division of the Social Sciences of the National Science Foundation. The author is indebted to C. C. Clogg for helpful comments and for the numerical results reported here (see, e.g., Table 1).