Attribute weighting methods and decision quality in the presence of response error: a simulation study

This paper uses a simulation approach to investigate how different attribute weighting techniques affect the quality of decisions based on multiattribute value models. The weighting methods considered include equal weighting of all attributes, two methods for using judgments about the rank ordering of weights, and a method for using judgments about the ratios of weights. The question addressed is: How well does each method perform when based on judgments of attribute weights that are unbiased but subject to random error? To address this question, we employ simulation methods. The simulation results indicate that ratio weights were either better than rank order weights (when error in the ratio weights was small or moderate) or tied with them (when error was large). Both ratio weights and rank order weights were substantially superior to the equal weights method in all cases studied. Our findings suggest that it will usually be worth the extra time and effort required to assess ratio weights. In cases where the extra time or effort required is too great, rank order weights will usually give a good approximation to the true weights. Comparisons of the two rank-order weighting methods favored the rank-order-centroid method over the rank-sum method. © 1998 John Wiley & Sons, Ltd.     

Reliability as a function of the number of item options derived from the “knowledge or random guessing” model

For (0, 1) scored multiple-choice tests, a formula giving test reliability as a function of the number of item options is derived, assuming the knowledge or random guessing model, the parallelism of the new and old tests (apart from the guessing probability), and the assumptions of classical test theory. It is shown that the formula is a more general case of an equation by Lord, and reduces to Lord's equation if the items are effectively parallel. Further, the formula is shown to be closely related to another formula derived from Lord's randomly parallel tests model.     

On the asymptotic standard error of a class of robust estimators of ability in dichotomous item response models

In item response theory, the classical estimators of ability are highly sensitive to response disturbances and can return strongly biased estimates of the true underlying ability level. Robust methods were introduced to lessen the impact of such aberrant responses on the estimation process. The computation of asymptotic (i.e., large‐sample) standard errors (ASE) for these robust estimators, however, has not yet been fully considered. This paper focuses on a broad class of robust ability estimators, defined by an appropriate selection of the weight function and the residual measure, for which the ASE is derived from the theory of estimating equations. The maximum likelihood (ML) and the robust estimators, together with their estimated ASEs, are then compared in a simulation study by generating random guessing disturbances. It is concluded that both the estimators and their ASE perform similarly in the absence of random guessing, while the robust estimator and its estimated ASE are less biased and outperform their ML counterparts in the presence of random guessing with large impact on the item response process.     

Best linear prediction of composite universe scores

The problem of predicting universe scores for samples of examinees based on their responses to samples of items is treated. A general measurement procedure is described in which multiple test forms are developed from a table of specifications and each form is administered to a different sample of examinees. The measurement model categorizes items according to the cells of such a table, and the linear function derived for minimizing error variance in prediction uses responses to these categories. In addition, some distinctions are drawn between aspects of the approach taken here and the familiar regressed score estimates.The author thanks Robert L. Brennan, Michael J. Kolen, and Richard Sawyer for helpful comments and corrections, and anonymous reviewers for suggested improvements.     

On the standard error of the modified biserial correlation

A paradoxical implication of Kraemer's expression for the large-sample standard error of Brogden's form of the biserial correlation is identified, and a new expression is given which does not imply the paradox. However, numerical evidence is presented which calls into question the correctness of the expression.     

初中生学业错误的实然观与应然观关系研究

运用《初中生学业错误实然观问卷》和《初中生学业错误应然观问卷》调查了453名初中生的学业错误观,结果表明：（1）初中生学业错误的实然观与应然观均表现出显著的年级特征和性别差异;（2）初中生学业错误的应然观对实然观具有显著正向预测效用。     

Alpha,Dimension-Free,and Model-Based Internal Consistency Reliability

As pointed out by Sijtsma (in press), coefficient alpha is inappropriate as a single summary of the internal consistency of a composite score. Better estimators of internal consistency are available. In addition to those mentioned by Sijtsma, an old dimension-free coefficient and structural equation model based coefficients are proposed to improve the routine reporting of psychometric internal consistency. The various ways to measure internal consistency are also shown to be appropriate to binary and polytomous items. Research supported in part by grants DA00017 and DA01070 from the National Institute on Drug Abuse. This paper is based in part on Bentler (2003).     

Structural analysis of subjective categorical data

A general approach to the analysis of subjective categorical data is considered, in which agreement matrices of two or more raters are directly expressed in terms of error and agreement parameters. The method provides focused analyses of ratings from several raters for whom ratings have measurement error distributions that may induce bias in the evaluation of substantive questions of interest. Each rater's judgment process is modeled as a mixture of two components: an error variable that is unique for the rater in question as well as an agreement variable that operationalizes the true values of the units of observation. The statistical problems of identification, estimation, and testing of such measurement models are discussed.The general model is applied in several special cases. The most simple situation is that underlying Cohen's Kappa, where two raters place units into unordered categories. The model provides a generalization and systematization of the Kappa-idea to correct for agreement by chance. In applications with typical research designs, including a between-subjects design and a mixed within-subjects, between-subjects design, the model is shown to disentangle structural and measurement components of the observations, thereby controlling for possible confounding effects of systematic rater bias. Situations considered include the case of more than two raters as well as the case of ordered categories. The different analyses are illustrated by means of real data sets.The authors wish to thank Lawrence Hubert and Ivo Molenaar for helpful and detailed comments on a previous draft of this paper. Thanks are also due to Jens Möller und Bernd Strauß for the data from the 1992 Olympic Games. We thank the editor and three anonymous reviewers for valuable comments on an earlier draft.     

More “tricks of the trade”: Reparameterizing LISREL models using negative variances

This paper shows how LISREL may be used to estimate simplex models which impose constraints on the variances of endogenous variables. This technique allows us to estimate both the parameters and the standard errors of the correlated measurement error model proposed by Wiley and Wiley (1974).We would like to thank Jim Wiley for his many helpful comments and suggestions on an earlier draft. We are grateful also to an anonymous reviewer for supplying the EQS program presented in Figure 4.     

Effects of Error Correction on Spanish Spelling Words for Students Whose Primary Language Is Spanish

The purpose of this study was to examine the effects of an error correction strategy on the spelling accuracy of Spanish words by elementary school students who come from a bilingual home environment, where the home language is Spanish. In the error correction strategy students wrote the word, viewed an accurate model of the word, and self-corrected any errors. In the traditional strategy students wrote each word three times on a paper after viewing an accurate model. All words and examples were presented in the student's home language (Spanish). Results showed that students spelled more words correctly under the error correction strategy.