Multiple-answer multiple-choice test items: Responding and scoring through bayes and minimax strategies

A multiple-answer multiple-choice test item has a certain number of alternatives,any number of which might be keyed. The examinee is also allowed to mark any number of alternatives. This increased flexibility over the one keyed alternative case is useful in practice but raises questions about appropriate scoring rules. In this article a certain class of item scoring rules called thebinary class is considered. The concepts ofstandard scoring rules and equivalence among these scoring rules are introduced in the misinformation model for which the traditional knowledge model is a special case. The examinee's strategy with respect to a scoring rule is examined. The critical role of a quantity called the scoring ratio is emphasized. In the case of examinee uncertainty about the number of correct alternatives on an item, a Bayes and a minimax strategy for the examinee are developed. Also an appropriate response for the examiner to the minimax strategy is outlined.Research partially supported under Grants N00014-67-A-0314-0022 from the Office of Naval Research and GS-32514 and MPS 75-07539 from the National Science Foundation.     

A rating formulation for ordered response categories

A rating response mechanism for ordered categories, which is related to the traditional threshold formulation but distinctively different from it, is formulated. In addition to the subject and item parameters two other sets of parameters, which can be interpreted in terms of thresholds on a latent continuum and discriminations at the thresholds, are obtained. These parameters are identified with the category coefficients and the scoring function of the Rasch model for polychotomous responses in which the latent trait is assumed uni-dimensional. In the case where the threshold discriminations are equal, the scoring of successive categories by the familiar assignment of successive integers is justified. In the case where distances between thresholds are also equal, a simple pattern of category coefficients is shown to follow.This work was conducted in part in the first half of 1977 while the author was on study leave at the Danish Institute for Educational Research. The Institute provided required research facilities while The University of Western Australia provided financial support.     

Tests of symmetry in three-way contingency tables

Three-dimensional contingency tables are analyzed, with one variable (e.g., sex) as a factor, and with a natural relation between the other variables (e.g. left and right eye vision). Models of special interest, like symmetry and proportional symmetry between the related variables, and homogeneity across the factor levels, are investigated. Maximum likelihood estimators of parameters and partitions of chi-square goodness-of-fit statistics are explicitly presented; the independence of certain models is noted, and an example is discussed.     

Reliability of multiple classifications

Cohen's kappa index is reformulated for multiple classifications based on exchangeable random variables. It is found that kappa is between 0 and 1 inclusive. Two characterizations for kappa are stated in terms of the relationship between such random variables. Within the normal test score model, kappa increases with test reliability and test length. Furthermore, when based on binary classifications, kappa is an inverse U-shaped function of the cutoff score. These trends also hold for the beta-binomial test score model.Paper read at the Spring meeting of the Psychometric Society, Bell Laboratories (Murray Hill, New Jersey), March 1976.The editorial assistance and helpful comments of Leonard S. Feldt, Sarah P. Seaman-Huynh, and the three referees are gratefully acknowledged.     

Robust estimation of ability in the Rasch model

Estimating ability parameters in latent trait models in general, and in the Rasch model in particular is almost always hampered by noise in the data. This noise can be caused by guessing, inattention to easy questions, and other factors which are unrelated to ability. In this study several alternative formulations which attempt to deal with these problems without a reparameterization are tested through a Monte Carlo simulation. It was found that although no one of the tested schemes is uniformly superior to all others, a modified jackknife stood out as the best one in general, it was also super efficient (more efficient than the asymptotically optimal estimator) for tests with forty or fewer items. It is proposed that this sort of jackknifing scheme for estimating ability be considered for practical work.This research was funded through a grant from the Law Enforcement Assistance Administration (78-NI-AX-0047) to the Bureau of Social Science Research, Howard Wainer, Principal Investigator. We would like to thank Ronald Mead, Anne Morgan and James Ramsay for kind, generous, and invaluable help at various stages of the project.     

Dynamic equilibrium on a cyclic-interval schedule with a ramp

Five human subjects pressed a panel for money on a cyclic-interval schedule that arranged recurring periods of linearly increasing reinforcement rates (ramps). Response rate versus time functions for all subjects showed recurring periods of linearly increasing response rates. The responding of four of the five subjects was in phase with the reinforcement input. The remaining subject showed a two-minute phase shift. These results suggest that organisms may act like simple amplifiers on cyclic-interval schedules, that is, the form of the input signal is not changed by the organism, but is returned with amplification. By analogy with the variable-interval case, the controlling variable on cyclic-interval schedules with rate ramps may be the constant reinforcement acceleration that is arranged by the schedule.     

Timing the second response in two-response avoidance.

Rats were trained on a free-operant avoidance task requiring two lever presses within R seconds, with the opportunity for each response distinguished by differing stimuli. Response latencies at a variety of response-shock intervals were found to be proportional to the time available for the response. These results are shown to be consonant with a scalar expectancy model of timing behavior.     

The Del Greco Assertive Behavior Inventory

Previous research in the area of assertiveness training has been examined with respect to the definitions of assertive, aggressive, and nonassertive behavior and the prominent assertion model. Logical inconsistencies have led to a refinement of the definitions of these behaviors, the inclusion of passive aggressive behavior, and the testing of a new two-dimensional model of assertion. The present study has developed a psychometric test to measure assertive, aggressive, nonassertive, and passive aggressive behavior in the college dormitory population. In addition, support was demonstrated for the validity of the two dimensional model of assertion. The instrument development was divided into four phases: item generation, item evaluation and revision, reliability testing and item analysis, and validity testing. Data analysis supported the two-dimensional model of assertion. In addition, an 86-item Del Greco Assertive Behavior Inventory designed for use with college dormitory students has weathered preliminary validity and reliability testing.     

Tests for stability in linear structural equation systems

Interpretations regarding the effects of exogenous and endogenous variables on endogenous variables in linear structural equation systems depend upon the convergence of a matrix power series. Convergence depends upon the eigenvalues of the structural coefficient matrix. The test for convergence developed by Jöreskog and Sörbom is shown to be only sufficient, not necessary and sufficient.     

Maximum likelihood estimation of the joint covariance matrix for sections of tests given to distinct samples with application to test equating

Consider an old testX consisting ofs sections and two new testsY andZ similar toX consisting ofp andq sections respectively. All subjects are given testX plus two variable sections from either testY orZ. Different pairings of variable sections are given to each subsample of subjects. We present a method of estimating the covariance matrix of the combined test (X ₁, ...,X _s ,Y ₁, ...,Y _p ,Z ₁, ...,Z _q ) and describe an application of these estimation techniques to linear, observed-score, test equating.The author is indebted to Paul W. Holland and Donald B. Rubin for their encouragement and many helpful comments and suggestions that contributed significantly to the development of this paper.This research was supported by the Program Statistics Research Project of the ETS Research Statistics Group.