Reliability of test scores in nonparametric item response theory

Three methods for estimating reliability are studied within the context of nonparametric item response theory. Two were proposed originally by Mokken (1971) and a third is developed in this paper. Using a Monte Carlo strategy, these three estimation methods are compared with four classical lower bounds to reliability. Finally, recommendations are given concerning the use of these estimation methods.The authors are grateful for constructive comments from the reviewers and from Charles Lewis.     

Preferred Difficulty and Subjective Probability in Different Action Phases

Drawing on Gollwitzer's deliberative–implemental mindset distinction (P. M. Gollwitzer, 1990), it was predicted that people who are deliberating on different actions or goals would be more cautious or more realistic in their expectation of success in subsequent tasks than people who are going to implement a chosen action or goal. Participants were given a choice between different test-materials. They were interrupted before (deliberative) or immediately after decision-making (implemental). They then either had to choose between various levels of difficulty within one type of task (Experiment 1) or they had to predict their own future performance (Experiment 2). The results showed that deliberative participants preferred less difficult tasks and overestimated their probability of success less than implemental participants. In addition, deliberative participants referred more than implemental participants to their past performance when selecting levels of difficulty or predicting future performance; however, the two groups did not differ in actual performance. Taken together, the findings suggest that people are more realistic in a deliberative than in an implemental state of mind. The present studies extend prior research because for the first time they document mindset effects on peoples' estimates concerning their future performance in the achievement domain.     

Parents' estimations of their own intelligence and that of their children: a comparison between English and Icelandic parents

In this study, 158 parents (79 fathers and 79 mothers) with a mean age of 38.3 yrs (SD = 8.2), estimated their own, and their children's, overall intelligence as well as their children's score on the 12 intelligence scales from the Wechsler's Intelligence Scale for Children (WISC-III). The sample included English (n = 122) and Icelandic parents (n = 36), and a comparison between them showed few differences except that Icelandic parents' estimates were lower than English parents' estimates. The results showed fathers estimated their own overall intelligence higher than mothers estimated theirs and sons were estimated higher than daughters on overall intelligence. Two factors (verbal, performance) of intelligence were identified through factor analysis of the ratings of the 12 WISC subscale score estimates. A hierarchical regression showed that these two factors explained most of the variance in the estimation of the child's overall intelligence; however, gender of child and parents' self-estimated own overall intelligence added incremental variance.     

Nonparametric IRT: Testing the bi-isotonicity of isotonic probabilistic models (ISOP)

Nonparametric tests for testing the validity of polytomous ISOP-models (unidimensional ordinal probabilistic polytomous IRT-models) are presented. Since the ISOP-model is a very general nonparametric unidimensional rating scale model the test statistics apply to a great multitude of latent trait models. A test for the comonotonicity of item sets of two or more items is suggested. Procedures for testing the comonotonicity of two item sets and for item selection are developed. The tests are based on Goodman-Kruskal's gamma index of ordinal association and are generalizations thereof. It is an essential advantage of polytomous ISOP-models within probabilistic IRT-models that the tests of validity of the model can be performed before and without the model being fitted to the data. The new test statistics have the further advantage that no prior order of items or subjects needs to be known.     

Testing whether independent treatment groups have equal medians

The paper suggests new methods for comparing the medians corresponding to independent treatment groups. The procedures are based on the Harrell-Davis estimator in conjunction with a slight modification and extension of the bootstrap calibration technique suggested by Loh. Alternatives to the Harrell-Davis estimator are briefly discussed. For the special case of two treatment groups, the proposed procedure always had more power than the Fligner-Rust solution, as well as the procedure examined by Wilcox and Charlin. Included is an illustration, using real data, that comparing medians, rather than means, can yield a substantially different conclusion as to whether two distributions differ in terms of some measure of central location.