Statistical inference for multiple choice tests

Finite sample inference procedures are considered for analyzing the observed scores on a multiple choice test with several items, where, for example, the items are dissimilar, or the item responses are correlated. A discrete p-parameter exponential family model leads to a generalized linear model framework and, in a special case, a convenient regression of true score upon observed score. Techniques based upon the likelihood function, Akaike's information criteria (AIC), an approximate Bayesian marginalization procedure based on conditional maximization (BCM), and simulations for exact posterior densities (importance sampling) are used to facilitate finite sample investigations of the average true score, individual true scores, and various probabilities of interest. A simulation study suggests that, when the examinees come from two different populations, the exponential family can adequately generalize Duncan's beta-binomial model. Extensions to regression models, the classical test theory model, and empirical Bayes estimation problems are mentioned. The Duncan, Keats, and Matsumura data sets are used to illustrate potential advantages and flexibility of the exponential family model, and the BCM technique.The authors wish to thank Ella Mae Matsumura for her data set and helpful comments, Frank Baker for his advice on item response theory, Hirotugu Akaike and Taskin Atilgan, for helpful discussions regarding AIC, Graham Wood for his advice concerning the class of all binomial mixture models, Yiu Ming Chiu for providing useful references and information on tetrachoric models, and the Editor and two referees for suggesting several references and alternative approaches.     

Exact and best confidence intervals for the ability parameter of the Rasch model

A commonly used method to evaluate the accuracy of a measurement is to provide a confidence interval that contains the parameter of interest with a given high probability. Smallest exact confidence intervals for the ability parameter of the Rasch model are derived and compared to the traditional, asymptotically valid intervals based on the Fisher information. Tables of the exact confidence intervals, termed Clopper-Pearson intervals, can be routinely drawn up by applying a computer program designed by and obtainable from the author. These tables are particularly useful for tests of only moderate lengths where the asymptotic method does not provide valid confidence intervals.     

Enhancement of conditioned reinforcement by uncertainty

Pigeons were trained in three conditions. In the baseline condition, the birds responded on a fixed-interval schedule with the response key white. When the interval was completed, the key turned either red or green for a delay interval that was terminated by a grain presentation dependent on no key pecks during the final 2 sec. In the uncertainty condition, no grain was presented at the end of the delay periods when the key was red. In the certainty condition, the white light appeared only on occasions when pecking would turn the key green and produce food. Otherwise, the key was illuminated red throughout the total time period. The highest response rate in white occurred in the uncertainty condition, the next highest in the certainty condition, and the lowest in baseline. The results suggest that uncertainty facilitated responding, although uncertainty is not a necessary condition for conditioned reinforcement.     

Shoplifting prevention: providing information through signs

Shoplifting is one of the most frequent crimes in the United States, yet there is no agreement about effective prevention procedures. Since most prevention strategies are aimed at either increasing public awareness of the severity of the consequences or increasing the threat of detection, procedures that contain these elements were evaluated. Posting signs around a department of a department store pointing out that shoplifting is a crime, etc., partially reduced shoplifting rates. When merchandise that was frequently taken was identified by signs and stars, shoplifting decreased to near zero. Publicity campaigns to inform the public of consequences for shoplifting may produce desirable results, but identifying likely shoplifting targets, which may increase the likelihood of detection, effectively reduces shoplifting rates.     

The teacher-pupil interaction scale: An empirical method for analyzing the interactive effects of teacher and pupil behavior

The Teacher-Pupil Interaction Scale was developed for use by educational consultants to provide teachers with systematic data for analyzing and improving the learning of pupils. The scoring protocol provides for the simultaneous rating of sequential teacher-pupil interactions; a single mark is used to designate both teacher and pupil behavior at each five-second interval. Observations of a target pupil are alternated with other pupils in the classroom to serve as a point of comparison with the target pupil and to provide a general assessment of the classroom climate. Using the sequential record of their verbal and nonverbal interactions with students, teachers can pinpoint conditions serving both as antecedents and consequents of current student behavior. Armed with this information, teachers can effect changes in their instructional program to increase the rate of desired behaviors and eliminate or reduce unwanted responses.     

Maximum likelihood estimation of multivariate polyserial and polychoric correlation coefficients

The method of finding the maximum likelihood estimates of the parameters in a multivariate normal model with some of the component variables observable only in polytomous form is developed. The main stratagem used is a reparameterization which converts the corresponding log likelihood function to an easily handled one. The maximum likelihood estimates are found by a Fletcher-Powell algorithm, and their standard error estimates are obtained from the information matrix. When the dimension of the random vector observable only in polytomous form is large, obtaining the maximum likelihood estimates is computationally rather labor expensive. Therefore, a more efficient method, the partition maximum likelihood method, is proposed. These estimation methods are demonstrated by real and simulated data, and are compared by means of a simulation study.     

Reinforcement of human observing behavior by a stimulue correlated with extinction or increased effort

Young men pulled a plunger on mixed and multiple schedules in which periods of variable-interval monetary reinforcement alternated irregularly with periods of extinction (Experiment 1), or in which reinforcement was contingent on different degrees of effort in the two alternating components (Experiment 2). In the baseline conditions, the pair of stimuli correlated with the schedule components could be obtained intermittently by pressing either of two observing keys. In the main conditions, pressing one of the keys continued to produce both discriminative stimuli as appropriate. Pressing the other key produced only the stimulus correlated with variable-interval reinforcement or reduced effort; presses on this key were ineffective during periods of extinction or increased effort. In both experiments, key presses producing both stimuli occurred at higher rates than key presses producing only one, demonstrating enhancement of observing behavior by a stimulus correlated with the less favorable of two contingencies. A control experiment showed that stimulus change alone was not an important factor in the maintenance of the behavior. These findings suggest that negative as well as positive stimuli may play a role in the conditioned reinforcement of human behavior.     

Features of the Expert-System-Shell SPIRIT

The inference process in a probabilistic and conditional environmentunder minimum relative entropy, permits the acquisition of basicknowledge, the consideration of - even uncertain - ad hoc knowledge,and the response to queries. Even if these procedures are wellknown in the relevant literature their realisation for large-scaleapplications needs a sophisticated tool, allowing the communicationwith the user as well as all relevant logical transformationsand numerical calculations. SPIRIT is an Expert-System-Shellfor these purposes. Even for hundreds of – consistent– facts about the involved variables’ dependenciesthe shell automatically generates the corresponding epistemicstate, thus permitting the derivation of conclusions from theacquired knowledge. These conclusions’ reliability orprecision can be checked, inviting the user to enrich the knowledgeby further facts, if desired. Any inconsistencies among providedfacts are detected, and their elimination will be supportedby the shell. Knowledge acquisition can come from provided factsby a knowledge engineer as well as from real world data; inductivelearning supports the use of such data. An important capabilityof the shell is the calculation of impacts upon ideas or conceptsfrom a given stimulus. This paper is a brief survey of theoreticalconcepts and the corresponding features of the system, whichare accompanied by illustrative examples.     

Inferences by exclusion in the great apes: the effect of age and species

This study investigated the ability of chimpanzees, gorillas, orangutans, and bonobos to make inferences by exclusion using the procedure pioneered by Premack and Premack (Cognition 50:347–362, 1994) with chimpanzees. Thirty apes were presented with two different food items (banana vs. grape) on a platform and covered with identical containers. One of the items was removed from the container and placed between the two containers so that subjects could see it. After discarding this item, subjects could select between the two containers. In Experiment 1, apes preferentially selected the container that held the item that the experimenter had not discarded, especially if subjects saw the experimenter remove the item from the container (but without seeing the container empty). Experiment 3 in which the food was removed from one of the containers behind a barrier confirmed these results. In contrast, subjects performed at chance levels when a stimulus (colored plastic chip: Exp. 1; food item: Exp. 2 and Exp. 3) designated the item that had been removed. These results indicated that apes made inferences, not just learned to use a discriminative cue to avoid the empty container. Apes perceived and treated the item discarded by the experimenter as if it were the very one that had been hidden under the container. Results suggested a positive relationship between age and inferential ability independent of memory ability but no species differences.This contribution is part of the special issue “Animal Logics” (Watanabe and Huber 2006).     

Learning a new human-computer alphabet: the role of similarity and practice

Two purposes motivated this study: (a) to quantify the difficulty in learning various symbols of the alphabet used to enter data into a personal digital assistant (PDA), and (b) to investigate the interaction of item difficulty with practice conditions that promote varying levels of cognitive effort. Levels of compatibility between members of the PDA alphabet and English were quantified through introspective ratings in Experiment 1 and objective performance measures in Experiment 2. Three levels of item compatibility were learned under conditions of proactive or retroactive augmented information in Experiment 3. Contrary to expectations, the item similarity effect did not interact with practice schedules—a retroactive augmented information condition resulted in degraded levels of acquisition performance, but superior retention levels, compared to the proactive condition. These findings are discussed in terms of the relative merits of cognitive effort in skill acquisition.