Statistical inference for multiple choice tests

Finite sample inference procedures are considered for analyzing the observed scores on a multiple choice test with several items, where, for example, the items are dissimilar, or the item responses are correlated. A discrete p-parameter exponential family model leads to a generalized linear model framework and, in a special case, a convenient regression of true score upon observed score. Techniques based upon the likelihood function, Akaike's information criteria (AIC), an approximate Bayesian marginalization procedure based on conditional maximization (BCM), and simulations for exact posterior densities (importance sampling) are used to facilitate finite sample investigations of the average true score, individual true scores, and various probabilities of interest. A simulation study suggests that, when the examinees come from two different populations, the exponential family can adequately generalize Duncan's beta-binomial model. Extensions to regression models, the classical test theory model, and empirical Bayes estimation problems are mentioned. The Duncan, Keats, and Matsumura data sets are used to illustrate potential advantages and flexibility of the exponential family model, and the BCM technique.The authors wish to thank Ella Mae Matsumura for her data set and helpful comments, Frank Baker for his advice on item response theory, Hirotugu Akaike and Taskin Atilgan, for helpful discussions regarding AIC, Graham Wood for his advice concerning the class of all binomial mixture models, Yiu Ming Chiu for providing useful references and information on tetrachoric models, and the Editor and two referees for suggesting several references and alternative approaches.     

Prospect theory analysis of guessing in multiple choice tests

The guessing of answers in multiple choice tests adds random error to the variance of the test scores, lowering their reliability. Formula scoring rules that penalize for wrong guesses are frequently used to solve this problem. This paper uses prospect theory to analyze scoring rules from a decision‐making perspective and focuses on the effects of framing on the tendency to guess. In three experiments participants were presented with hypothetical test situations and were asked to indicate the degree of certainty that they thought was required for them to answer a question. In accordance with the framing hypothesis, participants tended to guess more when they anticipated a low grade and therefore considered themselves to be in the loss domain, or when the scoring rule caused the situation to be framed as entailing potential losses. The last experiment replicated these results with a task that resembles an actual test. Copyright © 2002 John Wiley & Sons, Ltd.     

Total retrieval time and hypermnesia: Investigating the benefits of multiple recall tests

Hypermnesia is an increase in recall over repeated tests. A core issue is the role of repeated testing, per se, versus total retrieval time. Prior research implies an equivalence between multiple recall tests and a single test of equal total duration, but theoretical analyses indicate otherwise. Three experiments investigated this issue using various study materials (unrelated word lists, related word lists, and a short story). In the first experimental session, the study phase was followed by a series of short recall tests or by a single, long test of equal total duration. Two days later, participants took a final recall test. The multiple and single test conditions produced equivalent performance in the first session, but the multiple test group exhibited less forgetting and fewer item losses in the final test. In a fourth experiment, using a brief delay (15 min) between the recall sessions, the multiple recall condition produced greater hypermnesia as well as fewer item losses. In addition, final recall was significantly higher in the multiple than in the single test condition in three of the four experiments. Thus, single and repeated recall tests of equal total duration are not functionally equivalent, but rather produce differences observable in subsequent recall tests.     

Effects of time pressure on strategy selection and strategy execution in forced choice tests

We examined the effects of cognitive load on the strategy selection in the forced choice test (FCT) when used to detect hidden crime knowledge. Examinees (N = 120) with and without concealed knowledge from a mock crime were subjected to an FCT either under standard circumstances or cognitive load. Cognitive load was implemented through time pressure. The FCT distinguished examinees with concealed knowledge from those without better than chance in both conditions, but the counterstrategies did not differ between conditions. Further investigation revealed that time pressure did affect examinees' ability to follow their intended counterstrategy to produce randomized test patterns, which constitutes an effective counterstrategy in the FCT. Hence, time pressure lowered the success rate of effective counterstrategies, but not their incident rates. Further disambiguation of various cognitive load manipulations and their effects on strategy selection and execution is needed.     

A modified beta binomial model with applications to multiple choice and taste tests

A modified beta binomial model is presented for use in analyzing ramdom guessing multiple choice tests and certain forms of taste tests. Detection probabilities for each item are distributed beta across the population subjects. Properties for the observable distribution of correct responses are derived. Two concepts of true score estimates are presented. One, analogous to Duncan's empirical Bayes posterior mean score, is appropriate for assessing the subject's performance on that particular test. The second is more suitable for predicting outcomes on similar tests.This research was made possible by a grant from the Center for Food Policy Research, Graduate School of Business, Columbia University.     

Handling time and choice in pigeons

According to optimal foraging theory, animals should prefer food items with the highest ratios of energy intake to handling time. When single items have negligible handling times, one large item should be preferred to a collection of small ones of equivalent total weight. However, when pigeons were offered such a choice on equal concurrent variable-interval schedules in a shuttlebox, they preferred the side offering many small items per reinforcement to that offering one or a few relatively large items. This preference was still evident on concurrent fixed-cumulative-duration schedules in which choosing the alternative with longer handling time substantially lowered the rate of food intake.     

Probability effects in choice reaction time tasks

This paper contains a short review of the main results that were obtained by the author in a series of experiments that constituted a study of the effects of signal probability on choice reaction time. The effects of stimulus probability are shown to be influenced by the following variables: (1) differences in the method of varying stimulus probability, (2) differences in task complexity, (3) differences in S-R code, and (4) differences in Ss’ motivation. The data that are considered here are the overall mean RT for particular signals and the mean RT for sequential repetitions. Two questions, related to the psychological “nature” of the probability effects in choice RT are discussed: (1) The question of the relationship between the relative frequency and the number of alternatives as two different ways of determining the probability effect in choice RT; and (2) the question of identifying the main determinants of the trial-to-trial variability of RT in such experiments.

     

Temporal constraint on choice: Sensitivity and bias in multiple schedules

In multiple schedules of reinforcement, ratios of responses in successive components are relatively insensitive to ratios of obtained reinforcers. An analysis is proposed that attributes changes in absolute response rates to concurrent interactions between programmed reinforcement and extraneous reinforcement in other components. The analysis predicts that ratios of responses in successive components vary with reinforcer ratios, qualified by a term describing the reinforcement context, that is, programmed and extraneous reinforcers. Two main predictions from the analysis were confirmed in an experiment in which pigeons' responses were reinforced in the components of a multiple schedule and analog extraneous reinforcement was scheduled for an alternative response in each component. Sensitivity of response and time ratios to reinforcer ratios in the multiple schedules varied as a function of the rate of extraneous reinforcers. Bias towards responding in one component of the multiple schedule varied as an inverse function of the ratios of extraneous reinforcer rate in the two components. The data from this and previous studies of multiple-concurrent performance were accurately predicted by our analysis and supported our contention that the allocation of behavior in multiple-schedule components depends on the relative values of concurrently-available reinforcers within each component.     

Response-stimulus interval in choice serial reaction time: Interaction with sleep deprivation, choice, and practice

In choice serial reaction time (RT), Response-to-next-Stimulus Interval (RSI) was varied from 0 to 600 msec in 40-msec steps. In three experiments, RT fell and errors rose as RSI was increased to 480 msec; they remained unchanged thereafter. The effect of RSI on RT was not linear, was reduced by 6- as compared with 4- or 3-choice responding, and was unaffected by sleep deprivation, despite loss of sleep reducing RT overall. The effect of penultimate RSI on RT was similar to that current RSI, but smaller. Two explanations of RSI—response-generated kinaesthetic feedback blocking a “central processor” and a preparatory interval as in warned simple RT—are rejected. Instead, the idea of “relative refractory state” is revived but now, because of the RSI/error finding, biased more towards responding than stimulus reception and encoding. In all three experiments the influence of RSI on RT was reduced with practice. If practice encourages automatic rather than controlled processing (Shiffrin & Schneider, 1977), the prediction is that the former will show less refractoriness.     

Residence time and choice in concurrent foraging schedules

Five pigeons were trained on a concurrent-schedule analogue of the “some patches are empty” procedure. Two concurrently available alternatives were arranged on a single response key and were signaled by red and green keylights. A subject could travel between these alternatives by responding on a second yellow “switching” key. Following a changeover to a patch, there was a probability (p) that a single reinforcer would be available on that alternative for a response after a time determined by the value of λ, a probability of reinforcement per second. The overall scheduling of reinforcers on the two alternatives was arranged nonindependently, and the available alternative was switched after each reinforcer. In Part 1 of the experiment, the probabilities of reinforcement, ρ_red and ρ_green, were equal on the two alternatives, and the arranged arrival rates of reinforcers, λ_red and λ_green, were varied across conditions. In Part 2, the reinforcer arrival times were arranged to be equal, and the reinforcer probabilities were varied across conditions. In Part 3, both parameters were varied. The results replicated those seen in studies that have investigated time allocation in a single patch: Both response and time allocation to an alternative increased with decreasing values of λ and with increasing values of ρ, and residence times were consistently greater than those that would maximize obtained reinforcer rates. Furthermore, both response- and time-allocation ratios undermatched mean reinforcer-arrival time and reinforcer-frequency ratios.     

Goodness-of-fit tests for models of latency and choice

A goodness-of-fit test based on the maximum likelihood criterion is derived for use in evaluating models of choice reaction time that predict choice probabilities and means and variances of latency. Special cases of the test involving models that predict only one or two of these statistics are considered and shown to be asymptotically identical to the traditional goodness-of-fit tests appropriate for these special cases.     

Sequential effects in auditory choice reaction time tasks

This paper concerns sequential effects in choice reaction time tasks. Performance in two interleaved auditory tasks was examined, and two general types of sequential effects were revealed. First, a response repetition effect occurred: Subjects were facilitated in responding when both the stimulus and the response were immediately repeated. Generally, it appeared that subjects were operating according to the bypass rule—that is, repeat the response if the stimulus or some aspect thereof is repeated from the preceding trial; otherwise, change the response. In addition, the experiment also revealed a second type of sequential effect, known as a task-switching effect. Subjects were overall slower to respond when the task changed between adjacent trials than when there was no task change. A final result was that subjects were markedly impaired when the stimulus changed but the same response had to be repeated. This finding has been reported elsewhere when purely visual tasks have been used. Hence, it seems that particular difficulties arise, in such sequential testing situations, when type-distinct stimuli are grouped into the same response categories.     

Travel time and concurrent-schedule choice: retrospective versus prospective control

Six pigeons were trained on concurrent variable-interval schedules in which two different travel times between alternatives, 4.5 and 0.5 s, were randomly arranged. In Part 1, the next travel time was signaled while the subjects were responding on each alternative. Generalized matching analyses of performance in the presence of the two travel-time signals showed significantly higher response and time sensitivity when the longer travel time was signaled compared to when the shorter time was signaled. When the data were analyzed as a function of the previous travel time, there were no differences in sensitivity. Dwell times on the alternatives were consistently longer in the presence of the stimulus that signaled the longer travel time than they were in the presence of the stimulus that signaled the shorter travel time. These results are in accord with a recent quantitative account of the effects of travel time. In Part 2, no signals indicating the next travel time were given. When these data were analyzed as a function of the previous travel time, time-allocation sensitivity after the 4.5-s travel time was significantly greater than that after the 0.5-s travel time, but no such difference was found for response allocation. Dwell times were also longer when the previous travel time had been longer.     

Precueing response factors in choice reaction time: a word of caution

This note discusses the assumption researchers make when movement attributes are precued in a choice reaction time (RT) paradigm-i.e., that the precue only reduces the number of response alternatives, so that changes in RT reflect the programming time for the unprecued attributes. However, it is argued that since the precue reduces the number of stimulus alternatives, changes in RT reflect changes in both stimulus processing and response programming. Therefore, the interpretation made by Klapp (1977), using the precue paradigm, that response selection and response timing are not independent sequential stages in the programming sequence, is not necessarily correct.