Application of a Multidimensional Nested Logit Model to Multiple-Choice Test Items

Nested logit models have been presented as an alternative to multinomial logistic models for multiple-choice test items (Suh and Bolt in Psychometrika 75:454–473, 2010) and possess a mathematical structure that naturally lends itself to evaluating the incremental information provided by attending to distractor selection in scoring. One potential concern in attending to distractors is the possibility that distractor selection reflects a different trait/ability than that underlying the correct response. This paper illustrates a multidimensional extension of a nested logit item response model that can be used to evaluate such distinctions and also defines a new framework for incorporating collateral information from distractor selection when differences exist. The approach is demonstrated in application to questions faced by a university testing center over whether to incorporate distractor selection into the scoring of its multiple-choice tests. Several empirical examples are presented.     

Analysis of distractor difficulty in multiple-choice items

Two psychometric models are presented for evaluating the difficulty of the distractors in multiple-choice items. They are based on the criterion of rising distractor selection ratios, which facilitates interpretation of the subject and item parameters. Statistical inferential tools are developed in a Bayesian framework: modal a posteriori estimation by application of an EM algorithm and model evaluation by monitoring posterior predictive replications of the data matrix. An educational example with real data is included to exemplify the application of the models and compare them with the nominal categories model.This research was supported by the DGI grant BSO2002-01485.I would like to thank Eric Maris and Vicente Ponsoda for their advice, Juan Botella for providing the data for the empirical application, and three anonymous reviewers for their comments that were essential for improving the quality of the paper.     

The effects of response priming on the planning and execution of goal-directed movements in the presence of a distracting stimulus

Two models of selective reaching have been proposed to account for deviations in movement trajectories in cluttered environments. The response vector model predicts movement trajectories should deviate towards or away from the location a distractor of little or large salience, respectively. In contrast, the response activation model predicts that a distractor with large salience should cause movement deviations towards it whereas a distractor with little salience should not influence the movement. The precuing technique was combined with the distractor interference paradigm to test these predictions. Results indicate that when the target was presented at the precued (salient) location, movements were unaffected by a distractor. Conversely, when the distractor was presented at the precued location while the target was presented at an uncued (non-salient) location, participants demonstrated increased reaction times and trajectory deviations towards the location of the distractor. These findings are consistent with the model of response activation.     

Distractor selection ratios

A latent variable representation for multiple-choice item and option characteristic curves is presented. Under standard assumptions of conditional independence of item responses and monotonicity of item characteristic curves, a criterion for distractors is proposed based on distractor selection ratios. A connection is made between the proposed criterion and the theory of individual choice behavior, providing new insight. The main results allow for the testing of the criterion from observable data without first specifying a parametric form for the characteristic curves. A series of examples apply the method.     

Nested diagnostic classification models for multiple-choice items

This study proposes and evaluates a diagnostic classification model framework for multiple-choice items. Models in the proposed framework have a two-level nested structure which allows for binary scoring (for correctness) and polytomous scoring (for distractors) at the same time. One advantage of these models is that they can provide distractor information while maintaining the statistical properties of the correct response option. We evaluated parameter recovery through a simulation study using Hamiltonian Monte Carlo algorithms in Stan. We also discussed three approaches to implementing the proposed modelling framework for different purposes and testing scenarios. We illustrated those approaches and compared them with a binary model and a traditional nominal model through an operational study.     

On the fate of distractor representations

Research on the topic of distractor inhibition has used different empirical approaches to study how the human mind selects relevant information from the environment, and the results are controversially discussed. One key question that typically arises is how selection deals with the irrelevant information. We used a new selection task, in which participants sometimes had to respond to the distractors instead of the target. Importantly, we varied the time interval between stimuli onset and the cue that signaled participants to respond to the distractors. We analyzed RTs and error rates from responses to distractors as a function of how long the target had been processed (and the distractor ignored) before the cue required a response to the distractor (i.e., stimulus-cue SOA). The data are compatible with selection models assuming that distractor stimuli are initially activated and then deactivated. Thus, we argue for selection models assuming top down deactivation of distractor representations that work in parallel with top down activation of target representations.     

Evidence of preliminary response preparation from a divided attention task

In a divided attention situation, preliminary response activations produced by stimuli on one channel were revealed through their effects on responses to stimuli on a secondary probe channel. Subjects performed concurrent but independent four-choice reaction-time tasks using the same four response fingers (middle and index fingers on both hands). In the main task, targets were large and small Ss and Ts, and medium-sized Ss and Ts sometimes appeared as distractors. Targets in the probe task were squares differing in location. A response to a probe square was faster if a distractor letter presented just before it had the same name as the target letter corresponding to that square (i.e., assigned to the same response key) than if the distractor letter had a different name--a result indicating that distractor letters cause partial response preparation. The timecourse of the effect demonstrated that preparation was based on preliminary information about distractor name that was available before distractor size had been determined. The results support models in which response preparation can sometimes begin before stimulus recognition has finished.     

A Signal Detection Model for Multiple-Choice Exams

A model for multiple-choice exams is developed from a signal-detection perspective. A correct alternative in a multiple-choice exam can be viewed as being a signal embedded in noise (incorrect alternatives). Examinees are assumed to have perceptions of the plausibility of each alternative, and the decision process is to choose the most plausible alternative. It is also assumed that each examinee either knows or does not know each item. These assumptions together lead to a signal detection choice model for multiple-choice exams. The model can be viewed, statistically, as a mixture extension, with random mixing, of the traditional choice model, or similarly, as a grade-of-membership extension. A version of the model with extreme value distributions is developed, in which case the model simplifies to a mixture multinomial logit model with random mixing. The approach is shown to offer measures of item discrimination and difficulty, along with information about the relative plausibility of each of the alternatives. The model, parameters, and measures derived from the parameters are compared to those obtained with several commonly used item response theory models. An application of the model to an educational data set is presented.     

Perceptual fluency and affect without recognition

A dichotic listening task was used to investigate the affect-without-recognition phenomenon. Subjects performed a distractor task by responding to the information presented in one ear while ignoring the target information presented in the other ear. The subjects' recognition of and affect toward the target information as well as toward foils was measured. The results offer evidence for the affect-without-recognition phenomenon. Furthermore, the data suggest that the subjects' affect toward the stimuli depended primarily on the extent to which the stimuli were perceived as familiar (i.e., subjective familiarity), and this perception was influenced by the ear in which the distractor or the target information was presented. These data are interpreted in terms of current models of recognition memory and hemispheric lateralization.     

Interference From Multi-dimensional Objects During Feature and Conjunction Discriminations

Feature discrimination performance within an attended object and interference from irrelevant, multi-dimensional objects (distractors) were examined in a two-choice, response compatibility paradigm. Results showed that the amount of interference by multi-dimensional distractors was dependent on three factors: (1) the discriminability of the incompatible, task-relevant distractor features; (2) the number of incompatible, task-relevant distractor features; and (3) whether the task-relevant, incompatible features matched the task goals. The most interesting finding was that additive priming effects were found for multiple, task-relevant features that matched the task goals, whether these features were present in the attended object or in the ignored object. Models that assume that each task-relevant feature primes its corresponding decision/response asynchronously and that this priming is combined to meet a decision/response criterion (at least when attended) can account for distractor interference during conjunction discriminations. Implications of these findings for feature integration models, template models, and a response selection model are discussed.     

A high-loaded hemisphere successfully ignores distractors

We used a response competition paradigm to investigate whether a distractor is effectively rejected under conditions where it is projected to a highly-loaded hemisphere. In two experiments we asked right-handed participants to identify a target among five task-relevant letters while they ignored a distractor. We manipulated both the distractor visual-field (low-load/high-load) and the compatibility of the target and the distractor. In the low-loaded visual-field, we presented a distractor with one task-relevant stimulus to one visual-field and the remaining task-relevant stimuli to the opposite visual-field. In the high-loaded visual-field, we presented a distractor and task-relevant stimuli in reverse. In Experiment 1 (left/right), we found a compatibility effect for the low-loaded, but not for the high-loaded visual-field. In Experiment 2, this modulation of the compatibility effect did not appear when the upper/lower visual-field was manipulated. These findings demonstrate that a distractor is successfully ignored when it is presented to a highly-loaded hemisphere.     

When does visual attention select all features of a distractor?

What happens after visual attention is allocated to an object? Although many theories of attention assume that all of its features are selected and processed, there has been little direct evidence that an irrelevant feature dimension of an attended nontarget is processed. In 5 experiments presented here, the authors used a singleton paradigm to investigate the effect of attention on nontarget objects. Participants made a speeded feature discrimination of a target for which the response was either compatible or incompatible with an irrelevant feature dimension of a distractor. The results show that the irrelevant distractor features were processed to the point that they interfered with the response to the target. The response compatibility effect was observed even when the location of the target or the distractor was invariant, although it was much weaker when both locations were invariant. These results demonstrate that in many circumstances, an attended distractor is completely selected and fully processed, and the complete processing of distractors depends on a number of factors, many of which are related to the strength of attention to the distractor.     

Electrophysiological correlates of crossmodal visual distractor congruency effects: Evidence for response conflict

To investigate the basis of crossmodal visual distractor congruency effects, we recorded event-related brain potentials (ERP) while participants performed a tactile location-discrimination task. Participants made speeded tactile location-discrimination responses to tactile targets presented to the index fingers or thumbs while ignoring simultaneously presented task-irrelevant visual distractor stimuli at either the same (congruent) or a different (incongruent) location. Behavioural results were in line with previous studies, showing slowed response times and increased error rates on incongruent compared with congruent visual distractor trials. To clarify the effect of visual distractors on tactile processing, concurrently recorded ERPs were analyzed for poststimulus, preresponse, and postresponse modulations. An enhanced negativity was found in the time range of the N2 component on incongruent compared with congruent visual distractor trials prior to correct responses. In addition, postresponse ERPs showed the presence of error-related negativity components on incorrect-response trials and enhanced negativity for congruent-incorrect compared with incongruent-incorrect trials. This pattern of ERP results has previously been related to response conflict (Yeung, Botvinick, & Cohen, 2004). Importantly, no modulation of early somatosensory ERPs was present prior to the N2 time range, which may have suggested the contribution of other perceptual or postperceptual processes to crossmodal congruency effects. Taken together, our results suggest that crossmodal visual distractor effects are largely due to response conflict.     

Asymmetries in a unilateral flanker task depend on the direction of the response: the role of attentional shift and perceptual grouping

Four experiments were conducted using a flanker task with 1 distractor appearing either on the left or right side of a central target. Responses were made on a keyboard aligned parallel to the displays. A larger flanker effect was obtained when the distractor was on the same side as the response. Two factors account for this asymmetry. First, when the flanker and target are identical, the 2 form a group that is assigned a spatial tag, creating a form of the Simon effect on the basis of the compatibility between the response keys and the group. Second, preparation of a lateralized response appears to entail a shift of visual attention in the corresponding direction, thus enhancing processing of the flanker on the response side. Consistent with the 2nd hypothesis, participants were more likely to correctly recognize letters that were briefly presented at the distractor position on the same side as the response.     

Retrieval of event files can be conceptually mediated

Distractor-based retrieval of event files was assessed with a sequential priming experiment using a four-choice identification task. Pictures or sounds of four different animals (frog, chicken, lamb, singing bird) had to be categorized by pressing one of four keys. On each trial, a target and a distractor stimulus were presented simultaneously in different modalities. The relevant modality switched randomly between trials. Distractor repetition effects were modulated by the response relation between the prime and probe: Repeating the prime distractor in the probe produced facilitation if the response repeated, but not if a different response had to be given in the prime and probe. Repeating the prime distractor in the probe led to an automatic retrieval of the prime response. Importantly, this distractor-based response retrieval effect also emerged for those sequences in which the modality of the repeated distractor was switched between the prime and probe. This cross-modal priming effect indicates that distractors were integrated into event files on a conceptual level and that response retrieval processes were mediated by conceptual codes of the distractor stimuli.     

Spatial negative priming modulation: The influence of probe-trial target cueing,distractor presence,and an intervening response

Reaction times are slower when a target (T) appears at a location that has just contained a distractor (D) (ignored-repetition trial), relative to when it arises at a previously unoccupied spatial position (control trial), i.e., the spatial negative priming (NP) effect. In a typical spatial NP paradigm trials are presented in pairs, first the prime and then the probe. Validly cueing ignored-repetition trials, and/or reducing probe distractor probability, modulated the NP process under certain conditions following target-plus-distractor (prime response) but not after distractor-only (no prime response) primes. This supported the idea that the production of a prime (intervening) response meets the needs for producing NP modulation. Additionally, NP elimination, evident when the probe was randomly distractor-free, was not seen when the probe also contained a distractor event. This suggests that the removal of the NP effect is likely achieved by blocking the retrieval of prime distractor information, rather than by removing the NP cause. Seemingly, the presence of a probe distractor is able to bypass the retrieval block.     

The effects of salience on saccadic target selection

Two experiments were conducted to investigate the effects of saliency on saccadic target selection as a function of time. Participants were required to make a speeded saccade towards a target defined by a unique orientation presented concurrently with multiple nontargets and one distractor. Target and distractor were equally salient within the orientation dimension but varied in saliency in the colour dimension. Within the colour dimension, the target presented could be more, equally, or less salient than the distractor. The results showed that saliency played a large role early during processing while no effects of saliency were found in later processing. Results are discussed in terms of models on visual selection.     

On the role of intervening distractors in the attentional blink

The attentional blink (AB) refers to the decline in accurate report for a second target (T2) when presented within about 500 ms of a first target (T1) embedded in a rapid serial visual presentation stream of distractors. It is debated whether the distractors presented shortly after T1 cause the AB directly, as is proposed by distractor-based models, or can modulate its amplitude only indirectly by increasing T1 difficulty, as is proposed by capacity-based models. To investigate this issue, an intervening distractor was presented at lag 1 (T1 + 1), at lag 2 (T1 + 2), or at neither of these two lags (no distractor). T2 was presented at either lag 3 or 9. An AB was observed even in the absence of intervening distractors, indicating that distractors are not necessary to produce an AB. Nonetheless, the T1 + 2 distractor did modulate the AB directly, without influencing T1 performance. Neither theory can fully account for the results but can do so given some modifications.     

Attentional control settings prevent abrupt onsets from capturing visual spatial attention

When a visual distractor appears earlier than a visual target in a target-detection task, response time is faster if the distractor appears at the same location as the target. When a visual distractor appears concurrently with a visual target in a target-detection task, response time is slowed relative to when no distractor is presented. Both effects have been taken as evidence of the capture of visual spatial attention, yet capture by early distractors is contingent on top-down attentional control settings (ACSs), and capture by concurrent distractors is not. The present study evaluated whether this incongruity is attributable to the timing of distractors (earlier than vs. concurrently with the target), or to the employed comparisons (same location/different location vs. distractor/no distractor). Using a task that presented both early and concurrent distractors, we observed that, regardless of timing, capture was contingent on ACSs when assessed by the same-location/different-location comparison. This result suggests that, although irrelevant stimuli cause nonspatial purely stimulus-driven effects, the capture of visual spatial attention is contingent on ACSs.