Nested Logit Models for Multiple-Choice Item Response Data

Nested logit item response models for multiple-choice data are presented. Relative to previous models, the new models are suggested to provide a better approximation to multiple-choice items where the application of a solution strategy precedes consideration of response options. In practice, the models also accommodate collapsibility across all distractor categories, making it easier to allow decisions about including distractor information to occur on an item-by-item or application-by-application basis without altering the statistical form of the correct response curves. Marginal maximum likelihood estimation algorithms for the models are presented along with simulation and real data analyses.     

Nested diagnostic classification models for multiple-choice items

This study proposes and evaluates a diagnostic classification model framework for multiple-choice items. Models in the proposed framework have a two-level nested structure which allows for binary scoring (for correctness) and polytomous scoring (for distractors) at the same time. One advantage of these models is that they can provide distractor information while maintaining the statistical properties of the correct response option. We evaluated parameter recovery through a simulation study using Hamiltonian Monte Carlo algorithms in Stan. We also discussed three approaches to implementing the proposed modelling framework for different purposes and testing scenarios. We illustrated those approaches and compared them with a binary model and a traditional nominal model through an operational study.     

Semantic picture-word interference is a postperceptual effect

Naming a picture is slower while ignoring a semantically related versus an unrelated distractor word (semantic picture–word interference, or PWI). To locate the PWI effect in the word production processing stream (during perceptual encoding, response selection, or afterward), we used the psychological refractory period paradigm, in which participants identified a tone and then, at varying SOAs, named a picture while ignoring a semantically related or unrelated word (following Dell’Acqua, Job, Peressotti, & Pascali, 2007). As in results from the Stroop paradigm (Fagot & Pashler, 1992), we found equivalent PWI effects at short and long SOAs following tone identification in two experiments, indicating that semantic competition occurs at response selection or later. Our results suggest that it is premature to assume that competitive selection occurs at multiple levels in the word production system (van Maanen, van Rijn, & Borst, 2009) or that the Stroop and semantic PWI effects are fundamentally different (Dell’Acqua et al., 2007).     

A Signal Detection Model for Multiple-Choice Exams

A model for multiple-choice exams is developed from a signal-detection perspective. A correct alternative in a multiple-choice exam can be viewed as being a signal embedded in noise (incorrect alternatives). Examinees are assumed to have perceptions of the plausibility of each alternative, and the decision process is to choose the most plausible alternative. It is also assumed that each examinee either knows or does not know each item. These assumptions together lead to a signal detection choice model for multiple-choice exams. The model can be viewed, statistically, as a mixture extension, with random mixing, of the traditional choice model, or similarly, as a grade-of-membership extension. A version of the model with extreme value distributions is developed, in which case the model simplifies to a mixture multinomial logit model with random mixing. The approach is shown to offer measures of item discrimination and difficulty, along with information about the relative plausibility of each of the alternatives. The model, parameters, and measures derived from the parameters are compared to those obtained with several commonly used item response theory models. An application of the model to an educational data set is presented.     

On Compensation in Multidimensional Response Modeling

The issue of compensation in multidimensional response modeling is addressed. We show that multidimensional response models are compensatory in their ability parameters if and only if they are monotone. In addition, a minimal set of assumptions is presented under which the MLEs of the ability parameters are also compensatory. In a recent series of articles, beginning with Hooker, Finkelman, and Schwartzman (2009) in this journal, the second type of compensation was presented as a paradoxical result for certain multidimensional response models, leading to occasional unfairness in maximum-likelihood test scoring. First, it is indicated that the compensation is not unique and holds generally for any multiparameter likelihood with monotone score functions. Second, we analyze why, in spite of its generality, the compensation may give the impression of a paradox or unfairness.     

Distractor strength and selective attention in picture-naming performance

Whereas it has long been assumed that competition plays a role in lexical selection in word production (e.g., Levelt, Roelofs, & Meyer, 1999), recently Finkbeiner and Caramazza (2006) argued against the competition assumption on the basis of their observation that visible distractors yield semantic interference in picture naming, whereas masked distractors yield semantic facilitation. We examined an alternative account of these findings that preserves the competition assumption. According to this account, the interference and facilitation effects of distractor words reflect whether or not distractors are strong enough to exceed a threshold for entering the competition process. We report two experiments in which distractor strength was manipulated by means of coactivation and visibility. Naming performance was assessed in terms of mean response time (RT) and RT distributions. In Experiment 1, with low coactivation, semantic facilitation was obtained from clearly visible distractors, whereas poorly visible distractors yielded no semantic effect. In Experiment 2, with high coactivation, semantic interference was obtained from both clearly and poorly visible distractors. These findings support the competition threshold account of the polarity of semantic effects in naming.     

On the fate of distractor representations

Research on the topic of distractor inhibition has used different empirical approaches to study how the human mind selects relevant information from the environment, and the results are controversially discussed. One key question that typically arises is how selection deals with the irrelevant information. We used a new selection task, in which participants sometimes had to respond to the distractors instead of the target. Importantly, we varied the time interval between stimuli onset and the cue that signaled participants to respond to the distractors. We analyzed RTs and error rates from responses to distractors as a function of how long the target had been processed (and the distractor ignored) before the cue required a response to the distractor (i.e., stimulus-cue SOA). The data are compatible with selection models assuming that distractor stimuli are initially activated and then deactivated. Thus, we argue for selection models assuming top down deactivation of distractor representations that work in parallel with top down activation of target representations.     

When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT

Computerized adaptive testing (CAT) is a mode of testing which enables more efficient and accurate recovery of one or more latent traits. Traditionally, CAT is built upon Item Response Theory (IRT) models that assume unidimensionality. However, the problem of how to build CAT upon latent class models (LCM) has not been investigated until recently, when Tatsuoka (J. R. Stat. Soc., Ser. C, Appl. Stat. 51:337–350, 2002) and Tatsuoka and Ferguson (J. R. Stat., Ser. B 65:143–157, 2003) established a general theorem on the asymptotically optimal sequential selection of experiments to classify finite, partially ordered sets. Xu, Chang, and Douglas (Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada, 2003) then tested two heuristics in a simulation study based on Tatsuoka’s theoretical work in the context of computerized adaptive testing. One of the heuristics was developed based on Kullback–Leibler information, and the other based on Shannon entropy. In this paper, we showcase the application of the optimal sequential selection methodology in item selection of CAT that is built upon cognitive diagnostic models. Two new heuristics are proposed, and are compared against the randomized item selection method and the two heuristics investigated in Xu et al. (Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada, 2003). Finally, we show the connection between the Kullback–Leibler-information-based approaches and the Shannon-entropy-based approach, as well as the connection between algorithms built upon LCM and those built upon IRT models.     

Analysis of distractor difficulty in multiple-choice items

Two psychometric models are presented for evaluating the difficulty of the distractors in multiple-choice items. They are based on the criterion of rising distractor selection ratios, which facilitates interpretation of the subject and item parameters. Statistical inferential tools are developed in a Bayesian framework: modal a posteriori estimation by application of an EM algorithm and model evaluation by monitoring posterior predictive replications of the data matrix. An educational example with real data is included to exemplify the application of the models and compare them with the nominal categories model.This research was supported by the DGI grant BSO2002-01485.I would like to thank Eric Maris and Vicente Ponsoda for their advice, Juan Botella for providing the data for the empirical application, and three anonymous reviewers for their comments that were essential for improving the quality of the paper.     

predicting insurance agent turnover using a video-based situational judgment test

High turnover of newly contracted insurance agents is a wide-spread problem faced by many insurance companies. The objective of the present study was to examine the usefulness of a video-based selection test in the prediction of turnover for insurance agents. Newly contracted insurance agents (N=677) responded to items presented on a video-based situational judgment test. Each item presented a videotaped situation showing an agent interacting with client. Following the situation, the agents responded to a multiple-choice question indicating how they would handle the situation. Turnover information was gathered on the agents one year after contract. Using an empirical scoring method, data analyses indicated a significant relationship between test scores and the turnover criterion.This research was completed while the author was at LIMRA, International, Farmington, CT.     

Attention meets binding: Only attended distractors are used for the retrieval of event files

Response-irrelevant stimuli can be encoded with, and later on retrieve, a response given to a relevant stimulus, an effect that is called distractor–response binding. In three experiments using a prime–probe design, we investigated whether the allocation of attention modulates the processes contributing to distractor–response binding. Participants identified letters via keypresses while attending to one of two sets of simultaneously presented but response-irrelevant number stimuli. In different experiments, both spatial attention and feature-based attention were allocated to the response-irrelevant stimuli. The results showed that only attended response-irrelevant stimuli elicited effects of distractor–response binding. In particular, while the encoding of response-irrelevant stimuli and responses was not particularly affected by attention during prime processing, only attended response-irrelevant stimuli in the probe retrieved previous responses. Hence, we show that attention affects action regulation due to modulating the influence of stimulus–response binding on behavior.     

Distractor interference during smooth pursuit eye movements

When 2 targets for pursuit eye movements move in different directions, the eye velocity follows the vector average (S. G. Lisberger & V. P. Ferrera, 1997). The present study investigates the mechanisms of target selection when observers are instructed to follow a predefined horizontal target and to ignore a moving distractor stimulus. Results show that at 140 ms after distractor onset, horizontal eye velocity is decreased by about 25%. Vertical eye velocity increases or decreases by 1 degrees /s in the direction opposite from the distractor. This deviation varies in size with distractor direction, velocity, and contrast. The effect was present during the initiation and steady-state tracking phase of pursuit but only when the observer had prior information about target motion. Neither vector averaging nor winner-take-all models could predict the response to a moving to-be-ignored distractor during steady-state tracking of a predefined target. The contributions of perceptual mislocalization and spatial attention to the vertical deviation in pursuit are discussed.     

Distractor selection ratios

A latent variable representation for multiple-choice item and option characteristic curves is presented. Under standard assumptions of conditional independence of item responses and monotonicity of item characteristic curves, a criterion for distractors is proposed based on distractor selection ratios. A connection is made between the proposed criterion and the theory of individual choice behavior, providing new insight. The main results allow for the testing of the criterion from observable data without first specifying a parametric form for the characteristic curves. A series of examples apply the method.     

Retinal location and its effect on the processing of target and distractor information

Two experiments were conducted to study the perceptual facilitation and inhibition that occurs between foveal and parafoveal or peripheral regions of the eye. Experiment 1 used a Stroop task to compare color detection latencies of foveal and parafoveal targets. The target and distractor components of the Stroop stimuli were separated and presented with varied stimulus onset asynchronies. Experiment 2 used a Stroop-like task to replicate and extend the findings into the visual periphery. Subjects were found to process foveally presented distractor information while attending to targets presented in parafoveal or peripheral regions of the eye. Distractor information that was incompatible was suppressed while compatible information was used to facilitate target processing. When the targets were presented foveally along with the distractor information, subjects appeared to automatically process the distractor information. The findings are discussed within the framework of past studies that presented subjects with competing tasks across retinal location. The implications of these findings to a two-process theory of attention are also considered.     

Semantic interference from distractor pictures in single-picture naming: evidence for competitive lexical selection

Picture-naming studies have demonstrated interference from semantic-categorically related distractor words, but not from corresponding distractor pictures, and the lack of generality of the interference effect has been argued to challenge theories viewing lexical selection in speech production as a competitive process. Here, we demonstrate that semantic interference from context pictures does become visible, if sufficient attention is allocated to them. We combined picture naming with a spatial-cuing procedure. When participants’ attention was shifted to the distractor, semantically related distractor pictures interfered with the response, as compared with unrelated distractor pictures. This finding supports models conceiving lexical retrieval as competitive (Levelt, Roelofs, & Meyer, 1999) but is difficult to reconcile with the response exclusion hypothesis (Finkbeiner & Caramazza, 2006b) proposed as an alternative.     

A late locus of the distractor frequency effect in picture–word interference: Evidence from event-related potentials

In this study, we investigated how people deal with irrelevant contextual information during speech production. Two main models have been proposed. WEAVER++ assumes that irrelevant information is removed from the production system by an early blocking mechanism. On the other hand, the response exclusion hypothesis assumes a blocking mechanism that operates late, after lexical selection has finished. To delineate between these models, we focused on the distractor frequency effect (i.e., longer picture naming latencies in the context of low-frequency compared to high-frequency words) and measured ERPs concurrently. Behaviorally, the distractor frequency effect was replicated. In the ERPs, three effects were found. One effect occurred very early and is interpreted as an effect of low-level visual feature processing. The two other effects occurred after lexical access and are thus in line with the response exclusion hypothesis.     

Interference From Multi-dimensional Objects During Feature and Conjunction Discriminations

Feature discrimination performance within an attended object and interference from irrelevant, multi-dimensional objects (distractors) were examined in a two-choice, response compatibility paradigm. Results showed that the amount of interference by multi-dimensional distractors was dependent on three factors: (1) the discriminability of the incompatible, task-relevant distractor features; (2) the number of incompatible, task-relevant distractor features; and (3) whether the task-relevant, incompatible features matched the task goals. The most interesting finding was that additive priming effects were found for multiple, task-relevant features that matched the task goals, whether these features were present in the attended object or in the ignored object. Models that assume that each task-relevant feature primes its corresponding decision/response asynchronously and that this priming is combined to meet a decision/response criterion (at least when attended) can account for distractor interference during conjunction discriminations. Implications of these findings for feature integration models, template models, and a response selection model are discussed.     

The striking similarities between standard,distractor-free,and target-free recognition

It is often assumed that observers seek to maximize correct responding during recognition testing by actively adjusting a decision criterion. However, early research by Wallace (Journal of Experimental Psychology: Human Learning and Memory 4:441–452, 1978) suggested that recognition rates for studied items remained similar, regardless of whether or not the tests contained distractor items. We extended these findings across three experiments, addressing whether detection rates or observer confidence changed when participants were presented standard tests (targets and distractors) versus “pure-list” tests (lists composed entirely of targets or distractors). Even when observers were made aware of the composition of the pure-list test, the endorsement rates and confidence patterns remained largely similar to those observed during standard testing, suggesting that observers are typically not striving to maximize the likelihood of success across the test. We discuss the implications for decision models that assume a likelihood ratio versus a strength decision axis, as well as the implications for prior findings demonstrating large criterion shifts using target probability manipulations.     

The role of observer strategy in the single-target AB paradigm

Identification of the second of two targets (T1, T2) inserted in a stream of distractors is impaired when presented 200–500 ms after the first (attentional blink, AB). An AB-like effect has been reported by Nieuwenstein, Potter, and Theeuwes, Journal of Experimental Psychology: Human Perception and Performance, 35, 159-169, (2009, Experiment 2), with a distractor stream that contained only one target and a gap just before the target. Nieuwenstein et al. hypothesized that the gap enhanced the salience of the last distractor, causing it to be processed much like T1 in conventional AB studies. This hypothesis can also account for Lag-1 sparing (enhanced target performance when presented directly after the last distractor, without an intervening gap). We propose an alternative account of the Lag-1 sparing in the single-target paradigm based on observer strategy, and test it by presenting the single-target and dual-target conditions to separate groups (Experiment 2) instead of mixed across trials (Experiment 1 and Nieuwenstein et al.'s study). The single-target condition exhibited Lag-1 sparing when it was mixed with the dual-target condition, but Lag-1 deficit when it was done in isolation. This outcome is consistent with an observer-strategy account but not with an enhanced salience account of the Lag-1 sparing effect in the single-target condition.     

The effects of response priming on the planning and execution of goal-directed movements in the presence of a distracting stimulus

Two models of selective reaching have been proposed to account for deviations in movement trajectories in cluttered environments. The response vector model predicts movement trajectories should deviate towards or away from the location a distractor of little or large salience, respectively. In contrast, the response activation model predicts that a distractor with large salience should cause movement deviations towards it whereas a distractor with little salience should not influence the movement. The precuing technique was combined with the distractor interference paradigm to test these predictions. Results indicate that when the target was presented at the precued (salient) location, movements were unaffected by a distractor. Conversely, when the distractor was presented at the precued location while the target was presented at an uncued (non-salient) location, participants demonstrated increased reaction times and trajectory deviations towards the location of the distractor. These findings are consistent with the model of response activation.