Bias produced by fast guessing in distribution-based tests of race models

A comparison involving cumulative probability distributions of reaction time (RT) has been used to test race models of the redundancy gain observed in certain divided-attention paradigms. It has beenpointed out, however, that the presence of fast guesses wouldinterfera with this test, biasing it to accept race models. The present paper reports simulations carried out to determine the size of the bias introduced by fast guesses. In absolute terms, this bias can be quite large— exceeding 175 msec in some conditions. Simulations indicate that the bias increases with the percentage of fast guesses and with the latency difference between the lower tails of guess and nonguess RT distributions. Discarding and rerunning errors reduces bias somewhat, but a more elaborate “kill-the-twin” procedure reduces it much more.     

Testing the race inequality: A simple correction procedure for fast guesses

In speeded response tasks with redundant signals, parallel processing of the redundant signals is generally tested using the so-called race inequality. The race inequality states that the distribution of fast responses for a redundant stimulus never exceeds the summed distributions of fast responses for the single stimuli. It has been pointed out that fast guesses (e.g. anticipatory responses) interfere with this test, and a correction procedure (‘kill-the-twin’ procedure) has been suggested. In this note we formally derive this procedure and extend it to the case in which redundant stimuli are presented with onset asynchrony. We demonstrate how the kill-the-twin procedure is used in a statistical test of the race model prediction.     

Evaluating the unequal-variance and dual-process explanations of zROC slopes with response time data and the diffusion model

We tested two explanations for why the slope of the z-transformed receiver operating characteristic (zROC) is less than 1 in recognition memory: the unequal-variance account (target evidence is more variable than lure evidence) and the dual-process account (responding reflects both a continuous familiarity process and a threshold recollection process). These accounts are typically implemented in signal detection models that do not make predictions for response time (RT) data. We tested them using RT data and the diffusion model. Participants completed multiple study/test blocks of an "old"/"new" recognition task with the proportion of targets and the test varying from block to block (.21, .32, .50, .68, or .79 targets). The same participants completed sessions with both speed-emphasis and accuracy-emphasis instructions. zROC slopes were below one for both speed and accuracy sessions, and they were slightly lower for speed. The extremely fast pace of the speed sessions (mean RT=526) should have severely limited the role of the slower recollection process relative to the fast familiarity process. Thus, the slope results are not consistent with the idea that recollection is responsible for slopes below 1. The diffusion model was able to match the empirical zROC slopes and RT distributions when between-trial variability in memory evidence was greater for targets than for lures, but missed the zROC slopes when target and lure variability were constrained to be equal. Therefore, unequal variability in continuous evidence is supported by RT modeling in addition to signal detection modeling. Finally, we found that a two-choice version of the RTCON model could not accommodate the RT distributions as successfully as the diffusion model.     

Display size effects in visual search: Analyses of reaction time distributions as mixtures

In a reanalysis of data from Cousineau and Shiffrin (2004) and two new visual search experiments, we used a likelihood ratio test to examine the full distributions of reaction time (RT) for evidence that the display size effect is a mixture-type effect that occurs on only a proportion of trials, leaving RT in the remaining trials unaffected, as is predicted by serial self-terminating search models. Experiment 1 was a reanalysis of Cousineau and Shiffrin's data, for which a mixture effect had previously been established by a bimodal distribution of RTs, and the results confirmed that the likelihood ratio test could also detect this mixture. Experiment 2 applied the likelihood ratio test within a more standard visual search task with a relatively easy target/distractor discrimination, and Experiment 3 applied it within a target identification search task within the same types of stimuli. Neither of these experiments provided any evidence for the mixture-type display size effect predicted by serial self-terminating search models. Overall, these results suggest that serial self-terminating search models may generally be applicable only with relatively difficult target/distractor discriminations, and then only for some participants. In addition, they further illustrate the utility of analysing full RT distributions in addition to mean RT.     

PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS

This article has 2 goals. First, we discuss publication bias and explain why it presents a potential problem for industrial and organizational psychology. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (trim and fill), which has been developed in the medical literature but is largely unfamiliar to industrial and organizational psychology researchers. Second, we demonstrate trim and fill by applying it to validity information reported in the technical manuals of 4 test vendors. In doing so, we assess the likelihood that criterion-related validity information provided by test publishers may overestimate test validity. In our analysis of 18 validity distributions, we found evidence of either no or minimal bias for 2 of the vendors' distributions and evidence of moderate-to-severe bias in at least 1 distribution from each of the other 2 vendors. In both cases in which publication bias was found, we noted instances in which the publishers tended to report only statistically significant correlations and that this practice was detected using publication bias methodology.     

Testing models of the redundant-signals effect: a warning concerning the combination-rule regression analysis.

The redundant-signals effect is the observed RT advantage for trials presenting two or more targets, as compared with trials with only one target. Two general classes of parallel-processing model have been proposed to explain this effect: race models (e.g., Raab, 1962) and coactivation models (e.g., Miller, 1982). Various distributional analyses have been used in work aimed at discriminating between these two model classes. The present study reexamined one of these tests--the combination-rule regression analysis based on variable-criterion theory (Grice, Canham, & Boroughs, 1984)--by applying it to the data from two sets of simulated experiments. One set of simulations assumed coactivation; the other set assumed an independent race on redundant-target trials. Nearly identical combination-rule values were observed in the two sets of simulations. This finding shows that the combination rule of variable-criterion theory does not discriminate between models capable of explaining the redundant-signals effect. The implications of this finding are briefly discussed.     

Testing models of the redundant-signals effect: A warning concerning the combination-rule regression analysis

The redundant-signals effect is the observed RT advantage for trials presenting two or more targets, as compared with trials with only one target Two general classes of parallel-processing model have been proposed to explain this effect race models (e.g., Raab, 1962) and coactivation models (e.g., Miller, 1982). Various distributional analyses have been used in work aimed at discriminating between these two model classes. The present study reexamined one of these tests—the combination-rule regression analysis based on variable-criterion theory (Grice, Canham, & Boroughs, 1984)—by applying it to the data from two sets of simulated experiments. One set of simulations assumed coactivation; the other set assumed an independent race on redundant-target trials. Nearly identical combination-rule values were observed in the two sets of simulations. This finding shows that the combination rule of variable-criterion theory does not discriminate between models capable of explaining the redundant-signals effect The implications of this finding are briefly discussed     

The resurrection of Tweedledum and Tweedledee: Bimodality cannot distinguish serial and parallel processes

Simultaneously presented signals may be processed in serial or in parallel. One potentially valuable indicator of a system’s characteristics may be the appearance of multimodality in the response time (RT) distributions. It is known that standard serial models can predict multimodal RT distributions, but it is unknown whether multimodality is diagnostic of serial systems, or whether alternative architectures, such as parallel ones, can also make such predictions. We demonstrate via simulations that a multimodal RT distribution is not sufficient by itself to rule out parallel self-terminating processing, even with limited trial numbers. These predictions are discussed within the context of recent data indicating the existence of multimodal distributions in visual search.     

The emergence and reduction of bias in letters of recommendation

Despite the widespread use of letters of recommendation (LORs) in selection systems, research has largely failed to consider the potential emergence of bias in interpretations of LORs. The present study fills this void by examining both race and gender bias in evaluations of LORs and assessing the efficacy of elaboration as a strategy for reducing such bias. Undergraduate students (n = 423) rated four LORs that varied with regard to applicant race and gender. Results suggest that bias does exist in evaluations of LORs, but that requiring raters to expand on their evaluations (i.e., elaborate) reduces this bias. Implications include elaboration as a strategy organizations can implement to reduce bias from emerging when relying on LORs as a selection tool.     

Testing the race model inequality: An algorithm and computer programs

In divided-attention tasks, responses are faster when two target stimuli are presented, and thus one is redundant, than when only a single target stimulus is presented. Raab (1962) suggested an account of this redundant-targets effect in terms of a race model in which the response to redundant target stimuli is initiated by the faster of two separate target detection processes. Such models make a prediction about the probability distributions of reaction times that is often called the race model inequality, and it is often of interest to test this prediction. In this article, we describe a precise algorithm that can be used to test the race model inequality and present MATLAB routines and a Pascal program that implement this algorithm.     

A hierarchical model for estimating response time distributions

We present a statistical model for inference with response time (RT) distributions. The model has the following features. First, it provides a means of estimating the shape, scale, and location (shift) of RT distributions. Second, it is hierarchical and models between-subjects and within-subjects variability simultaneously. Third, inference with the model is Bayesian and provides a principled and efficient means of pooling information across disparate data from different individuals. Because the model efficiently pools information across individuals, it is particularly well suited for those common cases in which the researcher collects a limited number of observations from several participants. Monte Carlo simulations reveal that the hierarchical Bayesian model provides more accurate estimates than several popular competitors do. We illustrate the model by providing an analysis of the symbolic distance effect in which participants can more quickly ascertain the relationship between nonadjacent digits than that between adjacent digits.     

The salience of sex role instructions to mental health professionals

This experiment was concerned with the question of how gender might affect clinical judgment. Mental health professionals (MHP) were asked to rate a protocol (varied for gender) describing a depressed pseudopatient who displayed stereotypically masculine behaviors. The major hypothesis was that, when subjects were aware that the experimental purpose was to measure how patient gender affected clinical ratings, they would suppress bias. If awareness were not present, then bias would be apparent in the ratings. The results failed to support this hypothesis. However, serendipitous results had meaningful relevance for the methodology of gender research. Data from subjects, given a credibility check to determine whether they had guessed the intent of the research, showed that aware subjects were more likely to make incorrect guesses about the study when they rated a male pseudopatient, whereas unaware subjects were more likely to make correct guesses when they rated a female pseudopatient. The interpretation of the data indicated that MHPs differed in their vigilance concerning women's issues. As a result, there may exist a woman's role within the context of psychological experimentation.Thanks to Drs. Tom Morgan, Ed Fischer, and John Turner for their help and support in statistical analyses.     

Rasch-representable reaction time distributions

This article investigates properties of a representation based on the Rasch test model for reaction times (RT) that was proposed by Micko. Necessary and sufficient conditions for a set of RT distributions to be Rasch-representable are derived. It is shown that independent serial and independent parallel processing models cannot be reconciled with the representation. However, random extreme models compatible with the Reasch-representation exist that assume RT is determined by the longest or he shortest processing time of a random number of independent paraloel channels. Nonparametric properties of Rasch-representable distributions are derived that can be used for testing the model and for estimating its parameters. Conditions are presented for Rasch-representable distributions to form a scale family. Finally, Rasch-represent-able distributions are characterized interms of their hazard functions.For helpful discussions, we are grateful to Hans Irtel, Christoph Micko, Hartmann Scheiblechner, and Hans-Henning Schultz.     

Cognitive arithmetic: comparison of operations

Adults' performance of simple arithmetic calculations (addition, multiplication, and numerical comparison) was examined to test predictions of digital (counting), analog, and network models. Although all of these models have been supported by studies of mental addition, each leads to a different prediction concerning relations between the times required for addition, multiplication, and numerical comparison. Pairs of single-digit integers were presented and reaction times (RTs) for adding, multiplying, and comparing the stimuli were collected. A high correlation between RT for addition and multiplication of the same digits was obtained. This result is consistent with a network model, but presents difficulties for both analog and counting models. A "ties" effect of no increase in RT with increases in problem size for doubles such as 2 + 2 has been found in previous studies of addition using verification procedures, but was not found with the production task employed in the present study. Instead, a different kind of ties effect was found. Reaction time for both addition and multiplication of ties increased more slowly with problem size than did RT for non-tie problems. This ties effect, and the finding that probability of making errors contributes independently of problem size to RT support a distinction between location and accessibility of information in a network.     

Visual attention: Race models for selection from multielement displays

Summary A choice model for partial report from briefly exposed visual displays (Bundesen, Shibuya, & Larsen, 1985) is further investigated and related to a general class of selection models called independent race models. The choice model relates performance to the numbers of targets and distractors in the stimulus display by way of the choice axiom. In race models, the selection process is viewed as a race between items in the choice set toward a state of having been processed in that the first items reaching this state are the ones selected. If items are processed independently and processing times are exponentially distributed, selection occurs strictly in accordance with the choice axiom, so the race model is a choice model. The choice model also seems to work as a good approximation for independent race models based on other gamma distributions than the exponential one.     

Hindsight bias: how knowledge and heuristics affect our reconstruction of the past

Once people know the outcome of an event, they tend to overestimate what could have been anticipated in foresight. Although typically considered to be a robust phenomenon, this hindsight bias is subject to moderating circumstances. In their meta-analysis, Christensen-Szalanski and Willham (1991) observed that the more experience people have with the task under consideration, the smaller is the resulting hindsight bias. This observation is one benchmark against which the explanatory power of process models of hindsight bias can be measured. Therefore, we used it to put the recently proposed RAFT model (Hoffrage, Hertwig, & Gigerenzer, 2000) to another test. Our findings were consistent with the "expertise effect." Specifically, we observed-using computer simulations of the RAFT model-that the more comprehensive people's knowledge is in foresight, the smaller is their hindsight bias. In addition, we made two counterintuitive observations: First, the relation between foresight knowledge and hindsight bias appears to be independent of how knowledge is processed. Second, even if foresight knowledge is false, it can reduce hindsight bias. We conclude with a discussion of the functional value of hindsight bias.