The reliability of speeded tests

Some methods are presented for estimating the reliability of a partially speeded test without the use of a parallel form. The effect of these formulas on some test data is illustrated. Whenever an odd-even reliability is computed it is probably desirable to use one of the formulas noted in Section 2 of this paper in addition to the usual Spearman-Brown correction. Since the formulas given here involve the mean and the standard deviation of the “number unattempted score,” a method is given in Section 4 for computing this mean and standard deviation from item analysis data. If the item analysis data are available, this method will save considerable time as compared with rescoring answer sheets.     

Reliability formulas for independent decision data when reliability data are matched

A distinction is made between reliability data and decision data. Each of these sets of data may be matched or independent, depending on whether the same instruments (tests, judges, etc.) are applied to every individual in the group or the instruments to be applied to each individual are selected independently for him. Reliability formulas are developed (for both single observations and for composites ofk observations) for the case where reliability data are matched but decision data are independent. Formulas previously reported in the literature are inappropriate for this case.This paper was prepared under USPHS Grant M-1839, in which the author is associated with Lee J. Cronbach and Goldine C. Gleser, and is an outgrowth of a more comprehensive conceptualization of reliability being developed in that study. The author is indebted to Dr. Frederic M. Lord for numerous suggestions.An unpublished paper prepared in 1958 entitled A generalization of Kuder-Richardson reliability formula 21.     

Reliability formulas that do not assume experimental independence

Tautologies are established for the reliability coefficient ² _t of the sum ofn part scores. It is not assumed that the part scores are experimentally independent of each other nor that the parts are equivalent to each other. The tautologies show the exact role played by experimental dependence and nonequivalence of parts, respectively, in the reliability of the sum. The formal algebra is appropriate to reliability in the sense of repeated trials of the same test, as well as in the sense of a universe of parallel tests, although the empirical meanings are different. Emphasis is on practical formulas that require information from only a single experiment (or test). These can take the form only of lower bounds to ² _t , four of which are developed.     

Simulating naturalistic demands for speeded judgment

Responses to speeded judgment (e.g., “shoot/don’t shoot”) situations in naturalistic environments are influenced by many variables and psychological processes. We describe a custom-built simulator with head- and eye-tracking capabilities for basic and applied research on speeded decision making. The system allows the user to project life-sized, time-coded, videotaped, or computer-animated scenarios on a screen to collect reaction-time, shot-accuracy, and eye-movement data. An initial experiment is also reported to demonstrate the utility of the system for studying factors that affect speeded judgment under naturalistic situations.     

Perceptual and linguistic interactions in speeded classification: tests of the semantic coding hypothesis

We tested the semantic coding hypothesis, which states that cross-modal interactions observed in speeded classification tasks arise after perceptual information is recoded into an abstract format common to perceptual and linguistic systems. Using a speeded classification task, we first confirmed the presence of congruence interactions between auditory pitch and visual lightness and observed Garner-type interference with nonlinguistic (perceptual) stimuli (low-frequency and high-frequency tones, black and white squares). Subsequently, we found that modifying the visual stimuli by (a) making them lexical (related words) or (b) reducing their compactness or figural 'goodness' altered congruence effects and Garner interference. The results are consistent with the semantic coding hypothesis, but only in part, and suggest the need for additional assumptions regarding the role of perceptual organization in cross-modal dimensional interactions.     

Preparation for speeded action as a psychophysiological concept

Mental preparation aids performance and induces multiple physiological changes that should inform concepts of preparation. To date, however, these changes have been interpreted as being due to a global preparatory process (e.g., attention or alertness). The authors review psychophysiological and performance investigations of preparation. Concepts of the central regulation of action offer an integrative framework for understanding the psychophysiology of preparation. If people process multiple streams of information concurrently, then preparatory processing requires a form of supervisory attention- central regulation to maintain unity of action. This concept is consistent with existing psychophysiological results and links them to current views of information processing. Conversely, psychophysiological measures may provide indices to test concepts within theories of the central regulation of action.     

Information-accumulation theory of speeded categorization

A process model of perceptual categorization is presented, in which it is assumed that the earliest stages of categorization involve gradual accumulation of information about object features. The model provides a joint account of categorization choice proportions and response times by assuming that the probability that the information-accumulation process stops at a given time after stimulus presentation is a function of the stimulus information that has been acquired. The model provides an accurate account of categorization response times for integral-dimension stimuli and for separable-dimension stimuli, and it also explains effects of response deadlines and exemplar frequency.     

The role of familiarity in item recognition, associative recognition, and plurality recognition on self-paced and speeded tests

Four experiments compare the effect of familiarity on item, associative, and plurality recognition on self-paced and speeded tests. The familiarity of test items was enhanced by presenting a prime that matched the subsequent test item. On item and plurality recognition tests, participants were more likely to respond "old" to primed than to unprimed test items. In associative recognition, priming increased the proportion of old responses on a speeded test, but not on a self-paced test. This suggests that familiarity plays a larger role in item and plurality recognition than in associative recognition on self-paced tests. On speeded tests, priming has a similar effect on item, associative, and plurality recognition. Results suggest that item and associative recognition rely differentially on familiarity and recollection. They are also consistent with recent evidence suggesting that different processes underlie plurality and associative recognition.     

A mechanism for error detection in speeded response time tasks

The concept of error detection plays a central role in theories of executive control. In this article, the authors present a mechanism that can rapidly detect errors in speeded response time tasks. This error monitor assigns values to the output of cognitive processes involved in stimulus categorization and response generation and detects errors by identifying states of the system associated with negative value. The mechanism is formalized in a computational model based on a recent theoretical framework for understanding error processing in humans (C. B. Holroyd & M. G. H. Coles, 2002). The model is used to simulate behavioral and event-related brain potential data in a speeded response time task, and the results of the simulation are compared with empirical data.     

Using PPT to correct for inconsistency in a speeded task

Two experiments employed Potential Performance Theory (PPT) to correct for inconsistency in a speeded task. In Experiment 1, a search task was employed whereby participants searched for a target letter among distracters. In Experiment 2, the search task was more complex and naturalistic-participants searched for enemy weapons in aerial photographs. Not surprisingly, the results revealed that longer search times led to greater accuracy. However, this improvement was due only in part to improved search strategies. Much, if not most, of the improvement was due to increased consistency. The authors demonstrate and discuss some of the advantages to be gained by considering a speeded task within the context of a clear and mathematically precise theory (PPT).     

Effects of similarity and practice on speeded classification response times and accuracies: Further tests of an exemplar-retrieval model

Observers were tested in a perceptual category-learning experiment in which they were instructed to make classification decisions as rapidly as possible without making errors. Nosofsky and Palmeri's (1997b) exemplar-based random walk (EBRW) model of speeded classification was tested for its ability to fit the classification response times and accuracies. The authors demonstrated that the EBRW model provided good quantitative fits to the mean response times and accuracies associated with individual objects as a function of their locations in a multidimensional similarity space and as a function of practice in the task. Preliminary evidence was also obtained that stimulus-specific adjustments in the random walk response criteria may have occurred during the course of learning.     

A speech-rate intelligibility threshold for speeded and time-compressed connected speech

An automated threshold method has been developed for determining the maximum rate of speech understood by individual listeners. Two experiments were undertaken to determine whether the threshold was related to the comprehension of speech or to speech intelligibility. The first experiment compared thresholds of two types of rapid speech reportedly different in intelligibility: simple speeded speech and speech compressed by the sampling method. The second experiment sought to determine the relationship of the threshold to traditional comprehension measures. The results are discussed in terms of the intelligibility and comprehensibility of speech.     

A crossed random effects diffusion model for speeded semantic categorization decisions

Choice reaction times (RTs) are often used as a proxy measure of typicality in semantic categorization studies. However, other item properties have been linked to choice RTs as well. We apply a tailored process model of choice RT to a speeded semantic categorization task in order to deconfound different sources of variability in RT. Our model is based on a diffusion model of choice RT, extended to include crossed random effects (of items and participants). This model retains the interesting process interpretation of the diffusion model’s parameters, but it can be applied to choice RTs even in the case where there are few or no repeated measurements of each participant-item combination. Different aspects of the response process are then linked to different types of item properties. A typicality measure turns out to predict the rate of information uptake, while a lexicographic measure predicts the stimulus encoding time. Accessibility measures cannot reliably predict any component of the decision process.     

Emotion-induced impairments in speeded word recognition tasks

Recent studies show that emotional stimuli impair the identification of subsequently presented, briefly flashed stimuli. In the present study, we investigated whether emotional distractors (primes) impaired target processing when presentation of the target stimulus was not impoverished. In lexical decision, animacy decision, rhyme decision, and nonword naming, targets were presented in such a manner that they were clearly visible (i.e., targets were not masked and presented until participants responded). In all tasks taboo-sexual distractors caused a slowdown in responding to the subsequent neutral target. Our results indicate that the detrimental effects of emotional distractors are not confined to paradigms in which visibility of the target is limited. Moreover, impairments were obtained even when semantic processing of stimuli was not required.     

An exemplar-retrieval model of speeded same--different judgments

R. M. Nosofsky and T. J. Palmeri's (1997) exemplar-based random-walk (EBRW) model of speeded classification is extended to account for speeded same--different judgments among integral-dimension stimuli. According to the model, an important component process of same--different judgments is that people store individual examples of experienced same and different pairs of objects in memory. These exemplar pairs are retrieved from memory on the basis of how similar they are to a currently presented pair of objects. The retrieved pairs drive a random-walk process for making same--different decisions. The EBRW predicts correctly that same responses are faster for objects lying in isolated than in dense regions of similarity space. The model also predicts correctly effects of same-identity versus same-category instructions and is sensitive to observers' past experiences with specific same and different pairs of objects.     

Reliability of 16 balance tests in individuals with Down syndrome

To assess test-retest reliability scores on 16 balance tests of 21 individuals with Down syndrome whose ages ranged from 5 to 31 yr., participants performed a standing test on firm and soft surfaces with the eyes open and closed, the balance subset of the Bruininks-Oseretsky test, full turn, timed-up-and-go test, forward reach, and sit-to-stand. Each participant completed all 16 tests twice in one day and then again on a subsequent day for a total of 4 sessions. The interclass reliability correlation coefficients (ICC) value for each measure of balance varied considerably by age and sex. Based on having an ICC > .50, only 3 tests were reliable in young males and young females, whereas 5 tests could reliably be used in adult females and 9 tests could reliably be used in adult males. The results of this study raise suspicions as to the reliability of tests commonly used to assess balance and differences in reliability due to age and sex. Results of balance tests should be interpreted with caution in males and females with Down syndrome across the age span.