The art of model development and testing

Mathematical and computer simulation modeling are often computationally demanding procedures, so much so that even certain parts of these procedures, such as parameter estimation, exceed the capacities and speed of the best modern computer facilities. A good deal of effort has therefore been dedicated to speeding up and making more efficient programs such as those that are meant to find a global minimum of a parameter space. Our experience, however, is that such well-explored technical procedures in fact represent some of the shortest components of the total set of procedures by which models are developed. In this article we discuss what elements take up the lion’s share of development time and speculate on what lessons can be drawn concerning the role of high-performance computing in such enterprises.     

Model evaluation using grouped or individual data

Analyzing the data of individuals has several advantages over analyzing the data combined across the individuals (the latter we term group analysis): Grouping can distort the form of data, and different individuals might perform the task using different processes and parameters. These factors notwithstanding, we demonstrate conditions in which group analysis outperforms individual analysis. Such conditions include those in which there are relatively few trials per subject per condition, a situation that sometimes introduces distortions and biases when models are fit and parameters are estimated. We employed a simulation technique in which data were generated from each of two known models, each with parameter variation across simulated individuals. We examined how well the generating model and its competitor each fared in fitting (both sets of) the data, using both individual and group analysis. We examined the accuracy of model selection (the probability that the correct model would be selected by the analysis method). Trials per condition and individuals per experiment were varied systematically. Three pairs of cognitive models were compared: exponential versus power models of forgetting, generalized context versus prototype models of categorization, and the fuzzy logical model of perception versus the linear integration model of information integration. We show that there are situations in which small numbers of trials per condition cause group analysis to outperform individual analysis. Additional tables and figures may be downloaded from the Psychonomic Society Archive of Norms, Stimuli, and Data, www.psychonomic.org/archive.     

Retrieval strategies in recall of natural categories and categorized lists

Experiments 1 and 2 examined the effect of retrieval strategies on 3 or 12 min of recall from a natural category. Experiment 3 examined the effect of strategy on 6 min of recall from a subset of a category presented as a list. In Experiments 1 and 2, a large recall deficit was produced by retrieval strategies involving recall in alphabetic order and by size of the words' referents, relative to free recall. In Experiment 3, four strategies, alphabetic, size, serial order, and free recall, gave similar levels of recall after 6 min, though the growth rate of the cumulative output functions differed among the strategies. An extension of the search of associative memory (SAM) model of Raaijmakers and Shiffrin was developed to explain these results; the new model postulates attention sharing among probe cues and the use of idiosyncratic strategies for free recall from natural categories.     

Automatization and training in visual search.

In several search tasks, the amount of practice on particular combinations of targets and distractors was equated in varied-mapping (VM) and consistent-mapping (CM) conditions. The results indicate the importance of distinguishing between memory and visual search tasks, and implicate a number of factors that play important roles in visual search and its learning. Visual search was studied in Experiment 1. VM and CM performance were almost equal, and slope reductions occurred during practice for both, suggesting the learning of efficient attentive search based on features, and no important role for automatic attention attraction. However, positive transfer effects occurred when previous CM targets were re-paired with previous CM distractors, even though these targets and distractors had not been trained together. Also, the introduction of a demanding simultaneous task produced advantages of CM over VM. These latter two results demonstrated the operation of automatic attention attraction. Visual search was further studied in Experiment 2, using novel characters for which feature overlap and similarity were controlled. The design and many of the findings paralleled Experiment 1. In addition, enormous search improvement was seen over 35 sessions of training, suggesting the operation of perceptual unitization for the novel characters. Experiment 3 showed a large, persistent advantage for CM over VM performance in memory search, even when practice on particular combinations of targets and distractors was equated in the two training conditions. A multifactor theory of automatization and attention is put forth to account for these findings and others in the literature.     

A model for recognition memory: REM—retrieving effectively from memory

A new model of recognition memory is reported. This model is placed within, and introduces, a more elaborate theory that is being developed to predict the phenomena of explicit and implicit, and episodic and generic, memory. The recognition model is applied to basic findings, including phenomena that pose problems for extant models: the list-strength effect (e.g., Ratcliff, Clark, & Shiffrin, 1990), the mirror effect (e.g., Glanzer & Adams, 1990), and the normal-ROC slope effect (e.g., Ratcliff, McKoon, & Tindall, 1994). The model assumes storage of separate episodic images for different words, each image consisting of a vector of feature values. Each image is an incomplete and error prone copy of the studied vector. For the simplest case, it is possible to calculate the probability that a test item is "old," and it is assumed that a default "old" response is given if this probability is greater than .5. It is demonstrated that this model and its more complete and realistic versions produce excellent qualitative predictions.     

Short-term memory: A brief commentary

Over the years, a metatheoretical view of short-term memory has developed. This view, closely related to the “modal” model from the 1960s, is supported by an increasing base of neurophysiological data, and a wide variety of empirical findings. It treats short-term memory as (1) the temporary, above threshold, activation of neural structures (related in not-too-well-specified ways to various recency effects); (2) a work space for carrying out virtually all cognitive operations involved in human cognition; and (3) the source of capacity limitations, accounting for certain memory limitations and most attentional limitations. The main problem with this view is the fact that it encompasses virtually everything that we are concerned with in human cognition—asuccessful model would almost be a general model of cognition, something the field has not yet approached. This situation is not grounds for despair. Progress is being made on many fronts, notwithstanding the fact that the most successful models are focused on specific task domains. Recent advances include an increasing awareness of the necessity for detailed models of short-term retrieval, a theme reflected in a number of articles in the present collection.     

Prime diagnosticity in short-term repetition priming: is primed evidence discounted, even when it reliably indicates the correct answer?

The authors conducted 4 repetition priming experiments that manipulated prime duration and prime diagnosticity in a visual forced-choice perceptual identification task. The strength and direction of prime diagnosticity produced marked effects on identification accuracy, but those effects were resistant to subsequent changes of diagnosticity. Participants learned to associate different diagnosticities with primes of different durations but not with primes presented in different colors. Regardless of prime diagnosticity, preference for a primed alternative covaried negatively with prime duration, suggesting that even for diagnostic primes, evidence discounting remains an important factor. A computational model, with the assumption that adaptation to the statistics of the experiment modulates the level of evidence discounting, accounted for these results.