Generalization and similarity in exemplar models of categorization: Insights from machine learning

Exemplar theories of categorization depend on similarity for explaining subjects’ ability to generalize to new stimuli. A major criticism of exemplar theories concerns their lack of abstraction mechanisms and thus, seemingly, of generalization ability. Here, we use insights from machine learning to demonstrate that exemplar models can actually generalize very well. Kernel methods in machine learning are akin to exemplar models and are very successful in real-world applications. Their generalization performance depends crucially on the chosen similarity measure. Although similarity plays an important role in describing generalization behavior, it is not the only factor that controls generalization performance. In machine learning, kernel methods are often combined with regularization techniques in order to ensure good generalization. These same techniques are easily incorporated in exemplar models. We show that the generalized context model (Nosofsky, 1986) and ALCOVE (Kruschke, 1992) are closely related to a statistical model called kernel logistic regression. We argue that generalization is central to the enterprise of understanding categorization behavior, and we suggest some ways in which insights from machine learning can offer guidance.     

Social categorization and similarity in intergroup behaviour

The present study is one of a series exploring the role of social categorization in intergroup behaviour. It has been found in our previous studies that in ?minimal' situations, in which the subjects were categorized into groups on the basis of visual judgments they had made or of their esthetic preferences, they clearly discriminated against members of an outgroup although this gave them no personal advantage. However, in these previous studies division into groups was still made on the basis of certain criteria of ?real' similarity between subjects who were assigned to the same category. Therefore, the present study established social categories on an explicitly random basis without any reference to any such real similarity. It was found that, as soon as the notion of ?group' was introduced into the situation, the subjects still discriminated against those assigned to another random category. This discrimination was considerably more marked than the one based on a division of subjects in terms of interindividual similarities in which the notion of ?group' was never explicitly introduced. In addition, it was found that fairness was also a determinant of the subjects' decisions. The results are discussed from the point of view of their relevance to a social-cognitive theory of intergroup behaviour.     

The role of visual similarity in picture categorization

Categorization is usually assumed to require access to a concept's meaning. When pictures are categorized faster than words, they are assumed to be understood faster than words. However, pictures from the same category are more similar than pictures from different categories. The present article argues that the use of visual similarity as a cue to category membership may produce the picture advantage. The visual similarity hypothesis was tested in two experiments. In the first experiment, pictures showed a disadvantage for the visually similar categories of fruits and vegetables, but showed their usual advantage for the visually dissimilar categories of fruits and animals. In the second experiment, with a mixed list design, pictures were slower only for visually similar different decisions, but showed the usual advantage for all other decisions. The reliability of visual similarity as a cue to the decision accounted well for these results. Because visual similarity can be shown to have large effects on picture categorization, the use of categorization to compare speed of understanding of pictures and words is questionable.     

Comparing categorization models

Four experiments are presented that competitively test rule- and exemplar-based models of human categorization behavior. Participants classified stimuli that varied on a unidimensional axis into 2 categories. The stimuli did not consistently belong to a category; instead, they were probabilistically assigned. By manipulating these assignment probabilities, it was possible to produce stimuli for which exemplar- and rule-based explanations made qualitatively different predictions. F. G. Ashby and J. T. Townsend's (1986) rule-based general recognition theory provided a better account of the data than R. M. Nosofsky's (1986) exemplar-based generalized context model in conditions in which the to-be-classified stimuli were relatively confusable. However, generalized context model provided a better account when the stimuli were relatively few and distinct. These findings are consistent with multiple process accounts of categorization and demonstrate that stimulus confusion is a determining factor as 10 which process mediates categorization.     

The role of similarity in the development of categorization

Early in development, humans exhibit the ability to form categories and overlook differences for the sake of generality. This ability poses several important questions: How does categorization arise? What processes underlie category formation? And how are categories mentally represented? We argue that the development of categorization is grounded in perceptual and attentional mechanisms capable of detecting multiple correspondences or similarities in the environment. We present evidence that: (a) similarity can drive categorization early in development; and (b) early in development, humans have powerful learning mechanisms that enable them to extract regularities in the environment. We conclude that, despite remaining challenges, the similarity-based approach offers a promising account of the development of categorization.     

Predicting similarity and categorization from identification.

In this article, the relation between the identification, similarity judgment, and categorization of multidimensional perceptual stimuli is studied. The theoretical analysis focused on general recognition theory (GRT), which is a multidimensional generalization of signal detection theory. In one application, 2 Ss first identified a set of confusable stimuli and then made judgments of their pairwise similarity. The second application was to Nosofsky's (1985b, 1986) identification-categorization experiment. In both applications, a GRT model accounted for the identification data better than Luce's (1963) biased-choice model. The identification results were then used to predict performance in the similarity judgment and categorization conditions. The GRT identification model accurately predicted the similarity judgments under the assumption that Ss allocated attention to the 2 stimulus dimensions differently in the 2 tasks. The categorization data were predicted successfully without appealing to the notion of selective attention. Instead, a simpler GRT model that emphasized the different decision rules used in identification and categorization was adequate.     

Unitization,similarity, and overt attention in categorization and exposure

Unitization, the creation of new stimulus features by the fusion of preexisting features, is one of the hypothesized processes of perceptual learning (Goldstone Annual Review of Psychology, 49:585–612, 1998). Some argue that unitization occurs to the extent that it is required for successful task performance (e.g., Shiffrin & Lightfoot, 1997), while others argue that unitization is largely independent of functionality (e.g., McLaren & Mackintosh Animal Learning & Behavior, 30:177–200, 2000). Across three experiments, employing supervised category learning and unsupervised exposure, we investigated three predictions of the McLaren and Mackintosh (Animal Learning & Behavior, 30:177–200, 2000) model: (1) Unitization is accompanied by an initial increase in the subjective similarity of stimuli sharing a unitized component; (2) unitization of a configuration occurs through exposure to its components, even when the task does not require it; (3) as unitization approaches completion, salience of the unitized component may be reduced. Our data supported these predictions. We also found that unitization is associated with increases in overt attention to the unitized component, as measured through eye tracking.     

Response-time tests of logical-rule models of categorization

A recent resurgence in logical-rule theories of categorization has motivated the development of a class of models that predict not only choice probabilities but also categorization response times (RTs; Fifi?, Little, & Nosofsky, 2010). The new models combine mental-architecture and random-walk approaches within an integrated framework and predict detailed RT-distribution data at the level of individual participants and individual stimuli. To date, however, tests of the models have been limited to validation tests in which participants were provided with explicit instructions to adopt particular processing strategies for implementing the rules. In the present research, we test conditions in which categories are learned via induction over training exemplars and in which participants are free to adopt whatever classification strategy they choose. In addition, we explore how variations in stimulus formats, involving either spatially separated or overlapping dimensions, influence processing modes in rule-based classification tasks. In conditions involving spatially separated dimensions, strong evidence is obtained for application of logical-rule strategies operating in a serial-self-terminating processing mode. In conditions involving spatially overlapping dimensions, preliminary evidence is obtained that a mixture of serial and parallel processing underlies the application of rule-based classification strategies. The logical-rule models fare considerably better than major extant alternative models in accounting for the categorization RTs.     

Enemies and friends in the neighborhood: orthographic similarity effects in semantic categorization

Studies investigating orthographic similarity effects in semantic tasks have produced inconsistent results. The authors investigated orthographic similarity effects in animacy decision and in contrast with previous studies, they took semantic congruency into account. In Experiments 1 and 2, performance to a target (cat) was better if a previously studied neighbor (rat) was congruent (i.e., belonged to the same animate-inanimate category) than it was if it was incongruent (e.g., mat). In Experiments 3 and 4, performance was better for targets with more preexisting congruent neighbors than for targets with more preexisting incongruent neighbors. These results demonstrate that orthographic similarity effects in semantic categorization are conditional on semantic congruency. This strongly suggests that semantic information becomes available before orthographic processing has been completed.     

Synonym comprehension and the generality of categorization models

Memory & Cognition - When subjects classify a two-word display as representing the “same” category or two “different” categories, semantic similarity between the words...     

Comparing decision bound and exemplar models of categorization

The performance of a decision bound model of categorization (Ashby, J992a; Ashby & Maddox, in press) is compared with the performance of two exemplar models. The first is the generalized context model (e.g., Nosofsky, 1986, 1992) and the second is a recently proposed deterministic exemplar model (Ashby & Maddox, in press), which contains the generalized context model as a special case. When the exemplars from each category were normally distributed and the optimal decision bound was linear, the deterministic exemplar model and the decision bound model provided roughly equivalent accounts of the data. When the optimal decision bound was nonlinear, the decision bound model provided a more accurate account of the data than did either exemplar model. When applied to categorization data collected by Nosofsky (1986, 1989), in which the category exemplars are not normally distributed, the decision bound model provided excellent accounts of the data, in many cases significantly outperforming the exemplar models. The decision bound model was found to be especially successful when(1) single subject analyses were performed, (2) each subject was given relatively extensive training, and (3) the subject's performance was characterized by complex suboptimalities. These results support the hypothesis that the decision bound is of fundamental importance in predicting asymptotic categorization performance and that the decision bound models provide a viable alternative to the currently popular exemplar models of categorization.     

Evolutionary models of color categorization based on discrimination

Specifying the factors that contribute to the universality of color categorization across individuals and cultures is a longstanding and still controversial issue in psychology, linguistics, and anthropology. This article approaches this issue through the simulated evolution of color lexicons. It is shown that the combination of a minimal perceptual psychology of discrimination, simple pragmatic constraints involving communication, and simple learning rules is enough to evolve color-naming systems. Implications of this result for psychological theories of color categorization and the evolution of color-naming systems in human societies are discussed.     

Knowledge partitioning in categorization: constraints on exemplar models

The authors present 2 experiments that establish the presence of knowledge partitioning in perceptual categorization. Many participants learned to rely on a context cue, which did not predict category membership but identified partial boundaries, to gate independent partial categorization strategies. When participants partitioned their knowledge, a strategy used in 1 context was unaffected by knowledge demonstrably present in other contexts. An exemplar model, attentional learning covering map, was shown to be unable to accommodate knowledge partitioning. Instead, a mixture-of-experts model, attention to rules and instances in a unified model (ATRIUM), could handle the results. The success of ATRIUM resulted from its assumption that people memorize not only exemplars but also the way in which they are to be classified.     

Elemental versus configurai models of categorization in children and adults

In experiment 1, 7-and 8-year-old children learned to classify six features, A–F, as belonging to one of two artificial categories, or to neither category. Feature A and the compound BC were designated as members of Category 1, the compounds DE and EF were members of category 2, while D alone and the compound AB, belonged to neither category. Following successful learning, the participants were asked to rate two groups of test features, ABC and DEF, as likely members of their respective categories. Participants' certainty ratings of the categorization of the compound DEF were greater than for the compound ABC. Experiment 2 replicated the results of experiment 1 with adult participants. These data are at odds with predictions from an elemental associative theory, that suggested by Rescorla and Wagner (1972), which assumes that category judgements are made on the basis of associations between individual features and a category.     

Elemental versus configurai models of categorization in children and adults

In experiment 1, 7-and 8-year-old children learned to classify six features, A–F, as belonging to one of two artificial categories, or to neither category. Feature A and the compound BC were designated as members of Category 1, the compounds DE and EF were members of category 2, while D alone and the compound AB, belonged to neither category. Following successful learning, the participants were asked to rate two groups of test features, ABC and DEF, as likely members of their respective categories. Participants' certainty ratings of the categorization of the compound DEF were greater than for the compound ABC. Experiment 2 replicated the results of experiment 1 with adult participants. These data are at odds with predictions from an elemental associative theory, that suggested by Rescorla and Wagner (1972), which assumes that category judgements are made on the basis of associations between individual features and a category.     

Judgments of similarity and spatial models

Ss judged the similarity between all pairs of stimulus objects under 3 conditions: when the objects were (a) Munsell 5R color patches varying in value and chroma; (b) parallelorams varying in size and tilt; and (c) circles-with-radius varying in diameter and angle of radius. For each set of judgments, the pattern of deviations from the Euclidean model was used to diagnose the most appropriate spatial model. The results confirm previous findings that the Euclidean space is appropriate for judgments of color patches, but that the city block space is appropriate for judgments of geometrie forms which vary on perceptually distinct dimensions.     

Dissociations between categorization and similarity judgments as a result of learning feature distributions

A dissociation between categorization and similarity was found by Rips (1989). In one experiment, Rips found that a stimulus halfway between a pizza and a quarter was categorized as a pizza but was rated as more similar to a quarter. Smith and Sloman (1994) discussed these results in terms of the role of necessary and characteristic features. In two experiments, participants had to learn to categorize novel artificial shapes composed of a nonsalient necessary feature combined with a salient characteristic feature. Participants categorized stimuli on the basis of a necessary feature, whereas their similarity judgments relied on characteristic features. The role of deep (essential) features in dissociations is considered. Results are discussed in terms of the differences between requirements of categorization and similarity judgments.     

Comparison of two models of similarity judgment

The conventional multidimensional distance model of similarity judgment was compared with a new model in which component differences are weighted and then averaged. To evaluate the models, qualitative and quantitative predictions were derived from Romney and D'Andrade's (1964) componential analysis of American kinship terms, and these predictions were tested by having subjects rate the similarity (in experiment 1) and the difference (in experiment 2) between all possible pairs of 12 kinship terms. In both experiments, violations of qualitative predictions for both a simple distance model and a simple averaging model revealed that the componential analysis was not sufficient to account for the data. However the averaging model was able to account for the data when the dichotomous dimension of lineality used by Romney and D'Andrade was replaced by a continuous dimension of immediacy or closeness of kin. In contrast, no comparable elaboration under the distance model was successful. These results were discussed in terms of the likely psychological processes underlying similarity judgment.     

When parameters collide: a warning about categorization models

Similarity-choice (S—C) models of categorization contain two principal mathematical transformations: an exponential-decay similarity function and a choice rule. However, there is a tension between the psychological processes that models emulate and the mathematics they use to do so. To illustrate this, I show that in these models an unappreciated interaction occurs between the mathematical transformations so that the stages of the model essentially cancel each other out. The result is that the model’s output reflects its input linearly. This cancellation phenomenon has potentially serious implications regarding the interpretation and use of S—C models. The phenomenon also raises questions about the simplification and psychological grounding of categorization models. Modelers broadly might benefit from an internal analysis of their models, such as that described here.