On more powerful tests of judge agreement with a known standard

To establish the existence of his abilities, a judge is given the task of classifying each ofN=rs subjects into one ofr known categories, each containings of the subjects. An incomplete design is proposed whereby the judge is presented withb groups, each one containingn=rs/b<r subjects. Then different categories corresponding to members of the group are known. Using the total number of correct classifications, this method of grouping is compared to that in which the group size is equal to the number of categories. The incomplete grouping is shown to yield a more powerful test for discriminating between the null hypothesis that the judge is guessing the classifications and the alternative hypothesis that he has some definite abilities. The incomplete design is found to be most effective (powerful) when the number of subjects in a group is limited to two or three.The author is grateful for the suggestions of the referees and the editor, which greatly improved the paper.     

Using the conditional grade-of-membership model to assess judgment accuracy

Consider the case whereJ instruments are used to classify each ofI objects relative toK nominal categories. The conditional grade-of-membership (GoM) model provides a method of estimating the classification probabilities of each instrument (or judge) when the objects being classified consist of both pure types that lie exclusively in one ofK nominal categories, and mixtures that lie in more than one category. Classification probabilities are identifiable whenever the sample of GoM vectors includes pure types from each category. When additional, relatively mild, assumptions are made about judgment accuracy, the identifiable correct classification probabilities are the greatest lower bounds among all solutions that might correspond to the observed multinomial process, even when the unobserved GoM vectors do not include pure types from each category. Estimation using the conditional GoM model is illustrated on a simulated data set. Further simulations show that the estimates of the classification probabilities are relatively accurate, even when the sample contains only a small percentage of approximately pure objects.The authors thank Max A. Woodbury, Kenneth G. Manton and H. Dennis Tolley for their help and four anonymous Psychometrika reviewers (including an associate editor) for their beneficial expository and technical suggestions. This work was supported by the Dean's Fund for Summer Research, Owen Graduate School of Management, Vanderbilt University.     

Recent developments in non-traditional employment testing: An introduction

The purpose of this special issue of theJournal of Business and Psychology is to provide a review and update of several key types of non-traditional employment tests. Non-traditional employment tests can be divided into two categories. One category consists ofbiological/physiological tests. The second category consists ofnarrow-based personality tests. A framework for thinking about non-traditional employment tests is provided.     

Incremental Bayesian Category Learning From Natural Language

Models of category learning have been extensively studied in cognitive science and primarily tested on perceptual abstractions or artificial stimuli. In this paper, we focus on categories acquired from natural language stimuli, that is, words (e.g., chair is a member of the furniture category). We present a Bayesian model that, unlike previous work, learns both categories and their features in a single process. We model category induction as two interrelated subproblems: (a) the acquisition of features that discriminate among categories, and (b) the grouping of concepts into categories based on those features. Our model learns categories incrementally using particle filters, a sequential Monte Carlo method commonly used for approximate probabilistic inference that sequentially integrates newly observed data and can be viewed as a plausible mechanism for human learning. Experimental results show that our incremental learner obtains meaningful categories which yield a closer fit to behavioral data compared to related models while at the same time acquiring features which characterize the learned categories. (An earlier version of this work was published in Frermann and Lapata 2014 .)     

Contrast effects in typicality judgements: A hierarchical Bayesian approach

We examine the influence of contrast categories on the internal graded membership structure of everyday concepts using computational models proposed in the artificial category learning tradition. In particular, the generalized context model (Nosofsky, 1986), which assumes that only members of a given category contribute to the typicality of a category member, is contrasted to the similarity–dissimilarity generalized context model (SD-GCM; Stewart & Brown, 2005), which assumes that members of other categories are also influential in determining typicality. The models are compared in a hierarchical Bayesian framework in their account of the typicality gradient of five animal categories and six artefact categories. For each target category, we consider all possible relevant contrast categories. Three separate issue are examined: (a) whether contrast effects can be found, (b) which categories are responsible for these effects, and (c) whether more than one category influences the typicality. Results indicate that the internal category structure is codetermined by dissimilarity towards potential contrast categories. In most cases, only a single contrast category contributed to the typicality. The present findings suggest that contrast effects might be more widespread than has previously been assumed. Further, they stress the importance of characteristics particular of everyday concepts, which require careful consideration when applying computational models of representation of the artificial category learning tradition to everyday concepts.     

Contrast effects in typicality judgements: A hierarchical Bayesian approach

We examine the influence of contrast categories on the internal graded membership structure of everyday concepts using computational models proposed in the artificial category learning tradition. In particular, the generalized context model (Nosofsky, 1986), which assumes that only members of a given category contribute to the typicality of a category member, is contrasted to the similarity-dissimilarity generalized context model (SD-GCM; Stewart & Brown, 2005), which assumes that members of other categories are also influential in determining typicality. The models are compared in a hierarchical Bayesian framework in their account of the typicality gradient of five animal categories and six artefact categories. For each target category, we consider all possible relevant contrast categories. Three separate issue are examined: (a) whether contrast effects can be found, (b) which categories are responsible for these effects, and (c) whether more than one category influences the typicality. Results indicate that the internal category structure is codetermined by dissimilarity towards potential contrast categories. In most cases, only a single contrast category contributed to the typicality. The present findings suggest that contrast effects might be more widespread than has previously been assumed. Further, they stress the importance of characteristics particular of everyday concepts, which require careful consideration when applying computational models of representation of the artificial category learning tradition to everyday concepts.     

A sequential cognitive diagnosis model for polytomous responses

This paper proposes a general polytomous cognitive diagnosis model for a special type of graded responses, where item categories are attained in a sequential manner, and associated with some attributes explicitly. To relate categories to attributes, a category‐level Q‐matrix is used. When the attribute and category association is specified a priori, the proposed model has the flexibility to allow different cognitive processes (e.g., conjunctive, disjunctive) to be modelled at different categories within a single item. This model can be extended for items where categories cannot be explicitly linked to attributes, and for items with unordered categories. The feasibility of the proposed model is examined using simulated data. The proposed model is illustrated using the data from the Trends in International Mathematics and Science Study 2007 assessment.     

An experimental analysis of psychometric functions in a threshold discrimination task with four response categories1

An experiment on a threshold discrimination task with four response categories was conducted to check the implicit assumptions of a threshold discrimination task with three response categories. Consideration of a task with three response categories, for example, “stronger,”“don't know,” and “weaker,” leads us to anticipate a relation between position parameters and that between the sizes of scale parameters of a model for a task with four response categories, such as, “stronger,”“probably stronger,”“probably weaker,” and “weaker.” That is, position parameters will be equidistant between adjacent categories and the middle scale parameter will be the largest. The results of the experiment on a four‐category task did not support these anticipated relation, and a possibility of biases in the estimation of point of subjective equality (PSE), using a three‐category task is pointed out. As an alternative to a three‐category task, a four‐category task is suggested when more than two categories are required.     

General estimators for the reliability of qualitative data

We study a proportional reduction in loss (PRL) measure for the reliability of categorical data and consider the general case in which each ofN judges assigns a subject to one ofK categories. This measure has been shown to be equivalent to a measure proposed by Perreault and Leigh for a special case when there are two equally competent judges, and the correct category has a uniform prior distribution. We consider a general framework where the correct category is assumed to have an arbitrary prior distribution, and where classification probabilities vary by correct category, judge, and category of classification. In this setting, we consider PRL reliability measures based on two estimators of the correct category—the empirical Bayes estimator and an estimator based on the judges' consensus choice. We also discuss four important special cases of the general model and study several types of lower bounds for PRL reliability.Bruce Cooil is Associate Professor of Statistics, and Roland T. Rust is Professor and area head for Marketing, Owen Graduate School of Management, Vanderbilt University. The authors thank three anonymous reviewers and an Associate Editor for their helpful comments and suggestions. This work was supported in part by the Dean's Fund for Faculty Research of the Owen Graduate School of Management, Vanderbilt University.     

Similarity and dissimilarity as evidence in perceptual categorization

In exemplar models the similarities between a new stimulus and each category exemplar constitute positive evidence for category membership. In contrast, other models assume that, if the new stimulus is sufficiently dissimilar to a category member, then that dissimilarity constitutes evidence against category membership. We propose a new similarity-dissimilarity exemplar model that provides a framework for integrating these two types of accounts. The evidence for a category is assumed to be the summed similarity to members of that category plus the summed dissimilarity to members of competing categories. The similarity-dissimilarity exemplar model is shown to mimic the standard exemplar model very closely in the unidimensional domain.     

Inhibition of irrelevant category–response mappings

When switching tasks, performance tends to be worse for n – 2 repetitions than with n – 2 switches. This n – 2 repetition cost has been hypothesized to reflect task-set inhibition: specifically, inhibition of irrelevant category–response mappings involved in response selection. This hypothesis leads to divergent predictions for situations in which all tasks involve the same stimulus categories: An n – 2 repetition cost is predicted when response sets differ across tasks, but not when the response set stays the same. The authors tested these predictions by having subjects perform relative judgements with different reference points. In Experiment 1, the stimulus categories were the same across reference points, but the response set either differed or stayed the same (the multiple- and single-mapping conditions, respectively). An n – 2 repetition cost was found in the multiple-mapping condition but not in the single-mapping condition. Experiment 2 provided evidence against the possibility that these divergent effects reflected differences in memory load. These findings confirm predictions that link n – 2 repetition costs to inhibition of irrelevant category–response mappings.     

Null category-length and target–lure relatedness effects in episodic recognition: A constraint on item-noise interference models

There are two main classes of model of interference effects in recognition memory: item-noise and context-noise. Item-noise models predict that a loss of memory discriminability will occur with an increase in the number of studied items from the same taxonomic category (category length, CL) and that forced-choice recognition performance will be higher when the target and lure are related rather than unrelated. Context-noise models, however, predict null effects for both of these manipulations. Although results from some recent experiments suggest that CL and target–lure relatedness have a trivial or no effect on memory discriminability when the related items from the same taxonomic category are “not back to back in the study list but are separated (spaced) by interleaving items from other semantic categories,” these experiments have methodological limitations that were eliminated in the present experiment in which exemplars representing category lengths of 2, 8, or 14 were presented spaced apart within the same study list. Recognition was tested using a yes/no recognition test or a two-alternative forced-choice recognition test in which the target and lure were either related or unrelated. In yes/no recognition, d′ decreased as CL increased, replicating prior research. However, when the slope of the z-ROC function is less than 1.0, as is typically so and was so in the present results, d′ differences can arise due to criterion shifts and are not necessarily due to memory discriminability differences. When the more appropriate measure of memory discriminability, d _a , was computed, CL had no effect in yes/no recognition, nor did it have an effect in forced-choice recognition, which also was not affected by target–lure relatedness. Thus, the present results are congruent with context-noise models and pose a challenge for item-noise models.     

Order Matters! Influences of Linear Order on Linguistic Category Learning

Linguistic category learning has been shown to be highly sensitive to linear order, and depending on the task, differentially sensitive to the information provided by preceding category markers (premarkers, e.g., gendered articles) or succeeding category markers (postmarkers, e.g., gendered suffixes). Given that numerous systems for marking grammatical categories exist in natural languages, it follows that a better understanding of these findings can shed light on the factors underlying this diversity. In two discriminative learning simulations and an artificial language learning experiment, we identify two factors that modulate linear order effects in linguistic category learning: category structure and the level of abstraction in a category hierarchy. Regarding category structure, we find that postmarking brings an advantage for learning category diagnostic stimulus dimensions, an effect not present when categories are non-confusable. Regarding levels of abstraction, we find that premarking of super-ordinate categories (e.g., noun class) facilitates learning of subordinate categories (e.g., nouns). We present detailed simulations using a plausible candidate mechanism for the observed effects, along with a comprehensive analysis of linear order effects within an expectation-based account of learning. Our findings indicate that linguistic category learning is differentially guided by pre- and postmarking, and that the influence of each is modulated by the specific characteristics of a given category system.     

Overcoming default categorical bias in spatial memory

In the present study, we investigated whether a strong default categorical bias can be overcome in spatial memory by using alternative membership information. In three experiments, we tested location memory in a circular space while providing participants with an alternative categorization. We found that visual presentation of the boundaries of the alternative categories (Experiment 1) did not induce the use of the alternative categories in estimation. In contrast, visual cuing of the alternative category membership of a target (Experiment 2) and unique target feature information associated with each alternative category (Experiment 3) successfully led to the use of the alternative categories in estimation. Taken together, the results indicate that default categorical bias in spatial memory can be overcome when appropriate cues are provided. We discuss how these findings expand the category adjustment model (Huttenlocher, Hedges, & Duncan, 1991) in spatial memory by proposing a retrieval-based category adjustment (RCA) model.     

On the use of heterogeneous thresholds ordinal regression models to account for individual differences in response style

This paper proposes a general approach to accounting for individual differences in the extreme response style in statistical models for ordered response categories. This approach uses a hierarchical ordinal regression modeling framework with heterogeneous thresholds structures to account for individual differences in the response style. Markov chain Monte Carlo algorithms for Bayesian inference for models with heterogeneous thresholds structures are discussed in detail. A simulation and two examples based on ordinal probit models are given to illustrate the proposed methodology. The simulation and examples also demonstrate that failing to account for individual differences in the extreme response style can have adverse consequences for statistical inferences.The author is grateful to Ulf Böckenholt, an associate editor, and three anonymous reviewers for helpful comments, and Kristine Kuhn and Kshiti Joshi for providing the data.     

Statistical learning of phonetic categories: insights from a computational approach

Recent evidence ( Maye, Werker & Gerken, 2002 ) suggests that statistical learning may be an important mechanism for the acquisition of phonetic categories in the infant's native language. We examined the sufficiency of this hypothesis and its implications for development by implementing a statistical learning mechanism in a computational model based on a mixture of Gaussians (MOG) architecture. Statistical learning alone was found to be insufficient for phonetic category learning – an additional competition mechanism was required in order for the categories in the input to be successfully learnt. When competition was added to the MOG architecture, this class of models successfully accounted for developmental enhancement and loss of sensitivity to phonetic contrasts. Moreover, the MOG with competition model was used to explore a potentially important distributional property of early speech categories – sparseness – in which portions of the space between phonetic categories are unmapped. Sparseness was found in all successful models and quickly emerged during development even when the initial parameters favoured continuous representations with no gaps. The implications of these models for phonetic category learning in infants are discussed.     

A Bayesian Multi-Level Factor Analytic Model of Consumer Price Sensitivities Across Categories

Identifying price sensitive consumers is an important problem in marketing. We develop a Bayesian multi-level factor analytic model of the covariation among household-level price sensitivities across product categories that are substitutes. Based on a multivariate probit model of category incidence, this framework also allows the researcher to model overall price sensitivity (i.e., indicated by higher-order factor scores) as a function of household-level covariates. All model parameters are estimated simultaneously to circumvent the downward bias resulting from two-stage estimation. The modeling framework is illustrated using scanner panel data from multiple categories of instant coffee.     

On the Interaction of Theory and Data in Concept Learning

Standard models of concept learning generally focus on deriving statistical properties of a category based on data (i.e., category members and the features that describe them) but fail to give appropriate weight to the contact between people's intuitive theories and these data. Two experiments explored the role of people's prior knowledge or intuitive theories on category learning by manipulating the labels associated with the category. Learning differed dramatically when categories of children's drawings were meaningfully labeled (e.g., “done by creative children”) compared to when they were labeled in a neutral manner. When categories are meaningfully labeled, people bring intuitive theories to the learning context. Learning then involves a process in which people search for evidence in the data that supports abstract features or hypotheses that have been activated by the intuitive theories. In contrast, when categories are labeled in a neutral manner, people search for simple features that distinguish one category from another. Importantly, the final study suggests that learning involves an interaction of people's intuitive theories with data, in which theories and data mutually influence each other. The results strongly suggest that straight-forward, relatively modular ways of incorporating prior knowledge into models of category learning are inadequate. More telling, the results suggest that standard models may have fundamental limitations. We outline a speculative model of learning in which the interaction of theory and data is tightly coupled. The article concludes by comparing the results to recent artificial intelligence systems that use prior knowledge during learning.     

Simultaneous Two-Way Clustering of Multiple Correspondence Analysis

A 2-way clustering approach to multiple correspondence analysis is proposed to account for cluster-level heterogeneity of both respondents and variable categories in multivariate categorical data. Specifically, in the proposed method, multiple correspondence analysis is combined with k-means in a unified framework in which k-means is applied twice to partition the object scores of respondents and the weights of variable categories. In this way, joint clusters that relate a subgroup of respondents exclusively to a subset of variable categories are obtained. The proposed method provides a low-dimensional map of displaying variable category points and the centroids of joint clusters simultaneously. In addition, it offers joint-cluster memberships of variable categories as well as respondents. A Monte Carlo study is conducted to assess the parameter recovery capability of the proposed method based on synthetic data. An empirical application concerning Korean consumers' preferences toward various underwear brands and attributes is presented to demonstrate the effectiveness of the proposed method as compared with 2 relevant extant approaches.