A more powerful method for testing for agreement between a judge and a known standard

We assume that a judge's task is to categorize each ofN subjects into one ofr known classes. The design of primary interest is employed if the judge is presented withs groups, each containingr subjects, such that each group of sizer consists of exactly one subject of each of ther types. The probability distribution for the total number of correct choices is developed and used to test the null hypothesis that the judge is guessing in favor of the alternative that he or she is operating at a better than chance level. The power of the procedure is shown to be superior to two other procedures which appear in the literature.The authors are grateful for the suggestions of the referees and for computer funding provided by the Northeast Regional Data Center at the University of Florida.     

Hubert's multi-rater kappa revisited

There is a frequent need to measure the degree of agreement among R observers who independently classify n subjects within K nominal or ordinal categories. The most popular methods are usually kappa-type measurements. When R = 2, Cohen's kappa coefficient (weighted or not) is well known. When defined in the ordinal case while assuming quadratic weights, Cohen's kappa has the advantage of coinciding with the intraclass and concordance correlation coefficients. When R > 2, there are more discrepancies because the definition of the kappa coefficient depends on how the phrase ‘an agreement has occurred’ is interpreted. In this paper, Hubert's interpretation, that ‘an agreement occurs if and only if all raters agree on the categorization of an object’, is used, which leads to Hubert's (nominal) and Schuster and Smith's (ordinal) kappa coefficients. Formulae for the large-sample variances for the estimators of all these coefficients are given, allowing the latter to illustrate the different ways of carrying out inference and, with the use of simulation, to select the optimal procedure. In addition, it is shown that Schuster and Smith's kappa coefficient coincides with the intraclass and concordance correlation coefficients if the first coefficient is also defined assuming quadratic weights.     

Pragmatic versus form-based accounts of referential contrast: evidence for effects of informativity expectations

Characterizing the relationship between form-based linguistic knowledge and representation of context has long been of importance in the study of on-line language processing. Recent experimental research has shown evidence of very rapid effects of referential context in resolving local indeterminacies on-line. However, there has been no consensus regarding the nature of these context effects. The current paper summarizes recent work covering a range of phenomena for which referential contrast has been shown to influence on-line processing, including prenominal and postnominal modification, focus operators, and intonational focus. The results of the body of work suggest that referential context effects are not limited to situations in which the linguistic form of the utterance directly specifies the point of contact with context. Rather, context effects of a pragmatic, Gricean nature appear to be possible, suggesting the relationship between linguistic form and context in rapid on-line processing can be of a very indirect nature.     

A Signal Detection Model for Multiple-Choice Exams

A model for multiple-choice exams is developed from a signal-detection perspective. A correct alternative in a multiple-choice exam can be viewed as being a signal embedded in noise (incorrect alternatives). Examinees are assumed to have perceptions of the plausibility of each alternative, and the decision process is to choose the most plausible alternative. It is also assumed that each examinee either knows or does not know each item. These assumptions together lead to a signal detection choice model for multiple-choice exams. The model can be viewed, statistically, as a mixture extension, with random mixing, of the traditional choice model, or similarly, as a grade-of-membership extension. A version of the model with extreme value distributions is developed, in which case the model simplifies to a mixture multinomial logit model with random mixing. The approach is shown to offer measures of item discrimination and difficulty, along with information about the relative plausibility of each of the alternatives. The model, parameters, and measures derived from the parameters are compared to those obtained with several commonly used item response theory models. An application of the model to an educational data set is presented.     

A Test of the Adams-Creamer Decay Hypothesis for the Timing of Motor Responses

The Adams-Creamer hypothesis states that S uses the decay of proprioceptive feedback from an early portion of a movement to cue a timed response at some later time. This hypothesis was tested by creating passive left-arm movement in one group and withholding it from another, and having Ss make a right-hand response when exactly 2.0 sec. had elapsed since the end of the movement Ss with left-arm feedback had less absolute and algebraic error, and greater within-S consistency than did the no-movement control Ss and, when KR was withdrawn, Ss with left-arm movement regressed less than did Ss without the left-arm movement, which provided 2 lines of support for the decay hypothesis.     

Identifiability and Equivalence of GLLIRM Models

The generalized logit–linear item response model (GLLIRM) is a linearly constrained nominal categories model (NCM) that computes the scale and intercept parameters for categories as a weighted sum of basic parameters. This paper addresses the problems of the identifiability of the basic parameters and the equivalence between different GLLIRM models. It is shown that the identifiability of the basic parameters depends on the size and rank of the coefficient matrix of the linear functions. Moreover, two models are observationally equivalent if the product of the respective coefficient matrices has full column rank. Finally, the paper also explores the relations between the parameters of nested models. I would like to express my gratitude to the editor and three anonymous reviewers for their helpful suggestions on earlier versions of the paper. This work was supported by the Comunidad de Madrid (Spain) grant: CCG07-UAM/ESP-1615.     

On more powerful tests of judge agreement with a known standard

To establish the existence of his abilities, a judge is given the task of classifying each ofN=rs subjects into one ofr known categories, each containings of the subjects. An incomplete design is proposed whereby the judge is presented withb groups, each one containingn=rs/b<r subjects. Then different categories corresponding to members of the group are known. Using the total number of correct classifications, this method of grouping is compared to that in which the group size is equal to the number of categories. The incomplete grouping is shown to yield a more powerful test for discriminating between the null hypothesis that the judge is guessing the classifications and the alternative hypothesis that he has some definite abilities. The incomplete design is found to be most effective (powerful) when the number of subjects in a group is limited to two or three.The author is grateful for the suggestions of the referees and the editor, which greatly improved the paper.     

The Question of Questions: What is a Gene? Comments on Rolston and Griffths & Stotz

If the question “What is a gene?” proves to be worth asking it must be able to elicit an answer which both recognizes and address the reasons why the concept of the gene ever seemed to be something worth getting excited about in the first place as well analyzing and evaluating the latest develops in the molecular biology of DNA. Each of the preceding papers fails to do one of these and sufferrs the consequences. Where Rolston responds to the apparent failure of molecular biology to make good on the desideratum of the classical gene by veering off into fanciful talk about “cybernetic genes,” Griffiths and Stotz lose themselves in the molecular fine print and forget to ask themselves why “genes” should be of any special interst anyway.