Use and misuse of metrics in research evaluation

This paper addresses some critical issues in the applicability of quantitative performance measures (including bibliometric, economic, and co-occurrence measures) to the assessment of basic research. The strengths and weaknesses of metrics applied as research performance measures are examined. It is concluded that metrics have a useful role to play in the evaluation of research. Each metric employed, whether bibliometric, economic, co-occurrence, or others, brings a new dimension of potential insight to the complex problem of research assessment. However, when used in a stand-alone mode, metrics can be easily misinterpreted and result in misleading conclusions. Metrics should be an integral part of a more comprehensive approach to research evaluation, in which the leading role is assumed by expert peer review. Under this structure, the reviewers must be broadly constituted, so that the technical correctness of the research approaches being taken (doing the job right) can be evaluated by technical experts and the correctness of the research target (doing the right job) can be evaluated as well. This comprehensive approach insures quality in the implementation of existing research paradigms and allows the introduction of revolutionary new research paradigms as well. The views contained in this paper are solely those of the author and do not represent the views of the Department of the Navy. Ronald Kostoff received a Ph.D. from Princeton University, performed technical and economic research at Bell Laboratories, managed energy programs at U.S. Department of Energy, was Director of Technical Assessment at the Office of Naval Research for many years and presently manages the Navy’s In-House Laboratory Independent Research Program.     

Using the Web to facilitate international academic exchange

Use of the World-Wide Web among academics has grown explosively during the last few years. To date, most applications of the Web have focused on the storage and dissemination of relatively static information, in part because of the cost associated with manual collating and updating of data. This paper presents an example of a dynamic worldwide database devoted to maintaining inherently transient information: The database accepts international travel itineraries submitted by individual academic psychologists, and makes them publicly available to potential hosts on the basis of various search keys. The objective of the database (http://www.psy.uwa.edu.au/wave/) is to facilitate international exchange among academics.     

The organization of multidimensional selection on the basis of color and shape: An event-related brain potential study

In this paper, we examine whether color and shape, tied to a single object in space, (1) are identified and selected in series or in parallel, (2) are identified and selected in a dependent, self-terminating manner or in an independent and exhaustive manner, and (3) are conjoined by a feature integration process before or only after an initial stage of separate attribute analyses has finished. We measured response time and the selection negativity (SN) derived from event-related brain potentials when participants responded to a unique conjunction of color and shape in a go/no-go target detection task. The discriminability of the color and the shape of the conjunction was manipulated in three conditions. When color and shape were easy to discriminate, the SNs to color and shape started at the same time. When one attribute was less discriminable, the SN to that attribute started later, but not the SN to the complementary attribute. This suggests that color and shape are identified and selected in parallel. In all three discriminability conditions, the SNs to color and shape were initially independent but later interacted. This suggests that color and shape are initially selected independently and exhaustively, after which their conjunction is analyzed. The SN to local shape features started later than that to the conjunction of color and global shape features, which suggests that feature integration can start before the analyses of the separate attributes have finished.     

Asymmetries and errors in perception of depth from disparity suggest a multicomponent model of disparity processing

In three experiments, asymmetries between the processing of crossed and uncrossed disparities were investigated. The target was a luminance-defined circle concentric to a fixation mark, viewed stereoscopically on a computer monitor for 105 msec. Fifteen disparities were presented according to the method of constant stimuli. Observers indicated the apparent direction of target depth relative to fixation. All experiments measured both the accuracy and latency of this response. Experiment 1 showed fewer errors and shorter reaction times for identifying crossed disparities. Experiments 2 and 3 replicated Experiment 1 and also showed that observers may often perceive a target in the direction opposite that prescribed by the disparity information. We propose that the asymmetries and reversals result from differences in computation of sign, not of magnitude. This notion is consistent with a scheme of continuous disparity tuning and accounts for such asymmetries and errors without positing disparity pooling mechanisms.     

Tic douloureux, Parkinson’s disease and the herpes connection

Several cases encountered in psychiatric practice are described in which herpetic involvement of the trigeminal nerve appeared to be crucial to the pathophysiology of the patients’s illness. A brief history of herpes is discussed in terms of the anatomy and physiology of the trigeminal nerve. Particular attention is given to the effects of intraneuronal herpes on gasserian ganglion and spinal nuclei discharge threshoolds, as well as viral pathways directly impacting the substantia nigra. Evidence characterizing Parkinson’s disease as one of several important manifestations of CNS herpes is covered. This article will be presented, by invitation, at the Neuroscience Seminar, Department of Medicine, University of Otago, Dunedin, New Zealand, on 19 August 1997.     

Influences of Destructive Personality Information on Decision Making1

This study examined the effects of information about a landlord's personality on juror judgments for a landlord-tenant civil trial. The personality information manipulated was specifically relevant to destructive acts. Alternative versions of witness testimony were used to describe a landlord either high or low on need for power. treating people as objects, and negative life themes. This information strongly influenced aspects of the schema for this case constructed by the individual mock jurors. Destructive personality information caused dislike of the landlord and lowered the credibility of his story. If the landlord treated people as objects, the credibility of the tenant's story and positive evaluations of the tenant increased. Juror judgments about relative fault were strongly shifted by destructive personality information. An empirical model for juror decisions indicated a dynamic interplay of story components and fault judgments.     

The Ramsey Test and Conditional Semantics

Proponents of the projection strategy take an epistemic rule for the evaluation of English conditionals, the Ramsey test, as clue to the truth-conditional semantics of conditionals. They also construe English conditionals as stronger than the material conditional. Given plausible assumptions, however, the Ramsey test induces the semantics of the material conditional. The alleged link between Ramsey test and truth conditions stronger than those of the material conditional can be saved by construing conditionals as ternary, rather than binary, propositional functions with a hidden contextual parameter. But such a ternary construal raises problems of its own.     

Measurement and mismeasurement of mood: recurrent and emergent issues

The affective explosion in psychology has led to tremendous advances in mood measurement. Mood ratings reflect a hierarchical structure consisting of two broad dimensions-Positive Affect and Negative Affect-and multiple specific states. Brief scales have been developed that reliably assess Positive and Negative Affect across different populations and time frames, in both between- and within-subject data. We examine controversies related to (a) the content of these higher order scales and (b) the independence of Positive and Negative Affect. Regarding the latter, we show that Positive and Negative Affect scales remain largely independent across a wide range of conditions, even after controlling for random and systematic error. Finally, there remains little consensus regarding the lower order structure of affect. This lack of a compelling taxonomy has substantially slowed progress in assessing mood at the specific affect level.