Use and misuse of metrics in research evaluation

This paper addresses some critical issues in the applicability of quantitative performance measures (including bibliometric, economic, and co-occurrence measures) to the assessment of basic research. The strengths and weaknesses of metrics applied as research performance measures are examined. It is concluded that metrics have a useful role to play in the evaluation of research. Each metric employed, whether bibliometric, economic, co-occurrence, or others, brings a new dimension of potential insight to the complex problem of research assessment. However, when used in a stand-alone mode, metrics can be easily misinterpreted and result in misleading conclusions. Metrics should be an integral part of a more comprehensive approach to research evaluation, in which the leading role is assumed by expert peer review. Under this structure, the reviewers must be broadly constituted, so that the technical correctness of the research approaches being taken (doing the job right) can be evaluated by technical experts and the correctness of the research target (doing the right job) can be evaluated as well. This comprehensive approach insures quality in the implementation of existing research paradigms and allows the introduction of revolutionary new research paradigms as well. The views contained in this paper are solely those of the author and do not represent the views of the Department of the Navy. Ronald Kostoff received a Ph.D. from Princeton University, performed technical and economic research at Bell Laboratories, managed energy programs at U.S. Department of Energy, was Director of Technical Assessment at the Office of Naval Research for many years and presently manages the Navy’s In-House Laboratory Independent Research Program.     

Spline smoothing with model-based penalties

Nonparametric regression techniques, which estimate functions directly from noisy data rather than relying on specific parametric models, now play a central role in statistical analysis. We can improve the efficiency and other aspects of a nonparametric curve estimate by using prior knowledge about general features of the curve in the smoothing process. Spline smoothing is extended in this paper to express this prior knowledge in the form of a linear differential operator that annihilates a specified parametric model for the data. Roughness in the fitted function is defined in terms of the integrated square of this operator applied to the fitted function. A fastO(n) algorithm is outlined for this smart smoothing process. Illustrations are provided of where this technique proves useful.     

Encoding, repetition, and the mirror effect in recognition memory: Symmetry in motion

Attention/likelihood theory has been used to explain the mirror effect in recognition memory. The theory also predicts that any manipulation that affects the recognition of old items will also affect recognition of the new items—more specifically, that all the underlying distributions will move and that they will move symmetrically on the decision axis. In five experiments, we tested this prediction. The first two experiments used encoding tasks during study to change recognition performance for high- and low-frequency words. The results show symmetrical dispersion of the underlying distributions. The final three experiments used repetition to increase recognition performance. Repetition produced a symmetrical pattern of movement that was different from that produced by encoding task. This pattern is, however, also covered by attention/likelihood theory. A further extension of the theory was used to predict response times.     

Inattention-Hyperactivity and Reading Achievement in Children from Low-Income Families: A Longitudinal Model

This study examined how preschool inattention-hyperactivity is related to elementary school reading achievement. Prereading skills were hypothesized to be a link between them. This link was explored using longitudinal data on 105 low-socioeconomic-status (SES) children's inattentive-hyperactive behavior and prereading skills in Head Start and in kindergarten and their inattentive-hyperactive behavior and reading skills in first grade. A model of this relationship was tested using structural equation modeling. The results failed to show a significant path between inattention-hyperactivity and prereading skills at both the Head Start and kindergarten levels. A significant path was found between first grade inattention-hyperactivity and reading skills, confirming that the strong relationship between inattention-hyperactivity and poor reading achievement commonly found in children from other SES levels was also significant in this low-SES sample. Strong relationships were found between pre-reading skills and reading skills, as well as among hyperactivity levels at the three grades. The issue of the direction of the path of influence between attention-behavior and reading achievement is addressed briefly; however, the results indicate that further longitudinal work is necessary to resolve this issue.     

THE EFFECT OF SEX SIMILARITY ON RECRUITERS' EVALUATIONS OF ACTUAL APPLICANTS: A TEST OF THE SIMILARITY-ATTRACTION PARADIGM

The present study examined the effect of sex similarity on recruiters' evaluations of actual applicants in campus interviews and tested whether perceived similarity and interpersonal attraction mediated this effect. Structural equation modeling was performed on data from 476 campus interviews. The results revealed complex effects of sex similarity on recruiters' decision processes. Perceived similarity and interpersonal attraction mediated the effect of sex similarity on female recruiters' assessments of applicants' qualifications. Unexpectedly, female recruiters saw male applicants as more similar to themselves and more qualified than female applicants. For male recruiters, interview outcomes were not affected by sex similarity. Social identity theory, which suggests that the status associated with demographic characteristics may influence the nature of demographic similarity effects, provides a possible explanation for the findings.     

Resistance to reinforcement change in multiple and concurrent schedules assessed in transition and at steady state

Behavioral momentum theory relates resistance to change of responding in a multiple-schedule component to the total reinforcement obtained in that component, regardless of how the reinforcers are produced. Four pigeons responded in a series of multiple-schedule conditions in which a variable-interval 40-s schedule arranged reinforcers for pecking in one component and a variable-interval 360-s schedule arranged them in the other. In addition, responses on a second key were reinforced according to variable-interval schedules that were equal in the two components. In different parts of the experiment, responding was disrupted by changing the rate of reinforcement on the second key or by delivering response-independent food during a blackout separating the two components. Consistent with momentum theory, responding on the first key in Part 1 changed more in the component with the lower reinforcement total when it was disrupted by changes in the rate of reinforcement on the second key. However, responding on the second key changed more in the component with the higher reinforcement total. In Parts 2 and 3, responding was disrupted with free food presented during intercomponent blackouts, with extinction (Part 2) or variable-interval 80-s reinforcement (Part 3) arranged on the second key. Here, resistance to change was greater for the component with greater overall reinforcement. Failures of momentum theory to predict short-term differences in resistance to change occurred with disruptors that caused greater change between steady states for the richer component. Consistency of effects across disruptors may yet be found if short-term effects of disruptors are assessed relative to the extent of change observed after prolonged exposure.     

Why is 9+7 harder than 2+3? Strength and interference as explanations of the problem-size effect

In four experiments, the problem-size effect was investigated, using an alphabet-arithmetic task in which subjects verified such problems as A + 2 = C. Problem size was manipulated by varying the magnitude of the digit addend (e.g., A + 2, A + 3, and A + 4). The frequency and similarity of problems was also manipulated to determine the contribution of strength and interference, respectively. Experiment 1 manipulated frequency at low levels of practice and found that strength could account for the problem-size effect. Experiment 2 manipulated frequency at higher levels of practice, and found that strength alone could not account for the problem-size effect at asymptote. Experiment 3 manipulated frequency and similarity and found a substantial problem-size effect at asymptote, suggesting that both strength and interference contribute to the problem-size effect. Experiment 4 manipulated similarity, keeping frequency constant, and found no problem-size effect at asymptote, suggesting that interference alone is not responsible for the problem-size effect. The results are related to findings with number arithmetic.