Use and misuse of metrics in research evaluation

This paper addresses some critical issues in the applicability of quantitative performance measures (including bibliometric, economic, and co-occurrence measures) to the assessment of basic research. The strengths and weaknesses of metrics applied as research performance measures are examined. It is concluded that metrics have a useful role to play in the evaluation of research. Each metric employed, whether bibliometric, economic, co-occurrence, or others, brings a new dimension of potential insight to the complex problem of research assessment. However, when used in a stand-alone mode, metrics can be easily misinterpreted and result in misleading conclusions. Metrics should be an integral part of a more comprehensive approach to research evaluation, in which the leading role is assumed by expert peer review. Under this structure, the reviewers must be broadly constituted, so that the technical correctness of the research approaches being taken (doing the job right) can be evaluated by technical experts and the correctness of the research target (doing the right job) can be evaluated as well. This comprehensive approach insures quality in the implementation of existing research paradigms and allows the introduction of revolutionary new research paradigms as well. The views contained in this paper are solely those of the author and do not represent the views of the Department of the Navy. Ronald Kostoff received a Ph.D. from Princeton University, performed technical and economic research at Bell Laboratories, managed energy programs at U.S. Department of Energy, was Director of Technical Assessment at the Office of Naval Research for many years and presently manages the Navy’s In-House Laboratory Independent Research Program.     

Effects of route familiarity on drivers’ psychological conditions: Based on driving behaviour and driving environment

Route familiarity affects a driver’s mental state and indirectly affects traffic safety; however, this important factor is easily overlooked. Previous research on route familiarity has only analysed psychological states in terms of unfamiliarity and familiarity, the influence of driving behaviour and driving environment on psychological states has been ignored. As a result, the mechanisms through which the route familiarity influence driver psychological states, and vice versa, are unclear. This study proposes a quantitative framework for studying driver psychological condition and route familiarity using experimental data from a real driving task and driving environment data. The experimental data included 1022 observations obtained by 23 participants over 7 consecutive trials on 6 unfamiliar experimental routes with large differences in scenarios; environmental data were automatically extracted after segmenting a driving video through the Dilated Residual NetWorks model. The results reveal that (1) the relationship between the driver’s psychological condition and route familiarity is not monotonic and is different for straight and turning sections; (2) the driver’s psychological condition is influenced by the visual scene elements and the type of road section, and the results of the multivariate regression analysis quantified the variability of the influence; and (3) unlike a majority of findings on distracted driving, our study suggest that the driver’s attention to the external environment in the urban distracted driving state will gradually approach a ‘distraction threshold’, and the time and size of the ‘distraction threshold’ are influenced by the driver. This study can further the development of urban traffic safety research and help urban designers plan and improve urban landscapes to ensure drivers maintain stable mental states when they drive.     

Global Progress I: Empirical Evidence for ongoing Increase in Quality-of-life

This paper argues that both the relativist and the pessimist critiques of the idea of progress are inadequate. Progress is defined as increase in global quality of life (QOL). Such QOL is intrinsically subjective, but not relative. It can be reliably measured through life satisfaction-type questions. The World Database of Happiness provides extensive data on social, economic and psychological factors that correlate with overall QOL. They include wealth, health, security, knowledge, freedom and equality. Various statistical data suggest that all these QOL indicators have undergone significant improvements during the last half century, in most of the world. This gives strong support to the thesis that progress objectively occurs.     

Subjective well‐being and human welfare around the world as reflected in the Gallup World Poll

We present data on well‐being and quality of life in the world, including material quality of life such as not going hungry, physical health quality of life such as longevity, social quality of life such as social support, environmental health such as clean water, equality in income and life satisfaction, and levels of subjective well‐being (SWB). There are large differences between nations in SWB, and these are predicted not only by economic development, but also by environmental health, equality and freedom in nations. Improving trends in SWB are seen in many countries, but declining SWB is evident in a few. Besides average differences in SWB between nations, there are also large disparities within many countries. We discuss the policy opportunities provided by national accounts of SWB, which are increasingly being adopted by many societies. They provide the opportunity to inform policy deliberations with well‐being information that reflects not only economic development, but also other facets of quality of life as well. National accounts of SWB reflect the quality of life in areas such as health, social relationships and the natural environment, and therefore capture a broader view of societal well‐being than afforded by measures of economic progress alone.     

Introducing a forced-choice recognition task to the California Verbal Learning Test – Children’s Version

The importance of performance validity tests (PVTs) is increasingly recognized in pediatric neuropsychology. To date, research has focused on investigating whether PVTs designed for adults function similarly in children. The downward extension of adult cutoffs is counter-intuitive considering the robust effect of age-related changes in basic cognitive skills in children and adolescents. The purpose of this study was to examine the signal detection properties of a forced-choice recognition trial (FCR-C) for the California Verbal Learning Test – Children’s Version. A total of 72 children aged 6–15 years (M = 11.1 , SD = 2.6) completed the FCR-C as part of a larger neuropsychological assessment battery. Cross-validation analyses revealed that the FCR-C had good signal detection performance against reference PVTs. The first level of failure (≤14/15) produced the best combination of overall sensitivity (.31) and specificity (.87). A more conservative FCR-C cutoff (≤13) resulted in a predictable trade-off between sensitivity (.15) and specificity (.94), but also a net loss in discriminant power. Lowering the cutoff to ≤12 resulted in a slight improvement in specificity (.97) but further deterioration in sensitivity (.14). These preliminary findings suggest that the FCR-C has the potential to become the newest addition to a growing arsenal of pediatric PVTs.     

The importance of including quality-of-life indices in international social and economic development activities

Psychology has not been a visible player in international social and economic development efforts. Through its demonstrated commitment to the concept of quality of life, psychology has an opportunity to help shape foreign policy and to improve the lives of countless people around the globe. Poverty has reached unacceptable limits of humanitarian tolerance and political consequence throughout the world. The United Nations estimates that 20% of the world's population now lives in conditions of absolute poverty in which there is an absence of even the bare essentials for living. Social and economic development efforts have often failed despite good intentions because they have often concentrated on improving peoples' material level of living but not their quality of life. This article addresses the need to include quality-of-life (QOL) indices in international social and economic development efforts. In addition, the article calls attention to the need to use valid cross-cultural measurement strategies (i.e., culturally equivalent) when assessing QOL across cultural and national boundaries. Current approaches to social and economic development rely heavily on interventions that do not reflect the actual peoples' perceptions of life satisfaction and subjective well-being. Self-serving political and economic national interests have kept new approaches to development from being implemented. New interventions must be holistic, decentralized, integrated, empowering, participatory, and human-resource directed, and must include culturally equivalent objective and subjective quality-of-life indices as the arbiters of success.     

The Ecology of Adolescent Substance Abuse Service Utilization

This paper presents an ecological-community model toward the explanation of variation in patterns of substance abuse (SA) service utilization among adolescents who are enrolled in Tennessee's Medicaid program (TennCare). Guided by a theoretical framework that draws from the social ecology work of Bronfenbrenner and health services utilization models promoted by Aday and Andersen, we apply a social indicators approach toward explaining the impact of community ecology on identification of SA and treatment engagement. Both county-level rates and individual-level treatment utilization are examined and hierarchical linear modeling is incorporated to examine the individual-in-community phenomenon. This study is an expansion of previous service utilization research and suggests that explanations of youth's service utilization must necessarily include not only individual, familial, and service system characteristics, but community factors, as well.     

Bayesian Estimation for Item Factor Analysis Models with Sparse Categorical Indicators

Psychometric models for item-level data are broadly useful in psychology. A recurring issue for estimating item factor analysis (IFA) models is low-item endorsement (item sparseness), due to limited sample sizes or extreme items such as rare symptoms or behaviors. In this paper, I demonstrate that under conditions characterized by sparseness, currently available estimation methods, including maximum likelihood (ML), are likely to fail to converge or lead to extreme estimates and low empirical power. Bayesian estimation incorporating prior information is a promising alternative to ML estimation for IFA models with item sparseness. In this article, I use a simulation study to demonstrate that Bayesian estimation incorporating general prior information improves parameter estimate stability, overall variability in estimates, and power for IFA models with sparse, categorical indicators. Importantly, the priors proposed here can be generally applied to many research contexts in psychology, and they do not impact results compared to ML when indicators are not sparse. I then apply this method to examine the relationship between suicide ideation and insomnia in a sample of first-year college students. This provides an important alternative for researchers who may need to model items with sparse endorsement.