Toward More Clinically Relevant Assessment Research

The purpose of this article is to encourage the design and conduct of more clinically relevant assessment research. The discussion focuses on the distinction between research design elements that are appropriate to the examination of validity versus utility issues in assessment. This was addressed in 2 ways. The first part of the article summarizes the results of an informal methodological review of 108 empirically based articles published in 3 major assessment journals (Assessment, Journal of Personality Assessment, & Psychological Assessment) during the period of September 1998 through August 1999. The results indicated that studies published in these journals demonstrated appropriate sensitivity to some aspects of utility but not others. For example, none of the articles addressed the process of interpretation in clinical settings or the reactions of stakeholders to assessment. Recommendations for enhancing the integration of validity and utility issues in the design of research are discussed. These include methodological issues and topics in need of further study. A particularly important example of the latter is the need for research identifying factors that foster positive impressions of psychological assessment.     

Introduction to the special section on clinical applications of analogue behavioral observation

This Special Section addresses methods, validity, and utility of analogue behavioral observation. Separate contributions to the Special Section cover analogue behavioral observation of marital interaction, child behavior problems, parent-child interaction, and adult social functioning. Additional articles address psychometric foundations of analogue behavioral observations, general issues, and future directions in the development and evaluation of this assessment method. Many published studies were reviewed in detail and issues of validity, clinical assessment utility, and sources of variance in obtained measures are addressed.     

Introduction to the special section on incremental validity and utility in clinical assessment

This special section focuses on the incremental validity and utility of clinical assessment data. The lack of replicated incremental validity research limits the ability of psychologists to establish their assessment practices on a solid empirical footing. The articles in this section deal with conceptual, methodological, and content issues in the development and use of clinical measures. The authors addressed several aspects of incremental validity research, including (a) the evaluation of the magnitude of validity increments, (b) the use of incremental validity data for test development and validation, (c) the costs of assessment, (d) the use of multi-informant and multimethod assessments, and (e) the treatment utility of assessment. Obstacles limiting the use of incremental validity research to inform and guide clinical practice are also emphasized.     

A comparative view of the new journal: Assessment

The reference sections from all articles in the 1997 volumes of Assessment, Journal of Personality Assessment, and Psychological Assessment were entered into a database and analyzed. An article published in Assessment averaged almost 31 references. An article published in Journal of Personality Assessment contained an average of 33 references. Psychological Assessment averaged 38 references per article. The median age of the references in the three journals was 8 years with an interquartile range of 4 to 14 years. The Journal of Personality Assessment had the largest number of citations in this database of 5,316 references. Each of these received a relatively large number of their citations from articles published in the same journal (self-citations). Randomly selected articles from the 1997 volume of Assessment received fewer citations in the Social Science Citation Index than a similar set of articles from the other two journals. However, the data on Assessment, when compared with data available on other new scientific publications, suggests that Assessment is doing as well as other fledgling journals.     

Errando Discimus: Has Post‐Yugoslav Political and Social Science Neglected Religion?

This article explores the issue of neglecting religion in the post‐Yugoslav political and social science literature (1990–2018). Therefore, it offers a qualitative and quantitative analysis of the leading political and social science journals from Serbia, Croatia, Bosnia, and Herzegovina. The research question that this article will tackle is: Has post‐Yugoslav political and social science neglected religion? The hypothesis of this article is that, although there was an obvious need for in‐depth studies of the relationship between religion and politics in the post‐Yugoslav region, leading political and social science journals published from Serbia, Croatia, Bosnia, and Herzegovina have neglected religion. In order to test our hypothesis, we employ content analysis by examining the archives of the leading journals published in the above stated states and basic statistics in order to highlight the percentage and extent of published articles dealing with religion compared to the total volume of articles published.     

The Development of a Clinical Rating Scale the McMaster Model of Family Functioning

This article describes the development and validation of the McMaster Clinical Rating Scale (MCRS). The MCRS is a 7-item scale designed to be completed by a trained rater after completion of an in-depth interview of the family. We present data from four new studies and review previously published articles concerning the reliability, validity, and clinical utility of the MCRS. Adequate interrater reliability and rater stability were obtained. The MCRS was found to correlate significantly with the self-report Family Assessment Device and to discriminate between families in different phases of a depressive disorder.     

Introduction to the special section on developing guidelines for the evidence-based assessment (EBA) of adult disorders

The goal of this special section is to encourage greater awareness of evidence-based assessment (EBA) in the development of a scientifically supported clinical psychology. In this introductory article, the authors describe the elements that authors in this special section were asked to consider in their focused reviews (including the scope of available psychometric evidence, advancements in psychopathology research, and evidence of attention to factors such as gender, age, and ethnicity in measure validation). The authors then present central issues evident in the articles that deal with anxiety, depression, personality disorders, and couple distress and in the accompanying commentaries. The authors conclude by presenting key themes emerging from the articles in this special section, including gaps in psychometric information, limited information about the utility of assessment, the discrepancy between recommended EBAs and current training and practice, and the need for further data on the process of clinical assessment.     

THE PERCEPTIONS AND USAGE OF STATISTICAL POWER IN APPLIED PSYCHOLOGY AND MANAGEMENT RESEARCH

We first assess the current level of statistical power across articles in seven leading journals that represent a broad sample of applied psychology and management research. We next survey the authors of these articles to examine their perceptions and usage of statistical power analysis. Finally, we examine the perceptions and usage of power analysis in a survey of authors of regression-based research appearing in leading journals. Findings from the assessment of power and surveys of researchers indicate that power analyses are not typically conducted, researchers perceive little need for statistical power, and power in published research is low. We conclude by discussing implications of low power for the field and recommending avenues for improving researchers' awareness and usage of statistical power.     

Toward the more efficient use of assessment center technology in personnel selection

Empirical research has consistently supported the validity and business utility of the assessment center method as a selection instrument. Nonetheless, the method as typically applied may be unnecessarily costly and inflexible. This paper begins by describing how the model for assessment center design that is widely used today came to be accepted as a standard. Then modifications in design and operation are discussed. These suggested modifications are intended to enhance the utility and flexibility of assessment centers, while at the same time maintaining, or even increasing, the validity of the process.Seymour Adler is Associate Professor of Applied Psychology at Stevens Institute of Technology, Hoboken, New Jersey, and Vice President of Assessment Systems Incorporated, a New York-based consulting firm.     

A review of research methods in IO/OB work-family research

A methodological review was conducted of work-family (WF) research published in industrial-organizational psychology and organizational behavior journals over a period of 24 years (1980-2003). Content analysis was conducted on 225 individual studies published in 210 articles to categorize methodological features, including the research design, sources of data used, data analysis techniques, reliability and validity of measures used, and sociodemographic characteristics of the samples. Results support many of the criticisms of WF research and suggest that scholars publishing WF research in industrial-organizational psychology and organizational behavior journals could make greater use of longitudinal and experimental research designs, gather more multisource data, and move beyond the individual level of analysis. Adopting more diverse conceptualizations of family, including a greater proportion of racial and ethnic minorities, and studying workers in occupations other than managerial or professional positions also appear warranted. Finally, methodological trends varied across specific WF content areas, which suggests that distinct methodologies might be useful to advance knowledge of specific WF topics.     

Exploring MMPI-Rorschach relationships

Limitations exist in the current literature examining Minnesota Multiphasic Personality Inventory (MMPI)-Rorschach interrelationships. In this article I attempt to identify several issues to be addressed by researchers in this area. These include restricting conclusions about MMPI-Rorschach relationships to research using the Comprehensive System; considering the relative reliability and validity of these tests; comparing the predictive power of the Rorschach and MMPI; examining the effects extreme outliers have on statistical analyses; and determining the appropriate level of analysis for research, the level of individual test scores, or psychological constructs. The implications of these issues for research design, statistical analysis, and applied clinical work are discussed.     

Reporting ethical practices in journal articles

Little attention has focused on the reporting of ethical research practices in journal articles. In Study 1, published articles in 2 psychopathology journals were reviewed to ascertain the types of ethical research information that were reported. In Study 2, a survey was sent to authors in Study 1 to determine which ethical practices they engaged in, if they reported this information, and reasons for not including this information in their article. In general, there is a great variability regarding the types of ethical research practices reported in journal articles. Commonly cited reasons for not including ethical research practice information in the articles included the need for brevity, belief that it was common practice, and lack of relevance for the project. These results suggest that there is no standard practice for reporting research practices in journal articles and great variability in the implementation of procedures that are generally considered standard.     

Lesbian,gay, bisexual,and transgender youth: Limited representation in school support personnel journals

Many lesbian, gay, bisexual, and transgender (LGBT) youth experience harassment and discrimination in schools and these experiences lead to increased negative social-emotional outcomes. Youth who can identify at least one supportive adult at school report better outcomes than youth who cannot identify a safe adult. Yet, many educators report feeling uncomfortable or unprepared to support LGBT youth. One reason for educators' discomfort may be that content related to issues unique to LGBT youth is sometimes missing or covered minimally in university training programs. We hypothesized that LGBT content may be covered minimally in school support personnel journals, as well. This study analyzed eight school support personnel journals across the disciplines of school counseling, school nursing, school psychology, and school social work for LGBT content published between 2000 and 2014 to gain a better understanding of the visibility of LGBT issues in the research. Results suggested that there has been a lack of presence of LGBT issues in journals across disciplines. These results also suggest a need for an intentional focus on issues relevant to LGBT youth in school support personnel journals. Thus, the article concludes with an introduction to two articles in this special topic section, including Russell, Day, Ioverno, and Toomey's (in this issue) study on teacher perceptions of bullying in the context of enumerated school policies and other supportive sexual orientation and gender identity related practices and Poteat and Vecho's (in this issue) study on characteristics of bystanders in homophobic bullying situations. The broad goal of these three studies is to increase visibility of critical LGBT issues in school support personnel journals.     

Analysis of Citations to Biomedical Articles Affected by Scientific Misconduct

We describe the ongoing citations to biomedical articles affected by scientific misconduct, and characterize the papers that cite these affected articles. The citations to 102 articles named in official findings of scientific misconduct during the period of 1993 and 2001 were identified through the Institute for Scientific Information Web of Science database. Using a stratified random sampling strategy, we performed a content analysis of 603 of the 5,393 citing papers to identify indications of awareness that the cited articles affected by scientific misconduct had validity issues, and to examine how the citing papers referred to the affected articles. Fewer than 5% of citing papers indicated any awareness that the cited article was retracted or named in a finding of misconduct. We also tested the hypothesis that affected articles would have fewer citations than a comparison sample; this was not supported. Most articles affected by misconduct were published in basic science journals, and we found little cause for concern that such articles may have affected clinical equipoise or clinical care.     

Age and scholarly impact

The claim that older scientists generate research of lower quality than do younger scientists was tested through two analyses in which the age distribution of authors of frequently cited articles in psychology journals was compared with the age distribution of authors of low-impact articles published in the same journals. Most high-impact articles were published by relatively young psychologists, but so were most low-impact articles. When allowance was made for relative numerical representation, there was no evidence that publications from older scientists have less impact. Results are discussed in the context of methodological issues in evaluation of relations between age and scientific achievement.     

Vocational behavior and career development, 1983: A review

This article reviews the literature related to vocational behavior and career development published during 1983. Journals in the fields of psychology, sociology, and organizational behavior were examined, and 445 relevant articles published in 42 different journals were identified. The review is organized around issues pertinent to the counseling psychology perspective (i.e., career development, vocational choice, vocational behavior of women, assessment, intervention strategies) and the industrial/organizational psychology perspective (i.e., personnel functions, worker adjustment problems, work adjustment) on vocational behavior.     

Publication Ethics in Biomedical Journals from Countries in Central and Eastern Europe

Publication ethics is an important aspect of both the research and publication enterprises. It is particularly important in the field of biomedical science because published data may directly affect human health. In this article, we examine publication ethics policies in biomedical journals published in Central and Eastern Europe. We were interested in possible differences between East European countries that are members of the European Union (Eastern EU) and South-East European countries (South-East Europe) that are not members of the European Union. The most common ethical issues addressed by all journals in the region were redundant publication, peer review process, and copyright or licensing details. Image manipulation, editors’ conflicts of interest and registration of clinical trials were the least common ethical policies. Three aspects were significantly more common in journals published outside the EU: statements on the endorsement of international editorial standards, contributorship policy, and image manipulation. On the other hand, copyright or licensing information were more prevalent in journals published in the Eastern EU. The existence of significant differences among biomedical journals’ ethical policies calls for further research and active measures to harmonize policies across journals.     

Needs versus deeds: The unfulfilled research potential in school psychology

A nationwide sample of school psychology practitioners and one of authors of recent journal articles, from three main school psychology journals, were asked to rate the importance of the types of needed research in school psychology. Close agreement was found between the priorities assigned by the two groups. Prevention of school-related problems and improvement in the functioning of non-referred populations were the areas identified as most in need of research by both authors and practitioners. A comparison of the priorities for needed research with a previously published study of types of articles actually published in the school psychology press revealed little agreement between perceived needs and actual publications. The results are discussed in terms of implications for collaborative research between field-based practitioners and researchers, and possible actions by journal editorial boards and professional organizations that would facilitate greater research productivity in areas of great need.     

An evaluation of the quantity and quality of empirical research in three pastoral care and counseling journals, 1990-1999: has anything changed?

This article summarizes a review of all articles published in Pastoral Psychology, The Journal of Rleigion and Health, and The Journal of Pastoral Care between 1900 and 1999, identifying a total of 737 scholarly articles, of which 165 (22.4%) were research studies. The proportion of research studies, especially quantitative studies, increased significantly between the first and second half of the study period (p < .05). There was a significant positive correlation between compliance with three out of four criteria of internal validity. Three of five criteria of external validity were also positively related to one another. Compared to previous research using identical criteria to assess quantitative studies in the same journals in 1980-1989, the 1990-1999 sample showed improved compliance with respect to specifying the sampling method (p < .001), reporting the response rate (p < .05), and discussing the limitations of research studies (p < .001). However, the overall findings suggest that many researchers in the field do not have a sophisticated knowledge of statistical sampling, statistical analysis, or research design. Several recommendations for increasing the quality of quantitative research are offered.     

The Scientometric Bubble Considered Harmful

This article deals with a modern disease of academic science that consists of an enormous increase in the number of scientific publications without a corresponding advance of knowledge. Findings are sliced as thin as salami and submitted to different journals to produce more papers. If we consider academic papers as a kind of scientific ‘currency’ that is backed by gold bullion in the central bank of ‘true’ science, then we are witnessing an article-inflation phenomenon, a scientometric bubble that is most harmful for science and promotes an unethical and antiscientific culture among researchers. The main problem behind the scenes is that the impact factor is used as a proxy for quality. Therefore, not only for convenience, but also based on ethical principles of scientific research, we adhere to the San Francisco Declaration on Research Assessment when it emphasizes “the need to eliminate the use of journal-based metrics in funding, appointment and promotion considerations; and the need to assess research on its own merits rather on the journal in which the research is published”. Our message is mainly addressed to the funding agencies and universities that award tenures or grants and manage research programmes, especially in developing countries. The message is also addressed to well-established scientists who have the power to change things when they participate in committees for grants and jobs.