Ensuring the Quality,Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making.     

Authors and publication practices

Although authors are usually considered to be the main perpetrators of research and publication misconduct, any person involved in the process has the potential to offend. Editors may breach ethical standards particularly with respect to conflicts of interest. In the same way that authors are now required to declare competing interests, notably commercial affiliations, financial interests and personal connections, so must editors. Editors can influence the chances of acceptance or rejection of a paper by reviewer selection. Reviewers should also be ready to disclose conflicts of interest. They must ensure that their reviews are evidence based and free from destructive criticism driven by self interest. It seems likely that ultimately we will progressively move towards 'open' peer review in which both the authors and the reviewers are known to each other. There is an urgent need for increased transparency of the relationship between editors and owners. The events of the last few years indicate that unless this interface is fully understood by all parties, conflicts may arise. There is also a need for a radical overhaul in the relationship between journals, journal editors and the biomedical industry. It is now increasingly accepted that all clinical trials should be registered in a centrally held database and that protocols should include the primary and secondary outcome measures and the intended approach to data analysis thereby avoiding opportunistic post hoc analyses. However, the even more radical proposal that journals should cease to publish clinical trials sponsored by industry deserves wider debate.     

The selection of creative ideas after individual idea generation: Choosing between creativity and impact

It is commonly assumed that successful innovation depends on creative idea generation: the more ideas are generated, the higher the probability of selecting a very good idea should be. However, research has shown that people do not perform optimally at idea selection and that ideational output may not contribute much to creative idea selection. The present studies aim to explain this phenomenon. We identified the strong tendency of our participants to select feasible and desirable ideas, at the cost of originality, as the main reason for their poor selection performance. Two manipulations of participants' processing of the available ideas (exclusion instructions and quality ratings) had no effect on selection effectiveness. In contrast, explicitly instructing participants to select creative or original ideas did improve selection effectiveness with regard to idea originality, but at the same time decreased participants' satisfaction and the rated effectiveness of chosen ideas. Results are discussed in relation to an effectiveness‐originality trade off.     

The Development of Ethics: A Historical and Critical Study

Editor's Introduction When Oxford University Press sent us the three enormous volumes of Irwin's The Development of Ethics, we had two thoughts: First, the book is very important and demands a review; second, since human sacrifice is abolished in North America, it will be very difficult to find a reviewer. We handed the volumes to several interested persons, who in the end returned the books saying the task was beyond them. Then, my wife, a lifetime worker at that center of communal thought, the United Nations, suggested that we form a team to review the book. We put an announcement out on the Web, asking for reviewers to do a chapter each, at 250 words a review. We got several hundred volunteers, and chose 82 to review the 96 chapters of Irwin (reviewers got the chapters in Portable Document Format [PDFs], kindly supplied by Oxford). We got 81 of 82 reviews, 75 before the deadline, six slightly later. For purposes of completeness, I filled in the sole missing review. Would that the students in my seminars were so punctual! I would like to thank Elyse Turr at Oxford University Press for the PDFs, and my 82 reviewers for their expertise and diligence. I am grateful to all the volunteers who showed an interest in this strange and perhaps unprecedented project, and who patiently endured the vetting process. Special thanks is due to Laura di Summa, who coordinated all the pieces of this incredible puzzle. Did we accomplish something, something new, by mobilizing 82 minds to review one book? I hope so, but I can now only say what they say on television: “America, it's up to you.”     

Design and Evaluation of a Decision Aid for Inviting Parents to Participate in a Fragile X Newborn Screening Pilot Study

The major objectives of this project were to develop and evaluate a brochure to help parents make an informed decision about participation in a fragile X newborn screening study. We used an iterative development process that drew on principles of Informed Decision Making (IDM), stakeholder input, design expertise, and expert evaluation. A simulation study with 118 women examined response to the brochure. An independent review rated the brochure high on informational content, guidance, and values. Mothers took an average of 6.5 min to read it and scored an average of 91.1 % correct on a knowledge test. Most women rated the brochure as high quality and trustworthy. When asked to make a hypothetical decision about study participation, 61.9 % would agree to screening. Structural equation modeling showed that agreement to screening and decisional confidence were associated with perceived quality and trust in the brochure. Minority and white mothers did not differ in perceptions of quality or trust. We demonstrate the application of IDM in developing a study brochure. The brochure was highly rated by experts and consumers, met high standards for IDM, and achieved stated goals in a simulation study. The IDM provides a model for consent in research disclosing complicated genetic information of uncertain value.     

Group decision making by experts: field study of panels evaluating medical technologies

Decision-making processes and their outcomes were investigated in six consensus development conferences at the National Institutes of Health in which panels of experts evaluated new medical technologies. One hundred seventy-seven self-administered questionnaires were obtained from participants in these conferences. Questionnaire data were analyzed along with data derived from content analyses of the six consensus statements (CS) produced by the conferences. Results of these analyses provide considerable support for the hypotheses that the quality of the outcome (i.e., the CS) is determined by the existence of an interaction process, a decision procedure, and a chairperson, which facilitate the exchange of relevant information. Strong disagreements among the panelists appear to inhibit such exchange and harm the quality of the CS. Personal satisfaction appears to be more strongly related to the quality of the process and of the information disseminated than to the quality of the outcome. A clear relation was found between the panelists' status and expertise, their participation in the process, and their contribution to the CS. The pattern of these findings is quite similar to that obtained in laboratory studies. The role of preconference organizational factors, such as the selection of conference questions, panel, and speakers, and the characteristics of the technology are discussed.     

Using a dialectical scientific brief in peer review

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. The author researches, lectures and publishes in the area of physical and social aspects of environmental quality. He is also a practicing architect. This paper is based on a presentation at a workshop, “Advances in Peer Review Research”, American Association for the Advancement of Science Meeting, Baltimore, MD, February 9, 1996.     

Should Authors be Requested to Suggest Peer Reviewers?

As part of a continuous process to explore the factors that might weaken or corrupt traditional peer review, in this paper, we query the ethics, fairness and validity of the request, by editors, of authors to suggest peer reviewers during the submission process. One of the reasons for the current crisis in science pertains to a loss in trust as a result of a flawed peer review which is by nature biased unless it is open peer review. As we indicate, the fact that some editors and journals rely on authors’ suggestions in terms of who should peer review their paper already instills a potential way to abuse the trust of the submission and publishing system. An author-suggested peer reviewer choice might also tempt authors to seek reviewers who might be more receptive or sympathetic to the authors’ message or results, and thus favor the outcome of that paper. Authors should thus not be placed in such a potentially ethically compromising situation, especially as a mandatory condition for submission. However, the fact that they do not have an opt-out choice during the submission process—especially when using an online submission system that makes such a suggestion compulsory—may constitute a violation of authors’ rights.     

Rightly crossing the Rubicon: Evaluating goal self-concordance prior to selection helps people choose more intrinsic goals

We studied how people “cross the Rubicon” when making personal goal selections. In Studies 1 and 2 participants rated the self-concordance of four candidate goals, two with intrinsic and two with extrinsic content, before selecting two goals to actually pursue. Intrinsic goal content predicted higher self-concordance, as did matching between goal content and participant values and motives. Self-concordance in turn explained participants’ actual goal-selections. In longitudinal Study 2, intrinsic goal selection predicted increased well-being. In experimental Study 3, participants randomly assigned to rate candidate goals prior to selection made more intrinsic selections on average, compared to those not afforded this opportunity. We conclude that considering one’s motivations for various candidate goals prior to selecting among them can improve one’s goal choices.     

Standards on how to develop and report systematic reviews in Psychology and Health

Systematic reviews are secondary studies that summarize the best scientific evidence available by means of explicit and rigorous methods to identify, select, appraise, analyse and summarise the empirical studies that enable responding to specific questions. The aim of this theoretical study is to set out a series of standards and recommendations for the planning, development and reporting of a systematic review in the field of the health sciences. The article describes the systematic reviews in the context of practice based on scientific evidence, their rise, justification, applicability and differences compared to traditional literature reviews. Secondly, the methodology is set out for their development and guidelines are established for their preparation; the stages of the process and preparation of a protocol are described with emphasis on the steps to follow to prepare and report a systematic review. Finally, some additional considerations are set out for their preparation and publication in a scientific journal. This guide is aimed both at authors and reviewers of a systematic review.     

National attitudes towards mental health treatment: the importance of research evidence

Most conceptualizations of evidence-based practice view it as a “three legged stool” consisting of: the use of best available research evidence, clinical expertise, and client preferences. Although empirical evidence and clinical expertise have received greater empirical attention, relatively little research has systematically explored client preferences. The present study analyzed self-reported treatment preferences for various clinical and non-clinical presentations. Adult participants (n = 1262) residing in the United States were presented with diagnostic vignettes and rated their relative preferences among 5 treatment variables, including: use of an empirically supported treatment (EST), quality of the client–therapist relationship, therapist empathy, therapist experience, and client speaking for the majority of therapy sessions. Results indicated that participants endorsed significant preference for receiving an EST over other treatment variables for all clinical disorders, with effect sizes ranging from small to large depending on the diagnosis. There was slightly greater variability in treatment preferences for non-clinical issues, though participants generally reported greater preference for receiving an EST. Follow-up questions provided further evidence for EST preferences. The implications of these results are discussed.     

How Regulatory Orientation and Feelings of Gratitude Shape Online Review Helpfulness

Online review helpfulness ratings are an important indicator of the impact of online reviews. Often times, helpfulness is explained in terms of observable qualities of online reviews that predict helpfulness ratings. This research proposes that focusing on the psychological processes that underlie helpfulness voting informs a better understanding of what shapes review helpfulness ratings. Specifically, because goal orientation influences information processing, consumers’ regulatory orientation interacts with review valence to determine review helpfulness. When review valence and regulatory orientation match, consumers are more likely to express helpfulness through voting. The findings show that this effect occurs at least in part because matching review valence and regulatory orientation instills feelings of gratitude towards the reviewer. As a consequence, consumers are more likely to reward the reviewer with a helpfulness vote to express their feeling of gratitude through actions. However, when reviewers actively state expectations of reciprocal behavior by readers, gratitude is reduced and so is the likelihood that a review receives a helpfulness vote. Evidence from five studies using review data and online experiments show support for these effects.     

Using religious paradox to facilitate life review groups with shame-driven older adults

This article explains how pastoral counselors who lead life review groups with shame-driven older members can use religious paradox as a model for change. The article first elaborates upon the challenge to self-acceptance that poor self-esteem poses for older adults engaged in life review. It then shows how paradoxes enhance self-acceptance by drawing reviewers to a more holistic understanding of their past and evoking insights about the meaning of their personal history. It presents a model of group process in which members individually select religious paradoxes and relate such paradoxes to their life review. It concludes with examples of how three participants used religious paradoxes to own the life review instead of feeling threatened by it.