Ensuring the Quality,Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making.     

The Problem of Peer Review is the Most Important Philosophical Problem

As philosophers we should have as one of our aims to produce as much philosophical knowledge as possible. A lot of potential philosophical knowledge is lost because of the flaws of the peer review system, and so a lot of philosophical knowledge would be gained were the system improved. Accordingly, as authors we should write papers about how to fix peer review, and as editors we should accept such papers if they are good. This paper presents some familiar problems with peer review, elaborates on and motivates the argument just given, and replies to some objections to it, making the case that fixing peer review is both a philosophical problem and one that admits of a solution.     

A Method for Improving the Integrity of Peer Review

Peer review is the most important aspect of reputable journals. Without it, we would be unsure about whether the material published was as valid and reliable as is possible. However, with the advent of the Internet, scientific literature has now become subject to a relatively new phenomenon: fake peer reviews. Some dishonest researchers have been manipulating the peer review process to publish what are often inferior papers. There are even papers that explain how to do it. This paper discusses one of those methods and how editors can defeat it by using a special review ID. This method is easy to understand and can be added to current peer review systems easily.     

Should Authors be Requested to Suggest Peer Reviewers?

As part of a continuous process to explore the factors that might weaken or corrupt traditional peer review, in this paper, we query the ethics, fairness and validity of the request, by editors, of authors to suggest peer reviewers during the submission process. One of the reasons for the current crisis in science pertains to a loss in trust as a result of a flawed peer review which is by nature biased unless it is open peer review. As we indicate, the fact that some editors and journals rely on authors’ suggestions in terms of who should peer review their paper already instills a potential way to abuse the trust of the submission and publishing system. An author-suggested peer reviewer choice might also tempt authors to seek reviewers who might be more receptive or sympathetic to the authors’ message or results, and thus favor the outcome of that paper. Authors should thus not be placed in such a potentially ethically compromising situation, especially as a mandatory condition for submission. However, the fact that they do not have an opt-out choice during the submission process—especially when using an online submission system that makes such a suggestion compulsory—may constitute a violation of authors’ rights.     

Establishing Sensible and Practical Guidelines for Desk Rejections

Publishing has become, in several respects, more challenging in recent years. Academics are faced with evolving ethics that appear to be more stringent in a bid to reduce scientific fraud, the emergence of science watchdogs that are now scrutinizing the published literature with critical eyes to hold academics, editors and publishers more accountable, and a barrage of checks and balances that are required between when a paper is submitted and eventually accepted, to ensure quality control. Scientists are often under increasing pressure to produce papers in an increasingly stringent publishing environment. In such a climate, timing is everything, as is the efficiency of the process. Academics appreciate that rejections are part of the fabric of attempting to get a paper published, but they expect the reason to be clear, based on careful evaluation of their work, and not on superficial or unsubstantiated excuses. A desk rejection occurs when a paper gets rejected even before it has entered the peer review process. This paper examines the features of some desk rejections and offers some guidelines that would make desk rejections valid, fair and ethical. Academics who publish are under constant pressure to do so quickly, but effectively. They are dependent on the editors’ good judgment and the publisher’s procedures. Unfair, unsubstantiated, or tardy desk rejections disadvantage academics, and editors and publishers must be held accountable for wasting their time, resources, and patience.     

Reporters and a Free Press

ABSTRACT A necessary condition for news to be produced under conditions of 'negative' freedom is for newspapers to become 'reporter codetermined newspapers', where reporters, along with editors and publishers, have a collective 'positive' freedom to vote on news-policy, select editors and hire reporters. 'Publisher-controlled' newspapers systematically prevent reporters from reporting some news-stories and coerce and manipulate reporters into reporting others. It is argued here that all newspapers should be legally required to become reporter codetermined newspapers. This change is also required by Rawls' and Mill's arguments for freedom of the press. On the basis of empirical data it is unlikely that the demands of freedom or justice can be met within existing institutional structures.     

Referees, editors, and publication practices: Improving the reliability and usefulness of the peer review system

The documented low levels of reliability of the peer review process present a serious challenge to editors who must often base their publication decisions on conflicting referee recommendations. The purpose of this article is to discuss this process and examine ways to produce a more reliable and useful peer review system. The author is a Senior Research Scientist & Biostatistician, Child Study Center and Department of Psychiatry in the Yale University School of Medicine. Note: The main points of this report were presented at the Annual Meeting of the American Association for the Advancement of Science (AAAS) in Baltimore, Maryland, on February 9, 1996.     

Scientific Peer Review: A Case Study From Local And Global Analyses.

The dialog between Staddon (2001, 2004) and Baum (2004) raises general questions about the nature of scientific peer review. Their dialog displays effects on peer review of differences of opinion about the relative merits of local and global analyses. Baum (1995, 1997, 2001, 2002) favors global analyses as a paradigm different, newer, and better than the local, dynamic, real‐time approach that plays a significant role in Staddon (2001). According to the Kuhnian perspective (Kuhn, 1996) Baum advocates, we can better understand his review of Staddon (2001) by considering the implications for it of his commitment to the idea that a global analysis is a superior scientific paradigm. This commentary examines some characteristics of local and global analyses, as well as some of their possible implications for peer review in the context of a reviewer's belief in the Kuhnian idea of incommensurability: According to this idea, a reviewer who either is, or who believes he is, from one paradigm is unlikely, for better or worse, to understand or perhaps even tolerate work from a different paradigm. It is recommended that a process be developed to encourage “truth in peer reviewing” to reduce possible conflicts of interest embedded in the current conception of scientific peer review.     

Authors and publication practices

Although authors are usually considered to be the main perpetrators of research and publication misconduct, any person involved in the process has the potential to offend. Editors may breach ethical standards particularly with respect to conflicts of interest. In the same way that authors are now required to declare competing interests, notably commercial affiliations, financial interests and personal connections, so must editors. Editors can influence the chances of acceptance or rejection of a paper by reviewer selection. Reviewers should also be ready to disclose conflicts of interest. They must ensure that their reviews are evidence based and free from destructive criticism driven by self interest. It seems likely that ultimately we will progressively move towards 'open' peer review in which both the authors and the reviewers are known to each other. There is an urgent need for increased transparency of the relationship between editors and owners. The events of the last few years indicate that unless this interface is fully understood by all parties, conflicts may arise. There is also a need for a radical overhaul in the relationship between journals, journal editors and the biomedical industry. It is now increasingly accepted that all clinical trials should be registered in a centrally held database and that protocols should include the primary and secondary outcome measures and the intended approach to data analysis thereby avoiding opportunistic post hoc analyses. However, the even more radical proposal that journals should cease to publish clinical trials sponsored by industry deserves wider debate.     

Using a dialectical scientific brief in peer review

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. The author researches, lectures and publishes in the area of physical and social aspects of environmental quality. He is also a practicing architect. This paper is based on a presentation at a workshop, “Advances in Peer Review Research”, American Association for the Advancement of Science Meeting, Baltimore, MD, February 9, 1996.     

Conflict(s) of Interest in Peer Review: Its Origins and Possible Solutions

Scientific communication takes place at two registers: first, interactions with colleagues in close proximity—members of a network, school of thought or circle; second, depersonalised transactions among a potentially unlimited number of scholars can be involved (e.g., author and readers). The interference between the two registers in the process of peer review produces a drift toward conflict of interest. Three particular cases of peer review are differentiated: journal submissions, grant applications and applications for tenure. The current conflict of interest policies do not cover all these areas. Furthermore, they have a number of flaws, which involves an excessive reliance on scholars’ personal integrity. Conflicts of interest could be managed more efficiently if several elements and rules of the judicial process were accepted in science. The analysis relies on both primary and secondary data with a particular focus on Canada.     

Observations on Unacknowledged Authorship from Homer to Now

In this essay brief sketches of three historical cases of unacknowledged authorship are offered to remind readers that unacknowledged authorship has been and still may be viewed in different ways given different contexts and purposes. Reflecting on these cases and many others that come to mind, it seems that the contemporary scene concerning unacknowledged authorship does not indicate a huge deterioration of research or publishing integrity. Following the brief historical journey, overviews of two contemporary cases are presented to illustrate some difficulties that editors and publishers have today that those in earlier historic periods could not have had, as well as some suggested procedures for managing such difficulties.     

The cultural context of a grievance process: Employee values and expectations in dispute resolution

The nature of employer-employee relationships is influenced by organizational and work unit values and expectations, including the ways members believe they should treat one another and how conflicts should be resolved. This study explored the perceptions of participants in a grievance peer review process. Employees of a health care organization with a recently established peer review grievance process were interviewed about their views of organizational and unit values and expectations, and their experiences with this type of grievance process. Comparisons across groups of grievants, managers, and grievance panel members were discussed.The authors have received consent to discuss this case from the health care organization in which this study was conducted. The authors would like to thank the members of this organization for their cooperation and extraordinary responsiveness that made this research possible, and Julie Rousos Timmons for her valuable help on the fieldwork.     

Social Biases and Solutions for Procedural Objectivity

An empirically sensitive formulation of the norms of transformative criticism must recognize that even public and shared standards of evaluation can be implemented in ways that unintentionally perpetuate and reproduce forms of social bias that are epistemically detrimental. Helen Longino's theory can explain and redress such social bias by treating peer evaluations as hypotheses based on data and by requiring a kind of perspectival diversity that bears, not on the content of the community's knowledge claims, but on the beliefs and norms of the culture of the knowledge community itself. To illustrate how socializing cognition can bias evaluations, we focus on peer‐review practices, with some discussion of peer‐review practices in philosophy. Data include responses to surveys by editors from general philosophy journals, as well as analyses of reviews and editorial decisions for the 2007 Cognitive Science Society Conference.     

Peer review and innovation

Two important aspects of the relationship between peer review and innovation includes the acceptance of articles for publication in journals and the assessment of applications for grants for the funding of research work. While there are well-known examples of the rejection by journals of first choice of many papers that have radically changed the way we think about the world outside ourselves, such papers do get published eventually, however tortuous the process required. With grant applications the situation differs in that the refusal of a grant necessarily curtails the possible research that may be attempted. Here there are many reasons for conservatism and reservation as to the ability of a grant allocation process based on peer review to deliver truly innovative investigations. Other methods are needed; although such methods need not be applied across the board, they should constitute the methods whereby some 10–20% of the grant monies are assigned. The nomination of prizes for specific accomplishments is one way of achieving innovation although this presumes that investigators or institution already have available the money necessary to effect the innovations; otherwise it is a question of the selection and funding of particular individuals or institutions and requiring them to solve particular problems that are set in the broadest of terms.     

When reviews attack: ethics, free speech, and the peer review process.

The peer review process, whether formally applied in publication and grant review, or informally, such as exchange of ideas in scientific and professional newsgroups, has sparked controversy. Writers in this area agree that scholarly reviews that are inappropriate in tone are not uncommon. Indeed, commentators have suggested rules and guidelines that can be used to improve the review process and to make reviewers more accountable. In this paper, we examine the relevance and impact of ethical codes on the conduct of peer review. It is our contention that the peer review process can be improved, not by a new set of rules but through closer attention to the ethical principles to which we, as psychologists, already subscribe.