Tips and guidelines for being a good peer reviewer

Abstract

Publication is the key means by which science spreads. The purpose of scientific journals is to publish novel and quality articles. The editors of the journals evaluate the content of the manuscripts by submitting them to a process called “peer review”, considered today the gold standard to guarantee the adequate publication of scientific articles. A well-crafted and critical peer-review report is a treasure for both authors and editors. In the present manuscript we will examine the key aspects of the peer review process. We will begin by explaining what exactly this process consists of and since when it has existed, and then clarifying why it is so important. Then we will argue why we should want to be reviewers of scientific papers. We will then review what are the fundamental rules to carry out a good review of a manuscript and what aspects of it we should focus on. Later we will see what format a peer review report should have and how to write its different sections, as well as the options for its final resolution. We will pay special attention to commenting on the ethical aspects and the most frequent errors that are made in the evaluation of manuscripts. Finally, we will recognize what the fundamental limitations of peer review are, and we will end by proposing some suggestions for their improvement. Our ultimate goal is to stimulate researchers―and authors―to go one step further and undertake the challenge of being peer reviewers of scientific manuscripts.

Resumen

La publicación es el medio clave por el cual la ciencia se difunde. El propósito de las revistas científicas es publicar artículos novedosos y de calidad. Los editores de las revistas evalúan el contenido de los manuscritos sometiéndolos a un proceso denominado “revisión por pares” (peer review), considerado hoy en día el estándar de oro para garantizar la adecuada publicación de los artículos científicos. Un informe de revisión por pares crítico y bien elaborado es un tesoro, tanto para el autor como para el editor. En el presente manuscrito examinaremos los aspectos clave de la revisión por pares. Comenzaremos explicando en qué consiste exactamente este proceso y desde cuándo existe, para después aclarar por qué es tan importante. Luego argumentaremos por qué deberíamos querer ser evaluadores de artículos científicos. Repasaremos cuáles son las reglas fundamentales para llevar a cabo una buena revisión de un manuscrito y en qué aspectos de este nos deberíamos centrar. Posteriormente veremos qué formato debe tener un informe de revisión por pares y cómo redactar sus distintos apartados, así como las opciones de su dictamen final. Dedicaremos especial atención a comentar los aspectos éticos y los errores más frecuentes que se comenten en la evaluación de manuscritos. Finalmente, reconoceremos cuáles son las limitaciones fundamentales de la revisión por pares y terminaremos proponiendo algunas sugerencias para su mejora. Nuestro objetivo final no es otro que estimular a los investigadores―y autores―a dar un paso más y acometer el reto de ser revisores de manuscritos científicos.

Introduction

Research is an activity aimed at obtaining new knowledge and applying it to solve problems. Scientific research, in particular, uses the scientific method to study a certain aspect of reality, either theoretically or experimentally.^1,2 The key means by which science is spread is through publication. The purpose of scientific journals is to publish high-quality articles and pass on novel information to their readers. The editors of these journals assess the scientific content of manuscripts by submitting them to a process called “peer review”. Peer review involves the participation of not only the editorial board of the journal in question, but more importantly, reviewers who are considered to be experts in that particular field of science. The reviewers assess the strengths and weaknesses of the study in light of the existing literature and send their valuable comments, and criticisms, to the editors. The combination of comments submitted by various reviewers helps editors to make their final decision on the suitability of the manuscript for publication in the journal in question.³ We can therefore appreciate how essential the role of the reviewers is in guaranteeing the quality and originality of the manuscripts published in scientific journals.

Researchers tend to be trained in critical analysis of the literature, and even in the writing of scientific manuscripts, but they do not usually receive specific training on how to correctly review a scientific article. However, as soon as their academic/scientific career takes off, there will come a time, sooner rather than later, when they will be invited to act as assessor of the work of their colleagues. Despite peer review is a very important part of the scientific process, it is rarely taught at universities (at either postgraduate or postdoctoral stages) and can be an intimidating or daunting task for beginners.⁴ Although reviewing someone else's work may seem relatively easy, compared to preparing a research project² or writing a scientific manuscript,⁵ reviewing a scientific article properly requires training, effort and time.

In this manuscript, we examine the key aspects of peer-reviewing a scientific article with a personal perspective based on the authors' experience. We begin by explaining what exactly this process consists of and how long it has existed and go on to clarify why it is so important. Then we ask (and answer) why we should want to be scientific article reviewers. We then go over the basic rules for carrying out a good manuscript review and which aspects we should focus on. We look at the format a peer review report should have and how to write the different sections and discuss the options for the final opinion of said review. Special attention is given to the ethical aspects and the most common errors made in assessing manuscripts. Finally, we acknowledge the fundamental limitations of peer review and conclude by suggesting some improvements. Our goal with this manuscript is simply to encourage researchers–and authors–to go one step further and take on the challenge of being scientific manuscript reviewers.

What exactly does the peer review process consist of?

Most journals in the biomedical field apply a pre-publication process which mainly involves the interaction of three elements around a research article: the editor, the reviewer, and the author. There are therefore three key roles in a journal's review process: the authors who write the article; the reviewers who provide expert opinions and advice; and the editors who make the final decisions on whether to accept or reject the manuscript. All three players share the same goal: to publish high-quality research results in a high-quality article in a high-quality journal.⁶

When an editor receives a manuscript, the first step is to check that it complies with the journal's rules on content and format. If it does, and the editor considers the article may be of interest to the journal, they move on to the next step, which is peer review. To do this, the editors select at least two experts on the subject (hence the name given to this evaluation system, as the manuscript is supposedly sent to researchers with similar skills to the authors) and ask them to assess the article. In reality, each journal has its policy on the usual number of reviewers per manuscript; most choose two or three, but it is uncertain how much additional and truly useful information is obtained by choosing more.^7–10

Peer review can be applied to assessing academic degrees, research projects or scientific articles¹¹ in a general context of what is known as evaluated science.¹² In the specific field of academic communication, it is the process by which manuscripts sent to scientific journals are assessed and approved or, if necessary, rejected.¹²

The reviewers will prepare a report assessing the manuscript and send it to the editor. The time required for a manuscript to complete the peer review process and receive an editor's decision varies, but is often around eight weeks after initial submission.⁸ After reading the reviewers' report, the editor will decide to do one of three things: reject the article, accept it outright; or ask the authors to review and resubmit the manuscript after responding to the reviewers' comments. If the authors resubmit the manuscript, the editors may ask the same reviewers to assess the manuscript again to confirm that their suggestions have been addressed.⁴

In short, peer review is the system commonly used to assess the quality and originality of scientific research before it is published. The peer review system judges the validity, relevance and originality of the study, rather than who carried it out.

How long has peer review existed?

Peer review has been an essential element in assessing academic credibility for centuries.¹³ There are records of physicians in the Arab world dealing with reviews of the efficacy of the treatments proposed by their colleagues as early as the 9th century.⁴ Peer review as a means of evaluating scientific research before publication emerged in 17th-century Europe. In this first stage, peer review was uncommon and informal, the aim being to select which information was most worthy of publication.¹⁴Philosophical Transactions, the first journal in the world dedicated exclusively to science, was also the first to systematically implement peer review of its articles in 1665.¹⁵ Later, during the 18th century, a number of journals started using peer reviews as a way of helping editors in the selection of manuscripts for publication.¹⁶ During the 20th century (especially after the Second World War), this editorial process developed into what we now recognise as conventional peer review.¹⁴ The first conference specifically dedicated to the subject of peer review was held in Chicago in 1989, and since then these conferences have been organised regularly every few years in different countries.¹⁶

Why is peer review so important?

Peer review is widely accepted as a standard criterion in medicine and other academic disciplines in assessing the validity, originality and integrity of scientific articles submitted for publication and circulation to the community in general.¹⁷ This process is considered crucial to maintaining the quality and reliability of scientific communication and, as such, of science itself.^4,12,18 Peer review is therefore, truly at the heart of the process, not just for medical journals but for all of science in general. It is the method by which grants are allocated, articles are published, academics are promoted, and even Nobel laureates are chosen.¹⁹

Peer review is an essential part of the modern scientific process; it is the formal expression of the principle that science works best in unrestricted criticism.²⁰ If science is to progress, research methods and findings must be thoroughly examined and verified. So reviewers should ask themselves the following three questions: a) does the manuscript address an important scientific problem and offer novel results? b) are the methods used to answer the research question valid? and c) do the results, and their interpretation contribute to future research or clinical practice?

After a paper has been peer-reviewed and accepted for publication, scientists and the general public can be (more) confident that the study has met certain standards and that the results are reliable. This scrutiny is especially vital in biomedical sciences, as the publication of biased or incorrect information can seriously compromise patient safety by steering clinical decision-making towards inappropriate diagnostic or therapeutic actions.²¹

In summary, peer review is a hallmark of the vast majority of scientific journals and is the basis for assessing the quality of their publications. Without these reviews, the publication system as we know it today could not exist.^22,23 In fact, only journals which use peer reviews are considered strictly scientific.

When done properly, peer review helps improve the clarity, validity and reliability of the research.^24,25 In the well-known journal Annals of Internal Medicine,²⁵ a group of authors described the process of manuscript “transformation” from initial submission to final publication. Of the 34 items reflecting the quality of the research report, 33 improved, with the greatest benefits for items considered especially important to editors, such as the discussion of the study's limitations or the tone of the conclusions.²⁵

When preparing a review of an article, the reviewer must take into account that there are two interested parties: on one hand, the authors, and on the other, the editor of the journal. The reviewer should try to meet the needs of both. The journal's editor needs the reviewer to give an unmistakable idea of the quality of the article and its suitability for publication in the journal. It is also crucial that the reviewer clarifies which aspects of the article, at their discretion, need to be altered to make it a better candidate for publication in the journal. The authors of a manuscript need constructive criticism from the reviewer to enable them to improve it so that it can be published in the journal or, if not possible, to advise them why its limitations make it unsuitable for publication. The editors will use the reviewers' comments on the relevance and originality of a manuscript to decide whether or not it is of a sufficient calibre to be published in their journal. Here, the reviewer's detailed comments are far more important than their simple recommendation; editors need to know why the reviewer thinks that an article should be accepted or rejected.^26,27

Finally, from the readers' perspective, the reviewer needs to identify areas needing clarification, to ensure that they (the reader) can easily understand the manuscript. As a reviewer, you can also help avoid wasting readers' time and frustration by contributing to keep irrelevant or inaccurate research out of the published scientific literature.

Why should we want to be scientific article reviewers?

As researchers progresses in their career, it is likely they will be asked to be involved in the peer review process. It is worth us asking, therefore, why a researcher would want to be a reviewer.²⁸ If we think about it quickly and not too deeply, it might seem that there are only negative aspects; it is hard work, unpaid, time-consuming, and usually anonymous.¹² However, taking a bit more time to consider, we could come up with some reasons why, in our opinion, a researcher would (or at least should) want to be an reviewer:

• 1

  First of all, although it may not be the most convincing argument for everyone, we should volunteer for this task because it is necessary for the sake of scientific progress.¹² Moreover, if we have been published thanks to colleagues reviewing our articles, the least we can do is return the favour to the academic community by agreeing to be reviewers too.^12,29 Without the dedication of a large number of volunteer reviewers, the peer review process which, as we said earlier, is what guarantees the quality of scientific publications, would not be possible.³⁰

• 2

  Receiving an invitation to carry out a peer review should be considered an honour, as it shows that the editors consider us to be an authority in the field.

• 3

  Acting as a reviewer develops critical thinking skills, which will help us in our own research. As Eleanor Roosevelt said, “Learn from the mistakes of others; you can't live long enough to make them all yourself”.

• 4

  Sharpening our reviewing skills will also help us improve our own manuscripts.^27,31,32

• 5

  Reviewing (manuscripts or research projects) is a skill which needs to be honed with practice and experience; being an expert in a given field of science does not automatically make a researcher a good critic.³³ It is therefore important that, from the very start of their professional career, researchers dedicate time to practising this art.

• 6

  By acting as a reviewer, we obtain privileged information through access to scientific material which will take months (or years) to become public knowledge. Obviously this is only going to benefit our personal growth, as a fundamental ethical norm, which we discuss later, prohibits making practical/external use of the information reviewed until the article is published.¹²

• 7

  Being involved in peer reviews helps build relationships with editors of scientific journals. Obviously, this does not mean that our articles will have more chance of being published than others, but the editors are also professional colleagues with whom we could work in the future. Moreover, positioning yourself as a good reviewer is the first step towards eventually joining the editorial board of a journal.

• 8

  Having acted as a reviewer becomes part of a researcher's curriculum vitae. Being able to show that we have been chosen as reviewers by certain journals means additional kudos in our academic career. Therefore, although certainly anonymous and unpaid, it is work which has a degree of reward career-wise. Journals provide certificates certifying that we have been reviewers and, as is discussed below, initiatives such as Publons offer reviewers a verified record of the work they do.³⁴

• 9

  Lack of time is often our main reason for turning down an invitation to act as a reviewer. In a survey of reviewers from five biomedical journals to find out why they accepted or rejected the invitation to review different manuscripts, the most important factors in the decision to accept were: the contribution of the article to the subject area, the relevance of the subject to one's work; and the opportunity to learn something new.³⁵ The most important factor in turning down the invitation was workload and the consequent lack of time.³⁵ Most respondents agreed that financial incentives would not be effective when time constraints are prohibitive. However, they considered that non-financial incentives, like formal recognition by academic institutions for example, could encourage reviewers to undertake an article review. In any event, reviewing a manuscript does not take that long. People often argue that they have no time to act as a reviewer. However, the task does not take as long as we may think, so that should not be an excuse.²² Every article is different, as is the experience of each reviewer, so there is no precise figure for the necessary review time.²⁷ A wide range has been described, from 30 min to 16 h,³⁶ but the estimates are usually between two and three hours for a standard article (with no particular complications in terms of length or content).¹² Other authors calculate higher averages, of five hours⁹ or even eight.³⁷ In short, assessing an article is a matter of hours, not days or weeks (but not minutes either), and the average rate per reviewer is less than once a month.⁹ Looking at it that way may make us less reluctant to agree when invited to review an article.

Do the reviewers know the who the authors are? And vice versa?

Depending on the type of blinding in the review process, there are three types of peer review:

• 1

  Single-blind. The reviewers know the identity of the authors, but the authors do not know who the reviewers are. At least in theory, this allows reviewers to voice their criticism without fear of “retaliation” from the authors. This is the most common form of blinding, at least in scientific journals.⁹

• 2

  Double blind. So called because neither do the authors know the identity of the reviewers nor the reviewers know the identity of the authors.³⁸ It reduces possible bias (of the reviewers) from knowing who the authors are or where the assessed work comes from and, again theoretically, increases the likelihood of a manuscript being reviewed based on its own merits.³⁹ Remember that the mission of the reviewer is to judge the work, not the researcher who has carried it out. Defenders of double-blind review consider that this system reduces the bias associated with institutional prestige and guarantees more objective assessments of the research of lesser-known authors and institutions.^15,40,41 However, some studies have found no differences between the single- and double-blind systems.^42–45 However, the double-blind system is also associated with potential drawbacks. For example, the manuscript has to be made anonymous before being sent out, so the journal may require certain information to be removed. This is not always easy, as for complete anonymity, the necessary changes could include removing references to the authors' institution (in the methods section when mentioning previous studies), information on funding sources (in the acknowledgements), self-citations (in the references section) or any mention of the author's place of work (for example in the discussion). In fact, various studies have shown that blinded reviewers are nevertheless capable of correctly identifying the authors on more than 40% of occasions.^43,46,47

• 3

  Open review. The identity of authors and reviewers is known by all participants in the process.⁴⁸ The potential advantages of this system are, perhaps, greater accountability and less likelihood of impropriety, and the fact that reviewers can receive public credit for their work. Possible drawbacks include that potential reviewers are less likely to agree to review manuscripts, and revealing the reviewer's identity may lead to animosity from the authors, or harm a personal or professional relationship.

Last of all, it is worth pointing out that there are very few studies comparing the different forms of peer review (open, single-blind or double-blind), and most suffer from significant methodological limitations. A recent meta-analysis of randomised studies comparing the impact of different interventions on the peer review process failed to demonstrate a significant difference between the various alternatives, either in terms of the quality of the review or in the likelihood of rejection.⁴⁹ In short, the best review strategy is still under debate.⁴⁹ Today, what really defines a scientific journal is the fact that it uses peer review, not the specific type adopted. As a result, journal editors may, at their own discretion and within reason, choose whether or not to blind reviewers.

What are the fundamental rules for a good manuscript review?

The golden rule of a good reviewer is the so-called law of reciprocity: review others as you would like to be reviewed yourself.²² This means putting yourself in the authors' shoes and thinking that they have done everything they can to prepare an excellent manuscript but still need objective outside reviewers to help them identify potential problems the article may have. It has been suggested that an effective way to apply this rule is to ask ourselves, as a reviewer, if we would change any aspect of the assessment report if it were to be made public along with our name.^12,50 This reflection has the effect of forcing us to be more rigorous, and avoid gratuitous or unfounded statements.¹²

As reviewers, we must always be courteous, polite, respectful and professional. Value judgements, disqualifications and, in general, using rude or inconsiderate language with the authors is out of place. The tone should be reasonably courteous, but not insincerely gentle.¹² The review has to be honest, but not brutal; there is no need to cause offence. The editor should never have to rewrite/edit reviewer comments for being overly aggressive. As some reviewers very graphically point out, “The criticisms issued should be aimed at achieving a constructive reform rather than a brusque demolition of the article”.¹³

It is important to keep the focus on the research, making comments about the manuscript, not about the authors personally. For this, impersonal phrases should be used; it is better to state that “the authors' discussion of the study findings is insufficient” rather than “their discussion of the study findings is insufficient”.⁵¹

Peer review should be a positive process and offer constructive criticism.^12,52,53 When the reviewer identifies problems in a manuscript, criticism should be made in such a way that helps the authors to improve it. In other words, if a problem is to be highlighted, a solution should be proposed. Even if the reviewer recommends that the editor rejects the manuscript, suggestions for modifications could help authors prepare the manuscript for submission to a different journal. In fact, a quality review has been defined as one which is useful for the authors.⁵⁴

Whether the reviewer recommends accepting or rejecting the manuscript, they should keep in mind that one of the goals of the review is to help authors improve this and future manuscripts, not to make them despair and end up giving up. It is important, therefore, to avoid excessively negative expressions or personal comments, and to point out the main strengths of the manuscript (in addition to its weaknesses, obviously). It is best to begin by mentioning the positive aspects of the article, not only if it is accepted, but also when rejection is proposed⁵⁵; if we start with the positive aspects, the authors are more likely to be more receptive and more disposed to accept our criticisms.

Be as explicit and detailed as possible about weaknesses in the manuscript and how they can be addressed, rather than making vague generalisations which are of no use to the authors.³⁰ The page and specific line numbers (if the manuscript has them) of the part of the study we are referring to should be included.⁵⁶ This will help both the authors and the editor, who will later be able to judge whether the authors have resolved the problems in their manuscript once it has been revised.⁴ The least useful reviewers' reports (both for the authors and the editor) are those of the type, “Very good article, I recommend publishing” or “Interesting article” and its opposite, “Does not add anything new to the literature” or a variant, “Needs to improve discussion”. Instead, you need to be specific. Why is the article so good? What does the reviewer mean by “interesting”? Why do you consider that the article adds nothing new to the existing literature? Why exactly does the discussion need to be improved?⁵⁷ It is also necessary to be clear; one of the worst scenarios for the author is to receive comments from the reviewer which they do not know how to interpret or respond to. For example, a comment such as, “I don't think the study design is appropriate” will likely leave the author confused and feeling helpless.²¹

What aspects of an article need to be reviewed?

The general aspects which should be addressed in every article by the reviewer are its originality, its relevance and its validity.^58,59 Originality in research means that it must be novel enough, offering something new to readers. We need to consider whether the study question is original and whether its results represent an advance in terms of current knowledge.¹² There is no point in answering a question that has already been repeatedly answered before. Obviously, it is not always necessary that the idea or concept of the study be completely original. The reviewers should assess whether, for example, an article focusing on a question which has already been posed previously by another study with a retrospective design could be considered novel if, this time, the aim is to answer the question through a study with greater methodological rigour (for example, a randomised clinical trial).

After considering originality, reviewers should assess the relevance of the study. The most relevant articles are those which have the potential to change clinical practice patterns and, most importantly, open up new areas of research.

Lastly, the validity of the manuscript refers to how the study question was formulated and whether the article follows a clear and logical progression from that research question to its conclusion. A good study question is one that is clearly defined. The precise definitions of the outcome measures and the variables influencing them show the authors’ clarity of thought. The methods and results sections are the heart of a manuscript and the reviewer should primarily focus on them. Other flaws become largely irrelevant if a major one is detected in the methods or results which weakens the foundation on which the article is built. In fact, the most common reason for rejecting a manuscript is the use of inappropriate methods, including for the statistics.²⁷

Once the general aspects of the manuscript, originality, relevance and validity, have been reviewed, it is time to make a detailed review of the more specific aspects of each of the sections in the article. It is useful to have a checklist for this.^22,60–63 The best way to learn to be a good reviewer is to be a good article writer, and vice versa (in fact, there is a great deal of overlap between the guidelines for writing and peer review). We should probably not accept an invitation to review a manuscript if we have never experienced the peer review process as an author. The ability to review a scientific article properly, first of all, requires us to know how to write it properly ourselves. Therefore, based on our previous recommendations on how to prepare and submit a manuscript to a biomedical journal,⁵ we have prepared a checklist for reviewing an article in detail from the reviewer's perspective (Table 1). To complement that, Table 2 summarises the most common errors authors make, which can lead to a manuscript being rejected.

Table 1.

Recommendations for preparing and submitting a manuscript to a biomedical journal.

Manuscript section	Recommendation
Title	Should reflect the content of the article
	Brief, explanatory and clear
	Avoid empty phrases
Abstract	Although in miniature, it should reproduce the structure of the full article: objectives, methods, results and conclusions
	Should be understandable without having to read the entire article
	Do not include abbreviations
	Keep to the word limit imposed by the journal
Introduction	It must be attractive, to make the reader want to continue reading the article
	Brief, focused on the subject
	Concisely set out knowledge gaps
	Explain the reasons for conducting the study
	Highlight the novelty of the study adequately
	Clearly state the hypothesis or research question
	Precisely define the study objectives
	Classify the primary (usually only one) and secondary objectives
	Moderate number of literature citations
Methods	Consistency between the objectives and the methods used
	Logical and ordered methodological sequence
	Describe the inclusion criteria in detail
	Provide enough detail to enable reproduction of the study by other researchers
	Describe novel methodology meticulously
	Adequately reference methods already used previously
	Include an adequate control group
	Describe how the sample size was calculated
	Use appropriate statistical tests
	Mention the ethical aspects of the study (Ethics Committee approval and informed consent of the patient)
Results	Order results consistent with the methods
	State results without comments
	Carry out all the analyses initially planned
	Do not perform unplanned analyses
	Include a flowchart describing the progression of patients through the study
	Quantify the findings and present them with the uncertainty measurements (confidence intervals for example)
	Include less important findings in the form of annexes (supplementary material)
Discussion	Organise the discussion following the order of the methods and results
	Use the results of the study as a common thread
	Discuss the most relevant findings of the study
	Focus on the main problem, without getting sidetracked
	Properly interpret and assign weight to the results
	Link the results of the study with the findings previously obtained by other researchers
	Cite the references of studies with both similar and opposite results
	Clearly highlight what information is new
	Mention the strengths and weaknesses of the study
	Formulate conclusions supported by study findings
	At the end of the discussion, clearly summarise the key messages of the study
	Specifically suggest the need for future studies
References	Follow the journal's style recommendations
	Do not exceed the number allowed by the journal
	Only include references that have actually been consulted
	Cite only the references necessary to support the key statements of the text
	Include updated references
	In general, prioritise references to original articles over reviews.
	Avoid citing abstracts from conference presentations when we have the full article
Tables and figures	Use tables to show the exact values of more data than could be summarised in a few sentences of text
	Use tables to display a series of data interrelated
	Include each table and figure on a separate page
	Do not repeat the same information in text and tables
	Do not duplicate information between tables and figures
	Use figures as an alternative to tables with a lot of data or when there is a relationship between the data or their sequence in time
	The reader should be able to understand the contents of a table or figure without having to consult the text
	Explain the abbreviations at the foot of the table or figure
Writing (qualities of good prose)	Flow
	Clarity
	Accuracy
	Sobriety
	Attraction

Table 2.

Common mistakes by authors when writing a scientific article.

Manuscript section	Mistake
Title	Does not reflect the content of the article
	Too long
	Unclear, vague or with empty phrases
Abstract	Using abbreviations
	Not accurately representing the main text
	Does not enable us to understand the content of the article without having to read the whole thing
	Containing excessive background text and insufficient essential information on methods and results
	Exceeding the word limit imposed by the journal
Introduction	Not attractive, does not capture the reader's attention
	Not clearly identifying the knowledge gap
	Not properly justifying the need to conduct the study
	Not adequately highlighting the novelty of the study
	Not clearly stating the research question
	Not precisely defining the study objectives
	Including too many objectives, not classified into primary and secondary
	Too long, with too detailed information
	Including an excessive number of references
Methods	Lack of consistency between the objectives and the methods used
	Methodological sequence illogical and not in order
	Not adequately describing the inclusion criteria
	Not providing enough detail to enable reproduction of the study by other researchers
	Not adequately describing new methods
	Not adequately referencing methods already used previously
	Not including a control group
	High risk of bias
	Not calculating the sample size
	Not using the statistical tests properly
	Not mentioning the ethical considerations
Results	Disorganised presentation of the results (without following the order of the methods)
	Discussing the results rather than simply presenting them
	Not carrying out all the analyses initially planned
	Performing analyses not previously planned
	Not providing a detailed description of the progression of patients through the study
	Repeating results included in tables and figures
	Not calculating the uncertainty measurements (confidence intervals for example)
	Mixing less important with more relevant findings (instead of including the less important as supplementary material)
Discussion	Disorganised structure (without following the order of the methods and results)
	Too long
	Content not concise, not focused on the main question
	Discussion not focused on the findings of the study in question
	Disproportionate number of literature references
	Not linking the results of the study with the findings previously obtained by other researchers
	Omitting evidence from other studies to make your own research seem more original
	Only citing references from studies with similar results
	Not clearly highlighting what information is new
	Repeating information stated previously in the introduction
	Not mentioning the strengths and, especially, the limitations of the study
	Speculating and reaching conclusions not supported by the study findings
	Not clearly summarising key messages, ideally at the end of the discussion
	Not concluding by suggesting new lines of research, or only doing so vaguely and ambiguously
References	Not following the journal's style recommendations
	Exceeding the number allowed by the journal
	Not including the key references of the most relevant studies
	Including irrelevant references
	Including references that are not updated
	Citing abstracts from conference presentations when the full published articles are available
Tables and figures	Including unnecessary tables (with very little unrelated data)
	Using tables when figures should be used (large amounts of data or relationship between them or their sequence in time)
	Including multiple tables or figures on the same page
	Including in table data already discussed in text
	Duplicating information between tables and figures
	Creating tables or figures that are difficult to interpret and require us to consult the text of the article
	Not explaining the abbreviations at the foot of the table or figure

What format should a peer review report take?

Peer review does not have a unique and unchanging format, as each journal has its own peculiarities, although in general the main guidelines are common to all. Some journals use an informal structure, while others have a more formal approach. Sometimes the reviewer is asked to answer specific questions through a questionnaire. Other times, the reviewer is required to rate the manuscript on various attributes, using a scoring system. Therefore, the first thing we have to do before starting the evaluation process is to find out whether or not the journal has specific review guidelines and follow them.

Entering fully into the actual review report, we need to separate our comments into different sections, following the structure of the article; typically title, abstract, introduction, methods, results, discussion, references, tables and figures. Both the list of problems and the list of recommendations (if applicable) should be set out as structured as possible. Ideally, number each observation and clearly specify which part of the manuscript we are referring to (for example, stating the page and even the line of text). We should, however, avoid simply presenting a list of criticisms, line by line, without any indication of which are the most important aspects to be improved (see below the section on major and minor problems).

Some evaluators have the bad practice of composing long rambling paragraphs with various considerations interlinked, so one paragraph may refer to several, sometimes quite diverse, problems. If the problems and the recommendations are presented numbered (and separated), it is much easier for the authors to apply the changes, and for the editors and the reviewers themselves to check that they are incorporated into the new version.¹²

How to write the summary section of the evaluation report?

Starting the evaluation report with a brief summary of the manuscript is recommended. The summary can be organised into two parts, and, as mentioned above, may need to be one to four paragraphs long. The purpose of the report is to “demonstrate” (both to the editor and to the author of the article) that the reviewer has understood the research and has been able to identify its most important aspects.¹² From the author's perspective, reading the reviewer's summary will let them know whether or not they have communicated their research well, and whether or not the reviewer has interpreted it correctly. For the reviewer, the need to summarise the most important elements of the article imposes a discipline that reduces the risk of only reading it superficially; having to write a coherent summary stating the research essentials is good insurance against making impatient or hasty interpretations. Lastly, the summary can help the editor to better understand the reviewer's verdict and also to determine the potential importance (or irrelevance) of the article for the journal's readers.¹² When writing the summary, it is helpful to think that, by providing a general description of the strengths and weaknesses of the manuscript, we are preparing a kind of “takeaway” message for the editor.

Usually the summary is structured into two parts. In the first, the basic characteristics of the research should be briefly described: the object of study; the methodology; and the results or conclusions.^12,55 For example, we could say, “This study assesses the efficacy and safety of treatment X versus placebo in patients with Crohn's disease by way of a randomised clinical trial; the conclusion is that said drug has a high degree of efficacy but is associated with frequent and significant adverse effects”. The second part of the summary should be dedicated to highlighting, equally briefly, the relevance of the research, meaning its originality, opportuneness or comparison with other previous studies, indicating whether its findings are novel or only confirmatory. The most important weaknesses or limitations should also be indicated here. For example, we could say, “The article is novel, as it is the first to assess the efficacy of drug X in Crohn's disease. The design (randomised double-blind clinical trial) is adequate. However, the sample size is very small, which limits the study's conclusions”.

How to write the (major and minor) problems section of the evaluation report?

The list of problems we identify in the manuscript should be divided into major issues and minor issues. Making this distinction is essential for establishing a clear hierarchy of how important the problems are, rather than combining aspects with different degrees of importance.¹²

Major issues include essential points that authors have to address before the manuscript can be published. These are issues that can potentially lead to the rejection of the article. They do not mean automatic rejection but do mean there is a need to make significant changes in the article and pass a second round of assessments. Obviously it is possible for a major issue to be found in a manuscript which is otherwise valuable and worth publishing. As the major issues tend to be mainly related to the research itself (and not to the format of the article), they often have to do with methodological aspects or data analysis; alternatively, they may refer to the fact that the conclusions presented by the authors are not derived from the results of their study. In fact, the major issues may centre on any important section of the manuscript.¹²

Minor problems tend to be mainly related to particular aspects of the article (structure or format), rather than the actual research. These may be aspects that are important in terms of improving the manuscript, but which do not affect the key conclusions, such as writing, structure or references. For example, minor issues could include spelling or syntax errors; parts that need more explanation or others with superfluous text that can be removed; lack of reference to an important article; list of references being too long; tables or figures difficult to interpret or superfluous. Minor issues do not necessarily require a new round of evaluations but, barring well-founded reasoning from the authors, they do require a new version of the article with all the improvements added.¹²

In theory, a properly identified issue implicitly incorporates the corresponding recommendation. For example, if the issue is that the article's abstract is poorly written, it is clear that, being explicit, the recommendation would be, “Improve the writing of the abstract”.¹² However, complementing the evaluation with a specific proposal or recommendation rather than simply pointing out the deficiency can be very useful for the authors, increasing the value of the review report for them.⁶⁴

What is the confidential report for the editors and what is it for?

In this section, as concisely and directly possible, the reviewer should explain the key points determining acceptance or rejection of the article to the editor. The reasoning usually has to do with the solidity, novelty, quality and importance of the manuscript and its scientific suitability for the journal. Editors consider many factors when deciding whether an article is appropriate for their journal or not, so providing evidence or reasoning for the recommendation we make as reviewers is very helpful to them.

Comments to editors in this confidential report should be restricted to those the reviewer does not wish the authors to see; anything important for researchers to read should be included in the comments to the authors. This space should not therefore be used to make criticisms or suggestions for improving the manuscript, as the comments included here will not be passed on to the authors. However, the fact that it is not going to be read by the authors does not give the reviewers permission to “stab the author in the back”; therefore, in the spirit of fairness, the reviewer should write their comments to the editors as if the authors were also able to read them.¹² Finally, information already included in the section addressed to authors should not be repeated unnecessarily, as the editor has access to it.

As this section is confidential, it is the right place to include sensitive information or express doubts about ethical or conflicting aspects of the article, such as the detection of potential conflicts of interest not recognised by the authors, suspicion of plagiarism or fraud, redundant or split publication, or failure to comply with the requirements of ethics committees.^29,65

The confidential report is also the appropriate section for the reviewer to comment to the editor about anything they feel limits their ability to evaluate a particular part of the manuscript (instead of writing it “in the open”, which could undermine the authors' trust in the reviewer). For example, the reviewer may recognise that they do not have sufficient experience in a complex statistical analysis technique used in the article, and honestly and humbly suggest that another expert be consulted.

What are the options for a reviewer's final opinion?

The aim of the peer review process can be stated as, “The valid article is accepted, the messy article improved, and the invalid article rejected.^66,67 Therefore, three options can be considered for a review opinion, and these are summarised below.

1) Accepted without changes. Means the article is accepted for publication as it was originally received. This situation is exceptional, as all articles can be improved and the reviewers should at least make some suggestion, even if it is minor.¹² If a reviewer is unable to suggest even a modification or improvement, they are probably not evaluating the article properly. If we have agreed to review an article, we have a moral obligation to do so with the detail it deserves; if we are not prepared to do so, we simply should not agree to be reviewers.

2) Rejected (definitively). This is usually because of insufficiencies in aspects such as the development, scope or interest of the article, or even conceptual or design errors, or detection of bad practices, or because it is considered to be poor quality in general.¹² Rejection rates in peer-reviewed journals vary hugely, from more than 90% to less than 10%, with the average rejection rate estimated to be around 50%.⁹ The decision cannot be appealed. Rejecting an article is, therefore, a great responsibility, but if the reviewer believes, in all conscience, that the article is unacceptable, they should not hesitate to make this recommendation.¹² When recommending the rejection of a manuscript, it may be tempting to write a very short text simply informing of our negative decision or only including a few general unhelpful comments. However, we have to remember that the fundamental purpose of the review process is to improve the quality of the articles, and we should therefore make an effort to prepare a report which is sufficiently detailed to be helpful to the authors (at least when they submit it for publication in another journal).²⁷

3) Accepted with changes. Means that it is accepted (or rather, that it could be accepted), but it requires a series of changes, which may be minor or involve redoing a good part of the article. Final acceptance will depend on the response of the authors, who may opt to accept all the proposed changes and incorporate them without further ado or accept only some and argue against the others if they do not consider them appropriate. The revision suggested by the reviewer may be minor or major; minor essentially involves format aspects, but does not require major changes in the content, and usually does not need to be checked again by the reviewer; while major does require changes in the content of the article, and usually does need to be reviewed again.¹² Therefore, if major revisions were requested, the editor will generally return the manuscript to the original reviewers unless they decline to review it again.

If the two reviewers (the minimum number required) agree on definitively rejecting the article, the process stops here, as the rejection cannot be appealed. However, if there is an impasse, with one of the reviewers accepting the publication (even with changes required) and the other rejecting it, the editors tend to go with the reviewer who made a more detailed and better-justified evaluation.²⁷ Other times, the editors bring in a third reviewer to break the deadlock.

It is important that the reviewer's comments are in line with their recommendation to accept/reject the article. If you are recommending (to the editors) that the manuscript be rejected, your comments (to the authors) should clearly state what the issues are and should not be overly positive.^12,21

Ethical aspects of the evaluation of scientific manuscripts

The Committee on Publication Ethics (COPE; http://publicationethics.org/) is an international forum for editors of peer-reviewed journals where all aspects of publishing ethics are discussed.⁶⁸ This group has developed best practice flowcharts for editors on handling instances of research and publication misconduct, including plagiarism and fraud, and guiding how editors can carry out peer review responsibly. Table 3 shows a summary of the most relevant recommendations from the COPE guidelines for reviewers.¹²

Table 3.

Ethical recommendations for manuscript reviewers.

Read the article in its entirety, including supplementary/additional material

Follow the journal instructions (if available) for the presentation of your information in terms of format and other characteristics

If, for any reason, you have refrained from evaluating parts of the manuscript for speciality or competence reasons, you must inform the editors when you submit your review.

Turn down the review if you are not sufficiently familiar with the subject or a substantial part of it

Turn down the review if there is a conflict of interest

Do not use the information you have accessed as part of the review before the article is published

Do not take advantage of your position to make recommendations for your own benefit, such as recommending including your own references when not fully justified

Do not contact the authors directly

Plagiarism in a scientific journal is bad practice and one of the most serious attacks that can be carried out against scientific integrity. Although the responsibility for recognising plagiarism generally rests with the editorial board of the journal, it can also be recognised by the reviewers.

One aspect of particular importance is potential conflicts of interest. Although there is no unified definition of “conflict of interest” within the scientific community, it is commonly described as anything that interferes (or may interfere) with full and objective presentation, peer review, editorial decision-making, or the publication of research. More concisely, it would be when there are interests that might affect the objectivity of an evaluation,²¹ in other words, when a “primary” professional interest is excessively influenced by a person's “secondary” interest.⁶⁹ To maintain the integrity of the peer review process, all persons involved in the research, writing and evaluation of a manuscript, including authors, editors and reviewers, must declare potential conflicts of interest. Although theoretically, any relationship, personal, professional or financial, can be classified as a conflict of interest, explicitly declaring a potential conflict of interest does not necessarily mean that the evaluation of the manuscript is biased.

In cases where the reviewer has a close relationship with the authors, such as an ongoing collaboration or active grants, or simply feels that they cannot fairly assess a particular manuscript, they should decline the editor's invitation. For example, we should not review an article written by a colleague from our own service or department. If we are employed at the same institution as any of the authors or have recently been mentors, close collaborators or mutual grant recipients, we should also not agree to review the manuscript, even if we have nothing to do with the current research project.

Last of all, if we identify aspects that seem in conflict with ethics in the article we are evaluating, we should contact the editor of the journal for instructions on how to proceed, not try to investigate on our own.

What should we not do when evaluating a scientific article? The 10 most common mistakes reviewers make

The 10 most common mistakes that every reviewer should avoid are summarised in Table 4; each is reviewed below.

Table 4.

Ten common mistakes that every reviewer should avoid when reviewing a scientific article.

Accepting an invitation to review an article when you shouldn't

Not respecting the journal review deadlines

Not respecting the confidentiality of the review process

Involving someone else in the review of the manuscript without first obtaining permission from the journal

Including non-specific, ambiguous or unclear review comments

Unfairly obstructing the publication of the reviewed article

Demanding additional experiments that are outside the scope of the reviewed study

Requiring authors to add reviewer citations that are not fully justified

Focusing on reviewing the composition of the manuscript

Including the recommendation of acceptance or rejection in the reviewer's comments to the author

Accepting an invitation to review an article when you shouldn't

The first thing before agreeing to review an article is to consider whether or not we really have the capacity to assess a paper on that topic, at least as suggested by the article's title and abstract (which is the information we initially have available).¹² It is not unusual to receive an invitation to review a topic for which we are not really qualified. Although declining an invitation under these circumstances seems like common sense, many of us have suffered ourselves as authors by the hand of reviewers who were glaringly ignorant of the subject before them. Although we do have to have competence in the main subject matter, we do not have to be an expert in all the sections of the manuscript, but we should point out any limitations we have for reviewing specific parts of the article to the editors (for example, a very specialised statistical analysis).²¹ If, after accepting the review based on the title and abstract, we receive the full text and realise we are not competent enough, we should have no qualms about declining the invitation.²² Refusing an invitation to review an article when we feel that we are not the right person, especially if we reply quickly, will be viewed favourably by the editor of the journal. In addition, explaining the reason for our rejection (lack of competence in the subject matter to be reviewed) to the editors will avoid us being sent similar articles from the same journal in the future.¹²

Not respecting the journal review deadlines

One of the most frustrating aspects, both for editors and authors, is the long delay there can be between the authors submitting the manuscript to the journal and the delivery of the evaluation report. Such delays can, moreover, have a very negative impact on the chances of the research being published, especially if the topic is very innovative, as it could become obsolete in a few months.²¹ One of the main causes of this delay is the time it takes the reviewers to respond to the editor's invitation to review the article. You therefore need to respond promptly when you receive such an invitation from an editor. If you feel you cannot or should not review that particular article, that is alright, but you should respond accordingly, rather than simply ignoring the invitation.³⁰ You should not agree to carry out a review if you are not sure of meeting the journal's deadlines⁷⁰; accepting a review also means committing to deliver on a set date. Obviously, unexpected circumstances can occur which justify a certain delay; in that case, we must notify the editor of the expected delay and request a deadline extension. Whatever the situation, the worst option is to accept the invitation, and simply not send the review report, as this will considerably delay the entire review process, forcing the editor to invite new reviewers.²² As is logical, most journals monitor the performance of their reviewers and in the end exclude those who repeatedly do not meet the established deadlines.⁵⁵

Not respecting the confidentiality of the review process

Reviewers must not use information obtained during the peer review process for their own advantage or that of others. It is essential to remember that all material to which the reviewer has access must remain confidential until the work is published. While it may be tempting to brag about a scoop concerning new information, we should not share it with others or use it for our personal gain. Furthermore, all copies of the manuscript, both printed and electronic, should be destroyed when the review is complete.

Involving someone else in the review of the manuscript without first obtaining permission from the journal

The names of all persons who have contributed to the review have to be acknowledged and included at the outset so that they are associated with the manuscript in the journal's records and can also receive due recognition for their efforts.⁵⁰

Including non-specific, ambiguous or unclear review comments

Reviewers' comments should be as clear, explicit, detailed and unambiguous as possible, rather than vague generalisations which are of no help to authors; if the author does not know how to interpret the reviewer's comments, they will not be able to respond to them either. We should not only indicate what needs to be improved, but also suggest how to do it.²²

Unfairly obstructing the publication of the reviewed article

It is unlikely that the authors of the article in question are the only ones doing research on that specific topic. Frequently, as is natural, the area of interest of the reviewers coincides with that of the authors, and it can happen, for example, that the former are preparing or have just submitted for publication an article very similar to that of the latter. The most serious breach of reviewer ethics would be unfairly obstructing the publication of the author's work or using the author's information for competitive advantage. If, in such a situation, we consider that we cannot carry out a fair, prompt and constructive review, we should declare a conflict of interest and notify the editor as soon as possible.^22,55

Demanding additional experiments that are outside the scope of the reviewed study

There is a fine line between being constructive and suggesting further experiments that extend the work being evaluated beyond its current scope, exceeding the scope of the study. We must not forget that our role as reviewers is to judge the work we receive, not to ask the authors to carry out experiments that go beyond or extend the objectives of the study.^55,71 Obviously, the reviewer is entitled to argue rigorously that further research is imperative for the paper to make any sense; but then, it may be that the study has a serious problem and perhaps it is more appropriate to reject it, always with the proper justification.¹² Either way, the reviewer should make it clear what additional research they consider essential to support the claims made in the manuscript and what would simply strengthen the work but is optional.

Requiring authors to add reviewer citations that are not fully justified

As a reviewer, we should not ask authors to add citations of our own authorship, especially when unjustified, to inflate our citation record or enhance the visibility of our own work.^12,72 Suggestions for expanding references must be based on purely scientific grounds.⁷³

Focusing on reviewing the composition of the manuscript

As a reviewer, we need to focus on the content of the research rather than the writing.¹³ The essential is the background, not so much the form. We should not waste time polishing grammar or spelling and pointing out individual typos; it is enough to specify that you recommend reviewing the composition of the manuscript.⁵⁵ The editors will make sure that the text is correctly written before publication. Therefore, it is enough for us to indicate, for example, that “the paragraphs of the discussion are excessively long, and it would be a good idea to split them up to make them easier to read”. However, if we detect grammatical errors that affect the clarity of the manuscript, it is important to highlight them.⁷⁴

Including the recommendation of acceptance or rejection in the reviewer's comments to the author

We should never write in the comments to the authors whether we recommend acceptance or rejection of the article; this is editor privilege.⁵⁵ They will make the final decision after weighing various factors, including our particular recommendation. The editors are more interested in our comments than in our opinions about whether the article should be published or not.

Predatory journals and peer review

A few years ago, the movement for open access to science revolutionised the process of scientific communication, demanding free access to published articles, without economic or copyright restrictions, proposing publication in open access journals and the deposit of articles in repositories.⁷⁵ In open-access journals, unlike in the traditional publishing system, the authors retain their patrimonial rights, assigning them to readers with Creative Commons licenses, which allow free access and reuse of their contents.⁷⁵ These journals lack the financing deriving from their commercialisation, and their survival depends on new economic models, such as the payment of fees for the processing of articles, moving from the usual model of multiple charging for reading to a single payment for publication.⁷⁵ In return, this model has opened a path for corruption, used by unscrupulous companies which create journals with the sole purpose of enriching themselves with the income from these fees, without providing any of the services traditionally provided by the normal biomedical publications.⁷⁵ Thus, the so-called “predatory” journals do not carry out a peer review process to guarantee the scientific quality of the published manuscripts or only do so superficially. One of the fundamental differences between “predatory” journals and quality open access journals with reliable information is the mention in the quality journal articles of the associated editor and the reviewers involved. To sum up, the proliferation of predatory journals, which, with their spurious interpretation of open access, operate with dubious or non-existent ethical standards, threatens the integrity of the scientific system, mainly because they compromise the quality of what is published due to their poor or sham peer review processes.^75–77 One of the consequences of missing or incomplete peer review is the publication of flawed research which confuses clinicians and potentially harms patients.⁷⁸

Limitations of peer review

All is not well in the peer review process.^79,80 This evaluation process is a human activity; reviewers, like editors, may be biased, zealous, ignorant, incompetent, malicious, corrupt or disabled by conflicts of interest.²⁰ The most common and important limitations of peer review are summarised below.

There is little evidence on the effectiveness of peer review

Scientific evidence supporting the effectiveness of the peer review process is very limited.⁷ One of the reasons for the small number of prospective, randomised studies could be the difficulty in evaluating complex human behaviour such as peer review.⁸¹ Another added difficulty is that peer review seems to fulfil a series of highly diverse functions, and there is no consensus on its main objective. Drugs are generally tested on strictly defined indications, with well-established endpoints. However, since the true role of peer review has not been clearly defined, it is difficult to measure the effectiveness of different interventions.⁸²

The Cochrane Collaboration reviewed the evidence for the effectiveness of this review strategy and concluded that, “Little empirical evidence is available to support the use of peer review as a mechanism to ensure the quality of biomedical research”⁸³; although, on the other hand, they point out that absence of evidence does not mean evidence of absence. In any case, the authors of this Cochrane review did find evidence that peer review, at the very least, makes articles more readable and improves the quality of review reports.

Peer review has limited ability to detect errors

Peer review is a system with a low capacity to detect errors in the articles evaluated. A study was carried out years ago in which eight errors were deliberately introduced into an article which was then reviewed by more than 200 reviewers; the average number of errors detected was two, no one detected more than five and approximately one sixth did not detect any.⁸⁴ In another study, investigators produced a fictitious manuscript which intentionally contained 10 major errors and 13 minor errors.⁸⁵ The article described a placebo-controlled clinical trial using propranolol for the acute treatment of migraine. Major errors included non-specification of the exclusion criteria, the lack of randomisation of the treatment groups and the lack of blinding. Additionally, although a figure included in the manuscript clearly showed that propranolol had no beneficial effect, the authors of the fictitious article concluded that this drug was effective for the treatment of migraine. Two hundred reviewers reviewed the manuscript and identified an average of only one-third of the 10 major errors and one-quarter of the 13 minor errors. Indeed no less than 70% of the reviewers missed that the article's conclusions were not supported by their results. Although most reviewers (58%) recommended that the article be rejected, a substantial proportion (one-third) advised accepting it with modifications, and a few (7%) even recommended acceptance without changes.

So unfortunately peer review does not always deliver a better product. Studies show that even after this review process, some articles still contain inaccuracies and that most rejected articles will be published elsewhere (without substantial changes), still with much room for improvement because authors tend to ignore reviewer comments if their manuscript has not been accepted.^86–88 It has been argued that almost any scientific manuscript, no matter how deficient, manages to find a journal in which it can be published if the author is persistent enough. As some authors have pointed out, the fact that an article is published does not mean that it is perfect.⁸⁹ Others have gone so far as to condemn peer review as an ineffective, slow, expensive, biased, anti-innovative and easily abused lottery.⁹⁰ However, residual errors after peer review are not reasons to abandon the strategy altogether; the errors would probably be worse without it.

Peer review can inhibit innovation

Peer review can be overly conservative for breakthrough articles. Significant agreement between reviewers is needed for an article to be accepted, and this can be difficult when the work being reviewed involves a paradigm shift.⁹¹ It has been suggested that peer review stifles innovation and perpetuates the status quo.⁹² Possible blocking of a work that is innovative or contrary to the opinions or perspective of the reviewer may cause an unnecessary delay in its publication.⁹²

A recent study published in Proc Natl Acad Sci USA (PNAS) tracked more than 1000 manuscripts submitted to three prestigious medical journals; of the 808 which were published at some point, the 2% most frequently cited had been rejected by those journals.⁹³

Nobel prize-winning ideas were not always easily accepted by the scientific community in their early days. Almost by definition, they are revolutionary ideas, which can often generate a certain amount of rejection among reviewers, especially among the more “traditional” ones. As a result, many of the advances currently found in our textbooks were initially rejected, if not ridiculed, by the scientific community. In fact, several articles by Nobel Prize-winning researchers were initially rejected by anonymous peer review.

An interesting example of disagreement between the opinion of a Nobel laureate and their reviewer was that of Albert Einstein and his angry reaction to his first peer review experience when this system was still in its infancy. Einstein was surprised to receive an evaluation report on his work, and even more so that it was anonymous, and withdrew the article from the journal with a statement addressed to the editor, which is quite an ode to the ignorance about the peer review process: “We (Mr Rosen and I) had sent you our manuscript for publication and had not authorised you to show it to specialists before it is printed. I see no reason to address the, in any case, erroneous comments of your anonymous expert. Based on this incident, I prefer to publish the paper elsewhere”.⁹⁴

Another paradigmatic example of total divergence between a Nobel Prize winner and their reviewer is the case of Robin Warren and Barry Marshall. In 1983, these researchers submitted an abstract to the Australian Gastroenterology Association describing the isolation, for the first time, of the bacterium Helicobacter pylori. After being reviewed, the paper, submitted to a merely national congress, was rejected. In the response letter, the scientific committee justified the rejection due to the numerical limitation of accepting 56 of the 67 papers received, citing the one by Warren and Marshal as lacking scientific interest. Years later, in 2005, the authors received the Nobel Prize in Medicine for precisely this discovery. It is clear that resistance to change and the difficulty in renouncing the official truth that “bacteria do not grow in an acid medium” prevented the initial acceptance of this breakthrough discovery by the scientific community.⁹⁵

Peer review can be biased

A number of different factors (other than the quality of the manuscript) can influence editorial decisions.^86,96 The peer review process is not free from potential bias, which may be intentional or subconscious.⁹⁷ It includes prestige bias, gender bias, confirmation bias, publication bias and language bias.⁹⁸ For example, as previously mentioned in the section dedicated to blind peer review design, reviewers are more likely to recommend acceptance of an article written by well-known authors belonging to highly reputable institutions. This may seem intuitive and at least partly appropriate, as it could be assumed that more experienced authors would produce higher quality articles. However, favourable reviews seem to also reflect a positive bias towards prestigious people and institutions. Okike et al.⁹⁹ submitted a fictitious manuscript purportedly written by two former presidents of the American Academy of Orthopaedic Surgeons, both from reputable institutions, to a group of reviewers. A number of inaccuracies were deliberately included in the manuscript, including numerical errors and erroneous conclusions. Half the reviewers knew who the author of the article was, and the other half did not. Reviewers who knew the authorship details were more likely to recommended acceptance of the article than blind reviewers. Similarly, non-blinded reviewers gave higher marks for methods, results, discussion, organisation and clarity. These findings strongly suggest that author and institutional prestige are factors influencing reviewers' perception of an article's value and quality.

The different reviewers often have opposing opinions

Various studies have shown that it is quite common for reviewers of the same manuscript not to agree with each other or even for their opinions to be completely opposing.¹⁰⁰ It is not unusual for an article to be praised by one reviewer and criticised by another. In fact, as mentioned above, a recent study found that the likelihood of two reviewers agreeing is only slightly greater than that explained by chance, and that to obtain a statistically reliable result, editors would need reports from six or more reviewers, rather than just two.¹⁰ In another classic study, two researchers selected a number of articles already accepted by highly regarded journals, removed their real names and academic affiliations and added false ones, and resubmitted the (identical) material to the same journals which had accepted them in the previous years; of the nine articles that continued with the review process, eight were rejected.¹⁰¹ It has been suggested that the decisions of peer reviewers are like diagnostic tests: false positives and false negatives are unavoidable.²⁴

It is increasingly difficult to find good reviewers

As the pressure to publish (publish or perish) continues to increase, so does the need for high-quality peer reviewers. However, availability is limited. In fact, it has been suggested that the number of reviewers, or their availability, is declining.^102–104 Possible reasons for this could be reviewer fatigue, the rapid increase in the number of journals available for manuscript submission, the demanding commitment to speed, or the lack of recognition. Ranking high among these factors is the recent explosion in the industry of scientific journals and the consequent exponential growth in the submission of articles. The number of spam emails received on a weekly basis requesting the submission of manuscripts to countless journals is certainly suffocating. Another factor to consider is that editors tend to abuse a reviewer's efficacy, which can lead to burnout syndrome¹⁶; a reviewer who has provided insightful feedback and criticism, particularly if also on time, is often “rewarded” with another article for review not long afterwards.¹⁰³ It is not therefore surprising that there is a marked imbalance in the distribution of the peer review effort in the scientific community; 20% of researchers carry out between 70% and 95% of the reviews.¹⁰⁵ Nevertheless, only reviewers knowledgeable about the subject are qualified to evaluate the article of interest, but, being competitors, they are often disqualified due to the potential conflict of interest. Finally, the lack of recognition (personal and financial) is another additional limitation to the number of reviewers available.

Peer review has a substantial cost

Peer review is not without its costs, both in financial and human terms. Financial costs borne by journals include creating and maintaining a reviewer database, staff time to identify reviewers and follow up on manuscripts, correspondence with reviewers and authors, and editors' time for dealing with external reviews. It has been estimated that peer review represents around 5% of the total costs of a scientific journal.⁷ Lastly, there is another less tangible cost. Peer review delays the publication of research results which could improve clinical practice and public health.⁷ There is a marked variation in the time it takes for journals to carry out peer review, from a few weeks to several months.

Suggestions for improvement of peer reviewing

As an initial reflection, it is not easy to make proposals for improvement of peer review without first agreeing on the objectives of the process,⁹ whether it is to select the best articles to publish in a journal, minimise fraud, improve the quality of published articles, improve the quality of research, act as a filter, rejecting poor work or decide where rather than if an article is to be published? There is, therefore, an urgent need to clarify the objective of the peer review process and the definition of what constitutes a good quality evaluation report and the results that should be taken into account to evaluate the process. We need more studies to help identify interventions which could make improvements.^49,83 All interested parties, editors, authors, reviewers and readers need to be aware of the limitations of peer review and strive to improve the process. Some suggestions for improvement are provided below.

Journals should include new, younger reviewers

The best reviewers are traditionally considered to be seasoned, accomplished professionals because they have accumulated the wisdom and experience to give good advice. However, studies of reviewer performance suggest that this is not necessarily the case. Various studies have shown that the best reviews (measured both by their quality and by their critical capacity and speed) were carried out by the youngest evaluators; in fact, one of the reviewer characteristics that was independently associated with the quality of the assessment was being under the age of 40.^106–108 Accordingly, in another study conducted over 14 years, researchers found that reviewers' quality ratings deteriorated slowly over time,¹⁰⁹ perhaps due to their progressively more limited time availability. These findings suggest that journals should not automatically recruit the most experienced researchers for the review process, as there is little evidence that they produce higher quality reviews. It is, therefore, important that journals continually recruit new, well-trained young reviewers and monitor their effectiveness over time.

Educate and train in the peer review process

Although reviewers are experts in the subject area they evaluate, they have not usually received specific training to carry out the peer review process. This explains why, when the quality of manuscript reviews is assessed, it is often found to be deficient.⁷⁸ A reasonable strategy would therefore be for training sessions to be provided for reviewers (by the journals or the publishers for example), including a variety of formats: text; face-to-face; webinars; podcasts; and videos. Some organisations, such as Committee on Publication Ethics and Sense about Science, have developed training materials for reviewers (available at: http://publicationethics.org and http://senseaboutscience.org/activities/peer-review-the-nuts-and-bolts/). Better training should be incorporated into doctoral and postdoctoral programmes, and mentoring of novice reviewers should be encouraged.^110,111 Medical societies should play a more active role here, organising educational programmes. However, training alone seems to have a modest and fleeting impact¹¹²; isolated educational events have little or no effect in terms of behaviour change.^113,114 To be most effective, educational programmes need to be multimodal (interdisciplinary) and longitudinal (over time).

Use structured review templates

The goal is for reviewers to write comments and rate the key elements of the manuscript. One strategy, developed by some journals, is to make the process more systematic and consistent through the use of review forms or templates, checklists and rating scales.¹⁹ Journals should at least encourage reviewers to write comments and rate the critical aspects that most influence the editorial decision, such as the relevance of the research question, the originality of the article, the strengths and weaknesses of the methodology, the cohesion between the different sections of the manuscript and how well it is organised, the quality of the writing and the utility of the tables and figures.^86,115 As an example, some materials that include these checklists for different types of studies are available at http://www.equator-network.org/toolkits/peer-reviewing-research/.

Give reviewers feedback

Providing feedback to reviewers can improve the quality of peer reviews. A relatively simple strategy, which can also increase reviewers' satisfaction, is to inform them of the editorial decision at the end of the manuscript review process and share the criticisms submitted by all the reviewers. Another alternative is (for editors) to rate reviewers on the quality of their evaluation (including content, structure and timeliness) and share this feedback with them periodically to promote learning; it has been suggested that creating a kind of scorecard for reviewers would enable them to compare their performance against other reviewers.¹¹⁶ Obviously, this score could also be helpful for editors when re-selecting or deciding not to use a reviewer in the future. However, we must not forget that peer review is voluntary and altruistic, so editors need to be careful with potential negative comments, to prevent reviewers from feeling undervalued or offended.

Remove the option for authors to suggest reviewers for their articles

A common practice of some scientific journals is to allow authors to suggest potential reviewers.¹¹⁷ Authors can take advantage of this option and suggest colleagues or collaborators who are likely to be more “receptive” to their manuscripts.⁶⁹ In fact, several studies have shown that reviewers suggested by the authors tend to be kinder in rating the manuscripts than reviewers proposed by the editors, making it more likely that the article will be accepted.^97,118,119 As an example, one study examined nearly 800 reviewer reports from the top 10 biomedical journals and found that author-suggested reviewers made significantly more favourable publication recommendations than editor-suggested reviewers, although there were no differences between the two assessments in quality.¹²⁰ This practice of recommending reviewers who are allies has allowed unscrupulous authors to suggest their own relatives, friends and colleagues, or to provide fake email addresses; in one particularly outrageous case of editorial manipulation, a Korean researcher wrote most of his own reviews, in the end arousing the suspicion of the editor because they were completed far too quickly (often within 24 h).¹²¹

Include incentives for reviewers

Examples of incentives for reviewers include free access or subscription to journal content, annual recognition in the reviewer list on the journal's website, letters of appreciation sent to reviewers by editors, holding awards for the reviewer of the year or the appointment of the best reviewers to form part of a journal's editorial committee in recognition of their excellence. Financial incentives, in the form of payment to reviewers for their time, is not generally accepted, as peer review is considered an eminently altruistic activity.

One possible strategy to turn peer review into a more rewarding activity is to promote recognition of involvement by creating certificates which can be included in academic or professional records for future assessments or promotions.^122–124 With this in mind, the Publons initiative (http://publons.com/) quite recently set up by the Web of Science group works with reviewers and publishers to give credit to the altruistic activity of peer review and ensure that the hard work of the reviewers does not go unnoticed. Publons' mission is to provide reviewers with an official record of their work, turning peer reviews into measurable activities. Reviewers can effortlessly create and maintain a personal profile which, along with a list of their publications, includes a verified “peer review history” listing the reviews they have done for the different journals.

Final thoughts

All researchers will encounter the peer review process sooner or later in their careers, either as authors submitting their articles to a journal for publication or as reviewers when a journal editor asks them to assess a manuscript⁴; in fact, the same researcher can, and often does, play the role of author one day and reviewer the next. Peer review is based on the trust and cooperation of the scientific community and is today considered the gold standard, the cornerstone, for guaranteeing proper publication of scientific articles. It is an essential part of the quality control process, ensuring that published research is valid, meaningful and original. A well-crafted critical peer review report is a treasure, a real luxury, for both the author and the editor. It serves as a road map for the transformation from an imperfect original article to another final version of better quality and greater value to readers. Although peer review is not infallible and is far from perfect, it is probably the least worst option, and in any event we have no better alternative. Sometimes peer review can be seen merely as a hurdle to be overcome. However, the reviewer's mission is not to prevent an article from being published, but rather to get the best out of those who deserve to be published. We, therefore, need to encourage authors to see reviewers as helpful advisers rather than hostile adversaries. Writing a constructive, objective review takes time and effort, but there are some simple tips that we have shared in this manuscript and which we hope will be of use. As reviewers, we need to strive to be methodical, constructive, objective, informed, considerate and clear when preparing our evaluation reports. In short, we must do everything possible to put ourselves in the author's shoes, always following the golden rule of reciprocity: review others as we would like to be reviewed ourselves. Contributing to the peer review process makes us better researchers and better authors.

Conflicts of interest

Javier P. Gisbert has provided scientific advice for and/or received support for research or training activities from MSD, Abbvie, Pfizer, Kern Pharma, Biogen, Mylan, Takeda, Janssen, Roche, Sandoz, Celgene/Bristol Myers, Gilead/Galapagos, Lilly, Ferring, Faes Farma, Shire Pharmaceuticals, Dr. Falk Pharma, Tillotts Pharma, Chiesi, Casen Fleet, Gebro Pharma, Otsuka Pharmaceutical, Norgine, Vifor Pharma, Mayoly, Allergan, Diasorin, Gebro Pharma and Richen.

María Chaparro has provided scientific advice for and/or received support for research or training activities from MSD, Abbvie, Hospira, Pfizer, Takeda, Janssen, Ferring, Shire Pharmaceuticals, Dr. Falk Pharma and Tillotts Pharma.