How to Write Constructive Peer Review Comments: Tips every journal should give referees

Image Credit: Loic Leray

Like the art of tightrope walking, writing helpful peer review comments requires honing the ability to traverse many fine lines.

Referees have to strike a balance between being too critical or too careful, too specific or too vague, too conclusive or too open-ended — and the list goes on. Regardless of the stage of a scholar’s career, learning how to write consistently constructive peer review comments takes time and practice.

Most scholars embark on peer review with little to no formal training. So a bit of guidance from journals before taking on assignments is often welcome and can make a big difference in review quality. In this blog post, we’re rounding up 7 tips journals can give referees to help them conduct solid peer reviews and deliver feedback effectively.

You can incorporate these tips into your journal reviewer guidelines and any training materials you prepare, or feel free to link reviewers straight to this blog post!

Take steps to avoid decision fatigue

Did you know that some sources suggest adults make upwards of 35,000 decisions per day? Hard to believe, right?!

Whether that stat is indeed the norm, there’s no question that we humans make MANY choices on the regular, from what to wear and what route to take to work to avoid construction to which emails to respond to first and how to go about that really tricky research project in the midst of tackling usual tasks, meetings — and, well, everything else. And that’s all likely before 10 AM!

In his blog post “How to Peer Review,” Dr. Matthew Might, Professor in the Department of Medicine and Director of the Hugh Kaul Precision Medicine Institute at the University of Alabama at Birmingham, explained that over time the compounding mental strain of so much deliberation can result in a phenomenon known as decision fatigue. Decision fatigue is a deterioration in decision-making quality, which for busy peer reviewers can lead to writing less than articulate comments at best and missing critical points at worst.

In order to avoid decision fatigue, Might said scholars should try to work on peer reviews early in the day before they become bogged down with other matters. Additionally, he advises referees to work on no more than one review at a time when possible, or within one sitting at least, and to avoid reviewing when they feel tired or hungry. Taking steps to prevent decision fatigue can help scholars produce higher quality comments and, ultimately, write reviews faster because they’ll be working on them at times when they’re likely to be more focused and productive.

Of course, referees won’t always be able to follow every one of the above recommendations all of the time, nor will journal editors know if they have. But, it’s worth it for editors to remind reviewers to take decision fatigue into account before accepting and starting assignments.

Be cognizant of conscious and unconscious biases

Another decision-making factor that can cloud peer reviewers’ judgment that all editors should be hyper-attuned to is conscious and unconscious biases. Journal ethical guidelines are, of course, the first line of defense for preventing explicit biases. Every journal should have conflict of interest policies on when and how to disclose potential competing interests (e.g., financial ties, academic commitments, personal relationships) that could influence reviewers’ (as well as editors’ and authors’) level of objectivity in the publication process. The Committee on Publication Ethics offers many helpful guides for developing conflict of interest / competing interest statements, and medical journals can find a “summary of key elements for peer-reviewed medical journal’s conflict of interest policies” from The World Association of Medical Editors (WAME) here.

But what about unconscious biases that could have potentially insidious impacts on peer reviews?

Journals can help curb implicit bias by following double-anonymized peer review processes. Though, as the editors of Clinical and Translational Gastroenterology acknowledged in an announcement about their decision to move to double-anonymized peer review, even when all parties’ identities are concealed “unintentional exposure of author or institution identity is sometimes unavoidable, such as in small, specialized fields or subsequent to early sharing of data at conferences.”

Truly tackling unconscious biases requires getting to their roots, starting with acknowledging that they exist. Journals should remind reviewers to be cognizant of the fact that everyone harbors implicit biases that could impact their decision-making, as IOPScience does here and Cambridge University Press does here and provide tips for spotting and addressing biases. IOP advises reviewers to “focus on facts rather than feelings, slow down your decision making, and consider and reconsider the reasons for your conclusions.” And CUP reminds referees that “rooting your review in evidence from the paper or proposal is crucial in avoiding bias.”

Journals can also offer unconscious bias prevention training or direct referees to available resources such as this recorded Peer Reviewer Unconscious Bias webinar from the American Heart Association.

Null or negative results aren’t a basis for rejection

Speaking of forms of bias that can affect the peer review process, “positive results bias” — or the tendency to want to accept and publish positive results rather than null or negative results — is a common one. In a Royal Society blog post on what makes a good peer review, Head of the Department of Population Health at the London School of Hygiene & Tropical Medicine, Dr. Rebecca Sear, spoke to how positive results bias can throw a wrench in peer review. Speaking from the perspective of an author, editor, and reviewer, Sear said, “at worst, this distorts science by keeping valuable research out of the literature entirely. It also creates inefficiencies in the system when publishable research has to be submitted to multiple journals before publication, burdening several reviewers and editors with the costs of evaluating the same research. A further problem is that the anonymity typically given to peer reviewers can result in unprofessional behavior being unleashed on authors.”

Journals can help prevent positive results bias by clearly stating that recommendations regarding manuscript decisions should be made on the basis of the quality of the research question, methodology, and perceived accuracy (rather than positivity) of the findings. Remind reviewers (and editors) that null and negative results can also provide valuable and even novel contributions to the literature.

List the negatives and the positives

When it comes time to write peer review comments, some scholars may intentionally or not lean heavily towards giving criticism rather than praise. Of course, peer reviews need to be rigorous, and that requires a critical eye, but it’s important for reviewers to let authors know what they’re doing right also. Otherwise, the author may lose sight of the working parts of their submission and could end up actually making it worse in revisions.

Journals should remind reviewers that their goal is to help authors identify what they are doing correctly as well as where to improve. Reviews shouldn’t be so negative that the author ends up pulling apart their entire manuscript. Additionally, it’s worth reminding reviewers to keep snarky comments to themselves. As Dr. Might noted in his blog, the presence of sarcasm in peer review may nullify any useful feedback provided in the eyes of the author.

Give concrete examples and advice (within scope!)

No author likes hearing that an area of their paper “needs work” without getting context as to why. It’s essential to remind reviewers to back up their comments and opinions with concrete examples and suggestions for improvement and ensure that any recommendations they’re making are within the scope of the journal requirements and research subject matter in question.

Remind reviewers that if they make suggestions for authors to provide additional references, data points, or experiments, they should be within scope and something the reviewer can confirm they would be able (and willing) to do themselves if in the author’s position.

One of the best ways to help train reviewers on how to give constructive feedback is to provide them with real-world examples. These “Peer Review Examples“ from F1000 are a great starting point.

Another way editors can help reviewers give more concrete commentary is by advising them to log their reactions and responses to a paper as they read it. This can help reviewers avoid making blanket criticisms about an entire work that are, in fact, only applicable to some sections. It may also encourage reviewers to recognize and point out more positives!

Providing reviewers with detailed feedback forms and manuscript assessment checklists is another surefire way to help them stay on track.

Don’t be afraid to seek support

Journals should also remind prospective reviewers that it’s OK to ask for support when working on peer reviews. For example, an early-career researcher might want to seek a mentor to co-author their first review with them or provide general guidance on how to determine whether an experiment was conducted in the best manner possible (keeping manuscript information confidential, of course).

To help new referees get their footing, journals can assist them in identifying mentorship opportunities where applicable and offer peer reviewer training or links to external resources. For example, Taylor & Francis has an “Excellence in Peer Review“ course, and Sense About Science has a “Peer Review Nuts and Bolts“ guide.

For journals dealing with specialized subject matter, it’s also critical to be prepared to bring in expert opinions when needed. Editors should let reviewers know not to hesitate to suggest bringing in an expert if they feel it’s necessary.

Follow the Golden Rule

Finally, perhaps the best piece of advice journals can give reviewers is to follow the Golden Rule. You know it, “do unto others as you would have them do unto you.”

In his “How to peer review” guide, Dr. Matthew Might provided a clear barometer for referees to determine if they’ve prepared a thorough and fair review. “Once you’ve completed your review, ask yourself if you would be satisfied with the quality had you received the same for your own work,” he said. “If the answer is no, revise.”