Regardless of how many times scholars have peer reviewed manuscripts in their careers, you can be sure that most appreciate getting reviewer guidelines from the journals they work with. Even for experienced peer reviewers, remembering all of the points to address in review comments can be challenging, since scholars have countless other obligations on their minds.
While offering reviewers any form of guidance is better than none, being thorough and creating a reviewer checklist is by far the best way to help reviewers know the expectations of your journal. Reviewer checklists, which enumerate the areas of a manuscript review comments should focus on and the type of feedback journal editors are looking for, can either be used as required responses from reviewers or simply a guide for them to follow. If you do plan to create a reviewer checklist and make all or parts of it required responses, avoid making it too long or cumbersome for the reviewer. One idea to help reduce response time but still get the feedback you want from reviewers is to include a mix of likert scale questions as well as open-ended questions.
When compiling a reviewer checklist, make sure it addresses both the needs of your editorial team and those of submitting authors. Editors want reviews to clearly indicate a recommendation as to whether a manuscript should be accepted or rejected and why, or how it should be revised to warrant reconsideration. For editors reviewer checklists must be specific enough to keep reviewers on track and avoid meandering towards concerns that are immaterial to the editorial team in the manuscript vetting process (for example: the layout of tables or images, which will be addressed after the paper is accepted). For authors, peer reviews should offer detailed explanations of problem areas within a submission and suggestions for improvement. Authors should be able to learn from their reviews rather than being bombarded with blanket statements of criticism.
Here are common sections journals include in their reviewer checklists:
Brief 1-2 sentence summary of the paper
One of the fastest ways to assess the quality of a manuscript is to ask each of your reviewers to write a 1 to 2 sentence synopsis of what it’s all about. Once you’ve received all of the review comments for a manuscript read the reviewer synopses side-by-side to see how well they align. If the reviewers’ descriptions don’t match up that can be a sure sign that the paper lacks coherent form and focus.
While academic research is often multifaceted and papers can deal with complex topics, ultimately a quality paper should have a central purpose that is clearly reflected to all readers. The Journal of Micro/Nanolithography, MEMS, and MOEMS lists a manuscript synopsis among the most critical information in its reviewer checklist guide because it is most telling of a submission’s overall quality.
Overview of the abstract and introduction
Many reviewer checklists tend to follow the order in which scholars read manuscripts, so that reviewers can pause as they read each section of a paper to recall how they’re supposed to be assessing it. The first place to start is, of course, the abstract. In the abstract portion of your reviewer checklist you’ll want to direct reviewers to consider:
- Is the abstract brief, precise, and to the point?
- Does the abstract clearly reflect the findings of the paper, without inflating or diminishing any of them?
- Does the abstract convey the impetus for the research and why it is significant?
- Does the abstract overview the approach used to reach the goals of the research?
- Does the abstract state the main results and conclusions of the study?
Following the abstract, you’ll want your reviewers to also comment on the quality of the manuscript’s introduction. Ask reviewers to consider whether the introduction gives a detailed explanation of the significance of the study, states the experimental goal, and references any relevant findings from previous studies. The last point is especially important. Scholars can leave out research that their paper relates to either intentionally or accidentally. Be sure to request note of any prior research relevant to the article that isn’t addressed.
Assessment of research process and presentation
It’s imperative that your reviewer checklist have a section devoted to examining the quality of how the research was conducted and how data are presented. Your checklist should ask reviewers to describe the research methods used, whether they feel they were given enough information to reproduce the research, and whether the methods used make sense to them.
Additionally, the methods portion of your checklist should cover the following:
- Were there any flaws in the research methods?
- Were there any flaws in the way the data was presented and analyzed?
- Is the research data used statistically significant and representative for the purposes of the study?
- Does the author address any limitations in the study? Is there a need to do so?
- Does the conclusion of the paper have statistical and/or substantial significance?
- Do the findings clearly relate back to the methods used in the study?
Depending on your journal’s field, you may need fewer or more in-depth questions related to research methods and presentation. If your articles tend to include figures and tables, it’s a good idea to include some checklist bullets or questions regarding their presentation, also.
Structural analysis of the paper
Part of your reviewer checklist should address the organization of the manuscript. Here reviewers can voice concerns with the logical structure of the argument or the overall structure of the manuscript.
You’ll likely want to limit the scope of this portion of the reviewer checklist in order to avoid allowing scholars to get too carried away with structural analysis, as this is an area your editorial team will be focusing on. Also, depending on your discipline and submission topics, you may find that some reviewers can get caught up in making subjective comments about how they would have structured a paper suggesting unnecessary rewrites based on personal preferences, which you want to avoid as it can upset authors and drag out the review process.
Assessment of research significance
In addition to asking reviewers to evaluate the clarity, methods, and structure of a paper, you’ll want to ask reviewers to assess the paper’s significance to the field. Ask reviewers to comment on:
- If the significance of the research is clearly stated in the paper
- Whether or not they believe the research is novel
- How they believe the research will contribute to its field
Reviewers’ assessment of a paper’s significance will likely play a large role in your decision to accept or reject it, so be sure to ask detailed questions that require reviewers to display their thought process rather than making a blanket statement that a paper is or isn’t significant with no backing.
Examples of reviewer checklists from other journals
Depending on the nature and discipline of your journal, you’ll have varying reviewer checklist needs. Be sure to take the time to compile your own list that addresses what matters most to your editorial board. That said, having examples to work from always helps, so here are a couple:
- Academic Medicine, Checklist of Review Criteria
- Preventing Chronic Disease, Peer Review Checklists
Help your reviewers help you
As you compile your reviewer checklist, you may want to take the time to also incorporate some basic best practices into your reviewer instructions. It never hurts to remind reviewers to be kind in their commenting, especially if you plan to share reviews with authors. Also, remind your reviewers of the importance of specificity. In all aspects of their review, from commenting on the quality of a paper’s abstract to the research methods used, encourage reviewers to not only state what they think but why they think it. Reviewer insights can be invaluable tools to authors when they clearly address why flagged areas of a manuscript are problematic and suggestions for improvement. Peer review is meant to be a learning process, make sure your reviewers are keeping that in mind!
