3 Pillars of quality peer review at academic journals

Image Credit: Peter Aschoff on Unsplash

Since it became an established part of the academic publishing process, peer review has been employed to ensure research accuracy, originality, and significance. However, like any human process, peer review is imperfect. There have been mounting concerns expressed in recent years about quality in peer review, including biases against null or negative results, the potential for research spin, and the ongoing replication crisis.

As the “gatekeepers” of research, journals and publishers are arguably on the frontlines of quality in peer review and have the potential to lead the way in addressing many of the challenges faced by the community. In this post, we look at three pillars of peer review at academic journals that can help to build a stronger foundation for quality research:

1. Clear peer review policies and standards
2. Peer review performance tracking
3. Transparent publishing and data sharing policies

Journals and publishers should work to fortify these pillars by assessing their performance in these three areas and looking for ways to improve based on past performance and new developments in the publishing landscape. Below are specific steps that journals and publishers can take to improve quality in peer review.

1. Clear peer review policies and standards

The first pillar of quality peer review is establishing clear peer review policies and standards. Journal policies and standards ensure that all parties involved in peer review — editors, authors, and reviewers — know what is expected of them and that all manuscripts are handled in the same way. To ensure quality in peer review, consistency is key! Journals should have:

• Established peer review policies: Peer review policies are a statement on the peer review guidelines and processes the journal follows for all manuscript submissions. Peer review policies should include an overview of the journal’s peer-review process (e.g. blindness, reviewers per manuscript, rounds of review allotted) and the anticipated peer review timeframe, as well as statements on publication ethics. Some guidelines, such as the Plan S implementation criteria, also require journals to make available annual statistics on their peer review performance to demonstrate adherence to their policies.
• Standardized submission guidelines: Authors should know what is expected of them when preparing their manuscript for a journal, from layout requirements to the citation and data reporting standards they’re expected to follow. All of this information should be included in the author guidelines section of the journal website. Journals should also require authors to provide statements of originality and disclosures with their submission. We provide examples of journal statements of originality and disclosures in this blog post.
• Standardized reviewer feedback: Journals should establish clear guidelines for peer reviewers to follow, including reminders of the duties of reviewers — to be objective, diligent, and confidential in their reporting — as well as a standardized reviewer feedback form. You can only expect reviewers to answer the questions that you ask them, required feedback forms ensure greater consistency and quality in referee reports.

And, of course, all journal policies and standards must be actionable. For example, all of the process steps explained in a journal’s peer review policies must be carried out. So if it states all original research manuscripts will have two external reviewers, this should always be the case. Additionally, journals must have plans in place for enforcing all ethical policies and standards. If an article is found to have a conflict of interest or if there is an allegation of misconduct post-publication, the journal must have processes in place to address the situation. Additionally, if the journal requires authors to follow certain reporting guidelines, it should have a process in place to check for adherence.

It is the duty of journal publishers to follow and enforce publishing best practices, such as the Committee on Publication Ethics’ Core practices and the Plan S implementation guidelines, as well as disciplinary research reporting guidelines like those compiled by equator, such as the CONSORT checklist for randomised trials.

2. Peer review performance tracking

The next pillar of quality peer review is performance tracking. Peer review quality depends on editors following journal policies and processes and reviewers completing reports in a timely and thorough manner. Journals should track peer review data in the following areas:

• Editorial team performance metrics: Journals should track their editorial performance including the average manuscript acceptance rate at the editor- and journal-level, average days to decision at the editor- and journal-level, and average manuscripts per editor. These metrics can help journals ensure they are accepting manuscripts at a reasonable rate and that editors are working within the peer review timeline outlined in the journal’s policies without being overburdened. If you find that some editors are managing more manuscripts than others or that some editors are struggling to make manuscript decisions within the established peer-review timeframe you know you have some areas to work on.
• Reviewer performance metrics: Journals should track reviewer performance metrics, including their average time to complete a review and number of reviews completed. Journals can use these metrics to know which reviewers are most reliable and spot if a reviewer may need a break from review requests.
• Manuscript stages: Journals should also track data around each manuscript’s stage in review, including reviewers per manuscript and rounds of revisions to ensure each submission has enough reviewers and is moving through peer review efficiently and effectively (when manuscripts get caught in manuscript R&R cycles it can often have diminishing quality outcomes).

In addition to tracking peer review data, journals should also have processes for assessing the quality of the peer reviews they receive. One proposal for assessing peer-review quality is the review quality instrument (RQI), which “assesses the extent to which a reviewer has commented on five aspects of a manuscript (importance of the research question, originality of the paper, strengths and weaknesses of the method, presentation, interpretation of results).” Ultimately, harkening back to the importance of standardized reviewer feedback forms, journals should set expectations for referee reports that they can communicate to reviewers in the same way in order to get more standardized feedback and, ultimately, to also be able to review the quality of referee feedback (e.g. did each reviewer provide a comprehensive analysis of the strengths and weaknesses of the research method as asked).

3. Transparent publishing and data sharing policies

Finally, the third pillar of quality peer review at academic journals — transparent publishing and data sharing policies — addresses key concerns around research biases and reproducibility. Journals can facilitate the reporting of null and negative results as well as research reproducibility by enabling and encouraging authors to share their manuscripts and data pre- and post-publication. This includes having preprint policies and data sharing policies or requirements.

One recent initiative to encourage data sharing and reproducibility is the Center for Open Science’s Open Science Badges, which are designed to incentivize researchers to share research data and materials and to preregister study and analysis plans. Journals can even opt to follow the “Registered Reports“ framework, in which peer review occurs before study results are known. The Registered Reports approach is designed to emphasize the quality of research methodology over the findings of the final outcomes to eliminate peer review biases, particularly around null and negative results.

Putting it all together: assessment and ongoing improvement

The peer-review process is, well, a process — meta, we know. As stated by Todd Carpenter in the recent Scholarly Kitchen article, “Ask The Chefs: Peer Review Quality,” when assessing the quality of peer review, “the question shouldn’t be whether the process is perfect. Rather, as with all scientific processes, does this process yield a better result than might otherwise be expected without it.” At the journal and publisher level, this means ensuring that peer review is improving research quality through clear assessment standards, consistent processes, and policies to promote research transparency. Journals should regularly audit their peer review processes and performance and look for opportunities to improve based on lessons learned and new developments in scholarly publishing. In all of this, it’s also vital for diverse voices to be part of the discussion of quality in peer review, to bring varied perspectives and solutions.