Progressing towards shared peer review terminology: Interview with Lois Jones

Photo by William Felker on Unsplash

The use of peer review to evaluate scholarly manuscripts is a pillar of journal publishing across research disciplines. But definitions of different peer review practices and participant roles can vary significantly from publisher to publisher, journal to journal, and even paper to paper — making communicating process specifics tricky.

With the introduction of new peer review models in recent years, it’s only getting more complex. For example, what does “open” peer review mean? While one publisher or journal might define it as revealing the identities of authors, reviewers, and editors to each other during review, another might narrow or expand the scope to publishing referee reports alongside articles. Still, others might define “open” peer review as a model that allows public commentary before and/or after publication.

To help address these language disparities and make peer review processes more transparent, STM launched a Peer Review Taxonomy working group in 2019 tasked with developing standard definitions for the different aspects of peer review. Now set to be formalized as an ANSI/NISO standard, the Taxonomy is progressing fast, with over 40 publishers involved in its pilot program. We caught up with Lois Jones, a member of the STM working group and Peer Review Manager at the American Psychological Association (APA), to learn more about the emerging Taxonomy. In the interview below, she discusses the project goals and latest updates. Thanks to Lois for taking the time for this interview!

Q&A with Lois Jones:

During your Peer Review Taxonomy update at STM’s Annual Meeting, you talked about the need to “open the black box of peer review.” How have peer review processes become opaque in recent years (or have they always been)? And what are the challenges there?

LJ: As soon as you said in recent years, my reaction was — I don’t think concerns about peer review processes being opaque is a new thing. I think it’s just something publishers have been considering more recently as they are working to connect with different types of authors. For example, in many fields, publishers are implementing new DEI (Diversity, Equity, and Inclusion) initiatives to ensure they’re taking different author and reviewer identities and experiences into account, and terminology is essential to those efforts.

One of the things we think about a lot at the APA is that there are many non-traditional authors submitting to journals, and we’re always talking about how to involve them more. In our case, we have practitioners, early-career psychologists, and others who haven’t necessarily been publishing a long time. When you’re coming from a non-traditional publishing background, the whole notion of peer review can be very unclear, let alone specific processes. In the traditional research model you might have an advisor or classes that teach you what it means to work with reviewers and journals. But, even then, there are still many ambiguities as far as what peer review entails at different publishers.

The Peer Review Taxonomy is an opportunity to demystify this process. Because it does seem that for many authors who submit a paper, the reviews and a decision letter come back, and it’s not always clear what happened in peer review to get to that point. As we began talking about this in the Taxonomy working group, it became clear that it can even be hard to distinguish process specifics between publishers. We’re often using different words to communicate the same things, so how are these poor authors figuring it out?

The Taxonomy applies to any peer review model, but I know “open peer review” has been a focus area in communication about the initiative. Why is that?

LJ: It’s interesting — when conversations about this working group began, the initial idea was to create a Peer Review Taxonomy for open peer review models. But after initial conversations we realized the reality was that we needed more clear standard terminology for all models. And I think the nice part is there was a recognition among this large group of publishers that things like open peer review are likely to get very confusing fast, so we all wanted to pull back and start with a baseline.

As far as applying the Taxonomy to open peer review specifically, we’ve been having conversations about different open review models — and there are many. Open can mean that everyone’s identities are known while doing peer review. It can also mean that the manuscript and comments are public for people to read while peer review is happening, or it can mean making review or reviewer information public afterward. There are different needs for different industries, publishers, and points in the open journey.

I think having precise language makes it easier to get a sense of the various models and what they might mean for specific journals rather than spending so much time trying to figure out what open even means.

With COVID-19, preprints have become a greater focus in publishing, as has open peer review. How has the pandemic informed the way you’re thinking about these areas in the Taxonomy?

LJ: I think conversations about preprints have definitely been a huge focus in the industry this year and likely accelerated just because of the absolute importance of preprints in terms of COVID-19 advances. One thing we’ve seen a lot at the APA is research on psychological trauma related to the impacts of the pandemic. I don’t know if we’ve discussed the Taxonomy in terms of the pandemic specifically. But it did become clear that when you have such a point of pressure in scholarly publishing, clarity around peer review expectations is even more crucial for authors. I know for our authors at the APA, clarity in expectations makes a big difference in terms of authors’ experience regarding their submissions and peer review speed. I also think open peer review will likely accelerate for things like open commenting on preprints while under review due to the pandemic.

You recently completed the first Peer Review Taxonomy pilot — what were those initial findings, and how has the Taxonomy progressed since?

LJ: The big update is, in July, we announced that we’re working with NISO to form a new workgroup to formalize the terminology as an official NISO standard. That’s going to be an exciting next step in terms of getting more publishers involved. NISO has such a wide-reaching community, and I think being part of their guidelines will help move this initiative forward so that it has the opportunity to be adopted across the board.

As far as initial outcomes, at the journal level, we’ve been focused on working with publishers to start including Peer Review Taxonomy information on their journal-specific websites. Those Taxonomy pilots have been going well so far. We’re getting a lot of great feedback from editors and authors who say that this is making things more clear. The editors I worked with all had most of this information on their journal submission pages already but said the Taxonomy helps make it more obvious to authors. You don’t generally hear from authors who are successful in finding the information they need, so it’s a little harder to measure the impact from that side. But from some of the user surveys publishers have done, it does seem that authors are saying the language is easy to understand and helpful.

Now we’re going to start moving into the next pilot phase, adding Taxonomy information to publication and article-level metadata. It’s a smaller lift compared to what we were initially thinking since NISO’s JATS XML standard is already widely used. That is very exciting because it means there isn’t the need for a massive new development project.

As you noted, phase two of the Peer Review Taxonomy pilot will involve implementing it at the article level, including within machine-readable metadata. Why is this so important, and where are you in those efforts?

LJ: We talk about metadata so much it’s easy to forget that not everybody knows about it. The importance of having the Peer Review Taxonomy applied at the article level is to have information about the review process attached to each article in a way that can be used in future years.

Collecting Taxonomy information and making it open at the journal level gives people a way to know how that publication handles peer review when they visit its website. But the publishing and reading experience is very often less about bound issues, even for those still publishing in print. The reality is you have all these articles online and, often, readers will go right to the articles. Having Taxonomy information travel with metadata means it can also be available within the articles themselves. This introduces the ability to someday potentially have mechanisms to search for articles based on their peer review details. It also means if a publisher were to migrate its content to a different database, or maybe move from PDF into a reflowable, mobile-friendly format, for example, that review information can stay with articles instead of being lost. The idea is that information should be able to travel along with the content itself to make it truly transparent and open.

In recent years, Peer Review Week themes have focused on the role of transparency, trust, and now identity in peer review. How is the Peer Review Taxonomy supporting efforts in these areas?

LJ: Obviously, the theme of “Trust in Peer Review” from last year was very relevant because the Taxonomy initiative is about increasing trust. So much of this is also about identities not being visible, so this year’s Peer Review Week theme is also extremely timely. One of the things I’ve discussed with people when thinking about open peer review, in particular, is if revealing reviewers’ identities could open those individuals up to public debate and if researchers have concerns about that. But I’ve actually seen a lot of interesting commentary on this where newer researchers or traditionally excluded researchers from underrepresented groups have said having peer reviews published is a way for them to get credit for hours of work that would historically go unrecognized.

I know some publishers are experimenting with different models for this kind of post-publication openness in peer review. And some are giving reviewers the option to decide whether to have their identity be open or publish their review anonymously. That’s why it’s so important to have more established terminology because there are so many variations. The idea that somebody would want their peer review to be published to show how they worked to help make a piece of content better was an identity element in this project that I know I had not thought about much before. So as identity conversations become more prevalent, as they should be, it’s fascinating to see the new findings and perspectives that emerge.