blog

This is the third in a series of posts in which we go into more detail about some of the concepts that F1000Research is based on. In previous installments, we looked at open access and open peer review. Here, we turn to post-publication peer review. What are the different types of post-publication peer review and which challenges are associated with different forms?

Researchers discussing published papers. Photo taken at a workshop in the Netherlands, November 2013

In the past few years, the phrase “post-publication peer review” has popped up in various discussions about scientific publishing, as either an add-on to, or a replacement of, pre-publication peer review. As is becoming increasingly apparent, pre-publication peer review doesn’t pick up all problems that may exist with a manuscript. But what is post-publication peer review, and how can it address these issues? Confusingly, the term can refer to a number of different models, which each work in a different way – some by introducing a new peer review system within a journal, others by providing a platform to discuss any published articles.

 

Types of post-publication peer review:

• Review by formally invited reviewers, after publication of the un-reviewed article.

This type of post-publication peer review is used by journals such as F1000Research and Copernicus journals. Here, peer review is carried out by invited reviewers, like it’s done at most journals, but the article is already published online (after an editorial check) before the peer review process starts. Articles that pass peer review are clearly marked as such and are indexed in scholarly databases.

 

• Review by volunteer reviewers, after publication of the un-reviewed article

This is also a publisher-driven method of post-publication peer review, and also involves articles being published online before peer review, but in this case the reviewers are not invited by the journal. Each publisher may use different criteria to determine who can review, and whether the reviews change the status of the published article. At Science Open, a reviewer must have at least five of their own published articles in their ORCID profile. At The Winnower, any registered user can leave a review on any of their published articles.

This system closely resembles the commenting system that several journals have implemented in addition to a formal (invited) peer review system, but journals may ask their volunteer reviewers to address specific aspects of the article, as with invited review.

 

• Comments on blogs or third party sites, independent of any formal peer review that may have already occurred on the article

In the past year, most of the discussions surrounding post-publication peer review have been about new platforms that allow researchers to comment on published research articles. PubPeer allows anonymous researchers to comment on any article with a DOI, or those published as preprints in arXiv. PubMed Commons gives authors with at least one of their own publications indexed in PubMed the ability to comment on any other articles in the database but here they cannot be anonymous. Both services were launched to encourage online discussion about scientific articles – a practice already taking place on blogs and on social media, and to a much smaller extent in the comments sections of journal articles itself.

This sort of discussion can be very valuable and highlights some of the problems of traditional anonymous pre-publication peer review. Discussions on social media and on PubPeer successfully identified issues with the STAP paper that was published earlier this year, and will probably continue to bring to light other issues with high-profile papers in the future.

Sometimes F1000Prime is also mentioned in the context of post-publication peer review. This service uses a network of six thousand international Faculty Members to recommend articles from the life sciences. However, F1000Prime focuses on recommendations only, as a service to highlight important articles, whereas the other methods focus more on criticism and debate.

 

Why does the distinction matter?

Post-publication peer review is still new, and is facing several challenges. However, different types of post-publication peer review are not all affected in the same manner. This is highlighted in the list of challenges included below.

Challenges for post-publication peer review

• Participation – Not all published articles receive comments via systems such as PubPeer and PubMed Commons, although they have shown to serve as a useful platform for discussion of controversial articles. But for journals that use voluntary post-publication peer review (such as Science Open or The Winnower), low levels of participation might mean that articles remain unreviewed. Journals that invite reviewers for post-publication peer review (such as F1000Research and Copernicus journals) use a system quite similar to “traditional” peer review, and can make sure that all articles are seen by reviewers.
• Reviewer expertise – Inviting reviewers also allows journals to ensure their reviewers have the adequate expertise to review each particular paper. Some voluntary review systems have also built in a checkpoint to control expertise level: for example, Science Open requires reviewers to have five articles in ORCID, and PubMed Commons requires one article in PubMed. However, neither system is able to check that the previously published work of the reviewer is in the field of the article they’re commenting on.
• Fragmentation of discussion – One critique of the variety of post-publication peer review systems is that discussion happens in multiple places. The same article can have comments on the article itself (if that feature is available), in PubPeer, on PubMed Commons, on Researchgate, on blogs, on Twitter, on F1000Prime and elsewhere.

As these examples show, the different versions of post-publication peer review all deal with different types of challenges, so it is important to clearly distinguish between them.

 

How to tell which type of post-publication review you’re dealing with

In a previous post we looked at open peer review, and saw that it can mean named reviewers, or public referee reports, or both. In all cases, though, “open peer review” refers to review by invited reviewers. “Post-publication peer review”, on the other hand, can be named or anonymous, and reviews can in some cases be written by uninvited reviewers who may not necessarily be literal “peers” in the field.

The many different uses of the phrase are confusing, and currently the only way to know which is being used is to look into each specific case to find out what is meant. For example, at F1000Research we noticed that many people assume that “post-publication peer review” means that anyone can provide the formal peer reviews on our articles. In fact, all our peer review is carried out by invited reviewers.

When you come across a mention of post-publication peer review, always check which “flavour” it is: are reviewers invited or is review voluntary? Is there a check for reviewer expertise? Are reviews published on the article itself or on a third-party site? Are reviewers anonymous? Does the post-publication review replace traditional peer review or is it an add-on service?

All types of post-publication peer review serve a purpose, but they don’t all serve the same purpose.

 

References and links

On issues with pre-publication peer review:

• The Guardian view on the end of peer review. Guardian editorial, July 6, 2014
• STAP retracted: Two retractions highlight long-standing issues of trust and sloppiness that must be addressed. Nature editorial, 2 July 2014

Discussions about post-publication peer review of the kind that happens on external sites (eg. PubPeer, Pubmed Commons, blogs)

• What is post-publication peer review? By Richard Smith on BMJ blog, 6 April 2011
• Post-publication peer review mainstreamed. By Aimee Swartz in The Scientist, 22 October, 2013
• Post publication peer-review: Everything changes, and everything stays the same, by Bonnie Swoger on the Scientific American blog Information Culture, 26 March 2014

Other:

• An F1000Research article that has both formal post-publication reviews and comments. You can see formal (invited) reviewer comments, as well as reader discussion. All comments were left after the manuscript was published.
• Notes and tweets from the “post-publication peer review” session at the Science Online 2014 conference.
• STAP paper discussion on PubPeer