What is open peer review?
| 21 May, 2014 | Eva Amsen |
|
|
This is the second in a series of posts in which we go into more detail about some of the concepts that F1000Research is based on. In the first instalment, we looked at open access. Here, we turn to open peer review. What is open peer review? What are the benefits and challenges?
History of traditional (closed) peer review
Even though scientific publishing has been around since the 17th century, formal peer review of submitted articles by external academics is relatively new. The journals Science and JAMA, for example, introduced formal peer review in the 1940s, and Nature didn’t introduce it until 1967.
The peer review system adopted in the 20th century has now become the norm for many journals. It involves an editor (usually a practising researcher, but sometimes a journal staff member in the case of journals like Nature) sending out a paper to a few experts in the field, who then provide comments for the paper’s authors. Although the reviewers can generally see who the authors are, they themselves remain anonymous to the author, and only the editor knows everyone’s identity.
Problems with traditional, semi-blind, peer review
This “single-blind” system is not without problems. Anonymous reviewers can be biased against the authors of the paper, and lean toward rejection or acceptance for unscientific reasons. Often, the closest “peers” in someone’s area of research are also that researcher’s direct competitors! One solution is to remove the authors’ names from the manuscript, but this double-blind system is not fool-proof, and a reviewer will still often recognize which lab a paper comes from. In addition, any bias towards competitors of the reviewer still remain, even if that competitor is anonymized.
Another drawback of traditional peer review is that the referee reports are visible only to the authors and the editor. Nobody else can see what the reviewers thought of the paper. Especially in situations where reviewers disagree, and a single editor makes the final decision, it can be very informative to see what the reviewers thought of an article, and whether the editor’s decision was in line with their opinion. Reviewers are usually in a position to put the work in a broader context of the field, and often mention this context in their reports. They can also point out where the work could be expanded into new areas, and may still have some lingering questions. All of this is useful for everyone to read – not just the authors. It’s also important to remember that not all journals use the same criteria for publication. Some journals may turn a great paper down just because it doesn’t fit the scope of the journal. Other journals publish all sound science, including some papers that get extremely high praise in the referee reports.
A timeline of open and transparent review
Within the life sciences in particular, several journals have opened their peer review process to address some of the issues discussed above. Sometimes this involves publicly naming reviewers and/or editors. Other journals publish some or all reviewer comments.
| 1999 | After studying various peer review models, BMJ starts revealing reviewer names to authors |
| 2000 | BioMed Central launches, and soon after that starts including reviewer names and pre-publication history for published articles in all medical journals in their BMC series of publications |
| 2001 | Atmospheric Chemistry and Physics introduces a system where manuscripts are placed online as a “discussion paper”, which is archived with all comments and reviews, even before approved and peer-reviewed articles appear in the journal. |
| 2006 | Launch of Biology Direct, which includes reviewer comments and names with published articles. |
| 2007 | Frontiers launches, and includes reviewer names with articles. |
| 2010 | EMBO journal starts publishing review process file with articles. Editors are named, but referees remain anonymous. |
| 2011 | BMJ Open launches, and includes all reviewer names and review reports with published articles. |
| 2012 | Several journals launch with an open peer review model:
|
At F1000Research our goal has been to champion transparency in the peer review process: Each article we publish includes all peer review reports, reviewer names, and author responses – even for articles that are still under review or revision.
Benefits of open review
Benefits for authors and readers
- Author can see who reviewed their work
- Reviewer comments put paper in context which is useful additional information for readers
- Reduces bias among reviewers
- More constructive reviews
- Published reports can serve as peer review examples for young researchers.
Benefits for reviewers
- Shows the reviewer’s informed opinion of the work
- Demonstrates experience as a reviewer
- Can take credit for the work involved in conducting the review
To make it easier for referees to take credit for their work, some journals, including F1000Research, now provide unique identifiers (DOIs) for referee reports. In addition, F1000Research is co-chairing a working group investigating how to include peer review output in ORCiD profiles.
Challenges
Although open peer review is becoming more common, and addresses several of the issues of anonymous review, a few challenges still remain. A study in the early days of open review suggested that naming referees slightly reduced the likelihood of finding reviewers but did not affect the quality of review. Conversely, other studies suggest that open review provides more constructive reports.
References
- Three myths about scientific peer review, by Michael Nielsen (January 2009)
- Nature timeline, 1960s
- The history of peer review, in Planned Obsolence: publishing, technology, and the future of the academy by Kathleen Fitzpatrick.
- Working double-blind. Nature 451, 605-606 (7 February 2008)
- Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial Van Rooyen S, Godlee F, Evans S, Black N, Smith R, BMJ 318, 23-7 (2 Jan 1999)
- A comparison of the quality of reviewer reports from author-suggested reviewers and editor-suggested reviewers in journals operating on open or closed peer review models. Maria K Kowalczuk, Frank Dudbridge, Shreeya Nanda, Stephanie L Harriman, Elizabeth C Moylan F1000Posters 2013, 4: 1252
- ORCID & CASRAI Kick-off New Standards Project on ‘Peer Review Services’.
- Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. Van Rooyen S, Delamothe T, Evans SJW, BMJ 341 (16 November 2010)
- Opening peer-review: the democracy of science, Shanahan DR, Olsen BR, Journal of Negative Results in BioMedicine 13:2, 2014
- Making Reviewers Visible: Openness, Accountability, and Credit, Fiona Godlee, JAMA 287(21):2762-2765 (5 June 2002)
- Plus additional editorial policies and references linked in the timeline above.
- Transparent laptop image via Martin Eckert on Flickr
Further reading
- A guide to peer review in ecology and evolution Good overview of different types of peer review, published by the British Ecological Society.
- Peer review: the nuts and bolts. Resource for researchers about peer review, published by Sense About Science.
- How to review a paper – RajLab. Mentions as tip to pretend review is not anonymous (even when it is).
- Open Peer Review group on Mendeley. Add examples of good open peer review reports.
- The ups and downs of peer review. Dale Benos et al., Advances in Physiology Education, Vol. 31no. 145-152 (1 June 2007)
- Are we training pit bulls to review our manuscripts? Virginia Walbot, Journal of Biology 8:24 (9 March 2009)
|
|
previous post
