blog

We have published our first articles on the F1000 Research beta site and have already learned a lot about what works, what doesn’t quite, and what we need to consider as we move towards our full launch at the end of the year.  Based on the experience of the past couple of weeks, we have made a few adjustments to our interface and refereeing processes that I want to let you know about.

A third referee status

Once referee reports started to come in, we quickly discovered the need for a third referee status beyond ‘Approved’ (which represents ‘seems OK’, displayed as a green tick) and ‘Not Approved’ (representing ‘does not seem OK’, displayed as a red cross). There are many instances where an article may not be scientific nonsense but a referee has significant concerns about a key element, for example the methods used or the conclusions drawn. We have therefore added an ‘Approved with Reservations’ status category (displayed as a green tick with a grey question-mark).

The first use of this ‘Approved with Reservations’ status can be seen from Prof Alan Schechter on the article NIH Portfolio Allocation, Lemmings, and the Silent Spring: A Time-Capsule Commentary; Its Update by Mark Boothby. This is turning out to be a very topical and quite controversial commentary, with strong views on both sides, with a similar but shorter editorial Iceberg Alert for NIH) published in Science magazine a couple of days later.

On a related issue, we are pleased to see a couple of articles with ‘Not Approved’ red crosses alongside other referee responses of ‘Approved’ green ticks. This shows the value of opening up access to a breadth of views from different researchers and counters the concern some have raised that complete openness will discourage referees from being honest about what they think.

Referee name checks

Many of our first authors and referees have been very engaged in providing feedback to us about our model and our processes.  One very interesting comment we received related to our process of checking referee suggestions.  Our authors are asked to suggest three to five appropriate referees for their article and can also tell us about any referees they feel would not be appropriate, giving reasons (this is a common procedure on many journals).  We had been asking an Editorial Board member to approve (or otherwise) these suggestions based on a set of criteria that include whether potential referees had collaborated with the authors recently, whether they were of appropriate standing, and whether there was sufficient international breadth to the set of referee suggestions.

What the author pointed out is that our sole reliance on the advice of an Editorial Board member can, of course, bias the refereeing process because we are setting up a single individual to essentially act as a gatekeeper on referee approval.  Furthermore, the Editorial Board member may not know the field as well as the author, and therefore may not recognise perfectly valid referee names and, in turn, may suggest referees who are actually less suitable in terms of their knowledge of the field in question. It’s a good point; we probably should not rely so exclusively on the opinion of one Editorial Board member.  However, equally, we cannot include referee suggestions from authors without any checks.

We are therefore proposing to amend our system to conduct some initial checks internally: for co-authorship of papers by authors and referees; for referees based at the same institutions as authors; and to ensure that the referees suggested have been the leading author on at least one article in the relevant field. We will then ask the Editorial Board to tell us if they have any significant reservations about the referees proposed.

This is a tricky issue and we are not sure our proposed solution is entirely right so we would welcome your feedback.

Referee competing interests

Another important issue raised by a couple of our authors was that of referee competing interests.  We now ask all our referees to tell us about any competing interests, and it is interesting to note that almost no one so far has felt that they have anything to disclose.  We had anticipated that most referees would have something to disclose as they are, by definition, close to the field being discussed in the article they are reviewing, and in some instances, that closeness could be construed by others as providing opportunities for bias.  Our sense is that we may need different guidance for referees when it comes to disclosure of interests beyond the more obvious financial ones. Again, comments and suggestions on this are welcomed.

More to come

August is likely to be a quietish time for us as authors and referees take time off for summer vacations.  In the meantime though, we have written some more on why we think open publication and peer review is key to future publishing for the London School of Economics Impact of Social Sciences blog, and watch out for an article in The Scientist coming soon on some of the other things we have learned from our first articles.