blog

The most comprehensive analysis to date finds that F1000Prime-recommended articles receive more citations compared to other articles, and that a higher F1000Prime score is associated with higher numbers of citations. Assuming these findings stand the test of peer review, now is the time to focus on new questions about how we assess the impact and quality of research with F1000Prime ratings – and other metrics.

Dr Lutz Bornmann’s article – deposited this week in arXiv – primarily investigates “convergent validity” (whether F1000Prime recommendations correlate with citation impact) and the consistency of the ratings and tags of the same article assigned by Faculty Members (“inter-rater reliability”). Bornmann, in this study, analyzed recommendations of around 100,000 articles (the majority of the F1000Prime database as of January 2014).

As well as agreeing with previous studies on the correlation between F1000Prime recommendations and citations, Bornmann, of the Max Planck Society, combines these results in a meta-analysis providing a more definitive answer. And finds, “the proportion of highly cited papers among those selected by the Faculty Members is significantly higher than expected.” The results also show an increasing – and statistically significant, due to the large sample size – likelihood of an article being highly cited when it is awarded 1 (40%), 2 (60%) or 3 (73% stars).

Another aspect of Bornmann’s analysis concerns consistency of Faculty Members’ assessments and this is found to be lacking, which is consistent with studies on pre-publication (journal) peer review. In discussions about the failings of peer review we often hear that it’s the least-worst system for filtering science we have (frequent misquoting of Winston Churchill aside). Bornmann’s analysis finds that, in this aspect of the system, F1000Prime’s expert review is no worse than pre-publication review – which demonstrates equivalence between pre- and post-publication review.

One might expect Faculty Members to be in greater agreement due to the openly published nature of their recommendations, but Faculty Members “reach their judgements in a similarly independent manner as they do in journal peer review.” This highlights the independent nature in which article recommendations are made by the F1000 Faculty. Our Faculty Members often tell us that they identify articles they want to recommend for F1000Prime while they are conducting journal peer review – and this is one of the reasons why F1000Prime identifies most articles soon after publication.

As we’ve said before, all expert peer review is subjective. But by conducting review openly – and post-publication – we have more opportunities to innovate and improve the system. We already know that open (signed) reviews are of equivalent quality to blinded reviews, and are of equivalent quality if these open reviews are to be posted online. The F1000 experience with post-publication review across our services, in particular at F1000Research, shows it is far more rapid than pre-publication review.

Now is an exciting time for the assessment of research outputs – in all its digital forms – with more and more data and metrics available on the (re)use of research products. CrossRef have begun using PLOS’s article-level metrics (ALMs) software to produce ALMs for their DOIs (used by most scholarly publishers) for a much broader collection of papers than PLOS articles alone. F1000 partnered with PLOS in 2013 to enhance ALMs with F1000Prime scores.

With these new data and tools, and with a number of studies already linking non-citation (alt)metrics with citations (such as F1000Prime scores and tweets), research-on-research-assessment should  evolve. Let us ask new questions. Rather than whether an article or alternative metric has or is associated with more citations, we should design studies to consider more broadly, “does this research have impact?” F1000 is open for collaboration.

Disclosure: F1000 provided Lutz Bornmann with the F1000Prime dataset for the analysis, and provided feedback on an earlier version of the article. F1000 had no involvement in the analysis or interpretation of the data or writing of the manuscript.