blog

It makes much more sense in fact to publish everything and filter after the fact

so says our friend Cameron Neylon, a senior scientist at the Science & Technology Facilities Council.

I’m not going to get into that argument here, but I am going to archly raise my eyebrow at the piece in Nature whence that quote comes—Peer review: Trial by Twitter.

First, I take objection to the statement that papers are being analysed—’taken apart’ if you like—in the social media world “rather than at small conferences or in private conversation”. I’m pretty sure they’re also getting shredded in private and at conferences just as much as they always were. It’s simply that there are now additional mechanisms: blogs, twitter, etc.

Maybe that’s picky, but stay tuned because it’s going to get worse.

The Nature reporter seems to think that the world of online commenting on published articles is ‘chaos’, and needs ‘a new set of cultural norms’ and ‘an online infrastructure to support them.’ To me, that sounds like she’s scared of social media and what it can do. While supporting the need for a clear record of scientific findings (and indeed, scientific process), I happen to think that this brave, new world we find ourselves in can only enrich the process. The authors of the two papers referenced in the article can be, I think, excused from grappling with criticisms in this arena simply because it is new, and they were (rightly!) scared. Hell, I’d be scared. But that doesn’t mean it’s wrong: simply that we haven’t got used to it yet (and David Goldstein seems to take a similar view).

Anyway, we can argue about such things till the cows come home. There are more serious errors in the piece—at least from our point of view.

It is good to see people like Liz Allen at the Wellcome saying nice things about us, and that our ranking is the only ‘alternative metric’ that the Wellcome uses ‘in any systematic way’. However, I am wondering who are the critics who ‘note that F1000 rankings tend to correlate closely with traditional citations’? I would like to see their data set, because while some papers correlate well with citations (and you’ll be excused a ‘well, duh’ moment, there), there are quite surprising differences. (Yes, I know, I’ve been promising to write that paper for months now).

Another ‘duh’ moment comes here: ‘most papers never attract the attention of the faculty members, so that they are never ranked at all.’ Well, quite: one of our selling points is that, depending on the field, only 1-2% of all biology/medicine articles ever make it into F1000. And yes, the longevity paper only had one evaluation—maybe that was because, on reflection, it might not have been all that good?

You’ll notice the writer doesn’t mention the arsenic paper. Probably because that has three dissents and therefore doesn’t jibe well with the critique.

Finally, there’s a ‘very brave’ definition of the word ‘common’:

For comparison, the currently highest-ranked paper on the site has an aggregate score of 62, and scores of 20 or more are common.

There are over 100,000 evaluations on the site—a little short of 80,000 papers. I just checked, and 174 have a Factor of 21 or more (you can verify that number easily enough). About 230 have a Factor of 20 or more. That’s less than one third of a percent. Oh hum.

And finally finally, being lumped together with thirdreviewer.com? A site that hasn’t had a post since October and where half of the latest 50+ comments are anonymous?

Please. That hurt.