rpg
stevepog
Callum Anderson
…what dreams may come
When we have shuffled off this mortal coil,
Must give us pause
Thomson, in a commentary in the Journal of the American Medical Association, reckon there ain’t nowt wrong with the Journal Impact Factor:
The impact factor has had success and utility as a journal metric due to its concentration on a simple calculation based on data that are fully visible in the Web of Science. These examples based on citable (counted) items indexed in 2000 and 2005 suggest that the current approach for identification of citable items in the impact factor denominator is accurate and consistent.
Well, they would say that.
And they might well be right, and you and I and Thomson Reuters might argue the point endlessly. But there are a number of problems with any citation-based metric, and a pretty fundamental one was highlighted (coincidentally?) in the same issue of JAMA.
Looking at thre general medical journals, Abhaya Kulkarni at the Hospital for Sick Children in Toronto (shout out to my friend Ricardipus) and co. found that three different ways of counting citations come up with three very different numbers.
Cutting to the chase, Web of Science counted about 20% fewer citations than Scopus or Google Scholar. The reasons for this are not totally clear, but are probably due to the latter two being of wider scope (no pun intended). Scopus, for example, looks at ~15,000 journals compared with Web of Science’s ten thousand. Why? The authors say that Web of Science ‘emphasized the quality of its content coverage’: which in English means it doesn’t look at non-English publications, or those from outside the US and (possibly) Europe, or other citation sources such as books and conference proceedings. And that’s before we even start thinking about minimal citable units; or non-citable outputs; or whether blogs should count as one-fiftieth of a peer-reviewed paper.
Presumably some of the discrepancy is due to removal of self-cites, which strikes me as being just as unfair: my own output shouldn’t count for less simply because I’m building on it. It’s also difficult to know how to deal with the mobility of sciences: do you only look at the last author? or the first? I don’t know how you make that one work at all, to be honest.
That aside, I think curation of citation metrics is necessary: Kulkarni et al. report that fully two percent of citations in Google Scholar didn’t, actually, cite what they claimed to. That is a worrying statistic when you realize that people’s jobs are on the line. You have to get this right, guys.
But it’d be nice if we could all agree on the numbers to start with.
On the run-29Jan10
Posted by rpg on 29 January, 2010
Vitek quotes a Polish proverb,
In that vein, take a look at this graph (don’t look too closely; it’s deliberately a tiny bit obscure):
What I’ve been doing this week is mostly hacking away in Perl at some of the information in our database. As you may know, each evaluation in f1000 has a score associated with it, based on the rating given an article by Faculty Members. We’ve redone the scoring and I’ve worked out a way of combining those scores, or ‘article factors’ as we’ve taken to calling them, for each journal that is represented at f1000. This gives us a ‘journal factor’, ffJ. It’s our answer to the journal Impact Factor, in fact; and the graph above is the top 250 journals according to our ratings (in blue) with the appropriate Impact Factor in red.
You’ll notice right away that there isn’t a one-to-one correlation, and of course we’d expect that (the Impact Factor has serious problems, which I’ve talked about previously). I’m currently analysing our data a bit more deeply, and I’ll be writing a paper with that analysis, as well as talking about it at the March ACS conference in San Francisco.
Last Friday evening I went down the Driver with a couple of the dev team and a bunch of people from places as varied as BioMed Central, Nature Publishing Group, Mendeley and Mekentosj. We talked about what we’re variously calling cross- or federate-commenting. On the whole we’ve decided it’s a good idea, and we simply have to figure out how to do it. What this implies of course is that we’re actually going to allow user comments at f1000—and indeed that’s the plan. I’m looking forward to rolling out this functionality to you, not least because when people want to raise questions about articles evaluated on f1000, they’ll be able to.
While we’re on the mythical new site, we asked another web designer what they could come up with for us. And for the first time, all of us who saw the design liked it. So hopefully we’ll be able to get that implemented real soon now and I’ll be able to start showing you what you’re going to get, you lucky people. (Rumours that someone said “It looks like a proper website!” are completely unfounded.)
Interesting reviews
A couple of things you may have missed.
First, the (possible) mechanism behind photophobia in migraines. Turns out that people who are functionally blind, but sensitive to circadian input and pupillary light reflexes are susceptible to photophobia. Work in rats published in Nature Neuroscience implicates a (previously uncharacterized) multisynaptic or heavily networked retinal path.
In Biology, the problem of de novo assembly of DNA sequence reads into sensible contigs from massively parallel sequencing technologies has been addressed. This, if it works, would bring exciting concepts such as personal genomics that little bit closer. The paper is in Genome Research (subscription required) and you can read the evaluation for free.
And finally
Faculty of 1000 is big in Italy—or at least on Facebook. My post on the recycling of integrins drew an excited response from one Grazia Tamma, who was then mocked mercilessly by her brother!
Hang in there, Grazia; science is great and the cytoskeleton rocks.
Posted in Friday afternoon, Indicators, Metrics | Tagged: comment, cytoskeleton, ffJ, impact factor, Website | 4 Comments »