In this analysis the authors “examine whether the magnitude of the effect sizes of biomarkers proposed in highly cited studies is accurate or overestimated”; and their conclusion? “Highly cited biomarker studies often report larger effect estimates for postulated associations than are reported in subsequent meta-analyses evaluating the same associations.”

Biomarkers are often used in studies as surrogates for ‘hard’ clinical endpoints such as death (the ultimate endpoint), heart attack, etc. The rationale being that showing an effect of an intervention on the biomarker will have a corresponding effect on the associated clinical endpoint.

Not only that, many biomarkers have been hastily incorporated into clinical practice, adding to the expense of diagnosis and, in some cases, intervention (think of being prescribed a drug because an elevation in biomarker X has been linked to disease Y). I don’t think anyone would argue that the use of properly validated and extensively researched biomarkers isn’t an extraordinary advance in personalised medicine. Such tools to enable earlier diagnosis will benefit all, but practitioners and patients alike should demand proper validation.

This paper raises another issue: surely it would be better to encourage a system where all the data from a study are released for analysis rather than just the main finding in a journal. Currently results may be enshrined as doctrine because of the logistical barriers to others examining or attempting to reproduce the data.

!F1000 plug alert!

Dr Ioannidis also says in the interview on the news@JAMA site that “Clinicians should keep in touch with the literature to the extent they have time to do so”. And how better to do so than by visiting F1000 regularly. There, I’ve said it!

PS. Since writing this I have read Dr Ioannidis’s article in this month’s issue of Scientific American where he calls for the full disclosure of scientific data… and then some!

The New York Times reported on a paper in the 4 January issue of the Annals of Internal Medicine, which claims published clinical trials don’t cite previous trials. Now, there might be trivial explanations for that, but in the same vein Sir Iain Chalmers, editor of the James Lind Library, has some harsh words for scientists. He says there are fundamental and systemic things wrong with the way research, particularly clinical research, is done today.

It’s certainly misconduct

Among these, he accuses (some) researchers of not addressing questions that are of interest to patients and clinicians, of failing to contextualize new findings, and being clear about what they’ve actually discovered. He also takes aim at the failure of scientists to publish negative or ‘disappointing’ results. In Ann Intern Med last year there was a paper, recently evaluated at Faculty of 1000, scrutinizing the reliability of and inherent bias in clinical trials. And today, Nature published a Correspondence arguing that it’s critical to publish negative results.

Interesting times. Are the criticisms Sir Iain makes fair? If so, is this fault of the scientists themselves or the system in which they find themselves working? If the latter, how can it be changed? What about publishing negative results, and reproducibility and publication bias?

Are we questioning Science enough?

[poll id="6"]

A statistician (or possibly biologist) friend of mine who writes at Nature Network yesterday published a paper entitled Do not log-transform count data in Nature Precedings. He and his coauthor have come up with a novel way of assigning authorship, of which I heartily approve:

The order of the authors was determined by the result of the South Africa – England cricket ODI on the 27th September 2009, which England won by 22 runs.

How do you solve this problem? I might offer a small prize for the most innovative method, provided it has actually been used for a published paper.

O’Hara, Robert and Kotze, Johan. Do not log-transform count data. <http://hdl.handle.net/10101/npre.2010.4136.1> Available from Nature Precedings (2010).

It’s a tough topic to tackle, especially in a time when celebrity deaths, marijuana usage and suicide are so closely linked by tabloid media (Marilyn Monroe being the newest revelation) and when fears that teenage brains getting destroyed by cannabis are high on the news agenda. But on the other hand, when a highly-respected scientist such as Professor David Nutt gets vilified by the government for his outspoken views on drugs policy, the media generally showed support for the sacked Professor while still being skeptical of his evidence-based comments (such as cannabis use being safer than horse riding).

Hall looked at the paper, Cannabis and suicide: longitudinal study, by Allebeck and Price at al. from Cardiff University UK, published in the British Journal of Psychiatry, which went beyond previous small cross-sectional studies to look at whether the cannabis/suicide attempt relationship took into account pre-existing suicide risk between young people who become regular cannabis users and their peers who do not.

In the study, more than 50,000 Swedish men aged 18-20 were followed up for 33 years using death registers to identify those who had died from suicide.

Hall says:

As in previous studies, self-reported cannabis use at conscription was positively related to suicide (odds ratio [OR]=1.62, 95% confidence interval [CI] 0.65-2.07) but this association was no longer significant when plausible potential confounders, such as problematic behavior during childhood, intelligence, alcohol abuse, parental psychiatric disorder, other drug use, and psychiatric diagnosis at conscription were statistically controlled for by logistic regression (OR=0.88, 95% CI 0.65-1.20).

The selection of confounders to control for did not affect the finding that the OR was no longer significant after adjustment for confounders. This study strongly suggests that the modest association observed between regular cannabis use and suicide in cross sectional studies reflects the fact that young people who are at marginally higher risk of suicide are more likely to become regular cannabis users than their peers.

And it’s the last point that is most pressing: those at a slightly higher suicide risk are more likely to become regular pot smokers, not the other way round. If you look at the citation rates on Google Scholar for “cannabis and suicide”, many academics seem to support the view that the two are  strongly linked.

Allebeck and Price’s earlier paper, published in 1990 at this study’s 15-year mark, even stated “the proportion of suicides increased sharply with the level of cannabis consumption“: their new study clarified that the “association was eliminated after adjustment for confounding” and the link was better explained by markers of psychological and behavioural problems.

No doubt long-term studies such as this will lend more weight to the Nutt debate and, if they are given adequate publicity, hopefully help to cut down on biased anti-drug journalism.

Well, this is 1998 information and back then, things were a LOT slacker and this kind of thing did go on. The last 5 years have seen a big change and the policy that Wyeth now has is pretty much in line with everyone else

and

Pharma has sorted this out and anyone behaving like that gets fired. In fact, not stating the source of funding for writing invokes the OIG Federal Anti-kickback Statute, and that is two years in chokey.

{REDACTED} have EXAMS on compliance and anyone breaching compliance in a way that results in negative press for us or our clients is fired.

Teapot, there’s a storm a-brewin’.

Anyway, I didn’t want to talk about that, except it’s a nice hook on which to hang examples of a different-but-similar kind of spin.

Via John Graham-Cumming (go sign his Turing petition) I found this wonderful, wonderful site that shows you just how medical comms, pharma companies, eco-terriers, homeopaths, publishers, bloggers, GPs, PR agencies, newspapers, organic interest groups and in fact just about anyone can lie to you without really lying. It’s precisely the sort of thing that Ben Goldacre tries to get the numpty public (and let’s face it, most scientists/medics) to understand, except with pretty graphs and funky webby clicknology.

2845 ways to spin the risk uses an interactive animation to show exactly how drugs, interventions, whatever can be made to look good, bad or indifferent, simply by displaying the same data in different ways.

Play with the animation for a while, then go read the explanations. The whole ‘relative risk/absolute risk/number needed to treat’ thing is pretty well explained, along with bacon butties:

Yet another way to think of this is to consider how many people would need to eat large bacon sandwiches all their life in order to lead to one extra case of bowel cancer. This final quantity is known as the number needed to treat (NNT), although in this context it would perhaps better be called the number needed to eat. To find the NNT, simply express the two risks (with and without whatever you are interested in) as decimals, take the smaller from the larger and invert: in this case we get 1/(0.06 – 0.05) = 100. Now the risks do not seem at all remarkable.

Mmm. Bacon.

Interesting tidbits abound:

One of the most misleading, but rather common, tricks is to use relative risks when talking about the benefits of a treatment, for example to say that “Women taking tamoxifen had about 49% fewer diagnoses of breast cancer”, while harms are given in absolute risks – “the annual rate of uterine cancer in the tamoxifen arm was 30 per 10,000 compared to 8 per 10,000 in the placebo arm”. This will tend to exaggerate the benefits, minimise the harms, and in any case make it unable to compare them. This is known as ‘mismatched framing’

which is quite intriguing, but then we find that it

was found in a third of studies published in the British Medical Journal.

Ouch.

It’s a splendid breakdown of all the things we need to understand when, for example—oh I don’t know—understanding clinical trial results, perhaps; and I certainly haven’t got to grips with it all yet.

I should, but it’s lunchtime and I now fancy some bacon…