blog

When we announced our campaign to waive the article processing fee for negative results articles until the end of August, we published a press release with a quote from Ben Goldacre, who has been fighting publication bias in medical research and clinical trials. He has talked about the issue in his book Bad Pharma and on other platforms, and recently co-founded the AllTrials campaign to call for the public availability of all information from clinical trials. In this brief interview he explains how medical research and patient treatment are affected by the lack of publication of negative data.

What usually happens with negative results in medical research?

Negative results very commonly go unpublished, and this happens at every possible level. It’s a fascinating research issue, from the field of “research about research”. Various studies have shown that basic science research often doesn’t get published if it produces negative results. For example, there’s a paper from March 2012 in Nature, which looked at 53 molecular biology papers that suggest a possible target treatment for cancer. When they tried to replicate all of those findings in their own lab, they found that they were only able to replicate six. So 47 out of these 53 published molecular biology studies, which suggested a potential treatment target for cancer, couldn’t be independently replicated. The authors of that study thought the main reason for this, for the publication of irreproducible data, was the pressure to publish “positive” studies, and they suggested that one way to fix this is to make it easier to publish negative results.

Beyond that, you can find traces of this problem in other parts of the literature. For example there’s a well-known hump in the P-values that people publish: there is an excess of findings published with a P-value just under 0.05, and fewer papers than you’d expect with a P-value just over 0.05. You can also look at the number of positive findings reported in a whole field, for example the brain scanning literature, and see that the number of positive findings reported is greater than expected from a power calculation on all of the studies that are being published.

What are the consequences of not publishing negative results?

Publication bias is a problem in basic research, but it’s a really important problem for clinical trials. The best currently available evidence on this comes from a review published by the HTA in 2010, which summarizes the results of dozens of studies, and shows that overall, the chances of a clinical trial being published are roughly 50:50. That’s a serious problem for medicine, especially when we also know that trials with positive results are about twice as likely to be published as trials with negative results.

To be clear, this problem would be bad enough if we were just missing results at random:  If results are left unpublished – from both positive and negative clinical trials to an equal extent – then you are still unnecessarily duplicating research, because you believe incorrectly that the questions about what works best are unresolved. But it’s worse than that: we know that what we see is a biased sample of clinical trials that have been done. It’s an exaggerated picture of the benefits of treatment. That has a serious impact on the treatment decisions that doctors, patients and researchers make every day, because we use the results of clinical trials to make decisions about which treatment is best, and we can’t make an informed decision when half of the trial data is withheld. So we need all of that information to be made available, and as long as it continues not to be, patients will continue to be harmed.

To add to this, I think it’s very uncommon for drugs on the market to be worse than useless, although that can happen. What’s much more common is that we’re misled about the relative benefits of different treatments for the same condition, or about the scale of the benefits of the treatment. But if you’re deprived of the best treatment, because you’re misled into believing that something else is better, then you suffer unnecessarily. Also, exaggeration matters, because we balance benefits against disadvantages. If we believe that a treatment is better than it really is, we might say it’s worth enduring side effects and inconvenience for a large treatment benefit. If in reality the treatment benefit is much more modest, then some people, if they were making an informed decision, might choose not to bother taking it. So again, it’s really important that we have all the evidence.

What are you doing to let people know about the problem of publication bias in medicine?

We’ve known about this problems since at least the 1980s, and it hasn’t been fixed behind closed doors. I feel very let down by the senior academic and medical professional bodies on this, as do many other people: many of them have acted and even written as if the problem doesn’t exist. That’s why we had to start a public campaign. I set up the AllTrials campaign in January with the British Medical Journal, the Centre for Evidence-based Medicine in Oxford, the Cochrane collaboration and Sense about Science, and we are calling for four things.

Firstly, we want people to recognize that this is an ongoing problem affecting patient care. That’s really important, because people have been trying to brush this under the carpet for many years. For example, the Ethical Standards in Health and Life Sciences Group, which is a body that includes people from industry and from medical and academic professional bodies, have produced documents which effectively pretend that this problem doesn’t exist. I think that’s very dangerous.

Beyond calling on people to recognize that this problem exists, we ask for three things. We ask for all trials to be registered, so that we know that the trial has happened. We also ask for the summary results of that trial to be published. A summary can be published in a regulatory filing, on clinicaltrials.gov according to certain specified standards, or in an academic journal article. Academic journal articles are considered summaries because they are often quite brief and haphazard descriptions of what happened in a trial, and they can be misleading. For example, they can report subgroup analyses that weren’t pre-specified, or switch the primary outcome: we know that academic journals have been very bad at monitoring that kind of thing, and ensuring that it doesn’t happen. So we consider academic publications a “summary” of a trial.

Finally, we also ask for something called the Clinical Study Report, where it’s available. A CSR is a very long and detailed document that is already produced for all industry-led trials. It provides an enormous amount of detail about the methods and the results, so it can be a thousand pages long or more, and it includes everything: the statistical analysis plan, copies of documents such as the patient record forms, and lots of very detailed information about exactly what has been done, and what was found. Over the course of the past three or four years we’ve just started to understand the importance of these documents. With papers published on Tamiflu and some Pfizer drugs, we can now see that the short brief story published in academic journal articles can be incomplete, inaccurate, or even misleading about what happened in clinical trials that have been conducted. The long CSRs give you the best possible description of all of the information about what happened in the trial. There can be some individual patient information in these documents, for example in a narrative description of one person’s potential adverse reaction during the trial, but the EU Ombudsman has ruled that removing this – where necessary – presents no great administrative burden.

So those are the four things that we’re asking for. We don’t ask for individual patient data. That’s not because we think that’s not an important area, but individual patient data is a slightly more complicated issue, with more interesting privacy concerns.  What we do ask for is for trial registration, trial summary results, and clinical study report – where available – to be put in the public domain. Many naively have believed that these problems were all already fixed. They were wrong. But there is certainly an inevitable policy trajectory towards this level of transparency, and I hope people in industry and academia will recognise that by showing leadership and embracing this new ethical approach they will be benefitting themselves and patients.