blog

It has been half a year since the Preclinical Reproducibility & Robustness (PRR) channel was launched. PRR provides a venue for researchers to publish both confirmatory and non-confirmatory studies to help improve reproducibility of results, mitigate publication bias towards positive results and to promote open dialogue between scientists.

A number of invaluable replication attempts have been published in the channel on topics as diverse as PTSD in re-located adolescents, uptake of miRNAs in breast milk, pharmacogenomic agreement in cancer and the influence of sadness on color perception (which we also covered in a blog here).

There have been three important developments that have all coincided this week.

PRR Advisory Board announced

We are honoured to welcome Brian Nosek, David Glass, Deborah Berry, Dorothy Bishop, Elizabeth Iorns, Glenn Begley and John Ioannidis to the PRR Advisory Board. They join Channel Advisors Bruce Alberts and Alexander Kamb, who were both instrumental in launching the PRR initiative. All are dedicated to the cause of improving the reproducibility and replicability of life science research and we appreciate the wealth of experience they bring to this initiative.

Extensive mislabelling of gene expression data

Genetic data is not having a good week. First, Ziemann et al. reported that roughly 20% of articles contain supplementary gene sheets with incorrect gene names due to Excel’s over-eagerness to convert certain names to dates and floating-point numbers.

An article published in PRR today reports the potentially more serious and widespread issue: the extensive mislabelling of gene expression data.

Many variables can influence the expression of any given gene, which is why it is necessary to carefully annotate each sample with important information such as the species, strain and tissue type sampled. It is critically important that this information is included in articles describing the results of gene expression studies, as well as in gene expression data repositories such as GEO that enable other researchers can check and re-use the data.

Considering that the conclusions of a study can be affected by the integrity of these annotations, it is clear how important accurate labelling is. But how can you determine whether a sample has been labelled correctly? In their PRR paper Toker, Feng and Pavlidis describe an elegant approach to this problem; by assessing the expression levels of sex-linked genes the sex of the donor organism can be inferred. Females should have high expression levels of female-specific genes and males should have high expression levels of male-specific genes. These expression levels can then be compared to whether the sample is labelled as male or female.

The team found that of the 70 datasets (comprising 4043 samples in total) listed in GEO that they studied, almost half contained at least one discrepancy between the sex stated on the label and the sex-specific gene expression levels. They estimate that at minimum, 33% of gene expression samples are labelled with the wrong sex.
As the authors mention, they likely underestimate the extent of the issue, as their sex-specific expression method can’t detect cases where two female or two male samples have been mixed up.

What’s causing these mistakes? It doesn’t look like these errors creep in when the data is submitted to GEO as the authors found that the associated publications contained the same discrepancies. This leaves actual, physical test tube mix-ups as the primary suspect.

As misannotation has the potential to seriously undermine the validity and utility of gene expression data, the authors argue that the test they performed should be routine. The paper is currently awaiting peer review.

STAP replication

An attempt to replicate the results of the stimulus-triggered acquisition of pluripotency (STAP) protocol published in PRR passed peer review earlier this week. The STAP method was tremendously exciting when it was first published by a team of researchers from RIKEN, as it promised a means of creating pluripotent stem cells from regular differentiated simply by stressing them with acid or mechanical stimulation, but it soon became apparent this was too good to be true.

There have already been a number of unsuccessful replication attempts including one published in F1000Research shortly before the two papers describing the protocol were retracted, as well as recently published attempt by one of the co-authors on the original papers. However, this most recent PRR paper is especially notable as Haruko Obokata, the lead author on both STAP articles, performed much of replication experiments herself, under supervision of the RIKEN Scientific Validity Examination team in specially designated rooms. Like the others, Haruko was also unable to successfully create pluripotent stem cells through this method.

The paper was solely authored by group leader Shinichi Aizawa as Haruko, who resigned from her role at RIKEN, could not be contacted after completion of the study. Although the Shinichi notes that there are limitations with the replication, not least because of the intense media scrutiny and the lack of technical assistance from a key lab member, the fact that Haruko herself could not replicate the phenomenon together with her departure from academia means that this might be the final chapter in the sad STAP saga.