blog

Late last year, we published the first Stage 1 Registered Report on F1000Research. Publishing Registered Reports is believed to increase the rigor around the scientific method, which supports reproducibility. We interviewed the lead author, Julius Emmrich (JE), Charité, Germany, and one of the article’s reviewers, Ádám Dénes (ÁD), Institute of Experimental Medicine, Hungarian Academy of Sciences, to get an insight into their views on this way of improving experimental design.

Can you tell us a little bit about the areas that you both research?

JE: I am interested in the mechanisms of inflammatory neurodegeneration in the brain. I am also interested in the regulation of apoptosis and its interaction with phagocytosis. My current research focuses on investigating the mechanisms of phagocytosis in the brain and its role in pathologies such as stroke, sepsis, and autoimmune encephalitis.

 

ÁD: The research of my lab focuses on the role of inflammatory mechanisms in brain injury. We are particularly interested in microglia-neuron communication. By using various in vivo and in vitro models and histological, cell biological and high resolution imaging approaches we aim to understand the contribution of microglia, systemic inflammatory actions and brain-immune interactions to acute neuronal injury and neurodegeneration.

 

As our first Registered Report author and reviewer, what motivated you to participate? What were the differences, if any, in how you approached this paper?

JE: Over the past years and in particular in neuroscience, difficulties concerning the reproducibility and replicability of scientific results have increasingly come into focus. Research practices such as p‐hacking and hypothesizing after results are known – HARKing – in combination with selective publication of significant over non-significant results hamper robustness and accountability of research data and lead to publication bias. My co-authors and I believe that a clear definition of research questions, experiments, and data analysis plan before a study is undertaken protects from engaging or being asked to engage by reviewers in these harmful practices.

Writing a registered report before the data collection was a new step in the workflow for us. Defining the hypothesis, experiments, and statistical tests are certainly steps that everyone has to take at some point. But doing so in detail before the actual data collection helped us to improve the study design and planning of this study.

“Registered reports could accumulate valuable feedback early on, making it possible to deal with major issues on time and generate robust experimental data.” Ádám Dénes

ÁD: As a reviewer I liked the initiative and tried to be as supportive as possible. The main purpose of a review at this stage, i.e., before doing the actual work, is to identify the potential caveats in the experimental design and the approaches to be used. This does not sound very different from reviewing a paper with all the experimental findings in there. However, there seems to be an agreement on that although the quality of an actual research paper is improved by the peer review process, the vast majority of studies cannot be fundamentally improved by revision once the work is done. This is one of the reasons why most studies including those with really poor design will eventually get published somewhere. Registered reports could accumulate valuable feedback early on, making it possible to deal with major issues on time and generate robust experimental data. I doubt that this strategy can be applied uniformly for all studies (just remember how many opposing opinions the same paper can get from reviewers), but the direction is good.

 

Julius’ papers highlights the value of Registered Reports in strengthening the robustness and accountability of data. Are there any other measures you have taken or new initiatives you would like to explore to improve data robustness and accountability?

JE: Our group has recently begun to make raw data available in online open data repositories. We believe that making data Findable, Accessible, Interoperable, and Reusable (FAIR) by others increases the credibility of our data. In addition, data sharing allows for aggregation across multiple groups and investigations and big data analysis.

ÁD: I agree. As an example, the availability of complex data sets from ‘omics studies has already made possible to cross-check experimental data and to perform in depth statistical analysis before collecting additional experimental data. Such opportunities increase the credibility of research findings and save money and time for everyone.

 

In Ádam’s peer review report, he mentions that it is a ‘great idea to share the design of […] proposed studies with the scientific community’. What benefits do you think Registered Reports offer neuroscientists over the traditional publishing model?

ÁD: Study design is a very important factor for the overall success of research and largely determines the interpretation of the results. This is also a topic frequently discussed during peer review to assess scientific quality. Since low reproducibility of experimental studies is one of the key issues that compromises the successful translation of research findings into clinical benefit – among many others – it would be a great advance if research protocols would be shared and openly discussed before the actual research is performed. For many reasons, it is not always possible, but we should aim for it. Beyond collecting feedback from others, such process can also facilitate in depth thinking about how exactly the proposed research will answer the questions raised.

“Registered reports provide an opportunity to refine the study design based on reviewers’ comments before a study is undertaken.” Julius Emmrich

JE: I agree with Adam on the importance of a well-designed study. Rather than having reviewers point out flaws in the hypothesis, methodology or statistical approach after all results are collected, registered reports provide an opportunity to refine the study design based on reviewers’ comments before a study is undertaken. That may save us time and grey hair down the line. In addition, by providing a detailed plan of experiments, variables, and data analysis registered reports may help to prevent practices such as p‐hacking and HARKing. Registered reports also require reporting a sound power analysis, thus reducing the probability of false positive results.

 

What have you taken away from this experience?

ÁD: As a matter of fact, reproducibility of research findings is generally low. Unfortunately, top scientific journals, which largely determine research directions in the respective fields of science (and hence the associated funding), are not immune to this problem either. In addition, the peer review process as we have it today is flawed, but criticism is often neglected by saying that we have no better one. In fact, it is relatively easy to identify problematic points but seems rather difficult to change these. At least concerning quality issues there is some good progress since the publication of comprehensive experimental and reporting guidelines (STAIR, ARRIVE, etc.).

“Parallel research should be openly discussed as early on as possible and hence even opposing results from different labs would become much easier to interpret.” Ádám Dénes

Still, there is a lot to do. Thus, it is worth trying whether transparency in general and tools like open peer review or registered reports in particular would do any good here. It is likely that we could improve the success and the reliability of individual studies by carefully discussing them with the wider scientific community before they are executed. In case such ideas turn out to be successful, some of these might then be considered further by scientific journals, funding bodies and so on, to increase the transparency of research in general.

For example, it is a major problem that many excellent, comprehensive studies that require large budgetary resources (mostly public money) and take 5-6 years to complete become extremely difficult to publish if someone else comes up with a similar story slightly earlier. In fact, we should do the opposite: parallel research should be openly discussed as early on as possible and hence even opposing results from different labs would become much easier to interpret.  This would not compromise naturally advantageous competition, as shown by the success of some excellent back-to-back papers in leading scientific journals.

Initiatives like bioRxiv (a preprint server for biology) also appear to be quite useful for even cutting-edge stories to increase their visibility before the lengthy publication procedure is completed. Concerning quality issues, external validation of research findings is also crucial: for example, in the stroke field, low reproducibility of studies and the failure of most clinical trials have triggered the initiation of pre-clinical multicenter trials, which are likely to eventually help selecting drugs with the highest potential to be tested in the clinic. These advances generally show how we can learn from previous pitfalls, and give hopefully appropriate responses to the problems associated with experimental research.

JE: Like Adam, I am very keen on seeing open peer review, registered reports and open science in general improve transparency, accessibility, and reproducibility of results. I believe that these approaches are necessary to challenge the current status quo of translational roadblocks and to increase the impact and value of biomedical research in general. They may also help to create awareness to rethink current research incentive schemes.

For further details about Registered Reports, please see our reviewer and author guidelines.