blog

Following a two-day workshop hosted by the Indiana University School of Public Health, Justin Otten et al. reflect on ways of improving rigor, reproducibility, and transparency (RRT) in scientific research. Their Opinion Article on the same topic was published in October 2020 and is included in the Future of Research collection on F1000Research.

---

Meet the research team

Danny Valdez is an Assistant Professor at the Indiana University School of Public Health. His area of research leverages Natural Language Processing (NLP) methods to draw inferences about population-level health behaviour during natural experiments.

---

Andrew Brown is an Assistant Professor with the nutrition program within the Department of Applied Health Science. In addition to primary research in nutrition and obesity, his research investigates rigor, reproducibility, and transparency with a particular focus on methodological and statistical choices that may lead to ambiguous or misinterpreted results. 

---

Evan Mayo-Wilson is an Associate Professor at the Indiana University School of Public Health. His research focuses on evaluating the effectiveness of health and social interventions, methods for conducting and synthesizing intervention studies, and improving research transparency using methods that promote openness and transparency.

---

Colby Vorland is a Post-Doctoral Research Fellow at the Indiana University School of Public Health. His research activities focus on RRT within nutrition and obesity research, including error correction, research reporting, and methods development.

---

David Allison is Distinguished Professor, Provost Professor, and Dean of the Indiana University School of Public Health-Bloomington. As a methodologist, he is committed to rigorous research in all domains of science. Dr Allison’s research ranges over many aspects related to energetics, from basic model organism studies, all the way through applications and policy analysis. 

---

Justin Otten is Director of Global Health Affairs at Indiana University. Dr Otten also serves in numerous administrative capacities as the RRT liaison between the IU School of Public Health Bloomington and national/international RRT leaders. Dr Otten has convened several RRT related events, including seminars, full-scale workshops and conferences directed at improving scientific rigor, and white papers in the scientific literature.

---

Introducing the Indiana University School of Public Health Rigor, Reproducibility, and Transparency team

Our team comprises faculty, staff, and graduate students who have expressed interest and published in areas related to rigor, reproducibility and transparency (RRT).

The lead authors (Drs Valdez, Brown, Mayo-Wilson, Vorland, and Allison) each study RRT in unique ways. Dr Valdez studies RRT from the perspective of writing education and Comparative Survey Research (CSR) ethics. Dr Brown and Dr Vorland study RRT issues primarily within the domains of nutrition and obesity research. Dr Mayo-Wilson focuses on RRT with respect to health and social intervention research. Dr Allison operates from a statistical perspective, which has often led to recognition of statistical challenges in research and subsequent development of new methods. Another branch of his research activity involves developing, evaluating, and disseminating novel statistical methods. Dr Allison also actively works to promote RRT within the School, focusing on instilling ethical principles among his faculty and students, and dispelling misinformation in scientific research.

Together, our team at the Indiana University School of Public Health-Bloomington (IUSPH-B) value and prioritize the need for RRT across scientific disciplines. Our unique backgrounds combine  together to create a well-rounded understanding of RRT.

What motivated you to write the Opinion Article?

In October 2019, the Indiana University School of Public Health hosted a two-day workshop with international leaders to discuss ongoing and future RRT challenges.

The goal of this workshop was to identify research priorities in three domains:

1. Improving education and training
2. Reducing statistical errors and increasing analytic transparency
3. Increasing truthfulness and accuracy of research communications

Workshop participants were evenly distributed across each group and tasked with identifying RRT challenges and corresponding research opportunities. Our Opinion Article represents a culmination of this workshop, where we summarize findings from our discussions within the context of the scientific literature. Our paper also presents calls to action for the scientific community, including promoting ways to distil accurate, truthful, and more objective science.

10 research opportunities in summary

Within the ‘Improving education and training’ group, we identified several gaps in existing curricula across education levels that do not adequately address problematic scientific practices, or provide the necessary toolkits to circumvent these issues:

1. Can RRT-focused statistics and mathematical modelling courses improve statistical practice?
2. Can specialized training in scientific writing improve transparency?
3. Does modality affect the efficacy of RRT-related education?

For ’Reducing statistical errors’ we identified novel ways to detect errors using conventional and automated methods to answer RRT-related questions:

• Can automation help identify errors more efficiently?
• What is the prevalence and impact of errors?
• Do error prevention workflows reduce errors?
• How do we encourage post-publication error correction?

For ‘Increasing truthfulness’ we identified opportunities for research related to research reporting and dissemination, primarily highlighting the importance of accuracy and truthfulness when communicating research findings:

• How does ‘spin’ in research communication affect stakeholders’ understanding and use of research evidence?
• Do tools to aid writing research reports increase the comprehensiveness and clarity of research reports?
• Is it possible to inculcate scientific values and norms related to truthful, rigorous, accurate, and comprehensive scientific reporting?

Collectively, the ten research opportunities identified by our working group present a well-rounded agenda for meta-science as a rigorous branch of science in its own right. Each research opportunity can spark much-needed discussion about broad issues and concerns across scientific disciplines.

What issues are there in science that necessitate emphasis in RRT?

Over time, the pursuit of truthful and accurate science has been affected by systemic problems. For example, two commonly discussed problems are error correction stigma and perverse incentives.

First, errors are a common part of science. Errors can happen to even the most careful scientist. When found, errors should be corrected to ensure that reported findings are accurately communicated regardless of the title or prestige of the scientist or groups of scientists. Yet, our current scientific climate stigmatizes error correction and the scientific community sometimes reacts to attempts to correct errors as if they were attempts to tarnish the reputation of others. Ultimately, this stigma around error correction has led to much difficulty in correcting published science, which remains an ongoing problem today.

Similarly, perverse incentives(a type of incentive that has an undesirable result contrary to its intentions) may promote poor scientific practices. Although scientists can (and should) be incentivized for producing high-quality and impactful work, anecdotal examples point toward possible negative side-effects of these incentives. For example, some argue that pressure to publish leaves some scientists feeling as though quantity of work or publications is rewarded over quality of work. In turn, tenure and promotion – admittedly only pertaining to a fraction of the scientific community – which should reward scientists for their scientific contributions with job security, may incentivize others to stop producing once that benchmark is achieved.

We acknowledge that more research is needed in this area, and that these issues require thoughtful discussion in order to find paths forward. This can be accomplished through ongoing RRT-related studies, or other investigations intending to redirect scientific norms that embrace error, occasional failure, and incentives that squarely uphold high-quality science.

What is needed from the research community to facilitate these research opportunities, or improved open and rigorous science?

We must normalize and institutionalize conversations about the rigor, reproducibility, and transparency of science. By normalize, we simply mean that such discussions are common and openly accepted as part of the scientific process. If we normalize conversations on correcting errors or eliminating perverse incentives, for example, then perhaps we can collectively identify new approaches that more appropriately incentivize scientists to produce work that adheres to higher standards.

A larger and arguably more difficult charge is to reorient the scientific climate around norms that promote and adhere to such rigorous standards. This is where institutions can play a large role by steering incentive structures toward the promotion of scientific rigor.

We are hopeful that future RRT studies originating from IUSPH-B will both draw attention to the importance of open and rigorous science and provide avenues to better facilitate it.

What impact could open research principles have on improving the rigor, reproducibility, and transparency in scientific research?

The push for, and interest in, open science has increased over the last several years. Yet, there may be some confusion as to what is meant by open science to people who do not study RRT. Broadly, open science refers to the act of making the scientific process accessible to others to promote collaboration, scholarly communication, and verification and feedback on work.

We are proud that one of our invited panellists, Dr. Brian Nosek, is a pioneer for open science. As the Director of the Center for Open Science, a non-profit organization whose mission is to increase the openness, integrity, and reproducibility of scientific research, Dr. Nosek serves as a sounding board for research opportunities geared toward this specific facet of RRT.

Broadly, our objective in promoting open science is to underscore its numerous benefits, both for scientists and the general public.

Finally, do you have any comments you would like to add?

We are thankful to F1000Research for the invitation to contribute to their blog and willingness to promote the value of RRT in science.

Our paper only represents initial steps taken to identify research opportunities related to RRT. We hope in the coming months and years our team at the Indiana University School of Public Health, and Indiana University more broadly, will follow through with these opportunities to produce timely and relevant studies that address the discussions in this post and our article.

---

Read the full Opinion Article on F1000Research here >>> Improving open and rigorous science: ten key future research opportunities related to rigor, reproducibility, and transparency in scientific research