blog

Peer review is a standard practice in scholarly publishing, upholding the integrity of published research alongside editorial workflows. But with a culture of data sharing growing at pace, can we do more to uphold research integrity through data peer review? Here, Rebecca Grant (Head of Data and Software Publishing, F1000) shares why peer review of underlying research data is central to the F1000 publishing model and what it could mean for the future of peer review.

Peer reviewers consider the contents of a submitted manuscript, including its relationship to current literature in the field, its methodology, and the conclusions drawn by the authors. Although peer review of the research data underlying a paper’s findings is not yet standard across all publishing outlets, at F1000, we believe that it has a crucial role in supporting research integrity, upholding the principles of open science, and enhancing the replicability of research findings.

Research integrity is connected to values including:

• Honesty when performing and reporting research
• Transparency and openness in sharing methods and outputs
• Acknowledging biases or conflicts of interest
• Ethical approaches to working with research subjects, whether humans or animals
• Academic rigor

Open data sharing at F1000

At F1000, our Open Data Policy is key to our approach to supporting open research and research integrity. Many researchers believe there is a reproducibility crisis in science, with the majority of scientists reporting that they have experienced a failure to reproduce the experiments of another scientist or even their own experiments. Sharing well-formatted, well-described data in public repositories and with open licenses, alongside any software or code necessary to reproduce the experiment, is a fundamental step towards reproducible research. These requirements underpin the F1000 Open Data Policy across all our platforms and are crucial to ensure that others can review, replicate, reinterpret, and reuse research.

Integrating open peer review and open data

F1000 supports an open peer review model, with peer reviewers providing their reports and authors publishing revisions after the initial version of an article has been made public. Unlike many publishers, we also prompt our peer reviewers to assess the study’s research data as part of the peer review process. Openly shared research data supports transparency in methodology and allows validation of a study’s results, improving the reproducibility and quality of published research.

To support our peer reviewers in assessing datasets, we have developed new peer review questions to make the assessment criteria more straightforward. During the peer review process, we ask reviewers to consider the question: “Are all the source data underlying the results available to ensure full reproducibility?” Now, our new guidance also includes prompts to consider the article’s data availability statement and its clarity, as well as the metadata describing the dataset and its usefulness. This guidance aims to aid the peer reviewer in determining whether others could reuse the dataset and reproduce the study.

Although not all journals require data peer review, researchers have demonstrated support for extending peer review criteria to include it. Plus, some argue data peer review is a key feature publishers could incorporate into their data policies to encourage good practice in data sharing.

How to enable peer review of your data

For data to be peer reviewed, it needs to be openly accessible to reviewers, as well as anyone else looking to use it. Our editorial team ensures that each author has deposited their study’s research data at an appropriate data repository. Authors should provide metadata to the repository record to ensure that anyone accessing the dataset in the future will understand how it was generated and by whom.

We also require authors to apply an open license, either Creative Commons Attribution-Only (CC-BY) or a Public Domain Dedication (CC0), so that others can reuse the dataset with minimal restrictions. A license is necessary to ensure that other researchers can use the data to reproduce the experiment. In addition, we request authors to share the code or software they used in conducting their original experiment – and this must also be shared in an open repository, with an open, OSI-approved license applied.

The future of peer review

Demand for data sharing continues to increase, as do data sharing requirements. Just last month, the White House Office of Science & Technology Policy (OSTP) issued a memorandum to make federally funded research freely available without delay and laid out their plans to work with agencies on their public access and data sharing plans. Plus, in January 2023, the US National Institutes of Health (NIH) will begin requiring most of the researchers and institutions it funds annually to include a data-management plan in their grant applications — and to eventually make their data publicly available.

As more data becomes openly available and continues to play a vital role in supporting research integrity, how will expectations around peer review change? While our approach to reviewing data isn’t yet shared by the entire publishing community, it’s possible that data peer review might become a standardized practice across publishers, journals, and disciplines. At F1000, we hope that creating more specific guidance for data peer review will assist our peer reviewers in assessing research datasets and the articles connected to them to support research integrity today and in the future.

Can you play a role in driving research integrity? Explore our Peer Review Toolkit today to learn how to write genuinely helpful peer review reports,