blog

The IICARus study is seeking volunteers to help assess the impact of the ARRIVE guidelines on the quality of reporting of in vivo research. Contributors will be listed as authors on the resulting publications. If you are interested in helping you can sign up here.

For research involving animals to be as scientifically and ethically sound as possible, it is crucial that experiments are properly designed, that data are appropriately analysed, and that reports are written in sufficient detail to allow the reader to judge the design and the analysis and to seek to replicate the experiments.

A 2009 survey of 271 in vivo research articles found these conditions were rarely fully met. The following year the NC3Rs launched the ARRIVE guidelines, a structured checklist outlining all the information that authors need to disclose to ensure that others could properly interpret the findings and so maximize the use of any knowledge gained.

A number of publishers, including F1000Research and Wellcome Open Research, have mandated compliance with the ARRIVE guidelines as a condition for publication. But how do we know that adoption of the ARRIVE guidelines actually makes a difference? Just because journals mandate compliance in their instructions to authors doesn’t mean they effectively enforce such compliance; and even if articles are now better at reporting animal research, this might be due to factors unrelated to adoption of the guidelines.

Enter IICARus, a randomized controlled trial run by CAMARADES and funded by the NC3Rs, MRC, Wellcome and BBSRC. IICARus is designed to assess whether stricter enforcement of the ARRIVE guidelines improves the reporting of animal research. To answer these questions, IICARUS needs to measure outcomes – to assess the extent to which some 800 publications meet the requirements of ARRIVE. This is a considerable undertaking, so the study is recruiting volunteers to help assess the quality of reporting.

We spoke to Professor Malcolm Macleod (University of Edinburgh, CAMARADES) and Drs Nathalie Percie du Sert and Viki Hurst (NC3Rs) who told us more about the initiative.

Could you provide more information on the design of the project?

This is a randomised control trial, we have been working with a major journal which regularly published in vivo studies. To be included, a manuscript simply had to contain some form of in vivo research, even observational studies. At the first stage of editorial review, manuscripts submitted were randomised either to the normal publication process, or to having an extra step where authors were required to submit a completed ARRIVE checklist with their manuscript. If the checklist wasn’t submitted, the manuscript went no further. At randomisation, we used a process called minimisation to ensure that broad subject areas and country of origin were represented approximately equally in each group.

The integrity of the study outcome requires that peer review, and decisions to accept or reject for publication, were taken without knowledge of the group to which the manuscript belonged, or even that a study was taking place.

That’s interesting – tell me about the ethics?

Yes, we were concerned to have an ethical view as to whether it was acceptable to have manuscripts randomised to different processes without the consent of the authors or indeed without the knowledge of the reviewers. After a great deal of searching (it seems there are very few ethics committees who consider this within their remit) we were able to get an informal opinion from another publisher’s ethics committee. They agreed with our view that, since knowledge even of the existence of the study would be likely to change behaviour, and since the journal guidance to authors mentioned that manuscripts might be included in studies such as ours, it was reasonable to proceed. In effect, the potential benefits outweighed the potential harms.

This seems like a big project. How many articles need to be reviewed before the study is considered complete?

The primary objective of the study is to determine whether requiring compliance leads to an increase in the proportion of studies which are fully compliant with ARRIVE.

At the beginning, we had to estimate what proportion of submitted manuscripts would make it through to publication, with the aim of having around 300 publications in each group or 600 publications overall. In fact, it turned out that the acceptance rate was somewhat higher, and so we have 720 publications to assess twice, or 1440 assessments in total.

What is expected of volunteer reviewers?

Volunteers access an online training module, which includes a guide on how to rate publications against the ARRIVE guidelines, and a series of training papers. It takes roughly one hour to review each paper, and they can review as many as they like, provided that they reach a predetermined level of agreement (which is not as high as 100%) with the assessment of a panel of expert reviewers. The training is a good opportunity to deepen your understanding of the importance of good reporting practices to the research community. Reviewers who complete their training are provided with a certificate so that they can demonstrate continuing professional development.

Are reviewers assigned articles to review or are they expected to choose their own?

Every time a reviewer starts a review, that article is selected at random.

What if a reviewer makes a mistake when assessing a paper?

Reviewers are able to change their assessments right up until they press the final “Submit” button. We follow the same protocol as we use for systematic reviews, where papers are scored by two independent reviewers and any discrepancies in their scores resolved by a third person. This helps us make sure that if there have been mistakes made, they will be identified and corrected.

Can anyone sign up to the project? Are you looking for any skills in particular?

Anyone can sign up and take the training module. Only reviewers who successfully complete the training module will be allocated papers to review. And we are about a quarter of the way through the outcome assessments at present, so there is plenty of time to get involved.

Aside from being a part of an important project, are there other incentives to get involved?

We think that the structured training in assessing articles will be useful for people who do or are planning to do peer reviews, and for people writing their own papers. All volunteers that score manuscripts will be listed as collaborators on the resulting papers from the project, which means they will come up under their name in PubMed (in the same way that doctors who recruit patients to clinical trials are recognised). Reviewers can also earn IICARus prizes, including for instance a Kindle or iPad mini, or even the star prize of a special limited edition CAMARADES memory stick!

How do you see the results of this study being used?

This work will tell us if mandating the completion of an ARRIVE checklist improves the quality of reporting in published papers, which items on the ARRIVE guidelines are most frequently reported or missed in manuscripts, and which are most resistant to interventions attempting to improve compliance. So, it will guide future efforts to improve reporting.

If even a policy as direct as rejecting manuscripts which do not provide a completed ARRIVE checklist does not deliver full compliance with ARRIVE, we are interested in why this might be. NC3Rs is therefore planning a survey to the research community to explore the reasons for good or bad compliance with the ARRIVE guidelines. For example, is poor compliance solely due to poor reporting practices, or does it actually reflect that researchers are not considering some of these issues during the design and conduct of the experiment.

Taken together, data from the IICARus study and the survey will help the NC3Rs advance its strategy to increase dissemination and uptake of the ARRIVE guidelines and improve reporting of animal research. These data will also be used to assess whether the ARRIVE guidelines should be revised, and if so, where change is needed.