blog

How does the media report on health interventions? We spoke to F1000Research author Matt Oxman, a PhD candidate at the Norwegian Institute of Public Health, about the need for all parties to approach research (and news reports on it) critically. Read the interview below to learn more about his findings from a Systematic Review of the literature.

Matt, thank you for taking the time to talk to us about your work. Can you tell us about your background and your field of research?

I have an MSc in Evidence-Based Health Care from the University of Oxford, and a Bachelor of Journalism from Ryerson University.

Since 2014, I have worked on the Informed Health Choices (IHC) project, in different capacities. The project goal is to help people think critically about health care, so they avoid waste and unnecessary suffering. Previously, we have developed and evaluated primary school resources. Currently, we are developing secondary school resources.

What inspired you to review the quality of media reports on health interventions?

There is endless information about what is healthy or unhealthy, and a lot of it is unreliable. This is the first part of the problem that IHC addresses.

We knew that there were many primary studies of the quality of health information in mass media—especially information in news media about the effects of health interventions, including medical interventions. We often cited a couple of the larger studies when presenting the project. However, we were not aware of any systematic review of such studies.

We thought it was important to fill the gap, for several reasons. For example, it could inform new studies of health information in mass media. I was also motivated by my background in journalism.

What were some of your key findings?

Some of our findings were about the types of health news reports that researchers have studied. Some were about the methods that researchers have used to assess the quality of information in media reports about health interventions. And of course, some of the findings were about the quality of health news reports.  

We found more than 40 eligible studies. There was variation both in terms of the samples of news reports, and the methods used to assess the quality of information. However, most of the studies included reports from newspapers, reports from English-speaking countries, or reports about “modern” medicines.

What made the study particularly challenging and interesting is that researchers have used many different criteria to assess quality. Moreover, we disagreed with some of the criteria, and we thought some criteria were missing. For example, news reports sometimes include average differences between health interventions, which can be misleading, but we did not find any criteria about reporting average differences. 

We found that there are several prevalent and important problems with the quality of the information in news reports about the effects of health interventions. In many reports, the information is unbalanced or oversimplified. For example, reporters often fail to report absolute differences between health interventions (relative differences between health interventions can also be misleading) or fail to quantify effects at all.  

What are the implications of this for journalists?

Table 5 in the article summarizes the implications for different groups, including journalists, as well as editors and news outlets. For journalists, the main implication is that they should approach research critically.

However, this is often easier said than done due to systemic problems, such as limited time to prepare stories. This is part of why we argue that the most important implication of our study and related research is that the public learns to think critically about health information. 

For journalists and others who want to improve their ability to think critically about health research, I recommend starting with the short and free book Testing Treatments, or one of the resources on the IHC website, which is also free. 

You discuss the use of the education system to teach young people to critically assess information. How important are the humanities and social sciences here?

It is important to learn to think critically about information within whatever subject you are studying, including the humanities and social sciences. In fact, when it comes to claims or research evidence about the effects of interventions, we have found that many of the same fundamental concepts apply to a variety of fields, from agriculture to veterinary medicine. Teaching critical thinking across subjects might have a synergetic effect.

Do researchers also have a role to play in ensuring the quality of information?

Absolutely—and this is especially important when research is published openly and is accessible to the wider public The main implication for researchers, journals, academic institutions, and funders of science, is to reduce false findings and waste. This is no small feat either, but there has been important progress. 

One of your peer reviewers notes that the media landscape may have changed due to COVID-19. If you were to revisit this Systematic Review today, would you do anything differently?

I have seen a couple of studies of the quality of COVID-19 information in the mass media, but not focusing on information in news reports about the effects of interventions. In other words, I have not seen any studies published during the pandemic that would have been eligible for the review.

My biased impression is that news coverage of the effects of COVID-19 interventions has generally suffered the same problems as those we found in our review. This is what I would expect, given the underlying problems that remain unsolved, such as the incentive to sensationalize. Some coverage has been reliable and informative, but this was also the case with health news before the pandemic.

Why did you choose to publish on an open research publishing platform?

We chose an open research publishing platform so anyone can see the first version of the article and the peer reviews, as well as the coming responses to the peer reviews and second version of the article. Also, it is helpful to be able to read research quickly without the delays of traditional peer review, which is often a time-consuming process.

Want to learn more about the quality of media reports on health interventions? Read the full Systematic Review on F1000Research today.