Good healthcare policy must be based on high-quality evidence. But policymakers must often make decisions in fields where they may lack specialist expertise. So, how do they know what evidence to look for, and how to assess it? 

Systematic evidence reviews by professional researchers help policymakers navigate complex scientific topics by:

  1. Identifying relevant studies
  2. Evaluating their quality
  3. Summarising the evidence in a clear and practical way. 

The HRB Evidence Centre conducts such reviews for the Department of Health in diverse areas including community water fluoridation, electronic cigarettes, and assisted living for older people. Next, we explore seven key steps in the process. 

1.    Ask the right question 

Defining a clear question is the essential first step in producing a quality evidence review so you can get the answers you need. By using what is known as a PICO format, you can translate a free-form enquiry from a policymaker into a structured research question. PICO stands for: 

  • Population, patient, or problem: Who is the group or what is the problem we are looking at?
  • Intervention: What is the intervention or action being considered?
  • Comparator: What alternative can we compare with the intervention (note – this may be ‘doing nothing’!)
  • Outcome: What is the outcome we are interested in?

For example, the HRB’s recent review of research on the health effects of community water fluoridation asked: What is the impact on the systemic health (O) of the human population (P) for those exposed to artificially fluoridated water between 0.4 and 1.5 ppm (I), compared with non-fluoridated water (less than 0.3 ppm) (C)?

When formulating your PICO question, simultaneously doing an initial scope of the literature will ensure there is sufficient material available for a robust answer. 

2.    Methodology matters 

With your PICO question set, the next step is deciding on the best methodological approach to answer it. Each methodology has its own strengths, for example: 

  • Quantitative approaches such as systematic narrative review or systematic meta-analysis review: these are suitable for synthesising data presented statistically, like findings from clinical trials. 
  • Qualitative approaches such as thematic analysis review or meta ethnography review: these are suitable for synthesising research on themes found in qualitative studies of written texts or interviews. For example, research on patient experiences of health interventions or how culture influences healthcare choices.
  • Mixed methods review: this is a review of a mix of quantitative and qualitative research. It may cover patient experience of a complex intervention and the effectiveness of the intervention.
  • Umbrella review: this goes one step further to provide an overview of all systematic reviews on a given topic. 

Once you have selected the methodology, you should draw up a protocol and timeline detailing your PICO question, methodology and approach to analysis, and the process to be followed by the research team. 

3.    Sifting through scientific papers

If you imagine that producing an evidence review means hundreds of hours buried in literature… you’re right! To find all relevant research, you must do an extensive search of at least two (and usually many more) bibliographical databases, trawl relevant sources for grey literature (material produced outside conventional publishing channels), and carefully scan citations in the texts you consult for breadcrumbs towards additional research. 

PICO in hand – and fuelled by a strong coffee – next you will need to go through each piece of literature multiple times. Start with the abstract, then the full text, and then further in-depth readings as often as required. 

Web-based software such as Eppi-Reviewer can help you manage the data you gather (abstracts, full texts, inclusions, exclusions and their reasons, and data to prepare PRISMA flow charts). And an electronic reference management system such as Zotero or EndNote is indispensable to keep tabs on all literature and enable seamless referencing in your final evidence review.

4.     Digging out the data 

After identifying the relevant literature, it’s time to dig out the data you need from the studies to answer your PICO question. When developing your protocol, you will have created a data extraction form. This specifies the information you need to extract from each research paper, allowing you to gather the data for your review in a structured way, ready for evaluation.

Microsoft Excel or similar is essential to manage these large data tables, covering topics like: 

  • Characteristics of the included studies
  • Quality assessment of primary studies
  • Summary results
  • Certainty of evidence for outcomes or themes. 

5.    Evaluating the evidence – and busting the bias

The next step is to thoroughly evaluate and assess the quality of the data you have gathered. There is a range of tried and tested ‘risk of bias’ tools for evaluating the quality of the included research studies. However, bias in your data is not the only way that it can creep into your review – the reviewers themselves need to leave preconceptions at the door and follow where the evidence takes them. One way to keep this bias at bay is to have at least two researchers assess the quality of the research papers and accuracy of the data in parallel, but independently, reconciling any disagreements along the way. 

6.    Making sense of the mass of evidence

This is where the magic happens: it’s time to synthesise the vast volumes of information you have gathered and make meaning from it. You will have defined a method of analysis to do this when setting out your protocol, informed by the type of review. A meta-analysis may work best for quantitative data. Whereas for qualitative data, you may be looking at thematic synthesis or meta-ethnography. The skills to distil these findings into a coherent review with valid conclusions are hard-won over many years of academic training and professional research experience. 

7.    Spreading the word 

With your review completed, it’s time to share it. While we researchers may dream of our reports being read cover to cover, the reality is that most readers are short on time and need to see the key findings at a glance. This means a clear, concise and accurate executive summary is essential.

Whether you’re talking to policymakers, media, or the general public, your readers will only be scraping the surface of a subject you have been immersed in for months. So, keep your language straightforward, and avoid jargon and abbreviations. And if you are planning to share your findings more broadly with the public, seek out support from communications professionals who can help you translate your research into everyday language and share it on the appropriate channels. 

ENDS

Read more about the HRB Evidence Centre