#InsideTheBox is a column by Dr Andy Gray, a pharmaceutical sciences expert at the University of KwaZulu-Natal and Co-Director of the WHO Collaborating Centre on Pharmaceutical Policy and Evidence Based Practice. (Photo: Supplied)Comment & Analysis
18th June 2025 | Andy Gray
It is easy to claim one is “following the scientific evidence”, but what does it mean to actually do so? In his latest #InsideTheBox column, Dr Andy Gray considers how evidence-based health policy and guidelines can and should be made.
The recent firing of the US Centers for Disease Control and Prevention advisory body on immunisation policy, allegedly because of unmanaged conflicts of interest, has again raised the question of what truly constitutes evidence-based policies or guidelines. The US Secretary of Health and Human Services has been quoted as saying that a new committee was necessary “to re-establish public confidence in vaccine science” and that it would be “committed to evidence-based medicine, gold-standard science, and common sense”.
An oft-cited definition of evidence-based medicine is that proposed by David Sackett and colleagues in 1996. “Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients,” they wrote in an editorial in the British Medical Journal. The practice of evidence-based medicine, the authors added, required “integrating individual clinical expertise with the best available external clinical evidence from systematic research”. Although this definition focused on the management of individual patients, the same components should inform population-level decisions, expressed as official policies or clinical practice guidelines, such as the schedule of immunisations recommended across the lifespan.
However, what defines “the best available external clinical evidence from systematic research”? How should such evidence be identified, assessed for quality, and then incorporated into decision-making processes?
PICO questions
The first step is crucial – converting an information need into an answerable question. For example, a general question about who should receive COVID-19 booster vaccinations could be sharpened in the following manner using the PICO approach: In adults with additional risk factors (the “P” for patient or population), what is the effect of annual vaccination with COVID-19 vaccines adapted to circulating variants (the “I” for intervention), compared to placebo or no vaccination (the “C” for comparator) on the risk of hospitalisation following diagnosis of a COVID-19 infection (the “O” for outcome).
Using this PICO framework, the evidence can be sought in the medical literature. Evidence from clinical trials involving adult patients with the identified additional risk factors (such as chronic diseases), given such adapted vaccines, could be identified. In particular, clinical trials could be identified which measured the outcome of interest.
Where to look for evidence
The international medical literature is vast and growing. The US National Library of Medicine’s database, known as PubMed, now includes more than 38 million citations. Searching that many citations pose significant challenges. For example, simply searching for any literature on COVID-19 retrieves over 480 000 citations, limiting to papers about COVID-19 vaccines still shows over 55 000 citations. While more sophisticated search strategies can limit the number of sources to consider, a more deliberate set of methods is needed to evaluate the certainty of the evidence, and therefore the confidence with which decisions can be made based on that evidence.
In the past, simplistic hierarchies of evidence placed systematic reviews of randomised controlled trials (RCTs) above individual RCTs, and those above data from purely observational studies. But increasingly, policy and guideline development bodies are using the more nuanced Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach.
In a prospective RCT, participants are randomly assigned to receive either the intervention (a procedure, medicine or vaccine) or a comparator (either a placebo or the usual standard of care or a deliberately chosen alternative), and then followed for a pre-specified period in order to detect the outcomes of interest. Those outcomes may be in relation to the efficacy of the intervention (whether it works or not) as well as its safety (whether unwanted adverse effects are detected). Participants or the clinicians caring for them may be blinded to whether they have received the intervention or the comparator. In a double-blind trial, both participants and clinicians are blinded, thus avoiding biases in the detection of the effects, whether wanted or unwanted. This is particularly important when the endpoint of the trial is subjective (such as pain) and cannot be measured objectively with a laboratory or other test.
In the example provided above, the endpoint is admission to hospital following a diagnosis of COVID-19. Although hospitalisation records are objective and can be documented, the decision to admit a patient to hospital may vary between settings.
There are times, however, when RCTs are not the best way to detect important evidence. Participants in RCTs are usually carefully chosen, for example to exclude the very young and very old, those with pre-existing conditions and on other medication, and in particular, pregnant people. Although some RCTs may enrol thousands of participants at multiple trial sites in different countries, they are still unable to detect rare adverse effects that may occur in higher-risk groups of patients. Where the primary interest is in the evidence for safety, carefully conducted observational studies can provide high quality evidence generated in millions of patients exposed to the intervention of interest. For example, a cohort study conducted in 8 countries was able to track adverse events associated with three different COVID-19 vaccines in 99 million vaccinated people.
Related Posts
Observational studies involve the gathering of data from patients where the investigators did not determine the exposure to the intervention. Observational studies may also include comparator groups, not exposed to the intervention of interest. However, unlike RCTs they cannot ensure that all factors that may influence the outcome, both known and unknown, are equally distributed between the groups studied.
Using the GRADE approach, the validity of the evidence can be assessed, as well as its applicability outside of the particular settings in which the RCT or observational study was conducted. For example, an RCT conducted in a high-income country with no financial or other barriers to hospitalisation may represent a very different reality from the situation in a low-income country where hospital beds are limited and out-of-pocket payment is demanded.
An RCT with a small sample size and a very imprecise estimate of the effect of interest may be downgraded in quality. Where the evidence retrieved is rated as high quality, further research is unlikely to change the estimate of the effect, and a recommendation can be made with confidence. Where the evidence is rated as low or very low quality, further research is very likely to change the estimate of effect, and there can be less confidence in making a recommendation. Instead, a suggestion may be made, which could be revisited once more evidence is generated.
What to do
So, how can we tell whether policies or guidelines are truly evidence-based?
First, we can ask what the PICO question was, then how the literature was searched and appraised for validity and applicability, and then whether other relevant factors were considered in the evidence-to-decision process. Patients’ values and preferences, feasibility, resource demands and equity considerations are examples of such factors.
In addition, we can ask who funded the development of the document and how potential conflicts of interest were managed. No expert is ever devoid of all potential conflicts, especially if they are active researchers in the particular field. What matters, however, is how those potential conflicts are documented, disclosed and managed.
The more widely the policy or guidelines will be applied, the greater the need for high-quality, explicitly documented and transparent processes. In the South African context, the benefit packages, practice and reimbursement guidelines developed by the National Health Insurance Fund will need to meet the highest standards if they are to be trusted by patients, their families and caregivers, and health professionals.
*Dr Gray is a Senior Lecturer at the University of KwaZulu-Natal and Co-Director of the WHO Collaborating Centre on Pharmaceutical Policy and Evidence Based Practice. This is the fourth of a new series of #InsideTheBox columns he is writing for Spotlight.
Disclosure: Gray is a member of South Africa’s National Essential Medicines List Committee and co-chairs its Expert Review Committee.
Note: Spotlight aims to deepen public understanding of important health issues by publishing a variety of views on its opinion pages. The views expressed in this article are not necessarily shared by the Spotlight editors.
