A thorough and up-to-date reference to be consulted for vaccine safety assessment is the ADVANCE Report on appraisal of vaccine safety methods. Together with a large number of relevant references, it provides a brief description of a very wide range of direct and indirect methods of risk assessment for vaccines (listed in the Table of Contents) and evaluates them based on 9 criteria related to five domains: Effect Measure, Statistical Criteria, Timeliness, Restriction and Robustness, and Operational Criteria. It also emphasises the specificities of safety assessment for vaccines and how they differ from other pharmaceutical drugs, evaluates study designs, discusses perspectives of different stakeholders on risk assessment, describes experiences from other projects and systems, and provides recommendations. This document is highly relevant for all the topics covered in this chapter on vaccine safety.
Specific aspects related to vaccine safety are discussed in several other documents.
The Report of the CIOMS/WHO Working Group on Definition and
Application of Terms for Vaccine Pharmacovigilance (2012) provides
definitions and explanatory notes for the terms ‘vaccine pharmacovigilance’,
‘vaccination failure’ and ‘adverse event following immunisation (AEFI)’.
The CIOMS Guide to
Active Vaccine Safety Surveillance (2017) describes the process of
determining whether active vaccine safety surveillance is necessary, more
specifically in the context of resource-limited countries, and, if so, of
choosing the best type of active safety surveillance and considering key
The CIOMS Guide to Vaccine Safety Communication (2018) provides an
overview of strategic communication issues faced by regulators, those
responsible for vaccination policies and other stakeholders in introducing
current or new vaccines in populations. Building upon existing recommendations,
it provides a guide for vaccine risk safety communication
Collaboration provides resources to facilitate and harmonise collection, analysis and
presentation of vaccine safety data, including case definitions, an electronic
tool to help the classification of reported signs and symptoms, template
protocols and guidelines.
Module 4 (Surveillance) of the e-learning training course Vaccine Safety
Basics of the World Health Organization (WHO) describes phamacovigilance
principles, causality assessment procedures, surveillance systems and factors
influencing the risk-benefit balance of vaccines. In particular, in contrast to
the use of other medicines, vaccines are often used in healthy people and it is
not only important to identify possible risks but also to emphasize safety if it
does exist. For example a systematic review on influenza vaccination in
pregnancy and the risk of congenital anomalies in newborns did not find an
association, adding to the evidence base of influenza vaccination in pregnancy
Influenza Vaccination and Risk for Congenital Malformations: A Systematic Review
and Meta-analysis. Obstet Gynecol 2015;126(5):1075-84.).
Recommendations on vaccine-specific aspects of the EU pharmacovigilance system, including on risk management, signal detection and post-authorisation safety studies (PASS) are presented in the Module P.I: Vaccines for prophylaxis against infectious diseases of the Good pharmacovigilance practices (GVP).
Aside from a qualitative analysis of spontaneous case reports or case series, quantitative methods such as disproportionality analyses and observed vs. expected (O/E) analyses are routinely employed in signal detection for vaccines. Several documents discuss the merits and review the methods of these approaches.
GVP Module P.I: Vaccines for prophylaxis against infectious diseases describes issues to be considered when applying methods for disproportionality analyses for vaccines, including the choice of the comparator group and the use of stratification. Effects of stratification on data mining in the US Vaccine Adverse Event Reporting System (VAERS) (Drug Saf 2008;31(8):667-74) demonstrates that stratification can reveal and reduce confounding and unmask some vaccine-event pairs not found by crude analyses. However, Stratification for Spontaneous Report Databases (Drug Saf 2008;31(11):1049-52) highlights that extensive use of stratification in signal detection algorithms should be avoided as it can mask true signals. Vaccine-Based Subgroup Analysis in VigiBase: Effect on Sensitivity in Paediatric Signal Detection (Drug Saf 2012;35(4)335-46) further examines the effects of subgroup analyses based on the relative distribution of vaccine/non-vaccine reports in paediatric ADR data.
The article Optimization of a quantitative signal detection algorithm for spontaneous reports of adverse events post immunization (Pharmacoepidemiol Drug Saf 2013; 22(5): 477–87) explores various ways of improving performance of signal detection algorithms when looking for vaccines.
The article Adverse events associated with pandemic influenza vaccines: comparison of the results of a follow-up study with those coming from spontaneous reporting (Vaccine 2011;29(3):519-22) reported a more complete pattern of reactions when using two complementary methods for first characterisation of the post-marketing safety profile of a new vaccine, which may impact on signal detection.
When prompt decision-making about a safety concern is required and there is insufficient time to review individual cases, GVP Module P.I: Vaccines for prophylaxis against infectious diseases suggests the conduct of O/E analyses for signal validation and preliminary signal evaluation. The module discusses key requirements of O/E analyses: the observed number of cases detected in a passive or active surveillance systems, near real-time exposure data, appropriately stratified background incidence rates (to calculate the expected number of cases) and sensitivity analyses around these measures. O/E analyses for vaccines are further discussed in Pharmacoepidemiological considerations in observed‐to‐expected analyses for vaccines (Pharmacoepidemiol Drug Saf 2016;25(2): 215-22) and are also addressed in the review Near real‐time vaccine safety surveillance using electronic health records—a systematic review of the application of statistical methods (Pharmacoepidemiol Drug Saf 2016;25(3):225-37).
Simple ‘snapshot’ O/E analyses require near-real-time exposure data, appropriately stratified background incidence rates (to calculate the expected number of cases) and sensitivity analyses around these measures, and they may not be appropriate for continuous monitoring due to inflation of type 1 error rates when multiple tests are performed. Safety monitoring of Influenza A/H1N1 pandemic vaccines in EudraVigilance (Vaccine 2011;29(26):4378-87) illustrates that simple ‘snapshot’ O/E analyses are affected by uncertainties regarding the numbers of vaccinated individuals and age-specific background incidence rates.
Human papilloma virus immunization in adolescents and young adults: a cohort study to illustrate what events might be mistaken for adverse reactions (Pediatr Infect Dis J 2007;26(11):979-84) and Health problems most commonly diagnosed among young female patients during visits to general practitioners and gynecologists in France before the initiation of the human papillomavirus vaccination program (Pharmacoepidemiol Drug Saf 2012; 21(3):261-80) illustrate the importance of collecting background rates by estimating risks of coincident associations of emergency consultations, hospitalisations and outpatients consultations with vaccination. Rates of selected disease events for several countries also vary by age, sex, method of ascertainment and geography, as shown in Importance of background rates of disease in assessment of vaccine safety during mass immunisation with pandemic H1N1 influenza vaccines (Lancet 2009; 374(9707):2115-22). Moreover, Guillain-Barré syndrome and influenza vaccines: A meta-analysis (Vaccine 2015; 33(31):3773-8) suggests that a trend observed between different geographical areas would be consistent with a different susceptibility of developing a particular adverse reaction among different populations.
Sequential methods, as described in Early detection of adverse drug events within population-based health networks: application of sequential methods (Pharmacoepidemiol Drug Saf 2007; 16(12):1275-1284), allow O/E analyses to be performed on a routine (e.g. weekly) basis using cumulative data with adjustment for multiplicity. Such methods are routinely used for near-real time surveillance in the Vaccine Safety Datalink (VSD) (Near real-time surveillance for influenza vaccine safety: proof-of-concept in the Vaccine Safety Datalink Project. Am J Epidemiol 2010;171(2):177-88). Potential issues are described in Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data (Pharmacoepidemiol Drug Saf 2012;21(S1):62-71). A review of signals detected over 3 years with these methods in Vaccine Safety Datalink concluded that care with data quality, outcome definitions, comparison groups and length of surveillance is required to enable detection of true safety problems while controlling error rates (Active surveillance for adverse events: the experience of the Vaccine Safety Datalink Project (Pediatrics 2011;127(S1):S54-S64)). Sequential methods are, therefore, more robust but also more complex to perform, understand and communicate to a non-statistical audience.
A new self-controlled case series method for analyzing spontaneous reports of adverse events after vaccination (Am J Epidemiol 2013;178(9):1496-504) extends the self-controlled case series approach to explore and quantify vaccine safety signals from spontaneous reports. It uses parametric and nonparametric versions with different assumptions to account for the specific features of the data (e.g., large amount of underreporting and variation of reporting with time since vaccination). The method should be seen as a signal strengthening approach for quickly exploring a signal based on spontaneous reports prior to a pharmacoepidemiologic study, if any. The method was used to document the risk of intussusception after rotavirus vaccines (see Intussusception after Rotavirus Vaccination — Spontaneous Reports; N Engl J Med 2011; 365:2139).
A complete review of study designs and methods from hypothesis testing studies in the field of vaccine safety is included in the ADVANCE Report on appraisal of vaccine safety methods.
Traditional study designs such as cohort and case-control studies may be difficult to implement for vaccines where studies involve populations with high vaccine coverage rates, an appropriate unvaccinated group is lacking or adequate information on covariates at the individual level is not available. Frequent sources of confounding to be considered are socioeconomic status, underlying health status and other factors influencing the probability of being vaccinated. Control without separate controls: evaluation of vaccine safety using case-only methods (Vaccine 2004; 22(15-16):2064-70) describes and illustrates epidemiological methods that are useful in such situations. They are mostly case-only design described in Chapter 5.3.2 of the Guide:
The case-crossover design was
primarily developed to investigate the association between a vaccine and an
adverse event. In this design, control information for each case is based on own
past exposure experience and a person can ‘crossover’ between two or more
exposure levels. It is a retrospective design that requires the strong
assumption that the underlying probability of vaccination should be the same in
all defined time intervals, but this is unlikely to hold for paediatric vaccines
administered according to strict schedules or for seasonally administered
The self-controlled case series
(SCCS) design can be both prospective and retrospective and aims to estimate a
relative incidence, which compares the incidence of adverse events within
periods of hypothesised excess risk due to exposure with incidence during all
other times (baseline risk).
The case-coverage design uses exposure information on cases and population data on vaccination coverage to serve as control. It requires reliable and detailed vaccine coverage data corresponding to the population from which cases are drawn. This will allow control of confounding by stratified analysis. During vaccine introduction, it is also particularly important to address selection bias introduced by awareness of possible occurrence of a specific outcome. An example of a study using a case-coverage method is Risk of narcolepsy in children and young people receiving AS03 adjuvanted pandemic A/H1N1 2009 influenza vaccine: retrospective analysis (BMJ 2013; 346:f794).
The study Control without separate controls: evaluation of vaccine safety using case-only methods (Vaccine 2004; 22(15-6):2064-70) concludes that properly designed and analysed epidemiological studies using only cases, especially the SCCS method, may provide stronger evidence than large cohort studies as they control completely for fixed individual-level confounders (such as demographics, genetics and social deprivation) and typically have similar, sometimes better, power. Three factors are however critical in making optimal use of such methods: access to good data on cases, computerised vaccination records with the ability to link them to cases and availability of appropriate analysis techniques.
Several studies on vaccines have compared traditional and case-only study designs:
Epidemiological designs for vaccine safety assessment: methods and
pitfalls (Biologicals 2012;40(5):389-92) used three study designs (cohort,
case-control and self-controlled case series) to illustrate the issues that may
arise when designing an epidemiological study, such as understanding the vaccine
safety question, case definition and finding, limitations of data sources,
uncontrolled confounding, and pitfalls that apply to the individual
epidemiologic methods for active surveillance of vaccine safety (Vaccine
2008; 26(26):3341-3345) performed a simulation study to compare four designs
(matched-cohort, vaccinated-only (risk interval) cohort, case-control and
self-controlled case series) in the context of vaccine safety surveillance. The
cohort study design allowed for the most rapid signal detection, the least
false-positive error and highest statistical power in performing sequential
analysis. The authors highlight, however, that the chief limitation of this
simulation is the exclusion of confounding effects and the lack of chart review,
which is a time and resource intensive requirement.
Another simulation study (Four different
study designs to evaluate vaccine safety were equally validated with contrasting
limitations. J Clin Epidemiol 2006; 59(8):808-818) compared four study
designs (cohort, case-control, risk-interval and SCCS) with the conclusion that
all the methods were valid designs, with contrasting strengths and weaknesses.
The SCCS method, in particular, proved to be an efficient and valid alternative
to the cohort method.
Hepatitis B vaccination and first central nervous system demyelinating events: Reanalysis of a case-control study using the self-controlled case series method. Vaccine 2007;25(31):5938-43) describes how the SCCS found similar results as the case-control study but with greater precision as it used cases without matched controls excluded from the case-control analysis. This is at the cost of the assumption that exposures are independent of earlier events. The authors recommended that, if case-control studies of vaccination and adverse events are undertaken, parallel case-series analyses should also be conducted, where appropriate.
In situations where primary data collection is needed (e.g. a pandemic), the SCCS may not be adequate since follow-up time needs to be accrued. In such instances, the Self-controlled Risk Interval (SCRI) method can be used to shorten the observation time (see The risk of Guillain-Barre Syndrome associated with influenza A (H1N1) 2009 monovalent vaccine and 2009-2010 seasonal influenza vaccines: Results from self-controlled analyses. Pharmacoepidemiol Drug Saf 2012;21(5):546-52), historical background rates can be used for an O/E analysis (see Near real-time surveillance for influenza vaccine safety: proof-of-concept in the Vaccine Safety Datalink Project. Am J Epidemiol 2010;171(2):177-88) or a classical case-control study can be performed, as used in Guillain-Barré syndrome and adjuvanted pandemic influenza A (H1N1) 2009 vaccine: multinational case-control study in Europe. BMJ 2011;343:d3908).
Ecological analyses should not be considered hypothesis testing studies. See Chapter 5.5 of this Guide.
A systematic review evaluating the potential for bias and the methodological quality of meta-analyses in vaccinology (Vaccine 2007; 25(52):8794-806) provides a comprehensive overview of the methodological quality and limitations of 121 meta-analyses of vaccine studies. Association between Guillain-Barré syndrome and influenza A (H1N1) 2009 monovalent inactivated vaccines in the USA: a meta-analysis (Lancet 2013;381(9876):1461-8) describes a self-controlled risk-interval design in a meta-analysis of six studies at the patient level with a reclassification of cases according to the Brighton Collaboration classification.
The article Vaccine safety in special populations (Hum Vaccin 2011;7(2):269-71) highlights common methodological issues that may arise in evaluating vaccine safety in special populations, especially infants and children who often differ in important ways from healthy individuals and change rapidly during the first few years of life, and elderly patients.
Observational studies on vaccine adverse effects during pregnancy (especially on pregnancy loss), which often use pregnancy registries or healthcare databases, are faced with three challenges: embryonic and early foetal loss are often not recognised or recorded, data on the gestational age at which these events occur are often missing, and the likelihood of vaccination increases with gestational age whereas the likelihood of foetal death decreases. Assessing the effect of vaccine on spontaneous abortion using time-dependent covariates Cox models (Pharmacoepidemiol Drug Saf 2012;21(8):844-850) demonstrates that rates of spontaneous abortion can be severely underestimated without survival analysis techniques using time-dependent covariates to avoid immortal time bias and shows how to fit such models. Risk of miscarriage with bivalent vaccine against human papillomavirus (HPV) types 16 and 18: pooled analysis of two randomised controlled trials (BMJ 2010; 340:c712) explains methods to calculate rates of miscarriage, address the lack of knowledge of time of conception during which vaccination might confer risk and perform subgroup and sensitivity analyses.
In Harmonising Immunisation Safety Assessment in Pregnancy (Vaccine 2016;34 (49): 5991-6110; Vaccine 2017;35 (48), 6469-582), the Global Alignment of Immunization Safety Assessment in pregnancy (GAIA) project has provided a selection of case definitions and guidelines for the evaluation of pregnancy outcomes following immunization. The Systematic overview of data sources for Drug Safety in pregnancy research provides an inventory of pregnancy exposure registries and alternative data sources useful to assess the safety of prenatal vaccine exposure.
Few vaccine studies are performed in immunocompromised subjects. Influenza vaccination for immunocompromised patients: systematic review and meta-analysis by etiology (J Infect Dis 2012;206(8):1250-9) illustrates the importance of performing stratified analyses by aetiology of immunocompromise and possible limitations due to residual confounding, differences within and between etiological groups and small sample size in some etiological groups. Further research is needed on this topic.
There is an increasing interest in the influence of genetics on safety and efficacy outcomes of vaccinations. Understanding this influence may optimise the choice of vaccines and the vaccination schedule. Research in this field is illustrated by Effects of vaccines in patients with sickle cell disease: a systematic review protocol (BMJ Open 2018;8:e021140. doi:10.1136/bmjopen-2017-021140).
Vaccine effects and impact of vaccination programmes in post-licensure studies (Vaccine 2013;31(48):5634-42) reviews effectiveness of vaccine and of vaccination programmes, proposes epidemiological measures of public health impact, describes relevant methods to measure these effects and discusses the assumptions and potential biases involved.
Generic protocols for retrospective case-control studies and retrospective cohort studies to assess the effectiveness of rotavirus vaccination in EU Member States based on computerised databases were published by the European Centre for Disease Prevention and Control (ECDC). They describe the information that should be collected by country and region in vaccine effectiveness studies and the data sources that may be available to identify virus-related outcomes a vaccine is intended to avert, including hospital registers, computerised primary care databases, specific surveillance systems (i.e. laboratory surveillance, hospital surveillance, primary care surveillance) and laboratory registers. Based on a meta-analysis comprising 49 cohort studies and 10 case-control studies, Efficacy and effectiveness of influenza vaccines in elderly people: a systematic review (Lancet 2005;366(9492):1165-74) highlights the heterogeneity of outcomes and study populations included in such studies and the high likelihood of selection bias.
Non-specific effects of vaccines, such as a decrease of mortality, have been claimed in observational studies but generally can be affected by bias and confounding. Epidemiological studies of the 'non-specific effects' of vaccines: I--data collection in observational studies (Trop Med Int Health 2009;14(9):969-76.) and Epidemiological studies of the non-specific effects of vaccines: II--methodological issues in the design and analysis of cohort studies (Trop Med Int Health 2009;14(9):977-85) provide recommendations for vaccine observational studies conducted in countries with high mortality; these recommendations have wider relevance.
The screening method estimates vaccine effectiveness by comparing vaccination coverage in positive cases of a disease (e.g. influenza) with the vaccination coverage in the population from which the cases are derived (e.g., the same age group). If representative data on cases and vaccination coverage are available, it can provide an inexpensive and ready-to-use method that can be useful in providing early effectiveness estimates or identify changes in effectiveness over time. However, Application of the screening method to monitor influenza vaccine effectiveness among the elderly in Germany (BMC Infect Dis. 2015;15(1):137) emphasises that accurate and age-specific vaccine coverage rates are crucial to provide valid VE estimates. Since adjusting for important confounders and the assessment of product-specific VE is generally not possible, this method should be considered only a supplementary tool for assessing crude VE.
The indirect cohort method is a case-control type design which uses cases caused by non-vaccine serotypes as controls. Use of surveillance data to estimate the effectiveness of the 7-valent conjugate pneumococcal vaccine in children less than 5 years of age over a 9 year period (Vaccine 2012;30(27):4067-72) applied this method to evaluate the effectiveness of a pneumococcal conjugate vaccine against invasive pneumococcal disease (IPD) and compared the results to the effectiveness measured using a standard case-control study conducted during the same time period. The authors considered the method would be most useful shortly after vaccine introduction, and less useful in a setting of very high vaccine coverage and fewer vaccine-type cases. Using the Indirect Cohort Design to Estimate the Effectiveness of the Seven Valent Pneumococcal Conjugate Vaccine in England and Wales (PLoS One 6(12):e28435. doi:10.1371/journal.pone.0028435) describes how the method was used to estimate effectiveness of various numbers of doses as well as for each vaccine serotype.
Effectiveness of live-attenuated Japanese encephalitis vaccine (SA14-14-2): a case-control study (Lancet 1996;347(9015):1583-6) describes a case control study of incident cases in which the control group consisted of all village-matched children of a given age who were at risk of developing disease at the time that the case occurred (density sampling). The effect measured is an incidence density rate ratio.
The article The test-negative design for estimating influenza vaccine effectiveness (Vaccine 2013;31(17):2165-8) explains the rationale, assumptions and analysis of the test-negative study as applied to influenza VE. Study subjects are all persons who seek care for an acute respiratory illness and influenza VE is estimated from the ratio of the odds of vaccination among subjects testing positive for influenza to the odds of vaccination among subject testing negative. This design is less susceptible to bias due to misclassification of infection and the confounding by health care-seeking behaviour, at the cost of difficult-to-test assumptions.
Effectiveness of rotavirus vaccines in preventing cases and hospitalizations due to rotavirus gastroenteritis in Navarre, Spain (Vaccine 2012;30(3):539-43) evaluates effectiveness using a test negative case-control design based on electronic clinical reports. Cases were children with confirmed rotavirus and controls were those who tested negative for rotavirus in all samples. The test-negative design was based on an assumption that the rate of gastroenteritis caused by pathogens other than rotavirus is the same in both vaccinated and unvaccinated persons. This approach may rule out differences in parental attitude when seeking medical care and of physician differences in making decisions about stool sampling or hospitalisation. A limitation is sensitivity of antigen detection which may underestimate vaccine effectiveness. In addition, if virus serotype is not available, it is not possible to study the association between vaccine failure and a possible mismatch of vaccine strains and circulating strains of virus.
The article 2012/13 influenza vaccine effectiveness against hospitalised influenza A(H1N1)pdm09, A(H3N2) and B: estimates from a European network of hospitals (EuroSurveill 2015;20(2):pii=21011) illustrates a multicentre test-negative case-control study to estimate influenza VE in 18 hospitals. It is believed that confounding due to health-seeking behaviour is minimised since, in the study sites, all people needing hospitalisation are likely to be hospitalised. The study Trivalent inactivated seasonal influenza vaccine effectiveness for the prevention of laboratory-confirmed influenza in a Scottish population 2000 to 2009 (EuroSurveill 2015;20(8):pii=21043) applied this method using a Scotland-wide linkage of patient-level primary care, hospital and virological swab data over nine influenza seasons and discusses strengths and weaknesses of the design in this context.
This design is described in Chapter 10.2.1.3.
A generic study protocol to assess the impact of rotavirus vaccination in EU Member States has been published by the ECDC. It recommends the information that needs to be collected to compare the incidence/proportion of rotavirus cases in the period before and after the introduction of the vaccine. These generic protocols need to be adapted to each country/regions and specific situation.
The impact of vaccination can be quantified in children in the age group targeted for the vaccine (overall effect) or in children of other age groups (indirect effect). The direct effect of a vaccine, however, needs to be defined by the protection it confers given a specific amount of exposure to infection and not just a comparable exposure. Direct and indirect effects in vaccine efficacy and effectiveness (Am J Epidemiol 1991; 133(4):323-31) describes how parameters intended to measure direct effects must be robust and interpretable in the midst of complex indirect effects of vaccine intervention programmes.
Impact of rotavirus vaccination in regions with low and moderate vaccine uptake in Germany (Hum Vaccin Immunother 2012; 8(10):1407-15) describes an impact assessment of rotavirus vaccination comparing the incidence rates of hospitalisations before, and in seasons after, vaccine introduction using data from national mandatory disease reporting system.
First year experience of rotavirus immunisation programme in Finland (Vaccine 2012; 31(1):176-82) estimates the impact of a rotavirus immunisation programme on the total hospital inpatient and outpatient treated acute gastroenteritis burden and on severe rotavirus disease burden during the first year after introduction. The study may be considered as a vaccine-probe-study, where unspecific disease burden prevented by immunisation is assumed to be caused by the agent the vaccine is targeted against.
The study of vaccine effectiveness against diseases where immunity wanes over time requires consideration of both the within-host dynamics of the pathogen and immune system as well as the associated population-level transmission dynamics. Implications of vaccination and waning immunity (Proc Biol Sci 2009; 276(1664):2071-80) seeks to combine immunological and epidemiological models for measles infection to examine the interplay between disease incidence, waning immunity and boosting.
Studies of vaccine effectiveness rely on accurate identification of vaccination and cases of vaccine-preventable diseases but in practice diagnostic tests, clinical case definitions and vaccination records often present inaccuracies. Bias due to differential and non-differential disease- and exposure misclassification in studies of vaccine effectiveness (PLoS One 2018;15;13(6):e0199180) explores through simulations the impact of non-differential and differential disease- and exposure-misclassification when estimating vaccine effectiveness using cohort, case-control, test-negative case-control and case-cohort designs.
Misclassification can lead to significant bias and its impact strongly depends on the vaccination scenarios. A web application is publicly available to assess the potential (joint) impact of possibly differential disease- and exposure misclassification.