Comparative effectiveness of three common SARS-COV-2 vaccines: A network meta-analysis of randomized trials

Abstract

Background

Moderna, Pfizer, and AstraZeneca SARS-CoV-2 vaccines for preventing COVID-19 have regulatory approval in most countries. We conducted a network meta-analysis to compare their effectiveness.

Methods

We searched PubMed, Cochrane Central Register of Controlled Trials (CENTRAL), ICTRP, and Clinicaltrials.gov for the randomized controlled trials (RCTs) published between 1st January 2020 and 1st February 2024. Eligible RCTs evaluated the Moderna, Pfizer or AstraZeneca vaccines among healthy individuals and reported the effectiveness of vaccination versus control measured with the outcome occurrence of COVID-19. We performed study selection, data extraction, and quality (risk of bias) assessment in duplicate. Network meta-analysis with random effects models was used to generate odds ratios (OR) with 95% confidence intervals (CI), evaluating heterogeneity statistically using I² for direct comparisons and ranking vaccines hierarchically using the surface under the cumulative ranking curve (SUCRA). This study was registered on PROSPERO, CRD42023457957.

Findings

Of the 1954 initial citation, 18 RCTs (272,724 participants; 151,034 received one of the vaccines and 121,690 controls) that reported the outcome occurrence of COVID-19 were selected. Of these, 2 (11%) were moderate and 5 (28%) were high in quality. In network meta-analysis, all three vaccines were effective compared directly with control (Moderna OR 0.13, 95% CI 0.07–0.26, I² 97%; Pfizer OR 0.10, 95% CI 0.05–0.19, I² 78%; AstraZeneca OR 0.38, 95% CI 0.25–0.59, I² 63%). Indirect comparison of vaccines using control as the common comparator showed that AstraZeneca was less effective than Moderna (OR 2.84, 95% CI 1.32–6.12) and Pfizer (OR 3.94, 95% CI 1.80–8.60), while Moderna versus Pfizer showed no difference (OR 1.39, 95% CI 0.56–3.46). Vaccine SUCRA probabilities were higher for Pfizer than Moderna and AstraZeneca (92%, 75% and 33% respectively compared to control).

Interpretations

Pfizer ranks highest followed by Moderna (without a statistically significant difference) and AstraZeneca vaccines for preventing symptomatic COVID-19.

Resumen

Antecedentes

Las vacunas de Moderna, Pfizer y AstraZeneca SARS-CoV-2 para prevenir COVID-19 tiene aprobación regulatoria en la mayoría de los países. Elaboramos un network metaanálisis en red para comparar su efectividad.

Métodos

Se realizaron búsquedas en PubMed, Cochrane Registro Central de Ensayos Controlados (CENTRAL), ICTRP y Clinicaltrials.gov para ensayos clínicos (RCTs) publicados entre el 1 de Enero de 2020 y el 1 Febrero de 2024. Se incluyeron RCTs elegibles que evaluaron la efectividad de las vacunas Moderna, Pfizer o AstraZeneca vacunas comparado con control y administradas en individuos sanos medido con el desenlace de aparición de COVID-19 sintomático. La búsqueda, selección, extracción de datos y análisis de la calidad (riesgo de sesgos) se llevó a cabo en duplicado. Se utilizó un metaanálisis en red con modelos de efectos aleatorios para generar odds ratios (OR) con intervalos de confianza (IC) del 95%, se estimó la heterogeneidad estadísticamente mediante I² para comparaciones directas y las vacunas se clasificaron jerárquicamente utilizando la superficie debajo de la curva de ranking acumulativo (SUCRA). Este estudio fue registrado en PROSPERO (CRD42023457957).

Resultados

Se revisaron 1.954 citaciones, de las que 18 RCTs (272.724 participantes; 151.034 recibieron una vacuna y 121.690 controles) informaron la tasa de aparición de casos de COVID-19. De ellos, dos (11%) fueron moderados y cinco (28%) de alta calidad. En el metaanálisis, las tres vacunas fueron efectivas comparado directamente con el control (Moderna OR 0,13, IC 95% 0,07-0,26, I² 97%; Pfizer OR 0,10; IC 95%: 0,05-0-19, I² 78%; AstraZeneca OR 0,38; IC del 95%: 0,25-0,59, I² 63%). La comparación indirecta de vacunas utilizando el control como comparador común mostró que AstraZeneca fue menos efectiva que Moderna (OR 2,84, IC 95% 1,32-6,12) y Pfizer (OR 3,94; IC del 95%: 1,80-8,60), mientras que Moderna frente a Pfizer no mostró diferencia (OR 1,39; IC del 95%: 0,56-3,46). La clasificación de probabilidades (SUCRA) fue más alta para Pfizer que para Moderna y AstraZeneca (92%, 75% y 33%, respectivamente comparado con el control).

Interpretaciones

Pfizer presentó la eficacia más elevada seguida de Moderna (sin diferencias estadísticamente significativas) y de AstraZeneca para la prevención del COVID-19 sintomático.

Introduction

Vaccination can prevent coronavirus disease (COVID-19), an infectious disease caused by coronavirus SARS-CoV-2 (Severe Acute Respiratory Syndrome Coronavirus 2).¹ In the first year after the SARS-CoV-2 pandemic was declared a worldwide health emergency in early 2020, over 98 million cases of COVID-19 occurred with over 1.8 million deaths.² Since early 2021, vaccines have been the main preventative advance with the acquisition of herd immunity helping to control the pandemic.

Drug regulators have granted approval to vaccines worldwide. Moderna, Pfizer and AstraZeneca are among the most accepted vaccines, with Moderna approved by regulators in 144 countries, Pfizer in 165 countries, and AstraZeneca in 185.³ Vaccine efficacy varies, reported to be 98.1%, 91.2%, and 84.3% for Moderna, Pfizer, and AstraZeneca respectively in pairwise comparisons versus unvaccinated controls.^4,5 It has been claimed that mRNA vaccines are superior in their efficacy.¹ A network meta-analysis can evaluate the effects of various vaccines against each other, using control as the common comparator, and provide a rank order of effectiveness.⁶ When such an attempt was made previously,⁷ only five relevant trials of Moderna, Pfizer, and AstraZeneca were captured by the search and indirect comparisons were not reported. Now the evidence base has grown, and an updated evidence synthesis is required.

We conducted a comprehensive systematic review and network meta-analysis of all the available randomized controlled trials (RCTs) that investigated the Moderna, Pfizer and AstraZeneca vaccines in the prevention of COVID-19 among healthy individuals to compare their effectiveness.

Methods

This systematic review and network meta-analysis was prospectively registered on PROSPERO (CRD42023457957).⁸ It was conducted using recommended methods⁹ and reported adhering to the Preferred Reporting Items for Systematic Reviews and Network Meta-Analyses (PRISMA-NMA) guidelines.¹⁰

Search strategy and study selection

We searched PubMed, Cochrane Central Register of Controlled Trials (CENTRAL), International Clinical Trials Registry Platform (ICTRP), and ClinicalTrials.gov to conduct an exhausting search on our research question from 1st January 2020 to 1st February 2024, with no language restriction. Separate searches were conducted for all the vaccines in each database with the search terms combination adapted to its structure and vocabulary (Appendix Table S1). The search strings included a combination of indexing terms, free text terms and word variants covering the concept: COVID-19 and vaccination. The included RCTs comparing either of the three vaccines i.e. Moderna (mRNA-1273), Pfizer (BnT162b2, BnT162b1) or AstraZeneca (AZD1222/ChAdOx1-SARS-COV-2) versus non-active (placebo) and active (meningococcal vaccine) controls for their effectiveness and safety among healthy participants of any age given at least 2 doses of the same vaccine. Single-arm trials and trials including immunocompromised patients were excluded. Non-published data or published pre-prints were also excluded. Two reviewers (AAS and FE) independently assessed the titles and abstracts of the citations. If either reviewer found a particular citation to be potentially eligible, it was subjected to independent full-text review by the same two reviewers. In case, the opinions of the reviewers conflicted, the decision concerning eligibility was made with the input of a third reviewer (ARSS). The included RCTs had obtained ethical approval and consent individually. No approval was required by the ethical committee for this systematic review and network meta-analysis using the available published data.¹¹

Data extraction and risk of bias assessment

Two independent reviewers (ARSS and SJZ) extracted the data on the study characteristics, participants, interventions, and outcomes included in this review. The extracted data was rechecked by two independent reviewers (AAS and HRS) to minimize the chances of errors. The primary outcome of this meta-analysis was the prevention of symptomatic COVID-19 after vaccination as defined by CDC.¹² The secondary outcomes were safety (serious adverse events related to the trial intervention), and mortality. The following local and systemic adverse events were not part of the serious adverse event classification: fever, headache, fatigue, myalgia, arthralgia, pain at the site of injection, nausea, chills, itching, induration, swelling, warmth, redness, tenderness, lymphadenopathy, and erythema. The quality of the studies was assessed for risk of bias.¹³ Two independent reviewers (HRS and SJZ) assessed the following domains selection bias (random sequence generation and allocation concealment), performance bias (blinding of participants and personnel), measurement bias (blinding of outcome assessment), attrition bias (incomplete outcome data), and coherence with registry with respect to selective reporting. If a study was high risk in two or more domains, it was considered as low quality overall. If a study was high risk in any one domain, it was considered moderate quality overall. And if the study was low risk in all domains, it was considered high quality overall.

Data synthesis

We performed pairwise random effects meta-analyses for trials of each vaccine separately to generate odds ratios (OR) with 95% confidence intervals (CI), evaluating heterogeneity within each vaccine visually using forest plots and statistically using I² statistic where there were 3 or more studies.¹⁴ An I² value 0–40% was considered low heterogeneity, 40–60% moderate heterogeneity, and >60% high heterogeneity. We performed subgroup meta-analysis comparing the results of non-active (placebo) and active (meningococcal vaccine) controls since an active vaccine control is expected to cause a similar reaction at the site of the COVID-19 vaccine injection permitting better blinding compared to non-active placebo control, a feature that may help prevent performance and measurement bias in the estimation of effect. Publication and related biases were assessed in funnel plot analysis for each vaccine and overall. Funnel asymmetry was statistically assessed by the Eggar's test.¹⁵ The above information was summarized in tabulated form.¹⁶

We performed network meta-analysis first examining the geometry and connectivity of the network.^9,17 The node with control (active or non-active) was set as the reference. We performed an effectiveness network meta-analysis using multivariate methods following a frequentist approach as implemented in R.¹⁸ We fitted a treatment contrast model with the assumption of common heterogeneity for all comparisons. We assume that within all three vaccine trials, any participants could be equally likely randomized to any of the vaccines meeting the assumption of consistency.¹⁹ We planned to check consistency between direct and indirect sources of evidence was statistically assessed locally (i.e. for all the closed network loops) and globally. The relative ranking of vaccines was presented as the surface under the cumulative ranking (SUCRA) probabilities for the vaccine achieving the highest value.

Results

Selection, characteristics, and quality of RCTs

The initial electronic search yielded 1954 records of titles and abstracts, of which 33 RCT citations were selected for full-text assessment. As shown in Fig. 1, 24 RCTs^20–43 (276,139 participants; 153,557 received the vaccine and 122,582 controls) were included. Of these, 18 reported the effectiveness outcome occurrence of COVID-19. Additionally, 17 and 11 studies respectively reported the outcomes serious adverse events and mortality. The excluded studies after full-text review are presented with the reason for exclusion in Appendix 2. Study characteristics are presented in Table 1. The median follow-up period was 9 months (15 days to 25 months) after the last vaccination.

Figure 1.

Flow chart of the step-by-step approach to the randomized controlled trial (RCT) selection process in the network meta-analysis of effectiveness and safety of three common SARS-COV-2 vaccines (see Appendix 1).

Table 1.

Characteristics of the included randomized controlled trials (RCT) in network meta-analysis of effectiveness and safety of three common SARS-COV-2 vaccines.

Author (year)	Total no. of patients (n)	Design (phase)	Patients in each arm		Patient population	Maximum follow-up reported in the study	Arms		Doses used (intervention)	Outcomes assessed
			Vaccine	Control			Vaccine	Control
Moderna vaccine (mRNA-1273)
Laurence Chu (2021)	600	Phase 2 RCT	400	200	Healthy adult, 18 years of age or older.	57 days	mRNA-1273	Placebo	Doses – 50 μg and 100 μg; 2 doses of each	Mortality.
Lindsey R. Baden (2020)	30,351	Phase 3 RCT	15,181	15,170	Healthy adult, 18 years of age or older.	25 months	mRNA-1273	Placebo	100 μg × 2 doses	Primary: Efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection Secondary: Safety (Serious AEs up to 28 days after any dose vaccination).
Hana M. El Sahly (2021)	28,451	Phase 3 RCT	14,164	14,287	Healthy adult, 18 years of age or older.	25 months	mRNA-1273	Placebo	100 μg × 2	Primary: Efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection Secondary: Safety (Serious AEs up to 28 days after any dose vaccination).
Kashif Ali (2021)	3726	Phase 2–3 RCT	2486	1240	Male and female between 12 and 17 years.	12 months	mRNA-1273	Placebo	100 μg × 2	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection), Secondary: Safety (Serious AEs up to 28 days after any dose vaccination).
C. Buddy Creech (2022)	4002	Phase 2–3 Open-label RCT	3007	995	Healthy children with ages between 6 and 11 years.	12 months after the last injection	mRNA-1273	Placebo	50 μg × 2 doses	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection),Secondary: Safety (Serious AEs up to 28 days after any dose vaccination).
Evan J. Anderson (2022)	6388	Phase 2–3 Open-label RCT	4792	1596	Healthy children with ages between 6 months to 5 years.	12 months after the last injection	mRNA-1273	Placebo	25 μg × 2 doses	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection), Secondary: Safety (Serious AEs)
Pfizer vaccine (BNT162b2, BNT162b1)
Fernando P. Polack (2020)	37,706	Phase 2/3 RCT	18,860	18,846	Healthy people with ages 16 or above.	For up to 2 years after the second dose. Cross-over in placebo patients after 6 months	BNT162b2	Placebo	30 μg × 2 doses	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection), Secondary: Safety (Serious AEs up to 6 months after last dose vaccination), and mortality.
Stephen J. Thomas (2021)	44,047	Phase 2–3 RCT	22,026	22,021	Healthy people with age groups 12–15 years, >16 years.	6 months	BNT162b2	Placebo	30 μg × 2 doses	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection), Secondary: Safety (Serious AEs up to 6 months after last dose vaccination), and mortality.
Edward E. Walsh (2020)	195	Phase 1 RCT	156	39	Healthy participants between 18 and 55 years or 65 and 85 years of age	24 months	BNT162b1 – Groups (10 μg, 20 μg, 30 μg, 100 μg) for different age groups; BNT 162b2 – Groups (10 μg, 20 μg, 30 μg, 100 μg)	Placebo	10 μg, 20 μg, 30 μg, 100 μg × 2 doses for different age groups	Mortality.
Miwa Haranaka (2021)	160	Phase 1/2 RCT	119	41	Healthy male or female 20–85 years of age	1 month	BNT162b2	Placebo	30 μg × 2 doses	Safety (Serious AEs), and mortality.
Robert W. Frenck Jr (2021)	2260	Phase 3 RCT	1131	1129	Healthy people with age groups 12–15 years, 16–25 years.	2 months	BNT162b2	Placebo	30 μg × 2 doses	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection), Secondary: Safety (Serious AEs up to 6 months after last dose vaccination), and mortality.
Emmanuel B. Walter (2021)	2268	Phase 1 dose finding study; Phase 2–3 dose identification study RCT	1518	750	Healthy people with ages between 5 and 11 years.	2 years	BNT162b2	Placebo	30 μg × 2 doses	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection), Secondary: Safety (Serious AEs up to 6 months after last dose vaccination), and mortality.
Flor M. Muñoz (2023)	1776	Phase 2–3 RCT	1178	598	Healthy people with age <5years.	12–18 months	BNT162b2	Placebo	3 μg × 2 doses; 3rd dose of 3 μg based on immunogenicity results	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection), Secondary: Safety (Serious AEs up to 6 months after last dose vaccination), and mortality.
Ai-Min Hui (2022)	959	Phase 2 RCT	720	239	Healthy adults 18–85 years.	6 months	BNT162b2	Placebo	30 μg × 2 doses	Safety (Serious AEs up to 6 months after last dose vaccination).
Mark J. Mulligan (2022)	45	Phase 1/2 RCT	36	9	Healthy people with age between 19 and 54 years.	Up to 2 year	BNT162b1	Placebo	10 μg or 30 μg or 100 μg BNT162b1	Safety (Serious AEs up to 45 days after 1st dose vaccination).
AstraZeneca vaccine (AZD1222)
Ann R. Falsey (2021)	32,451	Phase 3 RCT	21,635	10,816	Healthy adult, 18 years of age or older.	65 days	AZD1222	Placebo	AZD1222 (5× 10^10 viral particles)	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 15 days after the second injection), Secondary: Safety (Serious AEs up to 28 days after any dose vaccination), mortality.
Anthonet L. Koen (2023)	1895	Phase 1b/2 RCT	935	960	Healthy adult, 18 years of age or older.	250 days	AZD1222	Placebo	NA	Efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection)
Shabir A. Madhi (2021)	1912	Phase 1b–2 RCT	961	951	Healthy adults aged 18–65 years.	156–160 days	ChAdOx1 nCoV-19	Placebo	5 × 10^10 viral particles (vaccine)	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 15 days after the second injection), Secondary: safety (Serious AEs)
Kensuke Ishikawa (2023)	256	Phase 1/2 RCT	192	64	Healthy adult, 18 years of age or older.	Approximately 1 year	AZD1222	Placebo	5 × 10^10 viral particles/dose (vaccine)	Safety (Serious AEs up to 365 days after any dose vaccination), mortality.
Magdalena E. Sobieszczyk (2022)	30,724	Phase 3 RCT	20,770	9954	Healthy adult, 18 years of age or older.	6 months	AZD1222	Placebo	AZD1222 (5 × 10,1^10 viral particles)	Primary: efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 15 days after the second injection), Secondary: Safety (Serious AEs up to 28 days after any dose vaccination), and mortality.
Sue Ann Costa Clemens	9433	Phase 3 RCT	4722	4661	Healthy adults, 18 years of age or older.	–	AZD1222	Placebo	3.5–6.5 × 10^10 viral particles	Efficacy (in preventing a first occurrence of symptomatic COVID-19 after last dose injection),
Merryn Voysey (2020)	11,636	Phase 2/3 & 3 RCT	5807	5829	–	Median 2.0 months (IQR 1.3–2.3)	ChAdOx1 nCoV-19 vaccine	MenACWY	5 × 10^10 viral particles or 2.2 × 10^10 viral particles	Efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection).
Natalie Gabrielle Marchevsky (2022)	15,164	RCT	7617	7547	Healthy adult, 18 years of age or older.	15 days after 2nd dose	ChAdOx1 nCoV-19	MenACWY	5 × 10^10 viral particles	Efficacy (in preventing a first occurrence of symptomatic COVID-19 with onset at least 14 days after the second injection).
Katherine R.W. Emary (2021)	8534	Phase 2/3 & 3 RCT	4244	4290	–	Median 2.0 (IQR 1.3–2.3)	ChAdOx1 nCoV-19 vaccine	MenACWY	5 × 10^10 viral particles or 2.2 × 10^10 viral particles	Efficacy (in preventing a first occurrence of symptomatic COVID-19).

Abbreviations: NA = not available; AE = adverse event.

As shown in Fig. 2, for quality assessment of all evidence, six RCTs (25%) were high quality as they showed low risk of bias in all domains,^{23,29,32,37,42,43} four RCTs (17%) were moderate quality as they showed high risk in any one of the domains,^24,33,34,40 and 14 RCTs (58%) were low quality as they showed high risk in at least two domains.^{20–22,25–28,30,31,35,36,38,39,41} Two RCTs were high risk in the random sequence generation domain,^28,31 two RCTs were high risk in the allocation concealment domain,^34,39 seven RCTs were high risk in the blinding of participants and personnel domain,^{22,24–26,30,36,41} nine RCTs were high risk in the blinding of outcome assessment domain,^{20,22,24,28,31,35,36,39,41} ten RCTs were high risk in incomplete outcome data domain,^{20,21,26,27,30,33,35,38,39,41} and four RCTs were high risk in selective reporting domain.^21,27,38,40 Detailed risk of bias assessments is provided in Appendix 3.

Figure 2.

Quality assessment of 18 randomized controlled trials (RCTs) evaluating the effectiveness in the prevention of symptomatic COVID-19 after the last vaccination of one of three common SARS-COV-2 vaccines versus control (see associated Appendix S3; data presented as a 100% stacked bar chart with numbers in bars representing numbers of RCT).

Effectiveness evidence summaries are presented in Fig. 3. The associated forest and funnel plots appear in Appendix 6. All three SARS-COV-2 vaccines were effective in the prevention of symptomatic COVID-19 after vaccination. The effectiveness of Moderna^21–25 and Pfizer^26,27,30,32 was evaluated versus non-active (placebo) control. The effectiveness of AstraZeneca was evaluated versus non-active^{35–37,39,40} and active^41–43 controls. Within AstraZeneca RCTs, there was no control-based subgroup difference (p = 0.22). The RCT quality per vaccine is shown in Fig. 2.

Figure 3.

Network meta-analysis of 18 randomized controlled trials evaluating the effectiveness in the prevention of symptomatic COVID-19 after the last vaccination of one of three common SARS-COV-2 vaccines versus control (see associated Appendix 6).

Effectiveness network meta-analysis

Vaccine effectiveness in the prevention of COVID-19 was reported in 18 RCTs (272,724 participants; 151,034 received the vaccine and 121,690 controls). Of these RCTs, 2 (11%) were moderate and 5 (28%) were high in quality. The treatment network evaluated vaccines by direct and indirect comparisons setting the control node (active or non-active) as the reference is shown in Fig. 3. Direct comparisons showed that all three vaccines were effective compared with control (Moderna OR 0.13, 95% CI 0.07–0.26, I² 97%; Pfizer OR 0.10, 95% CI 0.05–0.19, I² 78%; AstraZeneca OR 0.38, 95% CI 0.25–0.59, I² 63%). Indirect comparison of vaccines using control as the common comparator showed that AstraZeneca was less effective than Moderna (OR 2.84, 95% CI 1.32–6.12) and Pfizer (OR 3.94, 95% CI 1.80–8.60), while Moderna versus Pfizer showed no difference statistically (OR 1.39, 95% CI 0.56–3.46). Pfizer vaccine ranked highest followed by Moderna and AstraZeneca with SUCRA probabilities 92%, 75% and 33% respectively compared to control.

Serious adverse event and mortality network meta-analysis

Vaccine serious adverse events and mortality were reported in 17 (227,482 participants; 128,776 received the vaccine and 98,706 controls) and 11 RCTs (152,443 participants; 87,985 received the vaccine and 64,458 controls) respectively. Of the 17 serious adverse events RCTs, 3 (18%) were moderate, and 4 (24%) were high in quality, and of the 11 mortality RCTs, 0 (0%) were moderate and 2 (18%) were high in quality. The treatment network for the outcome of serious adverse events had sparse data to permit sufficiently robust network meta-analysis with SUCRA (Appendix 4). The treatment network for the mortality outcome lacked a direct comparison for the Moderna vaccine so some indirect comparisons and network meta-analysis with SUCRA were not feasible (Appendix 5). The direct and indirect comparisons all did not show any statistically significant differences except for the mortality outcome for AstraZeneca versus control (OR 0.42, 95% CI 0.18–0.96, I² not calculable).

Discussion

This systematic review of RCT evidence deploying network meta-analysis with direct and indirect comparisons found the Pfizer vaccine to be the most effective in the prevention of symptomatic COVID-19, followed by the Moderna (without a statistically significant difference) and AstraZenca (with a statistically significant difference) vaccines. The analyses for the serious adverse events and mortality outcomes were imprecise to draw firm conclusions.

Our systematic review was prospectively registered and deployed a comprehensive search. SARS-COV-2 trials took place in an environment where prospective registration was already well established as a publication requirement,⁴⁴ so the risk of missing studies should be negligible. We did not find any evidence of funnel asymmetry minimizing the risk of publication and related biases.¹⁵ Data extraction was challenging in our work as outcome or endpoint definitions varied between studies. We strictly followed the CDC defined COVID-19 diagnostic criteria.¹² If trials reported other outcome criteria,²² we excluded them from our evidence synthesis. Our approach has the advantage of consistency in outcome data extraction which in turn adds strength to the statistical synthesis.

Our network meta-analysis findings require careful interpretation in the light of RCTs of mixed quality with heterogeneity. The included RCTs had high risk of bias in various domains. However, RCTs offer the highest level of validity in the evidence hierarchy. There was substantial heterogeneity, something that requires clinical interpretation. For instance, the inconsistency among studies in the follow-up duration after the administration of the last vaccine dose could be regarded as a potential factor that could contribute to the variation observed. I² statistic does have limitations in capturing variation between studies.⁴⁵ High I² values do not automatically mean that effects are dispersed over a wide range, therefore the interpretation ought to be nuanced. I² does depend on study precision (linked to its sample size).⁴⁶ The included RCTs, especially those conducted for regulatory approval, were large. In this situation, even small differences in effects may be associated with a high I² value. Prediction intervals capture the issue of dispersion of effects better and were reported with our network meta-analyses. Some of the planned analyses could not be performed due to the unavailability of sufficient data in the RCTs. For example, consistency of network could not be formally assessed as there were no direct and indirect comparison pairs available for the same vaccines. Another limitation is that we did not plan to explore if the conflicts of interests declarations could have a role in vaccine effectiveness against the different variants. It is unlikely that direct vaccine comparisons will be carried out now that regulatory approval already exists for the vaccines. Given this scenario, our findings merit consideration.

The previously published meta-analyses of vaccine effectiveness have only focused on direct comparisons with placebos.^47–49 To the best of our knowledge, ours is the first evidence synthesis that has undertaken indirect comparisons of the vaccines in network meta-analysis taking account of both active and non-active (placebo) controls. AstraZeneca vaccine was evaluated versus both active (meningococcal vaccine) and non-active (placebo) controls.^41–43 Their subgroup comparison revealed no statistically significant difference, so it made sense to maximize statistical power by including more trials in the network meta-analysis. In another network meta-analysis concluded that Pfizer vaccine had the highest efficacy, it does not appear that indirect comparisons and formal SUCRA ranking was deployed.⁷ Other previous systematic review with meta-analysis focused only on older⁵⁰ or younger patients.⁵¹ Ideally, a comparison of age-based subgroups should be made within a single evidence synthesis. Individual patient data may be required from the original trials to undertake such an analysis. A large-scale network meta-analysis including other vaccines and outcomes are required including the question of comparative efficacy of the vaccines over longer follow-up against different virus variants. These topics need to be addressed in future research.

In conclusion, among the SARS-COV-2 vaccines approved by the European Medicine Agency,⁵² the Pfizer vaccine has the highest protective potential against COVID-19 after seven and fourteen days of last dose vaccination compared to the other vaccines. Moderna and AstraZeneca vaccines are both effective and they perform at a level lower than Pfizer (Moderna without a statistically significant difference and AstraZeneca with a statistically significant difference) based on formal probability ranking.

Funding

None.

Authors’ contribution

KSK, MF, AAS, FE, ARSS, SJZ, and HRS designed the study. AAS and FE did the literature searches and formed the data extraction form. ARSS and SJZ extracted the data, and AAS and HRS cleaned and cross-checked the data extraction form. MF did the statistical analyses of the study, and KSK supervised the statistical analyses. AAS, FE, ARSS, HRS, and SJZ wrote the manuscript. All authors critically revised and edited the manuscript, and KSK and MF did the final revisions. All authors had complete access to the data in the article and had final responsibility for the decision of submission.

Data sharing

All data starting from the search strategies used in all databases, a list of included and excluded RCTs, raw data extracted for the analysis, the analyzed data including the forest and funnel plots, and quality assessment are available as appendices with the article.

Acknowledgment

K.S.K. is a Distinguished Investigator at the University of Granada funded by the Beatriz Galindo (senior modality) program of the Spanish Ministry of Education. AI language improvement software was used.