or not vaccinated elderly. Data extraction. The studies were valued independently by four investigators with predefined criteria of validity, such as results comparing rates of disease caused by serotypes included in the vaccine, random allocation, double blind design, included subjects pertaining to the same study base, and losses of less than 10% in clinical trials and 20% in observational studies. Results. Eight clinical trials considered the relative risk (RR) of pneumococcal pneumonia, three did not make estimations on pneumonia originated by serotypes included in the vaccine and only one study fulfilled all the inclusion criteria. Vaccinated versus not vaccinated pneumococcal pneumonia RR was 0.86 (95% CI, 0.24 to 2.99). Vaccine effectiveness was 14% (95% CI, 199 to 76%). Ten studies performed estimations on the effectiveness of the vaccine on invasive disease by vaccine serotypes. Of these, two clinical trials and two observational studies fulfilled the required quality criteria. RR
of invasive disease was of 0.68 (95% CI, 0.39-1.18); vaccine effectiveness was 32% (95% CI, 1861%). Conclusions. No evidence was found supporting pneumococcal vaccine effectiveness to reduce or avoid S. pneumoniae disease in the elderly.
Introduction
The indications for pneumococcal vaccination in persons aged 65 years or more are controversial because of discordant results between different studies,1-4 and important differences have been noted in international recommendations for vaccination in this age group.2,5,6 The controversy first appeared at the beginning of the 1980s in the light of difficulties large clinical trials had in demonstrating the effectiveness of vaccination in older persons,7 and it was argued that the large numbers of participants required to guarantee sufficient statistical power to prove the protective efficacy of vaccination made such clinical trials impracticable.8 At the end of the 1980s, recommendations for vaccination in persons older than 65 years were based on the results of observational studies. It is therefore high time for a systematic, thorough review of the literature, which now includes clinical trials and observational studies. Methodological quality criteria can now be used to guarantee an unbiased estimate of the effectiveness of pneumococcal vaccination in older persons.
Methodology
To estimate the effectiveness of 23-valent pneumococcal capsular polysaccharide vaccination in preventing the disease caused by pneumococcal organisms in persons older than 65 years, we systematically reviewed the literature, with attention to both clinical trials and observational studies. Bibliographic databases were searched for articles published in Spanish, English or French between the years 1964 and 2000 in MEDLINE, and between 1988 and 2000 in EMBASE. The CD-ROM edition of the Cochrane Library database9 was searched up to and including the first disk issued in 2001. Previously unidentified items were searched for in the reference lists of retrieved articles and in those of four separate reviews (three of which were systematic) of the effectiveness of polysaccharide vaccines.1-4 Researchers were contacted to request unpublished information and other unidentified items.
Two search strategies were used: the sensitive strategy was based on the terms pneumoc* AND vaccin* AND elderly, and the specific strategy used the terms pneumoc* AND vaccin* AND elderly AND (effectiv* OR effica*). We retrieved clinical trials, cohort studies and case-control studies that reported results for the effect of vaccination on the risk of pneumococcal disease, defined as pneumococcal pneumonia or invasive disease caused by pneumococci of the serotypes included in the vaccine and identified by culture, in populations of adults aged 65 years or older. Invasive disease was defined by the isolation of pneumococcal organisms from a normally sterile anatomical site. 12.
Inclusion, exclusion and evaluation criteria for studies
Studies were evaluated independently by four researchers on the basis of predefined criteria for validity.10-13 Differences were resolved by consensus between the researchers. The inclusion criteria used to select studies were: a) information included on the administration of the vaccine in adults aged 65 years or more; b) risk measured for pneumonia or invasive disease caused by pneumococci belonging to the serotypes included in the vaccine; c) information included on the comparability of cases and controls or exposed and unexposed persons, ie, information that both of the groups being compared belonged to the same study base;14d) random allocation according to an adequately concealed double-blind procedure in clinical trials;15e) response rates higher than 80% for observational studies, or drop-out rates lower than 10% for clinical trials;16f) absence of classification bias or serious violations of blinding;15g) suitable control of confounding variables, and h) sufficient information to be able to repeat the analysis.17 Noncompliance with either of the first two criteria (a or b) led to exclusion of the study. The other criteria were used as parameters to evaluate the quality of the studies, which were classified according to the following matrix:12
We excluded from the quantitative analysis all studies classified as C, and included all those classified as A or B.
Analysis
A descriptive analysis was prepared based on the year of publication, age groups included, type of population, study design, number of serotypes included in the vaccine, type of results studied and number of participants; relative risks (RR) were estimated and their confidence intervals (CI) calculated. Because of the low frequency of the events of interest, the odds ratio was considered an unbiased estimator of the RR. For each study we recorded factors that compromised the validity of the estimates and the likelihood that the results were biased.
Before estimating the protective effect of vaccination, we evaluated the homogeneity of the studies with graphs that illustrated point estimates of RR for pneumococcal disease and the CI, and their overlap or divergence. The hypothesis of homogeneity was tested by estimating the chi-squared value;18 significant heterogeneity was considered to exist when P<.10. The possibility of publication bias was explored with the test of Egger et al.19 The degree to which heterogeneity of the estimates was influenced by study characteristics such as design, number of serotypes in the vaccine, year and biases was studied by regression (metaregression analysis) of these characteristics on the logarithm of RR.20 Only when the inclusion and validity criteria were satisfied and homogeneity could not be ruled out did we estimate the aggregate RR and its CI with the random effects method of DerSimonian and Laird18 for a 95% confidence interval. When the confidence interval of the estimated RR included 1, we interpreted this to mean that vaccination did not confer significant protection against pneumococcal disease. The aggregate effectiveness of vaccination was calculated as (1RR)*100.21
The number needed to treat (NNT) to avoid the outcomes of interest was also calculated.22,23 All calculations were done with MS-Excel, MS-Access, Epi Info v. 6.04b and STATA v. 5 software.
Results
Twenty-five items were identified that compared the effectiveness of pneumococcal vaccination in preventing pneumococcal disease versus a control group.7,24-47 Four articles were excluded because they dealt with young adults or Children,24,27,28,34 and a fifth item was excluded because it included individuals aged 10 years or more, did not supply information on age groups, and was done in a setting that made it difficult to extrapolate the results to western populations.26 Of the remaining 20 articles (Table 1), 11 were clinical trials7,25,29,31,33,35,36,38,44-46 and 9 were observational studies.30,32,37,39-43,47
Of the 11 clinical trials retrieved, eight analyzed the efficacy of the vaccine in preventing pneumococcal disease caused by serotypes included in the vaccine.7,25,29,31,33,35,45,46 Six studies (Table 2) had one or both of the following problems: appropriate random allocation of the participants to the experimental or control group was not used, or the allocation procedure was not adequately concealed from the researchers and participants in the treatment group.25,29,31,33,44,46 For the five remaining studies we estimated a low or moderate likelihood of bias, as participants were assigned randomly to one group or the other, and because researchers and participants were appropriately blinded to the procedure.7,35,36,38,45
Of the nine observational studies we retrieved, four analyzed the effectiveness of the vaccine in preventing pneumococcal disease caused by serotypes included in the vaccine,32,37,41,42 and six were judged highly likely to be biased (Table 3). The patients described in the study by Bolan et al.32 were included in the study by Butler et al.;42 the former did not provide any indication of the thoroughness of the information regarding exposure, whereas the second lacked information about exposure for 39% of the participants eligible for inclusion. We therefore considered the validity of these estimates to be insufficient. In the study by Sims et al.,39 107 participants (46%) of a total of 229 were excluded because information about exposure was missing. The two studies by Shapiro et al. published in 198430 and 199141 had characteristics that met the criteria for estimates of moderate validity, and also presented estimates that changed direction when the analysis was repeated after ignoring the case-control matching30 or after information for 122 participants vaccinated with the 23-valent capsule polysaccharide vaccine was excluded.41 The study by Gable et al.40 was based on administrative data, and the analysis was inadequate. Lastly, in the study by Nichols et al.47 exposed (vaccinated) and unexposed (unvaccinated) individuals could not be considered members of the same population at risk.
Effect of vaccination on pneumococcal pneumonia caused by serotypes included in the vaccine
Eight of the clinical trials retrieved (Table 1, Figure 1a) estimated the RR of pneumococcal pneumonia in vaccinated versus unvaccinated persons.7,25,29,31,35,36,45,46 Three studies36,45,46 did not estimate the number of cases of pneumococcal pneumonia caused by serotypes included in the vaccine (Table 2). Only the study by Kaufman et al.,25 who investigated a vaccine composed of three capsular serotypes (Table 1), reported a favorable estimate of the vaccine´s protective efficacy. As shown in Figures 1a and 1b, this study cannot be considered similar to the others carried out between 1980 and 1998. None of the other clinical trials found a clinically significant protective effect (RR) for vaccination (Figure 1b). Figure 1c shows the estimates from those three trials (from among the original eight studies) that fulfilled the criteria for validity of the estimates (Table 2). The trial published by Austrian et al.7 was excluded because samples for the identification of the serotype that caused pneumonia were obtained from fewer than 75% of the patients, althoughas shown in Figure 1 and Table 1none of their estimates showed vaccination to be protective. Lastly, only the clinical trial reported by Simberkoff et al.35 in 1986 fulfilled the inclusion criteria for validity of the estimates and for our definition of a case: these authors provided estimates of the risk of pneumococcal pneumonia according to each serotype included in the vaccine.
Figure 1. Relative risks of pneumococcal pneumonia. Part a shows all studies that measured the effectiveness of vaccination in preventing this disease; the test for homogeneity yielded a chi-squared value of 191.63 with 7 degrees of freedom and P<.001. Part b shows that after the study by Kaufman et al.25 was excluded, the test for homogeneity yielded a chi-squared value of 2.69 with 7 degrees of freedom and P<.85. Part c shows the three studies that calculated the RR for pneumococcal pneumonia in vaccinated versus unvaccinated persons and that fulfilled our methodological quality criteria. However, two studies (Davis et al.36 and Ortqvist et al.45) failed to investigate the serotypes that caused the disease. The areas of the boxes reflect the precision of the estimates obtained in different studies.
The best available estimate placed the RR for pneumococcal pneumonia in vaccinated individuals compared to unvaccinated individuals at 0.86, with a 95% CI of 0.24 to 2.99. Effectiveness in preventing pneumococcal pneumonia caused by the serotypes in the vaccine was 14%, with a 95% CI of 199% to 76%. The number of individual needed to vaccinate (NNT) to prevent one case was 1197. The CI indicated the possibility that one case of pneumococcal pneumonia could be caused for every 85 or more persons vaccinated, and that one case could be prevented for every 242 or more persons vaccinated; this CI also included zero Table 4).
Effect of vaccination on invasive disease caused by serotypes included in the vaccine
Ten studied provided estimates of the effectiveness of vaccination against invasive disease caused by serotypes included in the vaccine (Figure 2a). Only three studies (Kaufman et al.,25 Bolan et al.32 and Butler et al.42) concluded that vaccination was significantly superior to the placebo (Figure 2a). The study by Kaufman et al.25 had important limitations regarding random allocation of the participants and blinding of the investigators; the studies by Bolan et al.32 and Butler et al.42 reported results for the same series of patients and did not provide information on exposure for a large percentage of participants. The studies by Bentley et al.,29 Klastersky et al.33 and Honkanen et al.46 did not satisfy the criteria because of problems with random allocation of the participants to different interventions, or problems with blinding the investigators or the participants (Table 2).
Figure 2. a) Estimates of relative risk (RR) of invasive disease caused by the pneumococcal serotypes included in the vaccine and the 95% confidence intervals (95% CI). For all studies retrieved, the test for homogeneity yielded a chi-squared value of 16.12 with 9 degrees of freedom and P=.06. The x axis scale has been truncated on both ends to improve legibility. b) Estimates of RR and 95% CI for the studies that fulfilled our inclusion criteria. Global RR was 0.68, 95% CI was 0.39 to 1.18; P=.17. The test for homogeneity yielded a chi-squared value of 2.09 with 3 degrees of freedom and P=.55. The areas of the boxes reflect the precision of the estimates obtained in different studies.
Of the 10 studies we examined, the clinical trials by Simberkoff et al.35 and Ortqvist et al.,45 and the observational studies by Forrester et al.37 and Shapiro et al.41 fulfilled our methodological quality criteria well enough so that we could trust the validity of their results (Tables 2 and 3). The aggregate RR for invasive disease was 0.68, with a 95% CI of 0.39 to 1.18 and a P=.17 (Figure 2b). The chi-squared test used to check the hypothesis of homogeneity of the different studies yielded a value of 2.09 with 3 degrees of freedom and a P=.55. The test of Egger et al. used to determine the possibility of publication bias yielded a nonsignificant value of P=.79. Effectiveness of vaccination was 32%, with a 95% CI of 18% to 51%, a result that implied that vaccination had no significant effect.
The estimated number of individuals needed to vaccinate (NNT) to prevent one case of invasive disease was 785, with a CI that showed that one case of invasive disease might be caused for every 1399 or more persons vaccinated, and that one case might be prevented for every 412 persons or more vaccinated. The CI again included zero (Table 4).
Sensitivity analysis
Figure 1a illustrates the effectiveness of vaccination in preventing pneumococcal pneumonia according to different studies. The results of the study by Kaufman et al.25 explained the heterogeneity in the estimates of risk for pneumococcal pneumonia; this was further confirmed by metaregression analysis, which showed no significant alterations in the results according to the number of serotypes included in the vaccine, the year the study was done, or publication bias after the Kaufman et al. study was excluded.
Figure 2a shows that the results of studies that investigated the effectiveness of the vaccine in preventing invasive disease were markedly heterogeneous. Metaregression on the logarithm of RR and analysis of the influence of the year of the study, the study design, the serotypes included in the vaccine or publication bias showed that this last factor was the one that best explained (P=.009) the variability in our results (Figure 3). In fact, the point estimate of effectiveness of the vaccination was 2.6-fold greater in studies for which we considered validity of the estimates to be compromised than in studies we considered free from bias (classified as A or B). The RR increased from 0.26 to 0.68, and the CI for the latter value indicated that the effect of vaccination was null.
Figura 3. Results of metaregression analysis of the effect of publication bias on estimates of the effectiveness of vaccination in preventing invasive disease. Estimates for studies classified as A or B were significantly different (P=.004) from estimates obtained for studies classified as C. Interpretation of values along the x axis: A, A Low likelihood of bias; B, moderate likelihood of bias; C, high likelihood of bias (see Methods section).
Discussion
Twenty-five years after vaccination with capsular polysaccharide vaccines for pneumococci was first authorized, uncertainty remains as to their effectiveness, and some authors have warned that recommendations in favor of systematic vaccination for all persons aged 65 years or more need to be reviewed in the light of current knowledge.4 This situation has been fomented by proponents of vaccination who claimed that clinical trials would be impracticable and that observational studies were sufficient.8 The present sytematic review of the literature and meta-analysis included both observational studies and clinical trials, and obtained estimates of the effectiveness of vaccination in which the lower limit of the CI was a negative value. In other words, we found no proof of the effectiveness of pneumococcal vaccine in reducing or preventing pneumococcal disease in older persons. These results are in agreement with the findings of Fine et al.1 and Moore et al.,4 who included in their reviews only clinical assays.
Several factors may compromise the results of a systematic review of the literature.19 One possible source of bias is that studies eligible for inclusion may be overlooked because the predominance of publications in English makes studies in other languages less likely to be cited and retrieved. Another possible bias arises from the fact that the databases we used for our literature searches are selective in the references they include. Moreover, unpublished studiesmost of which report negative findingsare not retrieved. In the present study we did not limit our searches to studies in English. We searched MEDLINE, EMBASE and the Cochrane Library, and made appropriate efforts to retrieve unpublished studies such as those by Austrian et al.,7 a clinical trial with adequately concealed random allocation, double-blinding and a large number of participants (13 600). We contacted other researchers to obtain information on additional publications and scrutinized the reference lists of the items retrieved from database searches. In fact, the results of the test of Eggar et al., which we applied to studies that investigated invasive disease, supported our assumption that no bias was introduced by the noninclusion of studies that might have been overlooked.19
Another source of error is bias in the choice and application of inclusion and exclusion criteria, data extraction and quality control criteria. To avoid these pitfalls the aims and inclusion criteria were defined before the study was begun.12 Moreover, we included both clinical trials and observational studies. All items were reviewed and assessed independently by the authors, who all used the same methods of analysis.
With regard to the studies we included, possible sources of heterogeneity were reduced by adapting our selection process to the question that formed the basis of the present analysis. The use of age as an inclusion criterion led us to exclude studies of children or young adults, in whom the response to vaccination differs from that in older persons;1 in addition, we opted to include only those studies with unambiguous clinical outcomes that evaluated the protection conferred against pneumococcal pneumonia or invasive disease caused by S. pneumonia serotypes included in the vaccine.
A further potential source of heterogeneity between studies included in our analysis is the aggregation of results of clinical assays and observational studies. However, for studies designed and executed with adequate guarantees of methodological rigor, the results of clinical trials and observational studies concur.48,49 Heterogeneity of the findings from included and excluded studies (Figure 2a) was explained mainly by the presence or absence of factors that compromised the validity of the estimates (Figure 3), as opposed to factors such as the type of study design, year of publication or number of serotypes in the vaccine, which had no significant influence on the results.
In comparison to fixed effect models, our random effects statistical model can be considered an appropriate method for aggregating data from different studies. The former approach assumes that differences in the estimates are caused only by sampling error, whereas the random effects model also takes into account the fact that because the characteristics of the participants were not homogeneous across the studies we included, the magnitude of the effects of vaccination differed between studies. This was in fact the case in the present review: the studies we retrieved involved populations that differed in their risk profiles, regardless of the participants´ age18. The discrepancy between our findings and the results of the systematic review by Hutchison et al.3 can be explained by the fact that these authors pooled the results of studies involving different age groups. When we controlled for this factor, the statistical significance of the differences in the estimates disappeared.
In accordance with the terminology proposed by Altman,23 we present our results as estimates of the number needed to treat (ie, to vaccinate) and the confidence interval. We extrapolated our figures to the most plausible incidence of cases of invasive disease in the community50 50 per 100 000 persons aged 65 years or olderassuming that 90% of the cases are caused by one of the serotypes included in the vaccine. The estimated NNT to prevent one case of invasive disease is 6947, with a CI that includes, on one extreme, the chance of causing one case of invasive disease per 12 352 persons or more vaccinated, and on the other extreme, the chances of preventing one case per every 3644 or more persons vaccinated. These estimates mean that we cannot claim to have sufficient evidence to support vaccination. The actual estimate lies somewhere on a continuum of values that includes, in addition to the null effect, the possibility that vaccination itself might contribute to cases of the disease it is intended to prevent.
In conclusion, our review provides no proof that pneumococcal vaccination with nonconjugate capsular polysaccharide vaccines of S. pneumoniae is effective in reducing or preventing pneumococcal disease in older persons. In the light of the results of the present review and meta-analysis, we can only conclude that the systematic recommendation for vaccination in the population of persons 65 years of age or older should not be made in the absence of results of clinical trials that unequivocally demonstrate a protective effect.
Acknowledgments
We thank Dr. Soledad Márquez Calderón of the Andalusia Public Health School, Dr. Juan B. Bellido of the Epidemiolgy Section, Castellón Public Health Center, and Dr. Vicente Palop of the Geriatrics Unit, Internal Medicine Service, Hospital de la Ribera in Alzira, Valencia, for their comments on and contributions to earlier versions of this manuscript.
Correspondence: Joan Puig-Barberà. Avenida Ferrandis Salvador, 50. 12100 Castellón. España. E-mail: jpuig@terra.es
Based on a talk given at the XXI Congress of the Spanish Society of Family and Community Medicine, San Sebastián, 14-17 November 2001.
Awarded First prize by the PAPPS-semFYM/Almirall Prodesfarma at the XXI Congress of the Spanish Society of Family and Community Medicine.
Manuscript accepted for publication 20 February 2002.
