ABSTRACT
Quantifying the benefit of early antibiotic treatment is crucial for decision making and can be assessed only in observational studies. We performed a systematic review of prospective studies reporting the effect of appropriate empirical antibiotic treatment on all-cause mortality among adult inpatients with sepsis. Two reviewers independently extracted data. Risk of bias was assessed using the Newcastle-Ottawa score. We calculated unadjusted odds ratios (ORs) with 95% confidence intervals for each study and extracted adjusted ORs, with variance, methods, and covariates being used for adjustment. ORs were pooled using random-effects meta-analysis. We examined the effects of methodological and clinical confounders on results through subgroup analysis or mixed-effect meta-regression. Seventy studies were included, of which 48 provided an adjusted OR for inappropriate empirical antibiotic treatment. Inappropriate empirical antibiotic treatment was associated with significantly higher mortality in the unadjusted and adjusted comparisons, with considerable heterogeneity occurring in both analyses (I2 > 70%). Study design, time of mortality assessment, the reporting methods of the multivariable models, and the covariates used for adjustment were significantly associated with effect size. Septic shock was the only clinical variable significantly affecting results (it was associated with higher ORs). Studies adjusting for background conditions and sepsis severity reported a pooled adjusted OR of 1.60 (95% confidence interval = 1.37 to 1.86; 26 studies; number needed to treat to prevent one fatal outcome, 10 patients [95% confidence interval = 8 to 15]; I2 = 46.3%) given 34% mortality with inappropriate empirical treatment. Appropriate empirical antibiotic treatment is associated with a significant reduction in all-cause mortality. However, the methods used in the observational studies significantly affect the effect size reported. Methods of observational studies assessing the effects of antibiotic treatment should be improved and standardized.
Sepsis affects 1.1 to 2.4 per 1,000 people per year and 20 to 42% of these patients die in hospital, with these rates probably underestimating the contribution of hospital-acquired infections (3, 16, 61). Septicemia and pneumonia combined are the sixth most common causes of death in the United States (36). Antibiotic treatment for the first 24 to 48 h is largely empirical (i.e., provided without evidence on the causative pathogen or its susceptibilities), and it is common wisdom that appropriate empirical antibiotic treatment (i.e., matching the in vitro susceptibilities of the isolated pathogens) reduces mortality. Physicians thus strive to achieve appropriate empirical antibiotic treatment for inpatients with suspected infections, and many times this is at the cost of administering superfluous and unnecessary antibiotics. Such treatment is associated with resistance development (83, 97) and side effects with no benefit.
Estimates of the potential benefit of appropriate empirical antibiotic treatment vary widely in the literature between no effect (21, 22, 48, 70, 84, 88) and adjusted odds ratios (ORs) above 6 (39). The effects might be truly variable and dependent on infection severity, the patient's immune status, and the type of bacteria. Alternatively, heterogeneity might stem from methodological factors in observational studies, since assessment of the effects of early treatment relies by necessity on nonrandomized studies (34). These may include the covariates collected and used for adjustment of the effect of antibiotic treatment on mortality and the methods used for adjustment.
We conducted a systematic review with meta-analysis of studies assessing the effects of appropriate empirical antibiotic treatment on mortality. We aimed to investigate the reasons for heterogeneity in the magnitude of this effect and to obtain a better estimate of the true effect in general or specific clinical scenarios. Such an estimate is crucial to the decision making regarding antibiotic treatment.
(Preliminary results have been presented at the European Congress of Clinical Microbiology and Infectious Diseases, oral presentation, 17 May 2009, Helsinki, Finland.)
MATERIALS AND METHODS
Study selection. (i) Study design.We included prospective cohort studies, defined as those where cases were identified prospectively and data collection was started with identification. We judged that prospective data collection would result in the better and uniform availability of confounders for the adjusted analysis of mortality. We excluded studies published before 1975, using an arbitrary time point to denote an era in critical illness management that may be less relevant to current practice. We excluded studies that recruited less than 50 patients, assuming that with an average mortality of about 10%, an analysis including less than 5 outcomes has no power. We excluded studies assessing specifically meningitis and endocarditis, where treatment effects are expected to largely deviate from any common effect.
(ii) Patients.The patients included were adults (age, >18 years) with sepsis and microbiologically documented infections.
(iii) Intervention.The intervention was appropriate (versus inappropriate) empirical antibiotic treatment. “Empirical” treatment was defined as that administered prior to microbiological documentation of infection. “Appropriate” treatment had to be treatment matching the in vitro susceptibility of the pathogen. We permitted the inclusion of studies where up to 10% of pathogens were not tested in vitro (e.g., Mycoplasma pneumoniae); in these cases, the study definitions for appropriateness were accepted. We did not try to include antibiotic dosing, intrinsic antibiotic activity (e.g., vancomycin for methicillin-sensitive Staphylococcus aureus and aminoglycosides alone for Pseudomonas aeruginosa), or combination therapy in the definition of appropriateness, due to poor reporting of these definitions and the lack of evidence of their impact on mortality (73, 75), but we documented the definition and assessed its effect on outcomes.
(iv) Outcome.The outcome assessed was all-cause 30-day mortality. If 30-day data were not available, we used mortality at another fixed point in time or in-hospital mortality and documented the outcome assessed in the study.
Data sources and searches.We searched PubMed (January 1975 to November 2008) and references of all identified studies, using the following search strategy: ((antibiot* OR antimicrob* OR anti-bacter* OR antibacter*) AND (approp* OR inapprop* OR adequate OR inadequate) AND (mort* OR fatal* OR death OR dead OR alive OR survi*)). We did not include unpublished studies, since we needed a complete description of the study methods and analysis to investigate the reasons for heterogeneity. No language restrictions were applied.
Data extraction and quality assessment.Two reviewers independently inspected each reference identified by the search and applied inclusion criteria. In cases where the same population studied was analyzed in more then one publication, the study's results were accounted for only once. Trials fulfilling the review inclusion criteria were assessed for risk of bias by two reviewers, independently, using the Newcastle-Ottawa score (NOS) (96), adapted for our review (see the information on adapted NOS in the supplemental material). The score assigns a study a maximum of 8 points, with higher scores indicating a lower risk of bias. In addition, we documented the definitions of “appropriate” and “empirical,” the timing of mortality assessment, and the prospective components of the study (planning, patient detection, and data collection).
Two reviewers independently extracted the data. In case of disagreement between the two reviewers, a third reviewer extracted the data. Trial authors were contacted for clarification and to complete missing data. We collected the raw, unadjusted number of deaths among patients given appropriate versus inappropriate empirical antibiotic treatment. We extracted the adjusted effect estimate of appropriate empirical treatment for mortality with its variance and documented the method used for adjustment, the covariates assessed, and terms for inclusion in multivariable analyses, which were the variables finally included in the analysis and their significance. We collected descriptive data on setting, study years, follow-up duration, patient characteristics, types of pathogens, sources of infection, and presence of bacteremia.
Data synthesis and analysis. (i) Unadjusted (univariate) analysis.We computed odds ratios with 95% confidence intervals (CIs) for individual studies and pooled these in the meta-analysis. Null values precluding calculation of ORs were replaced by 0.5. We investigated heterogeneity through subgroup analyses and meta-regression on the basis of the study years; the prevalence of bacteremia, neutropenia, and pneumonia among the studied patients; the patients’ ages; the percentages of patients with septic shock and in an intensive care unit (ICU); the mean APACHE score; the prevalence of Pseudomonas aeruginosa, Staphylococcus aureus, and methicillin-resistant S. aureus (MRSA) infections; the study's adapted NOS score; and the other methodological variables assessed.
(ii) Adjusted analysis.Out of all 70 studies included, 22 did not report an adjusted analysis: in 13 the univariate results for appropriate empirical treatment were nonsignificant, and in 9 no adjusted analysis was conducted, despite the significance observed on univariate analysis, usually due to a small sample size. In the primary analysis, these 22 studies were excluded, since we could not impute adjusted ORs. All 48 studies reporting an adjusted effect of appropriate empirical treatment used multivariable regression analysis. Most studies provided the numerical results of appropriate empirical treatment in the final model, whether it was significantly associated with mortality or not. Six studies reported qualitatively that appropriate empirical treatment was not significantly associated with mortality, with no numerical values being given. In the main analysis we imputed an OR of 1 for these studies and used the standard error (SE) of the univariate analysis as the dispersion measure. Thus, the main adjusted analysis includes all studies that assessed the adjusted effect of appropriate empirical treatment on mortality, using either reported numerical results from a multivariable analysis (42 studies) or an OR equal to 1 when appropriate empirical treatment did not remain significant on multivariable analysis (6 studies). We conducted a sensitivity analysis, where studies that did not perform a multivariable analysis because the univariate appropriate empirical treatment results were nonsignificant (13 studies) were included in the analysis, with OR equal to 1 with the univariate analysis results’ SEs. Heterogeneity was investigated as for the univariate analysis, with an added assessment of the types of covariates being included in the multivariable analysis (e.g., disease severity and background conditions). Odds ratios were pooled with 95% confidence intervals or standard errors calculated from reported P values.
Statistical methods.All meta-analyses were conducted and reported using a random-effects model, assuming a priori significant heterogeneity resulting from diverse study populations and different models for adjusted analyses. Heterogeneity was assessed using a chi-square test of heterogeneity and the I2 measure of inconsistency. Subgroup analyses were performed using a mixed-effects analysis, where a random-effects model is used to combine studies within each subgroup and the study-to-study variance is computed within each subgroup. Mixed-effect univariate meta-regression was conducted using the unrestricted maximum-likelihood method to assess individual variables. The proportion of between-study variance explained by the covariates (R2) was assessed using random-effect multivariable meta-regression (35). A funnel plot of standard errors against log(ORs) was constructed for the univariate analysis that included all studies, to assess for the effect of small studies; significance (2-tailed) of the Begg and Mazumdar rank correlation test is reported. Analyses were performed using the Comprehensive Meta-Analysis (version 2.2) and Stata (version 10.1) programs.
RESULTS
Seventy individual trials (2, 5-9, 11-15, 17, 19, 20, 23-33, 37-40, 42-47, 50-60, 62, 64-67, 69, 71, 76-82, 85-87, 90-93, 95, 98, 99), out of 2,800 identified references, fulfilled the inclusion criteria (Fig. 1). Overall, 46.5% of patients were given inappropriate empirical antibiotic treatment, and the mortality among them was 35%. Study characteristics are shown in Table 1. Twenty-six studies were conducted in an ICU. Fifteen assessed one specific pathogen, while others assessed all bacteria. Forty-two studies addressed only bacteremic patients, and the rate of bacteremia in the other studies ranged from 0 to 70%. The mean adapted NOS score was 6.7 (standard deviation, 1.0).
Study flow. References to excluded studies are available from the authors upon request.
Characteristics of included studiesa
Unadjusted (univariate) analysis for mortality.All studies but one (76) reported unadjusted results for the effect of inappropriate empirical antibiotic treatment on all-cause mortality. The pooled OR was 2.11 (95% CI, 1.82 to 2.44, 69 studies, 21,338 patients; see the figure in the supplemental material). Considerable heterogeneity was observed between studies (P < 0.001, I2 = 72%). Three small studies (<70 patients each) were extreme outliers, with two reporting ORs of >70 (29, 31) and one reporting an OR of 0.046 (64). Excluding these, the OR in 66 studies was 2.10 (95% CI, 1.83 to 2.41), with heterogeneity being similar to that in all studies (P < 0.001, I2 = 69%). Sensitivity analyses were conducted on these 66 studies. Exclusion of the largest study (50) in the meta-analysis (OR = 2.07) did not alter the results or heterogeneity (OR = 2.11; 95% CI = 1.83 to 2.45, 65 studies, 17,742 patients, I2 = 69%).
Mortality was significantly higher with inappropriate empirical treatment in nearly all subgroups (Table 2 ). However, significant heterogeneity persisted in most subgroups, and none of the factors analyzed, except mortality time definition, yielded significantly different results between subgroups. Mortality defined at 28 to 30 days or some other fixed point of time was associated with lower ORs than in-hospital mortality or other time definitions, but the pooled ORs were statistically significant with all definitions. ORs were similar in studies conducted in or outside an ICU and with or without bacteremia. The OR was higher in studies assessing only P. aeruginosa infections and lower in studies assessing only MRSA infections compared to the OR in studies that assessed all bacteria; but only a few studies assessed individual pathogens, and the differences were not statistically significant.
Subgroup analysis to assess the effect of confounders on the association between appropriate empirical antibiotic treatment and all-cause mortalitya
Similarly, there was no association between the mean APACHE score, age, study year, or percentage of patients with septic shock or neutropenia in the meta-regression (Table 3). There was no significant association between risk ratios for mortality and the mortality rate in individual studies (ORs were not used for this analysis due to the inherent correlation between ORs and outcome rates). The funnel plot including all 69 studies was asymmetrical (P = 0.034), with small studies showing no benefit for appropriate empirical treatment possibly missing from the analysis (Fig. 2).
Funnel plot, unadjusted analysis. Included studies (open circles) are asymmetrically distributed around the pooled odds ratio (vertical line). A more symmetric funnel can be obtain by imputing values for missing studies (black circles), and it is apparent that the missing studies are small studies with ORs of <1, i.e., favoring inappropriate empirical antibiotic treatment.
Meta-regression analysis to assess the effect of confounders on the association between appropriate empirical antibiotic treatment and all-cause mortalitya
Adjusted (multivariable) analysis for mortality.All studies reporting adjusted risk factors for all-cause mortality performed multivariable analysis. Two studies included a propensity score for appropriate empirical treatment in the multivariable analysis (33, 54), and one study performed a propensity-matched analysis (23, 74). Propensity-adjusted effects were slightly smaller than those obtained by multivariable analysis, but only two studies permitted this comparison (54, 74).
The studies collected and assessed various risk factors for mortality for potential inclusion in the multivariable analysis (see the table in the supplemental material). Nearly all studies assessed age, place of acquisition, and source of infection. Formal scores for sepsis severity (e.g., the APACHE score) and underlying conditions (e.g., the Charlson score) were each used in only about 50% of the studies. The median ratio between the number of covariates included in the multivariable model and the number of deaths in the cohort was 8.1 (range, 2 to 51.1). Nine studies did not provide information on the number or type of covariates included.
The pooled adjusted OR of the main analysis was 2.05 (95% CI, 1.69 to 2.49; 48 studies; Fig. 3). Considerable heterogeneity also remained in the multivariable analysis (P < 0.001, I2 = 79.7%). In the sensitivity analysis, including “no-benefit” univariate studies, the OR was 1.79 (95% CI = 1.51 to 2.12, 61 studies, I2 = 78.9%). In 41 studies reporting both unadjusted and adjusted numerical results, the ORs were 2.35 (95% CI, 1.99 to 2.78) on univariate analysis and 2.32 (95% CI, 1.88 to 2.87) on multivariable analysis.
Adjusted analysis of the effect of appropriate empirical treatment on mortality, subgrouped by adjustment to sepsis severity and background conditions (0, no adjustment; 1, covariates representing sepsis severity and background conditions included in adjusted analysis).
As for the unadjusted analysis, a significant advantage to appropriate empirical treatment was maintained in most subgroups assessed. Significant differences between subgroups were observed for several variables, including the time point for mortality assessment, as above, where the advantage was smallest (though still significant) when 28- to 30-day mortality was assessed (Table 2). Studies specifically designed to assess the effects of appropriate empirical treatment were associated with higher ORs than other studies. When the study definition of appropriate empirical treatment included dosing, route, or duration considerations or when single-aminoglycoside therapy was considered inappropriate for Pseudomonas aeruginosa, ORs were lower than those for studies that defined appropriate empirical treatment only by in vitro matching. A high adapted NOS score (lower risk of bias) was associated with larger ORs, but there was little variability in the total score. Similarly, reporting and methods of the multivariable model were associated with the effects reported.
Twenty-eight and 26 studies reported on terms for inclusion of variables and the number of patients included in the model, respectively. Reporting was associated with significantly higher ORs. Only five studies reported on the methods of handling missing values for the variables included. This and the ratio between the number of covariates and the number of deaths were not significantly associated with ORs. Adjustment for background conditions in general and neutropenia in particular were significantly associated with lower ORs, while adjustment for sepsis severity was associated with nonsignificantly higher ORs. The setting (ICU versus non-ICU), assessment of bacteremic patients, pneumonia, or specific pathogens did not significantly affect ORs.
In meta-regression (Table 3), only septic shock was positively associated with ORs, with the ratio of ORs being 3.60 for every 1% increase in the prevalence of septic shock in the study population (95% CI, 1.11 to 11.65). There was a trend for ORs to increase with the study year, but this did not reach statistical significance. All variables significantly associated with ORs explained only a small proportion of between-study variance, where R2 was equal to 36.02% and rose to 52.5% in the set of studies that reported on the rate of septic shock at onset (Table 3, multivariable analysis). Only adjustment for background conditions was significantly associated with ORs in the multivariable meta-regression (coefficient, −0.53; standard error, 0.22).
Restricting the analysis to those trials that adjusted for background conditions (including neutropenia) and sepsis severity resulted in a pooled adjusted OR of 1.60 (95% CI, 1.37 to 1.86; 26 studies; Fig. 3), with moderate heterogeneity (46.3%).
DISCUSSION
Decision making regarding antibiotic treatment is unique. On one hand, no treatment equals the efficacy of antibiotics. To place the effect in context of other well-established interventions, the practice of administering aspirin in acute myocardial infarction is based on an OR of 1.30 (95% CI, 1.41 to 1.18) for 7 to 30 days of treatment (number of patients needed to treat [NNT] to prevent one fatal outcome, 41; 95% CI, 30 to 66 patients) (4, 41). The practice of administering low-molecular-weight heparin was estimated on the basis of an OR of 1.16 (95% CI, 1.05 to 1.28), and the NNT is 63 patients (95% CI, 37 to 193) (18). Most interventions in medicine are not based on improved crude survival (e.g., beta-blockers during acute myocardial infarction [1]). In comparison, the pooled odds ratio of appropriate antibiotic treatment during the first 48 h for all-cause mortality in our review was 1.60 (95% CI, 1.37 to 1.86), corresponding to an NNT of 10 (95% CI, 8 to 15), in the set of studies adjusting for background conditions and sepsis severity. Thus, the drive for prescription of antibiotics to patients with suspected infection is clear. On the other hand, there is no other instance in medicine where treatment given to the individual patient affects other patients and the society at large. Present prescription of an antibiotic or a policy to use an antibiotic might mean the loss of availability of this antibiotic and similar antibiotics for future patients (10). In an era of increasing antibiotic resistance, prescription of an antibiotic to one patient might mean no available treatment for future patients (83). The bulk of antibiotic consumption is empirical (72). The balance between preventing deaths from infections and using antibiotics judiciously to prevent resistance development is largely determined by our belief in the benefit of appropriate empirical antibiotic treatment and the magnitude of the benefit.
Estimation of this effect relies on observational studies, since a randomized trial would be unethical. It is difficult to predict the direction of bias caused by the nonrandom allocation of patients to appropriate versus inappropriate empirical treatment. Patients given appropriate empirical treatment might have been more critically ill and thus prescribed broader-spectrum treatment. Conversely, they might have been carriers of more susceptible bacteria and thus healthier (68). Patients with guarded short-term prognoses because of severe underlying conditions might be given inappropriate treatment because antibiotics (or broad-spectrum antibiotics) might be considered futile.
We observed considerable heterogeneity between the studies, with adjusted effects ranging between no effect and ORs above 15. We expected heterogeneity to stem from clinical variables related to patient and infection characteristics. However, only a few clinical variables could be shown to affect results. The percentage of patients with septic shock at onset of infection and adjustment for septic shock were associated with higher ORs, pointing at a larger benefit of appropriate empirical antibiotic treatment among patients with septic shock at infection onset. None of the other clinical variables affected the results, including the study year and setting, the patient's age, presence of bacteremia, source of infection, presence of neutropenia, and causative bacteria, although analysis of the last two variables was based on few studies.
Many methodological variables significantly affected the ORs. Prospective planning, intervention definitions, and follow-up duration impacted OR estimates. Less than half of the studies provided a clear description of the terms for inclusion of variables in the multivariable analysis and the number of patients included in the analysis, and nearly none described the methods used to deal with missing data. Adequate reporting was associated with higher ORs. The number of covariates was frequently high in relation to the number of outcomes in the cohort, and significance or the performance of the model was rarely presented (data not shown). The studies used different risk factors in the multivariable models. Adjustment for background conditions was the most significant variable affecting ORs, where adjustment was associated with smaller effects. It has previously been shown that adjustment for disease severity measures before infection onset (at admission and 24 h before infection onset) is associated with smaller effect estimates for the association between appropriate empirical antibiotic treatment and mortality (89). We could not assess the effects of disease severity measures before infection onset on the results because these were not reported (63), but our findings regarding background conditions probably reflect the same trend. The NOS, whose use is recommended for risk of bias assessment in cohort studies, was not very informative because of the small variability between the studies.
Several limitations of our analysis should be noted. We needed to use assumptions to be able to conduct the meta-analysis, such as the imputation of an OR of 1 for studies reporting qualitatively that appropriate empirical treatment was not significantly associated with mortality on multivariable analysis. For the main analysis, our assumptions were chosen to obtain a conservative effect estimate (it is likely that in these studies the OR was higher than 1 and statistically nonsignificant). Sensitivity analyses showed that results were robust with different assumptions. Publication bias was suggested in our analysis and is partially due to the fact that studies that did not find a significant effect of appropriate empirical treatment on mortality reported results qualitatively and could not be included because no numerical data were reported (22). Infections that are not typically documented microbiologically, mainly community-acquired pneumonia, are ill represented in our analysis. Finally, despite detailed analysis of clinical and methodological variables, we could not fully explain the observed heterogeneity between the studies.
In summary, we showed that, overall, inappropriate empirical antibiotic treatment is significantly associated with all-cause mortality in prospective studies. However, the estimated effect of appropriate empirical antibiotic treatment on mortality reported in observational studies is highly variable. The main determinants of the magnitude of the effect are methodological and relate to study design, outcome definitions, availability of risk factors for adjusted analysis, and the methods used in the multivariable analysis.
Future cohort studies should adhere to the Strengthening the Reporting of Observational Studies in Epidemiology guidelines for reporting of observational studies (94) and to existing guidance on reporting of multivariable logistic regression. Specifically, on the basis of our and previous analyses (63, 89), studies should assess 30-day mortality rather than in-hospital or other unfixed follow-up and adjust the effect of appropriate antibiotic treatment for underlying disorders, disease severity before infection onset, and sepsis severity at onset of infection. The same applies for randomized controlled trials of antibiotic or nonantibiotic treatments for sepsis. Future studies should attempt to quantify the negative ecological impact of unnecessary and superfluous antibiotic treatment using the same outcome measures by which appropriate empirical treatment is measured, loss-of-life years. The loss to both the individual treated and society should be accounted for (49).
ACKNOWLEDGMENTS
This study was supported in part by a grant from the Rothschild Caesarea Foundation, Optimal antibiotic treatment of moderate to severe bacterial infections: integration of PCR technology and medical informatics/artificial intelligence.
None of us has a conflict of interest to declare.
FOOTNOTES
- Received 6 May 2010.
- Returned for modification 6 July 2010.
- Accepted 14 August 2010.
- Copyright © 2010 American Society for Microbiology
REFERENCES
- 1.↵
- 2.↵
- 3.↵
- 4.↵
- 5.↵
- 6.
- 7.
- 8.
- 9.↵
- 10.↵
- 11.↵
- 12.
- 13.
- 14.
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.↵
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.↵
- 30.
- 31.↵
- 32.
- 33.↵
- 34.↵
- 35.↵
- 36.↵
- 37.↵
- 38.
- 39.↵
- 40.↵
- 41.↵
- 42.↵
- 43.
- 44.
- 45.
- 46.
- 47.↵
- 48.↵
- 49.↵
- 50.↵
- 51.
- 52.
- 53.
- 54.↵
- 55.
- 56.
- 57.
- 58.
- 59.
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.
- 66.
- 67.↵
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.↵
- 73.↵
- 74.↵
- 75.↵
- 76.↵
- 77.
- 78.
- 79.
- 80.
- 81.
- 82.↵
- 83.↵
- 84.↵
- 85.↵
- 86.
- 87.↵
- 88.↵
- 89.↵
- 90.↵
- 91.
- 92.
- 93.↵
- 94.↵
- 95.↵
- 96.↵
- 97.↵
- 98.↵
- 99.↵