Validation of a predictive model for bacteraemia (MPB5-Toledo) in the patients seen in emergency departments due to infections

Abstract

Objective

To validate a simple risk score to predict bacteremia (MPB5-Toledo) in patients seen in the emergency departments (ED) due to infections.

Methods

Prospective and multicenter observational cohort study of the blood cultures (BC) ordered in 74 Spanish ED for adults (aged 18 or older) seen from October 1, 2019, to February 29, 2020.

The predictive ability of the model was analyzed with the area under the Receiver Operating Characteristic curve (AUC-ROC). The prognostic performance for true bacteremia was calculated with the cut-off values chosen for getting the sensitivity, specificity, positive predictive value and negative predictive value.

Results

A total of 3.843 blood samples wered cultured. True cases of bacteremia were confirmed in 839 (21.83%). The remaining 3.004 cultures (78.17%) were negative. Among the negative, 172 (4.47%) were judged to be contaminated. Low risk for bacteremia was indicated by a score of 0–2 points, intermediate risk by 3–5 points, and high risk by 6–8 points. Bacteremia in these 3 risk groups was predicted for 1.5%, 16.8%, and 81.6%, respectively. The model’s area under the receiver operating characteristic curve was 0.930 (95% CI, 0.916−0.948). The prognostic performance with a model’s cut-off value of ≥5 points achieved 94.76% (95% CI: 92.97−96.12) sensitivity, 81.56% (95% CI: 80.11−82.92) specificity, and negative predictive value of 98.24% (95% CI: 97.62−98.70).

Conclusion

The 5MPB-Toledo score is useful for predicting bacteremia in patients attended in hospital emergency departments for infection.

Resumen

Objetivo

Validar un modelo sencillo de riesgo para predecir bacteriemia (5MPB-Toledo) en los pacientes atendidos en los servicios de urgencias hospitalarios (SUH) por un episodio de infección.

Métodos

Estudio observacional de cohortes prospectivo y multicéntrico de los hemocultivos (HC) obtenidos en 74 SUH españoles en los pacientes adultos (≥18 años) atendidos por infección desde el 1 de octubre de 2019 hasta el 29 de febrero de 2020. Se analizó la capacidad predictiva del modelo con el área bajo la curva (ABC) de la característica operativa del receptor (COR) y se calculó el rendimiento diagnóstico de los puntos de corte (PC) del modelo elegidos con los cálculos de la sensibilidad, la especificidad, el valor predictivo positivo y el valor predictivo negativo.

Resultados

Se incluyeron 3.843 episodios de HC extraídos. De ellos, se consideraron como bacteriemias verdaderas 839 (21,83%) y como HC negativos 3.004 (78,17%). Entre los negativos, 172 (4,47%) se consideraron contaminados. Se categorizó a los pacientes en bajo (0–2 puntos), moderado (3–5 puntos) y alto (6–8 puntos) riesgo, con una probabilidad de bacteriemia de 1,5%, 16,8% y 81,6%, respectivamente. El ABC-COR del modelo tras remuestreo fue de 0,930 (IC 95%: 0,916−0,948). El rendimiento diagnóstico del modelo con un PC ≥ 5 puntos consigue una sensibilidad de 94,76% (IC 95%: 92,97−96,12), especificidad de 81,56% (IC 95%: 80,11–82,92) y un valor predictivo negativo de 98,24% (IC 95%: 97,62–98,70).

Conclusión

El modelo 5MPB-Toledo es de utilidad para predecir bacteriemia en los pacientes atendidos en el SUH por un episodio de infección.

Introduction

Bacteraemia is defined as the presence of bacteria in blood, demonstrated by isolation of bacteria in blood cultures (BCs).¹ Despite new technologies for rapid detection (of pathogen DNA or through the use of mass spectrometry), BCs enable aetiological diagnosis of infection, provide information on micro-organism sensitivity and promote optimisation of antimicrobial treatment.^2–4

Recently, it was confirmed that around 15% of patients seen at hospital accident and emergency departments (HAEDs) in Spain are diagnosed with an infectious disease. In their initial care, they have samples taken for microbiological tests in 43% of cases. Predominant among them are BCs, which are performed in 14.6% of patients seen with suspected or confirmed infection in HAEDs themselves.⁵ The diagnostic performance of these tests is highly variable (2%–20%),^6,7 while “contaminated BCs” may account for up to 30%–50% of isolates.⁸ Moreover, BCs with significant isolation in patients discharged from HAEDs (B-PDHAEDs) may correspond to 3%–5% of BCs taken in HAEDs.⁹ These facts are genuinely problematic as they translate to larger numbers of diagnostic tests, longer hospital stays, unnecessary costs and administration of needless antibiotic treatments, as well as, where applicable, improper discharges in cases of B-PDHAEDs.^6,9,10

In addition, the incidence of community-acquired bacteraemia has increased to one to two per 1000 patients seen in HAEDs and to six to 10 episodes per 1000 hospital admissions from HAEDs.^6,7 The agents responsible are Gram-positive bacteria in 30%–45% of cases, Gram-negative bacteria in 55%–70% and anaerobic bacteria in around 1%–3%.^6–10 This proportion may change, if the incidence of contaminated BCs were excessive, in favour of Gram-positive bacteria.⁸ Among cases of true or significant bacteraemia, with regard to focus, urinary tract infection (45%–55%) and respiratory infection (10%–25%) are the most common foci, while bacteraemia with an unknown focus accounts for around 10% of cases in HAEDs. The most commonly isolated bacteria overall are Escherichia coli, Staphylococcus aureus and Streptococcus pneumoniae.^6,7,10

The 30-day mortality rates in patients with true bacteraemia have been estimated at around 10%–25% — higher than all other patients with infectious disease. This is linked to the seriousness of the clinical situation, the primary focus and the characteristics of the patients (age, comorbidities, etc.).^6,7,11

Therefore, suspicion and confirmation of true bacteraemia carry substantial diagnostic and prognostic value and necessarily influence some of the most important decisions to be made in HAEDs. Such decisions include indicating discharge or admission, performing BCs and administering the right antimicrobial agent early.¹² In this regard, preparing models for predicting true bacteraemia identifiable in HAEDs that aid in preventing improper discharges and unnecessary admissions, and the consequences thereof, has become a goal for many authors. These models include, in different proposed scales, clinical, epidemiological and laboratory variables.^13–19 A recently published study offered a model for predicting bacteraemia with five variables (the Toledo five-variable model for predicting bacteraemia [5MPB-Toledo]).²⁰ The model includes temperature >38.3 °C (1 point), Charlson Comorbidity Index ≥3 (1 point), respiratory rate ≥22 respirations per minute (1 point), leukocyte count >12,000/mm³ (1 point) and procalcitonin (PCT) levels ≥0.51 ng/mL (4 points). Patients are categorised as low-risk (0–2 points), moderate-risk (3–5 points) or high-risk (6–8 points), with a likelihood of bacteraemia of 1.1%, 10.5% or 77%, respectively. The model's area under the receiver operating characteristic (ROC) curve (AUC) after bootstrapping was excellent: 0.946 (95% CI: 0.922−0.969).²⁰

The primary objective of this study was to perform an external validation of the model for predicting bacteraemia (5MPB-Toledo) in patients seen for infection in HAEDs. The secondary objective was to analyse the scale's predictive performance and cut-off points.

Patients and methods

Study design

This was an observational, multicentre, prospective, descriptive and analytical study of BCs performed in 74 HAEDs in Spain in adult patients (≥18 years of age) seen due to clinical suspicion of an infectious disease who underwent follow-up for 30 days and maintained a diagnosis of infection after this period.

Study sites

The participating sites belonged to the Sociedad Española de Medicina de Urgencias y Emergencias [Spanish Society of Urgent and Emergency Medicine] Infection Group (INFURG-SEMES) (Appendix A).

Study periods and population included

Between 1 October 2019 and 29 February 2020, BCs obtained from patients clinically diagnosed with an infectious disease who, as a requirement, also had recorded data for the five variables of the 5MPB-Toledo (temperature, Charlson Comorbidity Index, respiratory rate, leukocyte count and PCT levels) were included in a process of opportunity sampling (when the investigators were on duty). Paediatric patients as well as obstetrics and gynaecology patients were excluded. BCs were indicated and ordered at the discretion of the physician on duty.

Definitions, techniques and methods established for the samples

BCs were performed using the standard technique, with blood drawn by percutaneous venipuncture. In each patient, two separate blood samples were drawn at different points in time (and care was taken to draw them from different venipuncture sites). If endocarditis was suspected, three pairs of BCs were obtained. For each BC, two bottles (BD BACTEC®) were spiked: one with a medium for aerobiosis and another for anaerobiosis. According to protocol, the incubation time for the BCs was five to seven days, except in suspected endocarditis or brucellosis or by order of the physician on duty, in which cases it was extended to up to 30 days.

True (or significant) bacteraemia was defined as isolation of normally pathogenic bacteria in one or both of the two BCs with consistent signs and symptoms. Contaminated BC was defined as isolation in a single bottle of BCs of coagulase-negative Staphylococcus (CoNS), Bacillus spp., Streptococcus from the group viridans, Micrococcus spp., Propionibacterium spp., Corynebacterium spp. or other Gram-positive bacilli when the absence of clinical significance was interpreted in these cases (confirmed based on the medical record and/or at the discretion of the physician on duty and/or microbiology). In other cases, if patients had two positive BCs and a clinical significance attributed to them (especially if they were immunosuppressed, had a vascular catheter placed or had undergone invasive tests), they were considered to have true bacteraemia and treated with antibiotics.

For biomarkers, the reference values and methodology used in other studies by the INFURG-SEMES were adopted, by consensus. For C-reactive protein (CRP) with a method of determination by quantitative enzyme immunoassay: 0−8 mg/l, with a sensitivity of 1 mg/l. For PCT by quantitative electrochemiluminescence immunoassay with reference values: <0.5 ng/mL, with a sensitivity of 0.02 ng/mL.

Variables collected

The outcome variable was the presence of true bacteraemia. The following independent variables were collected: socio-demographic data (age, sex and institutionalisation), antibiotics taken in the past 72 h and/or the past three months, admission in the past month and comorbidities (solid or haematological tumour disease, liver disease, kidney disease, diabetes mellitus, chronic heart or cerebrovascular disease, chronic obstructive pulmonary disease, peripheral artery or connective tissue disease, state of immunosuppression, treatment with corticosteroids, or human immunodeficiency virus infection). The Charlson Comorbidity Index²¹ was calculated (and dichotomised ≥3 points); the Barthel Index²² of performance in activities of daily living was determined (and dichotomised ≤60) as well.

In addition, the following clinical and seriousness-related data were recorded: temperature in degrees centigrade (°C); altered level of consciousness, defined as <15 points on the Glasgow Coma Scale (GCS); systolic blood pressure (SBP); criteria for sepsis, serious sepsis or septic shock; and the variables that define those types of sepsis according to the 2001 sepsis expert conference.²³ The criteria for prognostic screening of patients in the definitions of the quick Sepsis Related Organ Failure Assessment (qSOFA) ≥2 and the variables that constitute them according to the third sepsis conference (SEPSIS-3) were applied.²⁴ The following clinical course- and destination-related variables were included: days of prior signs and symptoms, initial destination of the patients, length of hospital stay in days, further visits to the HAED in the 30 days thereafter, hospital mortality and 30-day mortality. Finally, in relation to laboratory data, the following were recorded: leukocyte count (as well as leukocytosis >12,000/mm³, leukopaenia <4000/mm³ or band neutrophils >10%), CRP levels in mg/l (dichotomised for ≥9 mg/l and for ≥21 mg/l) and PCT levels in ng/mL (dichotomised for the CPs selected based on prior studies of ≥0.43, ≥0.51 and ≥1 ng/mL).^8,20,25

Statistical analysis

Means and their standard deviations (SDs) were used for quantitative variables, and absolute numbers and percentages were used for qualitative variables. The chi-squared test or Fisher's exact test, Student's t-test and the Mann–Whitney U test were used, as applicable, to investigate the relationship between true bacteraemia versus negative BCs (contaminated BCs and BCs with no isolates) and the independent variables (and those that were dichotomised). A P value <.05 was considered significant; comparisons were bilateral. A descriptive analysis (absolute numbers and percentages) of both groups (true bacteraemia versus negative BCs) in relation to type of pathogen was performed, as was a differentiated analysis based on isolation of Gram-positive, Gram-negative or anaerobic bacteria and based on the focus or clinical diagnosis made in the HAED.

First, for the analysis of the behaviour of the primary scale in the sample for this study, a risk-scoring system was constructed in which a score was assigned to each variable of the 5MPB-Toledo according to the original design.²⁰ The risk score was calculated for each patient by adding up the points for each factor present. Subjects were divided into low-risk, moderate-risk and high-risk groups, based on the predicted probabilities of the model. The discrimination capacity of the predictive model was analysed by calculating the AUC and its 95% confidence interval (95% CI). The calibration of the model was evaluated using the Hosmer–Lemeshow goodness-of-fit test. Subsequently, the result obtained was internally validated by means of bootstrapping with 1000 resamplings, and the AUC and its 95% CI were calculated. The standard errors of the AUCs were calculated using non-parametric methods. In certain subgroups of special clinical interest, the model's performance was likewise analysed through their AUCs and 95% CIs. In addition, a box plot was used to show the dispersion and the relationship between the different scores on the scale and the probability of having true bacteraemia.

Second, the cut-off point (CP) of the results of the model (from 1 to 8) with the greatest diagnostic capacity that maximised the difference between the rate of true positives and false positives was determined using Youden's J statistic. Thus, the following were calculated: the diagnostic performance of these CPs of the model selected with calculations of sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), the positive coefficient of probability (PCP) and the negative coefficient of probability (NCP) for each result studied, as well as their 95% CIs by exact binomial methods and by Taylor's method for the CPs.

In all the comparisons, the null hypothesis was rejected with an alpha level lower than 0.5. Statistical analysis was performed with the IBM-SPSS® Statistics software package, ver. 22 for Windows, and the Stata statics software package, ver. 12.0.

Ethical considerations

The study complied with all international and site-specific protocols and standards (the Declaration of Helsinki) for the use of patient data which were encoded to ensure the confidentiality thereof. The study was evaluated and approved by the Complejo Hospitalario Universitario de Toledo [Toledo University Hospital Complex] Independent Ethics Committee (IEC) (no.: 398/2109), as well as by the reference IECs/IECs for research involving medicinal products (mIECs) of the participating sites. All patients gave their written informed consent to participate in the study.

Results

During the study period, 4066 patients who met the inclusion criteria were selected by opportunity sampling. Of these, 223 were excluded (36 because they did not have a microbiological diagnosis confirmed in the BCs, 137 because they were lost to follow-up or switched to another diagnosis within the 30 days after their visit to the HAED and 50 because they had semiquantitative determination of PCT). Ultimately, 3843 patients from whom at least two pairs of BCs were obtained were included. The mean age of the patients was 67 (SD: 19) years with a range of 18–101 years. Of them, 59.5% (2287) were over 65 years of age and 2346 (61%) were male.

After microbiological testing, 2832 (73.69%) BCs were negative. Isolates were obtained in 1011 (26.31%) BCs; among them, 172 (4.47%) were considered contaminated BCs and 839 (21.83%) were considered true bacteraemia (13 [1.27%] were polymicrobial). The episode inclusion flow chart is shown in Fig. 1.

Fig. 1.

Flow chart of inclusion of cases.

*Percentage of all patients included in the study (N = 3843).

Finally, it should be noted that 70 (8.34%) of cases of true bacteraemia were classified as B-PDHAEDs in patients discharged directly after assessment in the HAED or after having remained under observation for several hours.

Aetiologies, grouped and by micro-organism, in cases of true bacteraemia and contaminated BCs are set out in Table 1. The most common isolates in cases of true bacteraemia were Escherichia coli with or without extended-spectrum beta-lactamases (ESBLs) (336/839; 40.04%) and Streptococcus pneumoniae (143/839 cases; 17.04%). Escherichia coli was also the most commonly isolated pathogen in B-PDHAEDs (31/70; 44.29%). Regarding contaminated BCs, the most common were coagulase-negative Staphylococcus (117/172; 68.02%).

Table 1.

Microbiological characteristics of the overall sample by isolate type (true bacteraemia versus contaminated blood cultures).

Micro-organisms	Total N = 1011 n (%)	True bacteraemia N = 839 n (%)	Contaminated blood cultures N = 172 n (%)
Gram-negative bacteria; 530 (52.42%)
Escherichia coli^a	336 (33.23)	336 (40.04)	0 (0.0)
Klebsiella pneumoniae^a	72 (7.12)	72 (8.58)	0 (0.0)
Proteus spp.	28 (2.77)	28 (3.33)	0 (0.0)
Pseudomonas spp.^a	24 (2.37)	24 (2.86)	0 (0.0)
Haemophilus influenzae	11 (1.08)	11 (1.31)	0 (0.0)
Klebsiella spp.	7 (0.69)	7 (0.83)	0 (0.0)
Neisseria meningitidis	5 (0.49)	5 (0.59)	0 (0.0)
Salmonella spp.	5 (0.49)	5 (0.59)	0 (0.0)
Serratia spp.	6 (0.59)	6 (0.71)	0 (0.0)
Enterobacter spp.	6 (0.59)	6 (0.71)	0 (0.0)
Other Gram-negative bacteria^b	30 (2.97)	30 (3.57)	0 (0.0)
Gram-positive bacteria; 462 (45.70%)
Streptococcus pneumoniae	143 (14.14)	143 (17.04)	0 (0.0)
Coagulase-negative Staphylococcus (CoNS)^c	118 (11.67)	1 (0.11)	117 (68.02)
Enterococcus faecalis	37 (3.66)	37 (4.41)	0 (0.0)
Enterococcus faecium	22 (2.18)	22 (2.62)	0 (0.0)
Propioniobacterium spp.	22 (2.18)	0 (0.0)	22 (12.79)
Micrococcus spp.	16 (1.58)	0 (0.0)	16 (9.30)
Corynebacterium spp.	15 (1.48)	0 (0.0)	15 (8.72)
Staphylococcus aureus	48 (4.74)	48 (5.72)	0 (0.0)
Streptococcus pyogenes	14 (1.38)	14 (1.67)	0 (0.0)
Streptococcus viridans	4 (0.40)	2 (0.24)	2 (1.16)
Listeria monocytogenes	5 (0.49)	5 (0.59)	0 (0.0)
MRSA	9 (0.89)	9 (1.07)	0 (0.0)
Other Gram-positive bacteria (Streptococcus spp.)	9 (0.89)	9 (1.07)	0 (0.0)
Anaerobic bacteria; 6 (0.59%)
Bacteroides spp.	2 (0.20)	2 (0.24)	0 (0.0)
Clostridium spp.	2 (0.20)	2 (0.24)	0 (0.0)
Other anaerobic bacteria^d	2 (0.20)	2 (0.24)	0 (0.0)
Mixed; 13 (1.29%)^e	13 (1.29)	13 (1.55)	0 (0.0)

MRSA: methicillin-resistant Staphylococcus aureus.

a

Includes pathogens that are either carriers or non-carriers of extended-spectrum beta-lactamases (ESBLs).

b

Other Gram-negative bacteria: Morganella morganii, Hafnia alvei, Acinetobacter baumannii and Stenotrophomonas maltophilia.

c

Coagulase-negative Staphylococcus (CoNS): Staphylococcus epidermidis, Staphylococcus hominis, Staphylococcus hominis-hominis, Staphylococcus haemolyticus, Staphylococcus lugdunensis, Staphylococcus capitis, Staphylococcus simulans and Staphylococcus warneri.

d

Other anaerobic bacteria: Prevotella spp. and Fusobacterium spp.

e

Mixed: Escherichia coli + Klebsiella spp.; E. coli + Enterococcus spp.; E. coli + Salmonella spp.; Enterococcus spp. + Pseudomonas spp. and Proteus spp. + Enterobacter spp.

Table 2 shows the presumptive focus or clinical origin in the HAED in cases of true bacteraemia and negative BCs.

Table 2.

Presumptive focus/clinical diagnosis in the accident and emergency department in the overall sample based on the presence or absence of true bacteraemia.

Focus/clinical diagnosis	Total N = 3843 n (%)	True bacteraemia N = 839 n (%)	Negative blood cultures^a N = 3004 n (%)
Respiratory infection	1574 (40.96)	176 (20.98)	1398 (46.54)
Urinary tract infection	1228 (31.95)	367 (43.74)	861 (28.66)
Abdominal infection	489 (12.72)	163 (19.43)	326 (10.85)
Fever with no clear focus	251 (6.53)	50 (5.96)	201 (6.69)
Skin or soft-tissue infection	176 (4.58)	42 (5.01)	134 (4.46)
Central nervous system infection	34 (0.88)	16 (1.91)	18 (0.60)
Other foci^b	91 (2.37)	25 (2.98)	66 (2.20)

a

Negative blood cultures: includes the 2832 with no isolates and the 172 defined as contaminated.

b

Other foci: otorhinolaryngological, suspected endocarditis due to external devices, etc.

Table 3 shows the characteristics of the patients in the overall study sample in terms of socio-demographics, epidemiology, comorbidities, performance in activities of daily living, signs and symptoms, seriousness, clinical course and destination. Table 1S (supplementary materials) shows those same characteristics with the data from the comparative study of cases of true bacteraemia versus negative BCs.

Table 3.

Clinical/epidemiological and clinical course-, destination- and laboratory-related characteristics of the overall sample.

	Total N = 3843 (%)	Missing values (%)
Demographic/epidemiological data
Age (years); mean (SD)	67.08 (18.58)	0 (0.00)
Age >65 years	2287 (59.5)	0 (0.0)
Male gender	2346 (61.0)	0 (0.0)
Institutionalised	329 (8.6)	0 (0.0)
AB use in past three months	1334 (36.1)	146 (3.80)
AB use in past 72 h	607 (15.8)	2525 (65.70)
Admission in past month	402 (10.5)	0 (0.0)
Comorbidities
Solid neoplasm	449 (11.7)	0 (0.0)
Leukaemia/lymphoma	153 (3.98)	0 (0.0)
State of immunosuppression	541 (14.07)	0 (0.0)
Treatment with corticosteroids	215 (5.59)	0 (0.0)
Liver disease	99 (2.6)	0 (0.0)
Chronic heart disease	688 (17.9)	0 (0.0)
Chronic kidney disease	376 (9.8)	0 (0.0)
Cardiovascular disease	391 (10.2)	0 (0.0)
COPD	644 (16.8)	0 (0.0)
Diabetes mellitus	741 (19.3)	0 (0.0)
Peripheral artery disease	253 (6.6)	0 (0.0)
Connective tissue disease	96 (2.5)	0 (0.0)
HIV	28 (0.7)	0 (0.0)
Charlson Comorbidity Index^a; mean (SD)	2.75 (2.53)	0 (0.00)
Charlson Comorbidity Index ≥3	1754 (45.6)	0 (0.0)
Barthel Index^b; mean (SD)	83.55 (30.28)	122 (3.17)
Barthel Index ≤60	692 (18.6)	0 (0.0)
Clinical and seriousness-related data
Temperature in degrees C; mean (SD)	37.86 (1.06)	0 (0.0)
Temperature >38.3 °C	1310 (34.1)	0 (0.0)
HR in bpm; mean (SD)	100.27 (20.46)	40 (1.04)
HR >90 bpm	2651 (69.7)	40 (1.04)
RR in rpm; mean (SD)	21.19 (7.12)	0 (0.00)
RR ≥22 rpm	1512 (39.3)	0 (0.00)
Altered level of consciousness (GCS) ≤14	717 (18.66)	76 (1.98)
SBP in mmHg; mean (SD)	122.26 (27.63)	13 (0.34)
SBP <100 mmHg	883 (23.1)	13 (0.34)
Criteria for sepsis (SRIS ≥ 2)	2540 (66.09)	42 (1.09)
Criteria for serious sepsis	1260 (32.78)	55 (1.43)
Criteria for septic shock	525 (13.66)	55 (1.43)
qSOFA ≥2	697 (18.14)	89 (2.32)
Clinical course- and destination-related data
Days since onset of signs and symptoms; mean (SD)	3.80 (8.24)	110 (2.86)
Initial destination of patients		0 (0.0)
Discharge	636 (16.55)	0 (0.0)
Observation	177 (4.60)	0 (0.0)
Short-stay unit	121 (3.14)	0 (0.0)
Hospital ward	2331 (60.65)	0 (0.0)
Operating theatre/admission to surgery	118 (3.07)	0 (0.0)
Intensive care unit	263 (6.84)	0 (0.0)
Hospital transfer	85 (2.21)	0 (0.0)
Death in accident and emergency department	34 (0.88)	0 (0.0)
Home hospitalisation	78 (2.03)	0 (0.0)
Hospital stay in days; mean (SD)	7.79 (8.20)	354 (9.21)
Further visits after discharge from the accident and emergency department	558 (14.52)	0 (0.0)
30-day mortality	396 (10.30)	0 (0.0)
In-hospital mortality	345 (8.97)	0 (0.0)
Laboratory findings
Leukocytes per mm3; mean (SD)	14,312 (11,942)	0 (0.0)
Leukocytosis >12,000/mm^3	2183 (56.76)	0 (0.0)
Leukocytes <4000 mm^3	159 (4.14)	0 (0.0)
Band neutrophils >10%	214 (7.5)	987 (25.68)
Platelets per mm3; mean (SD)	218,370 (105,067)	23 (0.60)
Procalcitonin in ng/mL; mean (SD)	3.39 (10.17)	23 (0.60)
Procalcitonin ≥0.43 ng/mL	1566 (40.7)	0 (0.0)
Procalcitonin ≥0.51 ng/mL	1406 (36.6)	0 (0.0)
Procalcitonin ≥1 ng/mL	1101 (28.7)	0 (0.0)
C-reactive protein in mg/l; mean (SD)	15.11 (12.16)	310 (8.2)
C-reactive protein ≥9 mg/l	2141 (60.6)	310 (8.2)
C-reactive protein ≥21 mg/l	995 (28.2)	310 (8.2)

AB: antibiotics; bpm: beats per minute; C: centigrade; COPD: chronic obstructive pulmonary disease; h: hours; HIV: human immunodeficiency virus; HR: heart rate; n: number; qSOFA: quick Sepsis Related Organ Failure Assessment; rpm: respirations per minute; RR: respiratory rate; SD: standard deviation.

*Negative blood cultures: includes the 2832 with no isolates and the 172 defined as contaminated.

a

Charlson Comorbidity Index: weighted by age.²¹

b

Barthel Index.²² Criteria for sepsis (SRIS ≥ 2) according to 2001 consensus conference.²³ Criteria for sepsis (qSOFA ≥ 2) according to the third consensus conference (SEPSIS-3).²⁴

In addition, Table 3 shows the overall study sample results of the significant laboratory determinations such as absolute leukocyte count; leukocytosis >12,000/mm³; band neutrophils >10% and leukopaenia <4000/mm³; platelet count and thrombocytopaenia <150,000/mm³; and PCT and PCR levels, both absolute and according to the predetermined CPs. Table 2S (supplementary materials) shows those same variables with the data from the comparative study of cases of true bacteraemia versus negative BCs.

The 30-day mortality of the patients diagnosed with infection was 10.30%, while in patients with true bacteraemia versus patients with negative BCs it was 15.97% versus 8.72% (P < .001).

Behaviour of the scale and analysis of the risk groups of the 5MPB-Toledo in the validation study sample

Fig. 2 shows the 5MPB-Toledo scoring scale (temperature >38.3 °C, Charlson Comorbidity Index ≥3, respiratory rate [RR] ≥22, leukocytosis >12,000/mm³ and PCT ≥ 0.51 ng/mL), as well as the value and weight of each variable of the model, and the probability of the original model and of the validation cohort based on category—low risk (0–2 points), moderate risk (3–5 points) or high risk (6–8 points) of bacteraemia—which was 1.1% versus 1.5%, 10.5% versus 16.8% and 77% versus 81.6%, respectively.

Fig. 2.

Probabilities of bacteraemia according to the risk groups of the 5MPB-Toledo.

The percentages of patients included in the low, moderate and high risk groups of the original 5MPB-Toledo and the validation cohort were 65.24% versus 52.5%, 23.44% versus 20.4% and 11.32% versus 27.1%, respectively. Among the cases included, 336 (8.7%) had 0 points, 860 (22.4%) had 1 point, 823 (21.4%) had 2 points, 348 (9.1%) had 3 points, 127 (3.3%) had 4 points, 309 (8.0%) had 5 points, 486 (12.6%) had 6 points, 409 (10.6%) had 7 points and 145 (3.8%) had 8 points.

Fig. 3 shows the distribution and relationship between the results for the different scores and the probability of having true bacteraemia.

Fig. 3.

Probabilities of true bacteraemia according to the distribution of the values of the 5MPB-Toledo.

Fig. 4 shows the AUC of the 5MPB-Toledo at the validation CP which was 0.932 (95% CI: 0.924−0.940; P < .001). The Hosmer–Lemeshow goodness-of-fit test had a p value of 0.631. The internal validation, using the bootstrapping technique, was 0.930 (95% CI: 0.916−0.948; P < .001).

Fig. 4.

Predictive capacity for bacteraemia of the validation cut-off point of the 5MPB-Toledo.

95% CI: 95% confidence interval; AUC: area under the receiver operating characteristic curve.

Analysis of the results according to the different cut-off points of the 5MPB-Toledo with the validation study sample

Table 4 shows the results for diagnostic performance of the different CPs analysed and the number and percentage of episodes corresponding to the groups according to the CPs established.

Table 4.

Performance for prediction of true bacteraemia in blood cultures taken in the accident and emergency department according to the cut-off points of the 5MPB-Toledo.

5MPB-Toledo score Patients included (%)	Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV, % (95% CI)	NPV, % (95% CI)	PCP (95% CI)	NCP (95% CI)
Score ≥1 N = 3507 (91.26%)	99.88 (99.23−99.99)	11.15 (10.06−12.34)	23.90 (22.50−25.35)	99.70 (98.09−99.98)	1.12 (1.11−1.14)	0.01 (0.00−0.08)
Score ≥2 N = 2647 (68.87%)	99.52 (98.70−99.85)	39.68 (37.93−41.46)	31.55 (29.78−33.36)	99.67 (99.08−99.89)	1.65 (1.60−1.70)	0.01 (0.00−0.03)
Score ≥3 N = 1824 (47.46%)	98.45 (97.29−99.14)	66.78 (65.06−68.46)	45.29 (42.99−47.60)	99.36 (98.87−99.64)	2.96 (2.81−3.12)	0.02 (0.01−0.04)
Score ≥4 N = 1476 (38.40%)	97.02 (95.57−98.02)	77.96 (76.43−79.42)	55.15 (52.57−57.70)	98.94 (98.42−99.30)	4.40 (4.11−4.71)	0.04 (0.03−0.06)
Score ≥5 N = 1349 (35.10%)	94.76 (92.97−96.12)	81.56 (80.11−82.92)	58.93 (56.25−61.57)	98.24 (97.62−98.70)	5.14 (4.76−5.55)	0.06 (0.05−0.09)
Score ≥6 N = 1040 (27.06%)	81.64 (78.82−84.17)	88.18 (86.96−89.30)	65.87 (62.88−68.73)	94.51 (93.58−95.31)	6.91 (6.23−7.66)	0.21 (0.18−0.24)
Score ≥7 N = 554 (14.41%)	51.13 (47.69−54.56)	95.84 (95.05−96.51)	77.44 (73.68−80.81)	87.53 (86.35−88.63)	12.29 (10.22−14.77)	0.51 (0.48−0.56)
Score 8 N = 145 (3.77%)	15.14 (12.82−17.78)	99.40 (99.03−99.63)	87.59 (80.83−92.28)	80.75 (79.43−82.00)	25.26 (15.51−41.14)	0.85 (0.83−0.88)

5MPB-Toledo: Toledo five-variable model predictive of bacteraemia; CI: confidence interval; N: number; NCP: negative coefficient of probability; NPV: negative predictive value; PCP: positive coefficient of probability; PPV: positive predictive value.

Finally, the CP ≥ 5 points of the model was selected after applying Youden's J statistic and due to clinical interest. This CP achieved a sensitivity of 94.76% (95% CI: 92.97−96.12), a specificity of 81.56% (95% CI: 80.11−82.92), a PPV of 58.93% (95% CI: 56.25−61.57), an NPV of 98.24% (95% CI: 97.62−98.70), a PCP of 5.14 (95% CI: 4.76−5.55) and an NCP of 0.06 (95% CI: 0.05−0.09).

Predictive performance for true bacteraemia of the 5MPB-Toledo in certain subgroups

Table 5 shows the results for both the AUC of the 5MPB-Toledo and the analysis of performance of the CP ≥ 5 points, applied to the different patient subgroups: 1) Patients who had taken antibiotic therapy in the past 72 h; 2) Patients discharged from the HAED; 3) Patients with immunosuppression (primary or secondary, including immunosuppressant and neutropaenic agents); and 4) Patients being treated with corticosteroids on an ongoing basis (≥10 mg of prednisone or equivalent for more than seven days).

Table 5.

Predictive capacity for true bacteraemia of the 5MPB-Toledo in certain subgroups and performance of the cut-off point ≥5 points.

Group/subgroup Patients included (%)	AUC (95% CI) of the 5MPB-Toledo	Performance of the cut-off point ≥5
		Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV, % (95% CI)	NPV, % (95% CI)
Overall sample N = 3843 (100%)	0.932 (0.924−0.940)	94.76 (92.97−96.12)	81.56 (80.11−82.92)	58.93 (56.25−61.57)	98.24 (97.62−98.70)
Patients with antibiotic therapy in the past 72 h N = 607 (15.8%)	0.916 (0.883−0.948)	93.00 (87.99−98.00)	90.13 (87.54−92.73)	65.00 (62.36−67.64)	98.5 (97.86−99.14)
Patients discharged from the HAED N = 813 (21.15%)	0.961 (0.945−0.977)	91.42 (84.87−97.98)	91.25 (89.22−93.28)	49.62 (47.00−52.24)	99.06 (98.44−99.68)
Patients with immunosuppression N = 541 (14.07%)	0.908 (0.883−0.933)	99.17 (97.56−100)	82.38 (78.73−86.02)	61.98 (59.50−64.46)	99.76 (99.58−99.94)
Patients being treated with corticosteroids N = 215 (5.59%)	0.921 (0.877−0.965)	95.91 (90.37−100)	80.93 (75.67−86.18)	53.39 (50.73−56.05)	98.88 (98.26−99.50)

CI: confidence interval; HAED: hospital accident and emergency department; N: number; NCP: negative coefficient of probability; NPV: negative predictive value; PCP: positive coefficient of probability; PPV: positive predictive value.

Discussion

The results of this study enabled external validation of a simple risk model for predicting bacteraemia in adult patients seen for an episode of infection in HAEDs. The 5MPB-Toledo scoring scale²⁰ includes variables that can easily be obtained right away in the care in patients with suspected serious infection in relation to examination (temperature and RR), comorbidities (Charlson Comorbidity Index) and laboratory results (leukocyte count and serum PCT levels). Therefore, it may represent a useful tool in predicting bacteraemia so as to optimise the most important and most immediate decisions to be made in HAEDs: indication for BCs, administration of suitable and early antimicrobial therapy, and hospital admission or discharge home, among others.^5,12

The excellent AUC obtained after bootstrapping (0.930 [95% CI: 0.916−0.948; P < .001]), scarcely lower than that published for the original model (0.946 [95% CI: 0.922−0.969; P < .001]),²⁰ and the distribution of the percentage of true bacteraemia in each risk group (1.5% in the low-risk group, 16.8% in the moderate-risk group and 81.6% in the high-risk group) classified patients according to three well-differentiated categories. The CP ≥ 5 points offered, with a sensitivity greater than 94% and an NPV greater than 98%, an obvious source of assurance for ruling out bacteraemia in a patient with a clinical diagnosis of infection. In addition, there were minor differences in the behaviour of the model (AUC 0.908−0.961) in the subgroups of patients studied individually (patients discharged directly from the HAED, patients who had taken antibiotics previously or corticosteroids and immunosuppressed patients) where the CP ≥ 5 also achieved a sensitivity greater than 91% and an NPV greater than 98%. All this meant—clinical judgement, disease type and patient characteristics permitting—that the patient could be discharged with suitable antimicrobial treatment and decision-making around BCs in the HAED could be optimised with safety and efficacy.²⁰

At present, the technique for taking BCs is well documented,^1,6 but significant debate persists concerning the indications for taking them in the HAED.^1,10,15 Despite this, BCs are increasingly common in the initial assessment of patients with suspected infection in the HAED.^5,6,15 In them, suspicion and confirmation of bacteraemia is of major diagnostic, prognostic and therapeutic significance. However, BCs are also taken in the HAED to ensure care continuity, since the management and subsequent clinical course of the patients at their final destination will depend on their results.^5,15,26

In this context, in the past decade, the study of factors predictive of bacteraemia has intensified and different predictive models for HAEDs of varying complexity have been proposed.^13–20,27 In them, the role that biomarkers, especially PCT,¹⁵ might play as independent factors predictive of bacteraemia has taken on a great deal of importance. The diagnostic performance thereof has been shown to equal or even exceed that of different models.^{8,12,16,20,27–31}

Shapiro et al.¹³ published a proposed model that classified risk of bacteraemia as low (<1%), moderate (7%–9%) or high (15%–26%), depending on several major criteria (temperature >39.4 °C, vascular catheter or suspected endocarditis) and several minor criteria (temperature >38.3 °C, age >65 years, chills, vomiting, SBP < 90 mmHg, leukocytosis >18,000/mm³, band neutrophils >5%, thrombocytopaenia <150,000/mm³ and creatinine >2 mg/dl). This scale has been validated¹⁴ and has served as the most important reference for HAEDs for many years.^15,26 According to this decision-making model, BCs would be indicated when one major criterion is met or at least two minor criteria are met. The Shapiro scale achieves an AUC of 0.83. Undoubtedly, it is a useful model with significant performance (although lower than that achieved by the original 5MPB-Toledo, which was 0.946 and, in our validation cohort, 0.932),²⁰ but it is too complex to use in HAEDs and does not take into account the contributions that biomarkers could most certainly make.^27–31 Therefore, other proposed models have been claimed to be quicker and simpler, such as that of Tudela et al.,¹⁶ which included clinical and laboratory values and the Charlson Comorbidity Index and, after the multivariate analysis, identified two significant variables: Charlson Comorbidity Index ≥2 and a PCT > 0.4 ng/mL (one and two points, respectively).¹⁶ With these two variables, four groups of increasing probability of bacteraemia were established, and an AUC of 0.80 and an NPV of 95.3% for “ruling out” bacteraemia were achieved. Comparison of the Tudela et al. model to the 5MPB-Toledo scale revealed that the latter included (with other CPs) the two variables from the former model plus temperature, RR and leukocytes (which are present in the seriousness prognostic scales and defining criteria for sepsis: qSOFA and systemic inflammatory response syndrome [SIRS] criteria). Hence, the 5MPB-Toledo, along with assessment of SBP, HR and altered level of consciousness, could aid in easily performing a comprehensive diagnostic assessment (of infection and bacteraemia) and prognostic assessment (seriousness and mortality) in patients with infection in HAEDs.⁵

Contenti et al.²⁸ recently achieved the same AUC as the Shapiro model (0.83) by using just one variable among those identified in our study—PCT—but raising the CP to levels exceeding 2.25 ng/mL. Therefore, we believe that including PCT in any model or as an individual factor should be considered in HAEDs today as suggested by various authors.^15,27–31 In our study and in the 5MPB-Toledo,²⁰ PCT is the most heavily weighted factor in the scale (corresponding to 4 points) and the CP for PCT ≥ 0.51 ng/mL is consistent with recent recommendations.¹²

At present, including PCT in the model should not represent any delay or lag in decision-making in the HAED. This is firstly because, in patients with a serious clinical situation (serious sepsis/septic shock), action must be taken immediately to obtain laboratory samples and BCs, and the first antimicrobial dose must be administered without waiting for the results (as is normally done). Moreover, in all other patients, PCT values are included among emergency laboratory results today, thus enabling emergency care, assessment and decision-making within the first hour of the patient's stay in the HAED.⁵

Other models that have included some of the factors identified in our study, though useful, have not managed to achieve the performance of Shapiro's model.¹³ However, some of them are easier to evaluate and implement in HAEDs.^18,19 For example, a model proposed by Su et al.¹⁷ included as variables temperature ≥38.3 °C, tachycardia ≥120 bpm, lymphopaenia <500/mm³ and PCT > 0.5 ng/mL with other laboratory results. Su et al.'s model¹⁷ achieved an AUC of 0.85—lower than the performance of the 5MPB-Toledo.²⁰

As is to be expected, a model as simple and quick to use in the HAED has been held up as an essential factor for success in recent meta-analyses and reviews.^18,19 However, astonishingly, it has been confirmed that none of the 15 models figuring in said reviews has been implemented in daily clinical practice—not even by their respective authors.¹⁹

Yet, in contrast to the above, another review article that analysed 35 studies was unable to identify independent factors predictive of bacteraemia.¹⁸ It therefore did not recommend systematic BCs in cases with only fever and leukocytosis, which for example could be absent in immunosuppressed patients and in patients being treated with corticosteroids,²⁷ and which for our model would total 2 points (low risk of 1.5%). The authors suggested that the search should continue for an ideal model that incorporates other variables such as biomarkers and clinical assessment of the seriousness of the patient's condition (with vital signs: temperature, HR, RR, SBP and level of consciousness).^18,26

Our study had some limitations that must be noted. BCs were indicated and ordered at the discretion of the physician on duty at each site. Therefore, along with this clinical variability, it must be remembered that 5.48% of BCs were not recorded as they did not meet the inclusion criteria (follow-up and confirmation after 30 days, PCT sensitivity), and that cases were included by opportunity sampling (when the investigators were on duty in the HAEDs); all this could have represent selection bias as not all episodes were considered. In addition, the selection of clinical variables of the model could have been more complete (some variables such as chills, shivering/shaking and nausea/vomiting were not included).^18,19,26

It must also be noted that there was a significant rate of contaminated BCs (4.47%), repeated in recent studies, but not representing an obstacle to analysing the results, having already been published.^7,8,20 However, despite these limitations, we believe that our results faithfully reflect the reality of HAEDs in Spain. Moreover, compared to the original model, ours had the strengths of being a multicentre prospective study with a suitable sample and no missing data for the variables comprising the model.

In conclusion, the 5MPB-Toledo is useful for stratifying risk of bacteraemia in adult patients with infectious disease in HAEDs, since it is capable of suitably predicting bacteraemia with readily available variables and, along with clinical judgement and other independent disease- and patient-related variables, facilitates decision-making in relation to the indication for taking BCs in HAEDs and the diagnostic and therapeutic strategy.

Ethical responsibilities

All the authors have confirmed the maintenance of patient confidentiality and respect for patient rights and the transfer of rights to Enfermedades Infecciosas y Microbiología Clínica. The study was evaluated and approved by the Complejo Hospitalario Universitario de Toledo [Toledo University Hospital Complex] Independent Ethics Committee (IEC) (no.: 398/2109), as well as by the reference IECs/mIECs of the participating sites. The patients gave their written informed consent to participate in the study.

Funding

This manuscript did not receive any funding from any public or private organisation.

Authors

The authors declare that they were responsible for the design, development and preparation of the article.

Conflicts of interest

AJJ has participated in scientific meetings organised by Roche, Thermo Scientific Biomarkers, B·R·A·H·M·S AG and bioMérieux.

The rest of the authors declare that they have no conflicts of interest in relation to this article.

None of the authors received financial compensation for conducting this study.

Acknowledgements

The authors would like to thank Pedro Beneyto Martín and Francisco Javier Martín Sánchez for their help with the statistical analysis of the data.