Prediction of surgical risk in patients with endocarditis: Comparison of logistic EuroSCORE, EuroSCORE II and APORTEI score

Abstract

Objectives

APORTEI score is a new risk prediction model for patients with infective endocarditis. It has been recently validated on a Spanish multicentric national cohort of patients. The aim of the present study is to compare APORTEI performances with logistic EuroSCORE and EuroSCORE II by testing calibration and discrimination on a local sample population underwent cardiac surgery because of endocarditis.

Methods

We tested three prediction scores on 111 patients underwent surgery from 2014 to 2020 at our Institution because of infective endocarditis. Area under the curves and Hosmer–Lemeshow test were used to analyze discrimination and calibration respectively of logistic EuroSCORE, EuroSCORE II and APORTEI score.

Results

The overall observed one-month mortality rate was 21.6%. The observed-to-expected ratio was 1.27 for logistic EuroSCORE, 3.27 for EuroSCORE II and 0.94 for APORTEI. The area under the curve (AUC) value of APORTEI (0.88 ± 0.05) was significantly higher than that one of logistic EuroSCORE (AUC 0.77 ± 0.05; p 0.0001) and of EuroSCORE II (AUC 0.74 ± 0.05; p 0.0005). Hosmer–Lemeshow test showed better calibration performance of the APORTEI, (logistic EuroSCORE: p 0.19; EuroSCORE II: p 0.11; APORTEI: p 0.56).

Conclusion

APORTEI risk score shows significantly higher performances in term of discrimination and calibration compared with both logistic EuroSCORE and EuroSCORE II.

Resumen

Objetivos

El APORTEI score es un nuevo sistema de predicción de riesgo para pacientes con endocarditis infecciosa. El mismo ha sido recientemente validado en una cohorte de pacientes procedentes de un estudio nacional multicéntrico español. El objetivo del presente estudio es comparar la discriminación y la calibración del APORTEI score con las del EuroSCORE logístico y del EuroSCORE II analizando una población sometida a cirugía cardiaca por endocarditis infecciosa.

Métodos

Analizamos las propiedades de 3 sistemas de predicción de riesgo sobre una población de 111 pacientes sometidos a cirugía cardiaca desde 2014 hasta 2020 en un único hospital terciario por endocarditis infecciosa. El área bajo las curvas y la prueba de Hosmer-Lemeshow se usaron para analizar la discriminación y la calibración, respectivamente, del EuroSCORE logístico, del EuroSCORE II y del APORTEI score.

Resultados

La mortalidad global observada a un mes fue del 21,6%. La relación mortalidad observada/mortalidad esperada fue de 1,27 para el EuroSCORE logístico, 3,27 para el EuroSCORE II y 0,94 para el APORTEI score. El valor del área bajo la curva (AUC) del APORTEI score (0,88 ± 0,05) fue significativamente mayor que los del EuroSCORE logístico (AUC 0,77 ± 0,05; p 0,0001) y del EuroSCORE II (AUC 0,74 ± 0,05; p 0,0005). La prueba de Hosmer-Lemeshow mostró un mejor rendimiento en cuanto a calibración del APORTEI score, (EuroSCORE logístico: p 0,19; EuroSCORE II: p 0,11; APORTEI score: p 0,56).

Conclusión

El sistema de predicción de riesgo APORTEI score muestra un rendimiento significativamente mejor en cuanto a discriminación y calibración en comparación con el EuroSCORE logístico y con el EuroSCORE II.

Introduction

Infective endocarditis (IE), especially in surgical setting, represents a complex condition associated with high early and long-term mortality and morbidity.¹ In fact, surgical mortality in patients with IE ranges from 15% to 45%.² To approach the IE complexity in terms of diagnosis and therapeutic strategies, building of multidisciplinary ad-hoc teams and use of statistical models for prognosis prediction have been extensively advocated³.

These are, in most cases, the product of sophisticated statistical models obtained from specific patients-based datasets such as logistic EuroSCORE⁴ and EuroSCORE II.⁵

Still, several concerns have been raised about the discrimination performances of risk models obtained from dataset where patients with IE are poorly represented.^6,7

Recently, a new prediction model tool, APORTEI score, obtained from an extensive meta-analysis, has been reported to provide a precise and easy tool to stratify mortality risk in patients underwent surgery because of IE.^8,9

The aim of the present study is to analyze discrimination and calibration proprieties of the three risk scores, logistic EuroSCORE, EuroSCORE II and APORTEI score, over a sample population undergoing cardiac surgery at our Institution because of IE.

Methods

The sample population was constituted by 111 patients underwent consecutively cardiac surgery because of infective endocarditis from January 2014 to March 2020 at our Institution, a tertiary-level hospital.

Data were collected prospectively through our local electronic database. Logistic EuroSCORE and EuroSCORE II were prospectively calculated using the specific online calculators (http://www.euroscore.org/calcsp.html). The APORTEI score was retrospectively calculated for each patient using excel sheet published, as supplementary material, by the score developers⁹ (https://academic.oup.com/ejcts/article/57/4/724/5647329#supplementary-data). Continuous variables following a normal distribution were expressed as mean ± standard deviation (SD). Continuous variables not normally distributed were expressed as median (25th–75th percentiles). Dichotomous categorical variables were indicated as absolute frequency (percentage). Predicted surgical mortality was obtained for each score. Spearman rank correlation analyses were carried out between each score. Discrimination was analyzed by receive operating characteristic analyses from which area under the curve (AUC) ± standard error (SE) values were obtained. The method of DeLong¹⁰ was used for the calculation of the standard error (SE) of the AUC and of the difference between two AUCs. Univariable logistic regression analysis was built introducing “30-day mortality” as dependent variable and separately each of the three scores as independent variable. From this model the Hosmer–Lemeshow test was obtained. The Hosmer–Lemeshow method, as test for goodness of fit for the three logistic regression models, was used to analyze the calibration of the scores. Statistical analyses were performed using MedCalc Statistical Software version 19.0.7 (MedCalc Software bvba, Ostend, Belgium; https://www.medcalc.org; 2019).

Results

Patients characteristics are listed in Table 1. Mean age was 58.9 years. Most frequent causative microorganism were Staphylococcus aureus (25.2%, n = 28), Staphylococcus epidermidis (14.4%, n = 16), and Streptococcus viridans group (11.7%, n = 13). The aortic valve was the most frequent localization (43.2%, n = 48), (Table 1).

Table 1.

Pre and intra operative patients data of the whole sample population.

Patients characteristics	All sample population (N = 111)
Female	30 (27.0%)
Age	58.9 (±13.7)
Body mass index	26.5 (± 4.9)
Diabetes mellitus	24 (21.6%)
Chronic lung disease	16 (14.4%)
Arterial hypertension	40 (36.0%)
Preoperative inotropic treatment	21 (18.9%)
Pulmonary hypertension (>60 mmHg)	15 (13.5%)
Congestive heart failure	54 (48.6%)
Invasive ventilation	25 (22.5%)
Left ventricle ejection fraction < 49%	12 (10.8%)
Cardiogenic or septic shock	29 (26.1%)
Emergency surgery	32 (28.8%)
Prosthetics endocarditis	30 (27.0%)
Peri annular abscess	27 (24.3%)
Nosocomial endocarditis	16 (14.4%)
Staphylococcus Aureus	21 (22.6%)
Logistic EuroSCORE	17.1 (7.8–41.7)
EuroSCORE II	6.8 (3.2–24.5)
APORTEI score	46.4 (30.6–70.0)
APORTEI predicted mortality	23.0 (13–39.8)
Location of endocarditis
Aortic valve	48 (43.2%)
Mitral valve	32 (28.8%)
Aortic and mitral valves	23 (20.7%)
Mitral and tricuspid valves	2 (1.8%)
Pulmonary valve	2 (1.8%)
Interventricular septum	1 (0.9%)
Tricuspid valve	1 (0.9%)
Aortic, mitral and pulmonary valves	1 (0.9%)
Aortic, mitral and tricuspid valves	1 (0.9%)

Thirty-day mortality was 21.6% (n = 24). Predicted surgical mortality by logistic EuroSCORE, EuroSCORE II and APORTEI score were 17.1%, 6.6% and 23.0%, respectively.

Therefore, the observed-to-expected ratio was 1.27 for logistic EuroSCORE, 3.27 for EuroSCORE II and 0.94 for APORTEI score.

According to Spearman's rank correlation analysis the highest coefficient of correlation was observed between logistic EuroSCORE and EuroSCORE II (rho 0.88, p < 0.0001). Still, significant correlation was documented between logistic EuroSCORE and APORTEI score (rho 0.58, p < 0.0001) and between EuroSCORE II and APORTEI score (rho 0.63, p < 0.0001). According to the receive operating characteristic analyses, AUC value of APORTEI score was 0.88 ± 0.05 which was significantly higher than those of logistic EuroSCORE (AUC 0.77 ± 0.05; difference between areas 0.17; p 0.0001) and EuroSCORE II (AUC 0.74 ± 0.05; difference between areas 0.11; p 0.0005) (Fig. 1).

Fig. 1.

ROC curves of Logistic EuroSCORE, EuroSCORE II and APORTEI score.

Hosmer–Lemeshow test of the regression logistic analysis showed better calibration performance of the APORTEI score, (logistic EuroSCORE: chi-squared 12.30, p value: 0.19; EuroSCORE II: chi-squared 12.98, p value: 0.11; APORTEI: chi-squared 6.77; p value: 0.56).

Discussion

The present study shows higher discrimination and calibration proprieties of APORTEI score compared with logistic EuroSCORE and EuroSCORE II. The three scores were tested on a sample population of 111 patients underwent surgery because of IE.

In the last decade, several risk scores have been built to predict patients’ outcomes after cardiac surgery.^8,11 Risk models have been used also to identify specific risk factors, to allow comparison of postoperative results and to carry out risk stratification analysis.¹²

Patients undergoing cardiac surgery because of infective endocarditis represents a complex subgroup of patients, with higher surgical mortality than that one of the general populations undergoing cardiac surgery and specific risk factors.^13–15 For this reason, several concerns have been raised about the proprieties of logistic EuroSCORE and EuroSCORE II when applied to IE populations.^6,7 In fact, EuroSCORE I prediction models⁴ have been built from a database containing 19,030 patients underwent cardiac surgery, but, of these, only 1.1% (173) suffered from endocarditis. EuroSCORE II⁵ was built on a database of 22,381 patients undergoing major cardiac surgery. Of these only 2.2% (497) presented with endocarditis. In both cases prediction scores were obtained by logistic modellings.

APORTEI score building process lied on a different methodology as it was not computed from a specific database of patients. Instead, it was obtained by meta-analysis data consisted on 16 studies and 7484 patients with IE.⁸ Through overall pooled odds ratio (OR) calculation, 11 risk factors were identified and posteriorly tested on a national Spanish multicenter database containing 1338 patients with IE.⁹

Our results show clearly than logistic EuroSCORE and especially EuroSCORE II underestimates surgical mortality risk in patients with endocarditis. Also, receive operating characteristic analyses demonstrated that APORTEI score has the largest AUC value and, hence, the best performance in term of discrimination.

Even if the present study was carried out on a small sample population, its results confirm those of a previous study⁹ in terms of better performance of the APORTEI score compared with logistic EuroSCORE. Also, our analysis shows that the APORTEI score has higher discrimination and calibration proprieties even when compared with EuroSCORE II as prognostic and risk stratification tool for patients undergoing surgery because of IE.

Funding

This research received no funding.

Conflict of interest

None.

Acknowledgments

The present authors sincerely thank all the remaining members of the Equipo Multidisciplinar de Endocarditis (EME) of the Hospital Universitario Dr. Negrín:

Eloy Gomez, Oscar Sanz, Rafael Granados, Xerach Bosh (Unidad de Enfermedades Infecciosas), Ana Bordes, Raúl Gilarranz (Microbiología), José Luis Romero, Adela Hernández, Sílvia Hipola, José María Gómez, (Medicina Intensiva), Verónica Quevedo, Marta Blanco, Irene Menduiña (Cardiología), Luis Ríos, Raquel Bellot (Cirugía Cardíaca).