Usefulness of Administrative Databases for Risk Adjustment of Adverse Events in Surgical Patients

Abstract

Purpose

The aim of this study was to assess the usefulness of clinical–administrative databases for the development of risk adjustment in the assessment of adverse events in surgical patients.

Methods

The study was conducted at the Hospital of Navarra, a tertiary teaching hospital in northern Spain. We studied 1602 hospitalizations of surgical patients from 2008 to 2010. We analysed 40 comorbidity variables included in the National Surgical Quality Improvement Programme (NSQIP) of the American College of Surgeons using 2 sources of information: The clinical and administrative database (CADB) and the data extracted from the complete clinical records (CR), which was considered the gold standard. Variables were catalogued according to compliance with the established criteria: sensitivity, positive predictive value and kappa coefficient > 0.6.

Results

The average number of comorbidities per study participant was 1.6 using the CR and 0.95 based on CADB (p<.0001). Thirteen types of comorbidities (accounting for 8% of the comorbidities detected in the CR) were not identified when the CADB was the source of information. Five of the 27 remaining comorbidities complied with the 3 established criteria; 2 pathologies fulfilled 2 criteria, whereas 11 fulfilled 1, and 9 did not fulfil any criterion.

Conclusion

CADB detected prevalent comorbidities such as comorbid hypertension and diabetes. However, the CADB did not provide enough information to assess the variables needed to perform the risk adjustment proposed by the NSQIP for the assessment of adverse events in surgical patients.

Resumen

Objetivo

Conocer la validez de las bases de datos clínico-administrativas para realizar ajustes de riesgo en el estudio de los efectos adversos que pueden sufrir los pacientes intervenidos quirúrgicamente.

Métodos

Se estudiaron 1.602 episodios de hospitalización de pacientes intervenidos quirúrgicamente durante los años 2008 y 2010 en un hospital terciario situado en el norte de España. Se analizaron 40 variables de comorbilidades que recoge el Colegio Americano de Cirujanos en el National Surgical Quality Improvement Programme (NSQIP) a través de 2 fuentes de información: el conjunto mínimo básico de datos (CMBD) y los datos extraídos de la historia clínica completa (HC), considerada como patrón oro. Se catalogaron las variables en función de que los valores de sensibilidad (S), valour predictivo positivo (VPP) y kappa (K) fueran superiores a 0,6.

Resultados

La media de comorbilidades por paciente fue de 1,6 mediante la HC y de 0,95 a través del CMBD (p < 0,0001). El CMBD no detectó ningún caso en 13 comorbilidades (estas supusieron el 8% de las comorbilidades detectadas con la HC). De las 27 comorbilidades restantes, 5 cumplieron los 3 criterios establecidos (S, VPP y k > 0,6), 2 enfermedades cumplieron 2 criterios, 12 cumplieron al menos uno y 8 no cumplieron ninguno.

Conclusión

Las bases de datos clínico-administrativas detectaron comorbilidades como diabetes o hipertensión arterial pero no aportaron suficiente información para recoger las variables necesarias para hacer el ajuste de riesgo propuesto por el NSQIP para la medición de los efectos adversos en pacientes quirúrgicos.

Introduction

Health care is increasingly effective. Nevertheless, current research indicates that it is not always safe. For example, it is probable that one out of every 10 hospitalised patients will suffer an adverse event (AE) while they are in hospital.^1,2 Half of these AE are preventable.^3,4 AE rates vary depending on the departments studied and the methodology used.^5–7 The AE incidence rate in a set of General and Digestive Surgery departments in Spanish hospitals was 10.5% (interval of confidence [IC]: 8.1%–12.5%). The presence of intrinsic risk factors such as diabetes and obesity, etc., increased the risk of AE (14.8% vs 7.2%; P=.001).⁵ The evaluation of the AE that occur in a healthcare institution is, therefore, of key importance for the quality of any organisation that works in this field.

Risk adjustment systems are prepared to evaluate the results obtained by healthcare systems, with the final aim of determining their effectiveness. As a result of this, risk adjustment may be useful to prevent the distortion that could arise in evaluation due to patient characteristics. These characteristics (which may be socio-demographic, prognostic or clinical) can affect the results, independently of the care provided and the treatments used, thereby influencing the results of statistical analysis.⁸

Comparison of the results obtained in the evaluation of AE observed after surgery, without having made the necessary adjustment according to the risk, may mask serious problems in the quality of care in institutions that treat low-risk patients. This may lead to the erroneous conclusion that hospitals with more complex patients offer lower quality care than is actually the case.⁹

To undertake risk adjustment it is necessary to gather data on the main characteristics of patients, including associated comorbidities. Nevertheless, obtaining these data often complicates the preparation of a research project and leads to the corresponding increase in the resources that are necessary to carry out the study.

Clinical–administrative databases are an alternative source of primary data. The data gathered are systematically stored in the minimum basic set of data (MBSD). The information included in this clinical–administrative database is public and accessible. It is in electronic format and is recorded continuously over time for a large number of patients. However, use of the MBSD is not free of risk, so that its validity has to be studied. This is one of the main aims of this study.¹⁰ Several authors have previously demonstrated the validity of the MBSD in preparing risk adjustments in the evaluation of mortality.^11,12 Nevertheless, there are few studies which prove the validity of the MBSD for risk adjustment according to surgical patient safety indicators.^13–15

The aim of this paper is to evaluate the utility of risk adjustments based on the MBSD in the analysis of surgical patient AE.

Methods

Study Design

This study was undertaken in a tertiary teaching hospital in the North of Spain which has 500 beds and 10 operating theatres (9 of them for scheduled operations and one for urgent and emergency surgery), where approximately 5300 surgical operations take place per year.

The reference population is composed of 10 121 patients who were surgically operated on from 2008 to 2010. The sample was composed of 1602 patients. The sample was randomly extracted and was stratified according to speciality. The specialities studied were ophthalmology, otorhinolaryngology, general surgery, orthopaedic surgery and trauma, urology, neurosurgery, cardiac surgery, thoracic surgery and vascular surgery.

An initial stratification according to speciality was used to ensure that the samples maintained the proportions of patients treated by each surgical department. Once the number of patients which corresponded to each department had been determined, the sample was extracted using simple randomisation.

Inclusion Criteria

The study includes all of the adult patients with a minimum hospitalisation time of 24 hours who were surgically operated on.

Data Gathering

The sources of information used were: on the one hand the complete clinical history (CH), which was considered to be the gold standard, and on the other hand the MBSD.

Table 1 shows the variables used to describe the general characteristics of the patients. 40 comorbidities or risk factors were analysed following the definitions set by the National Surgical Quality Improvement Programme of the American College of Surgeons (ACS NSQIP).¹⁶

Table 1.

General Characteristics of the Patients Studied (No.=1602).

Women, number (%)	660 (41.2)
Age, years	60.8
Emergency hospital admission, number (%)	436 (27.2)
Duration of hospitalisation, average	8.01
Diagnosis (MBSD), average	4.1
Procedures (MBSD), average	2.25
Weighting of groups connected by diagnosis	2.7
Surgical speciality, number (%)
Ophthalmology	6 (0.4)
Otorhinolaryngology	39 (2.4)
General surgery	508 (31.7)
Orthopaedic surgery and traumatology	425 (26.5)
Urology	187 (11.7)
Neurosurgery	111 (6.9)
Cardiac surgery	135 (8.4)
Thoracic surgery	63 (3.9)
Vascular surgery	128 (7.9)

MBSD: minimum basic set of data.

The MBSD is a clinical–administrative database that extracts information from CH at the moment patients are discharged. The key elements in the administrative data bases are the dates of admission and discharge, a set of diagnoses and procedures, and demographic variables.¹⁷ Information is recorded for each episode of hospitalisation (from admission to discharge). The clinical coding of diagnosis and the therapeutic and surgical procedures included in the discharge report used the clinical modification of the ninth revision of the International Classification of Diseases (ICD-9-CM) based on the Ninth Revision by the World Health Organisation. In Spain the official guide for coding and notification is revised and updated every 2 years. The MBSD used in this study includes 20 fields for diagnostic variables and 16 fields for procedure variables.

The comorbidities studied by the NSQIP were mapped onto ICD-9 codes (see Table 2). These codes were searched for in all diagnostic fields, primary as well as secondary. An expert in codification prepared the code mapping.

Table 2.

Comorbidity codes in the ICD-9-CM.

Comorbidities	ICD-9-CM codes	Description of the codes
General
Diabetes mellitus	250.XX	Different terms in connection with diabetes
Smoking	305.1	Disorders caused by tobacco consumption
Alcohol consumption	303.9X	Other alcoholic dependencies and unspecified alcoholic dependencies
	305.0X	Abuse of alcohol without dependency
Dyspnoea	786.0X	Dyspnoea and respiratory anomalies
State of “do not resuscitate”	V49.86	State of “do not resuscitate”
Functional state of the patient prior to the current disorder at the moment of the operation	V49.87	State of physical limitation
Respiratory
State of dependency on a respirator	V46.1X	State of dependency on a respirator (ventilator)
A history of serious chronic obstructive pulmonary disease	491.XX	Chronic bronchitis
	491.20	Unexacerbated chronic obstructive bronchitis
	491.21	Exacerbated (acute) chronic obstructive bronchitis
	491.22	Chronic obstructive bronchitis with acute bronchitis
	491.8	Other chronic bronchitis
	493.20	Obstructive unspecified chronic asthma
	493.21	Chronic obstructive asthma with asthmatic state
	493.22	Chronic obstructive asthma with exacerbation (acute)
	494.0	Bronchiectasia without acute exacerbation
	494.1	Bronchiectasia with acute exacerbation
	496	Chronic obstruction of the airways
Current pneumonia	480–486	Different types of pneumonia
Hepatobiliary
Ascitis	789.5X	Ascitis
Gastroesophageal
Oesophageal varices	456.0–456.21	Oesophageal varicose veins with or without haemorrhage
Heart
Congestive heart failure (ICC) in the 30 days prior to the surgical operation	428.0–428.9	Heart failure
	398.91	Rheumatic (congestive) heart failure
	402.01	Malign hypertensive cardiopathy with heart failure
	402.11	Benign hypertensive cardiopathy with heart failure
	402.91	Unspecified hypertensive cardiopathy with heart failure
	404.01	Malign hypertensive chronic heart and kidney disease with heart failure and with chronic stage I to IV or unspecified kidney disease
	404.03	Malign hypertensive chronic heart and kidney disease with heart failure and with chronic stage V or terminal kidney disease
	404.11	Benign hypertensive chronic heart and kidney disease with heart failure and with chronic stage I to IV or unspecified kidney disease
	404.13	Benign hypertensive chronic heart and kidney disease with heart failure and with chronic stage V or terminal kidney disease
	404.91	Hypertensive chronic unspecified heart and kidney disease without heart failure and with chronic stage I to stage V chronic kidney disease, or unspecified
	404.93	Hypertensive unspecified chronic heart and kidney disease with heart failure and with chronic stage V or terminal kidney disease
A history of myocardial infarction	412	Previous myocardial infarction
Previous percutaneous coronary operation	V45.82	State of percutaneous transluminal coronary angioplasty
		Coronary atherosclerosis
A history of angina pectoris in the month prior to the surgical operation	414.0.X, 414.8.Y	Other specified forms of chronic ischaemic cardiopathy
	414.9	Unspecified chronic ischaemic cardiopathy
	411.1	Intermediate coronary syndrome
	413.X	Angina pectoris
Hypertension that requires medication	401–405	Hypertensive disease
Blood vessels
A history of revascularisation/amputation due to a peripheral vascular disease	V49.7X	State of amputation of lower limb
Pain in repose/gangrene	729.5	Limb pain
	785.4	Gangrene
	440.22; 440.24	Atherosclerosis of native arteries in the limbs with pain in repose/with gangrene
Kidneys
Acute kidney failure	584.X	Acute kidney failure (acute renal insufficiency)
Now in preoperative dialysis	V56.0	Admission for dialysis and care of dialysis catheter
	V56.8	Admission for other dialysis
Central nervous system
Alteration of awareness in the 48 hours prior to the operation	780.02	Transitory alteration in awareness
Coma during>24 hours	780.01	Coma
Hemiplegia and hemiparesia	342.XX	Hemiplegia and hemiparesia
A history of transitory ischaemic attacks	435.X	Transitory cerebral ischaemia
Cerebrovascular accident/ictus with neurological deficit	438.XX	Late effects of cerebrovascular disease
Cerebrovascular accident/ictus without neurological deficit	V12.54	A personal history of transitory ischaemic attack and cerebral infarction without residual deficits
Tumour that affects the central nervous system	191–192	Malign neoplasia of the brain and unspecified parts of the nervous system
	225	Benign neoplasia of the brain and unspecified parts of the nervous system
Paraplegia, paraparesia	344.1	Paraplegia, paraparesia
Quadriplegia and quadriparesia	344.0	Quadriplegia and quadriparesia
Immune, nutritional and other conditions
Metastatic cancer	196–197–198	Secondary malign neoplasia in different locations
Open wound	870–894	Open wound
Using steroid for a chronic disorder	V58.65	Prolonged (current) use of steroids
Loss of body weight>10% in the previous 6 months	783.2	Abnormal loss of weight and low weight
Haemorrhaging disorders	286.X	Coagulation defects
Transfusion of ≥4 units of	99.0X	Blood and blood components transfusion
erythrocytes in the 72 hours prior to the surgical operation	V58.2	Blood transfusions, without a declared diagnosis
Chemotherapy for malign disease	V58.11	Admission for antineoplastic chemotherapy
in ≤30 days prior to the operation	V66.2	Convalescence after chemotherapy
Radiotherapy for malign disease in the last 90 days	V58.0	Admission for radiotherapy
Systemic preoperative sepsis	038.XX	Septicaemia
	995.91–995.92	Sepsis and severe sepsis
Pregnancy	V22.2	Stage of pregnancy, incidental
	V85.0	Body mass index below 19, adults
	V85.1	Body mass index from 19 to 24, adults
Body mass index	V85.2X	Body mass index from 25 to 29, adults
	V85.3X	Body mass index from 30 to 39, adults
	V85.4	Body mass index 40 and higher, adults

The extraction of the information from the CH corresponding to all 1602 episodes was carried out by just 2 individuals, under the supervision of a third person who took charge of evaluation in case of doubt, with the aim of reducing variability in data gathering. To prevent distortion each evaluator extracted information from approximately half of the cases, after which both evaluators revised all of the specialities and years in similar proportions. The information extraction procedure and all of the documentation on the doubts which arose and the decisions reached by the research team were included in the data gathering manual. This allowed the inspectors to apply the same rules to the same data.

Before starting the study, a pilot study was undertaken in which the information gathered by the evaluator was verified by an external expert (which is considered to be the gold standard) to corroborate its validity. The 2 inspectors analysed 36 CH (2 CH per department and year). For all of the variables studied the kappa values were from 0.7 to 1. The pilot study served as a guide for the research process, and data extracted in this study were not included in the final analysis.

Statistical Analysis

Comorbidity frequency was analysed using both of the above-mentioned information systems. Sensitivity (S) was determined together with the positive predictive value (PPV) and the Cohen kappa index (K). A value above 0.6 was considered acceptable.

In this study, S refers to the capacity of the MBSD to correctly identify patients with the comorbidity in question. A low S score indicates that the MBSD did not detect the patients with comorbidities. The PPV answers the following question: “What is the probability that this patient will have the comorbidity when the MBSD is positive?” A low PPV indicates that the MBSD identified something other than was expected. The S and PPV scores varied from 0 to +1.

The K index shows the degree to which the existing coincidence is above what could be expected at random. K values ranged from −1 to +1. A value lower than zero indicates that the coincidence is worse than could be expected at random, while a value above zero indicates a higher coincidence than could be expected at random.¹⁸

Comorbidities were classified into 3 groups according to the number of S, PPV and K values that were above 0.6. The following groups were established: 3 indicators >0.6 indicated good validity; 2 indicators >0.6 indicated moderate validity; and fewer than 2 indicators >0.6 indicated poor validity.

Bubble graphs were used to show the values of S and PPV of the analysed AE. Bubble size indicates the proportion of comorbidity compared to the total number of comorbidities detected. The rarer a comorbidity is, the smaller the surface of the bubble. The graph showed comorbidities with a prevalence higher than 1% according to the CH.

Student's t-test for dependent data was used to compare the 2 determinations; Fisher's exact test was used to compare the 2 proportions. We use the Open-Epi computer programme (version 2.3.1 of the Epidemiology Department, Rollins School of Public Health, Emory University, Atlanta, GA, USA) and the SPSS Statistics programme, version 20 for Windows (Chicago, IL, USA).

The study was approved by the Navarre Research Ethics Committee (project 55/2014).

Results

The average number of comorbidities per participant in the study was 1.6 (DE=1.5) when the CH was used, and 0.95 (DE=1.16) when the MBSD was used. In other words, the MBSD recorded 66 comorbidities less per 100 patients than the CH (95% IC: 0.60–0.72; P<.001).

No case of the following 13 comorbidities was identified with the MBSD: coagulation disorder, alteration of awareness in 48 hours, chronic use of steroids, chemotherapy in the previous 30 days, radiotherapy in the previous 90 days, loss of more than 10% of body weight, dialysis, respirator dependency, state of “do not resuscitate”, pregnancy, previous cardiac surgery, quadriplegia and quadriparesia and obesity. On the contrary, these diseases amounted to 8% of total patient comorbidities identified in the CH.

The prevalence values, S, PPV and the K index of the 27 comorbidities detected using the MBSD are shown in Table 3. The comorbidities are shown in order of prevalence as detected in the CH.

Table 3.

The Prevalence of Comorbidities Detected Using 2 Sources Different of information. Sensitivity, Positive Predictive Value and Cohen's Kappa Index.

Comorbidities	Prevalence in administrative database (%)	Prevalence in clinical histories (%)	P	Sensitivity	Positive predictive value	Kappa	Influence on predictive models^a
Hypertension	28.7	42.3	<.0001	66.2	97.6	0.68	No
Nicotinism	16	25	<.0001	41.9	65.4	0.39	No
Diabetes mellitus	9.7	14.8	<.0001	63.7	96.8	0.77	No
Alcohol consumption	3.2	10.6	<.0001	18.8	62.7	0.25	No
Dyspnoea	0.2	8	<.0001	78.1	33.3	0.01	Yes
Serious chronic obstructive pulmonary disease	3.8	7.6	<.0001	46.3	91.3	0.59	Yes
Pain in repose/gangrene	2.3	4.2	<.0001	48.6	89.2	0.62	Yes
Revascularisation/amputation due to peripheral vascular disease	0.3	4.2	<.0001	2.9	40	0.04	No
Transitory ischaemic attacks	0.1	3.4	<.0001	1.8	100	0.03	No
Hemiplegia/hemiparesia	0.1	3.3	<.0001	1.9	50	0.04	No
Cerebrovascular accident/ictus without neurological deficit	1.3	3.1	<.0001	89.8	73.3	0.80	No
Cerebrovascular accident/ictus with neurological deficit	0.7	3.1	<.0001	20.4	86.3	0.32	No
Previous percutaneous coronary intervention	1.5	2.9	<.0001	44.7	87.5	0.58	No
Metastatic cancer	4.9	2.7	<.0001	76.6	42.3	0.53	Yes
Central nervous system tumour	1.3	2.2	<.0001	60	100	0.74	No
Open wound/infection of wound	0.7	2.2	<.0001	31.4	91.7	0.46	Yes
Congestive heart failure	1.5	1.9	.33	26.7	33.3	0.28	No
Acute kidney failure (acute renal insufficiency)	1.5	1.2	.53	5	4.2	0.03	No
Angina pectoris in the month prior to the surgical operation	4.8	1.1	<.0001	66.7	15.6	0.24	No
Paraplegia, paraparesia	0.1	0.9	.001	14.3	100	0.25	No
Transfusion of >4 units of erythrocytes in the 72 hours prior to the surgical operation	7.2	0.6	<.0001	30	2.7	0.04	Yes
Acute myocardial infarction	2.3	0.6	<.0001	22.22	5.4	0.08	No
Ascitis	0.1	0.6	.02	0	0	−0.002	Yes
Current pneumonia	0.9	0.3	.02	40	14.3	0.21	Yes
Systemic sepsis in the 48 hours prior to the operation	0.7	0.2	<.0001	75	25	0.37	No
Oesophageal varices	0.2	0.1	.32	100	66.7	0.79	No
Coma during >24 hours	0.1	0.1	.56	0	0	−0.0008	Yes

a

Adjustment variables according to risk in at least 5 of the models prepared by the National Veterans Administration Surgical Quality Improvement Programme to evaluate complications and mortality in the 30 days after a surgical operation in 8 surgical specialities.

The values of P>.05 are shown in bold type.

Source: Best et al.¹³

Of the 27 comorbidities detected using the MBSD, 23 (85.2%) showed statistically significant differences in frequency between the 2 information systems (see Table 3). The prevalence of the detected diseases is higher using CH than it is using the MBSD. Nevertheless, there was an exception to this pattern in 7 variables, and this was due to the lack of a time limit for these variables when the MBSD was used. For example, for an acute myocardial infarct to be recorded, it had to have occurred in the 6 months prior to surgery. The MBSD does not specify times.

A total of 5 comorbidities (18.5%) fulfilled all 3 of the set criteria; 2 comorbidities (7.4%) fulfilled 2 criteria, 11 diseases (40.74%) fulfilled at least one, while 9 comorbidities (33.3%) did not fulfil any of the criteria (Table 4 and Fig. 1).

Table 4.

Comorbidities ranked according to the number of criteria they fulfil (S, PPV or K >0.6).

3 criteria	2 criteria	1 criterion	0 criteria
Diabetes mellitus	Serious COPD	Smoking	Revascularisation/amputation due to peripheral vascular disease
Cerebrovascular accident/ictus without neurological deficit	Pain in repose/gangrene	Alcohol consumption	Congestive heart failure
Central nervous system tumour		Dyspnoea	Acute kidney failure (acute renal insufficiency)
Hypertension		Transitory ischaemic attacks	Hemiplegia/hemiparesia
Oesophageal varices		Cerebrovascular accident/ictus with neurological deficit	Transfusion of >4 units of erythrocytes in the 72 hours prior to the surgical operation
		Previous percutaneous coronary intervention	Acute myocardial infarction
		Metastatic cancer	Ascitis
		Open wound	Current pneumonia
		Angina pectoris in the month prior to the surgical operation	Coma during >24 hours
		Paraplegia, paraparesia
		Systemic sepsis in the 48 hours prior to the operation

COPD: chronic obstructive pulmonary disease.

Fig. 1.

The validity of the MBSD in comparison with CH in the detection of comorbidities in surgical patients. Sensitivity and positive predictive value along the axis. Bubble size depends on comorbidity frequency.

The S, PPV and K values were analysed according to surgical specialities for the 5 most common comorbidities (hypertension, smoking, diabetes mellitus, alcohol consumption and dyspnoea). The number of indicators with values higher than 0.60 was found to be almost the same as it was in the analysis of the total set of data. However, for diabetes mellitus only 3 of the 9 specialities studied fulfilled all 3 criteria (general, cardiac and vascular surgery).

Discussion

Our results indicate that the source of information used had considerable influence on the number of comorbidities detected per patient. Using the MBSD led to an approximate estimate of 66 comorbidities fewer per 100 patients than was the case with the CH. Additionally, only 7 (17%) of the comorbidities studied fulfilled at least 2 of the quality criteria for evaluation when the MBSD was used.

Best et al.¹³ studied the variables of the NSQIP which influenced the predictive models used to evaluate complications and mortality during the 30 days after surgery. Of the comorbidities evaluated in our study, 9 were important for adjusting risk according to the study by Best,¹³ while only 2 of them (serious chronic obstructive pulmonary disease and pain in repose/gangrene) fulfilled 2 of the set criteria. In these 9 comorbidities, the S and PPV obtained in our study do not differ from the results presented by Best et al.¹³

Analysis according to specialities showed similar results to those of the overall analysis, and therefore indicated that the validity of the MBSD did not depend on the speciality studied.

S and PPV scores have often been used to evaluate the validity of the codes assigned at discharge,^19–21 while the K index is used to measure concordance between different information systems.¹⁸ Although no single reference standard has been set to fix a cut-off point above which S and PPV are considered to be acceptable, values below 0.06 are considered to show room for improvement. Some authors suggest that the MBSD is not a good source of information if the S and PPV differ substantially from 0.9.¹³ In this paper we established the threshold for S and PPV at 0.60, with the aim of making them comparable with the K index value, for which values above 0.6 are considered acceptable.²²

With respect to the prevalence of the identified comorbidities, our data differ from the results presented in the paper by Davis et al.¹⁸ This was foreseeable, as Davis et al. selected patients with certain characteristics for their audit (patients who died within 30 days after surgery, and patients with a high probability of complications, etc.) and who, in general, presented a higher rate of comorbidities than the patients in our study.

The MBSD has great potential in the preparation of epidemiological studies. Its advantages are its accessibility, low cost, broad coverage of samples of a range of population groups and the fact that it facilitates simple electronic processing.¹ Nevertheless, it has certain limitations such as data quality, variability in the extraction and coding of diagnoses in different hospitals, or its limited capacity to differentiate complications from comorbidities.¹⁷ Some of the problems inherent in coding when the ICD-9 is used have been resolved by certain organisation with the inclusion of the POA indicator (present on admission) which makes it possible to identify whether the patient already had a certain disease at the moment of admission, or if this arose during hospitalisation.²³

Nonetheless, one of the main problems with MBSD is that it lacks sufficient detail to make it possible to estimate an adjustment according to risk.¹⁷

We identify the ICD-9 codes of 40 characteristics that may influence mortality and morbidity at 30 days. But as the NSQIP definitions do not always correspond to ICD-9 codes, this leads to a certain degree of subjectivity in how the codes are interpreted.

Moreover, many hospitals use discharge reports as their main source of information for clinical, administrative and demographic data on hospital discharges. If a discharge report does not show all of the comorbidities of a patient, the MBSD will contain less information than the CH.

Another limitation on the use of MBSD is that, in general, ICD-9-CM codes do not take time into account. In fact, 14 of the comorbidities in the ACS NSQIP include some type of time limit in their definitions. These time limits may have influenced our estimates, in such a way that 7 comorbidities had a higher incidence when they were measured using the MBSD than they did in the evaluation based on the CH. This may lead to overestimation of other comorbidities when MBSD data are used.

We use complete CH as the gold standard. Nevertheless, the validity of this source of information is determined, among other things, by the degree of exhaustiveness which healthcare workers have recorded data. This in turn depends on many factors, such as ease of recording, current knowledge of the disease, how exhaustive workers perceive the information to be and the value of quality documentation. In spite of its limitations, we do not have a better source of information than a complete CH. On the other hand, the prevalence of the comorbidities detected depends on the capacity of the revision undertaken by the team responsible for extracting the information from the CH, as well as the characteristics of admitted patients. Davis et al.¹⁸ audited the extraction of information based on the CH of healthcare centres of the Veterans Department of the NSQIP. For 48 of the 52 discrete variables studied, K values varied from 0.61 to 1. None of the variables obtained K values lower than 0.20, which suggests that the data extraction was correct.

Analysis centres exclusively on morbidity variables. Other variables included in the MBSD (such as emergency or programmed admission) were not included in this study, while variables such as the American Society of Anaesthesiologists score, analytical results, degree of wound contamination and the duration of surgery, among other possible variables of interest, were not recorded in the MBSD.

This study has certain limitations. It was undertaken in a single centre. As a result of this, extrapolation to other centres or healthcare contexts will depend on the comparability of the information included in the discharge reports and the codification policy of centres.

The study was restricted to the determination of the presence of certain comorbidities, depending on the source of information used. Our aim was not to evaluate the importance of each one of the comorbidities on AE after a surgical operation. Nevertheless, other studies¹⁴ have shown that the variables recommended by the NSQIP to develop risk adjustment are better than other adjustment systems based on administrative data bases such as the Charlson Comorbidity Index^14,24,25 or the DxCG^®. On the contrary, the paper by Hall et al.¹⁵ showed similar risk adjustment results after using the variables included in the NSQIP or an algorithm that used an administrative data base.

In any case, in the light of the results obtained in other studies,²⁶ further improvements are required in risk adjustment using the variables proposed in the NSQIP.

A long time has passed since Fink⁹ stated that “there is no other valid method of risk adjustment apart from extracting information from clinical histories. We have to cease deceiving ourselves by trying to compare patient cases using administrative data bases”. Nevertheless, improvements to clinical–administrative databases, such as the inclusion of the POA, offer major opportunities for the use of this source of information to measure the quality of patient care.

The use of other sources of information and other methodologies, such as screening guides^5,7 or causal instruments,⁶ may be useful in reducing to a minimum the arduous task of searching for information using CH.

Additionally, we foresee that the increasing availability of clinical data in electronic form will change the role of clinical–administrative data bases, encouraging improvement in the measurement of the quality of healthcare services.²⁷

To conclude, the MBSD detected prevalent comorbidities such as hypertension or diabetes, but it is not supply sufficient data to obtain the variables which are necessary to apply the risk adjustment proposed by the NSQIP in the evaluation of AE in surgical patients.

Financing

This study was supported by the official Spanish Government biomedical financing agency, the Instituto de Salud Carlos III (ISCIII), through a subsidy granted to Dr. Isabel Rodrigo (PI10/02027).

Authorship/Collaborators

Isabel Rodrigo and Francisco J. Abad designed the study. Marta Martín, Belén Tirapu, Pedro Zabalza, Fabiola Oteiza and Asunción Merino took part in data gathering, analysis and interpretation.

Isabel Rodrigo wrote the paper and the other authors revised it critically. All of the authors gave their final approval to the versions presented.

Acknowledgements

The authors would like to thank Blanca Salcedo for helping in the code mapping, together with Sergio Santana and Cristina Eslava for their invaluable help during data gathering.