Analysis of congenital heart surgery results: A comparison of four risk scoring systems

Okan Yıldız¹, Taner Kasar², Erkut Öztürk², Behzat Tüzün¹, Hüsnü Fırat Altın³, İsmihan Selen Onan¹, Alper Güzeltaş², Sertaç Haydin¹

¹Department of Pediatric Cardiovascular Surgery, University of Health Sciences, Mehmet Akif Ersoy Thoracic and Cardiovascular Surgery Training and Research Hospital, İstanbul, Turkey
²Department of Pediatric Cardiology, University of Health Sciences, Mehmet Akif Ersoy Thoracic and Cardiovascular Surgery Training and Research Hospital, İstanbul, Turkey
³Department of Pediatric Cardiovascular Surgery, University of Health Sciences, Dr. Siyami Ersek Thoracic and Cardiovascular Surgery Training and Research Hospital, İstanbul, Turkey

DOI : 10.5606/tgkdc.dergisi.2018.15083

Abstract

Background: This study aims to evaluate the surgical results of our clinic according to presumption systems of Risk Adjustment in Congenital Heart Surgery, Aristotle Basic Complexity score, Aristotle Comprehensive Complexity score, and Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories and to compare the efficiency of these systems in predicting morbidity and mortality.

Methods: In the study, classification and the risk scoring were performed with the four different systems for 1,950 patients (1,038 males, 912 females; mean age 5.5 months; range, 1 day to 18 years) who were administered congenital heart surgery between 1 October 2012 and 31 December 2016. The hospital mortality and morbidity were calculated for each category from the four models. The discriminatory ability of the models was determined by calculating the area under the receiver operating characteristic curve and the receiver operating characteristic curves of the four models were compared.

Results: Median weight of the patients was 7.2 kg (range, 1.8-80 kg). Among the patients, 53% were males and 47.5% were younger than one year of age. Of totally 1,950 operations, mortality was observed in 149 (7.6%) and morbidity was observed in 541 (27.7%). Areas under the receiver operating characteristic curve for mortality were 0.803, 0.795, 0.729, and 0.712 for the Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories, Aristotle Comprehensive Complexity, Risk Adjustment in Congenital Heart Surgery, and Aristotle Basic Complexity scores, respectively. Areas under the receiver operating characteristic curve for morbidity were 0.732, 0.731, 0.730, and 0.685 for the Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories, Risk Adjustment in Congenital Heart Surgery, Aristotle Comprehensive Complexity, and Aristotle Basic Complexity scores, respectively.

Conclusion: Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories, Risk Adjustment in Congenital Heart Surgery, Aristotle Basic Complexity, and Aristotle Comprehensive Complexity score systems were effective in predicting the morbidities and mortalities of patients who underwent congenital heart surgery and evaluating the performance of the surgical centers. Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories were on the forefront due to high feasibility and performance. Aristotle Basic Complexity score system had the lowest performance. Combinations of systems will provide the most benefit during evaluation of results.

Abstract

Congenital heart disease is an important health problem in both developed and developing countries. Approximately nine of every 1,000 live births and onethird of every major congenital disease are congenital heart diseases.[1]

Congenital heart diseases have various types and numbers of pathologies. Patients undergo either corrective or palliative surgeries depending on the problem. In addition, many underlying factors such as low birth weight, genetic syndromes, or preoperative clinical conditions affect the success of surgery.

It is difficult to establish a risk stratification system of nomenclature that is universally accepted for congenital heart diseases.[2] There are different risk scoring systems for pediatric cardiac surgery to determine the risks and difficulty levels. In addition, healthcare centers may use scoring systems to predict their mortality and morbidity rates; thus, determine their current situation compared to national and international healthcare centers and prepare for necessary regulations.[3-6] Risk Adjustment in Congenital Heart Surgery (RACHS-1), Aristotle Basic Complexity (ABC) score, Aristotle Comprehensive Complexity (ACC) score, and Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories (STS-EACTS MC) systems are most commonly used for this purpose.[3-5] Although these systems are strongly related to hospital mortality and morbidity rates, studies show that they are different in terms of effectiveness.[5-8] Furthermore, studies that compare all four scoring systems are limited. Therefore, in this study, we aimed to evaluate the surgical results of our clinic according to presumption systems of RACHS-1, ABC score, ACC score, and STS-EACTS MC and to compare the efficiency of these systems in predicting morbidity and mortality.

Methods

This study included 1,950 congenital heart disease patients (1,038 males, 912 females; mean age 5.5 months; range, 1 day to 18 years) operated in University of Health Sciences İstanbul Mehmet Akif Ersoy Thoracic and Cardiovascular Surgery Training and Research Hospital between 1 October 2012 and 31 December 2016. The study protocol was approved by the University of Health Sciences İstanbul Mehmet Akif Ersoy Thoracic and Cardiovascular Surgery Training and Research Hospital Ethics Committee. The study was conducted in accordance with the principles of the Declaration of Helsinki.

The inclusion criterion was to have undergone operation due to a congenital or acquired heart disease (either corrective or palliative) under the age of 18. Cases that could not be categorized in any of the four scoring systems, such as permanent pacemaker implantations, and those that were transferred to another unit for any reason were excluded. Data from patient records regarding age, gender, weight, year of surgery, diagnosis, presence of syndromes, type of surgery, and use of cardiopulmonary bypass (CPB) were obtained. Cases were separated according to their age groups as newborn (1-28 days), infant (29 days-1 year), toddler (1-12 years) and adolescent (12-18 years).

A team consisting 11 pediatric cardiologists and pediatric cardiac surgeons from Boston Childrens Hospital created the RACHS-1 model, which contains 207 procedures. The RACHS-1 system scale ranges from one to six. Three additional clinical factors (age, prematurity and non-cardiac congenital structural abnormalities) complement the model.[3]

In 1999, 50 cardiac surgeons led by Lacour-Gayet from 23 different countries developed the ABC score, which is based on three factors: potential for mortality, potential for morbidity and anticipated technical difficulty. For the ABC system, the scale ranges from 1.5 to 15. The ACC further adjusts complexity according to specific patient characteristics. It includes two categories of complexity factors as procedure dependent and procedure independent factors and a corresponding basic complexity level between one and four (level 1 [1.5 to 5.9]; level 2 [6.0 to 7.9]; level 3 [8.0 to 9.9] and level 4 [10.0 to 15.0]).[4]

Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories are the latest system found in 2008. The mortality risk was estimated for 148 procedure types, using real data from 77,294 patients (33,360 patients from the EACTS and 43,934 patients from the STS) between 2002 and 2007. Using Bayesian statistics that fit the data for small denominators, mortality rates were calculated for each procedure. For the STSEACTS MC, the scale range from 0.1 to 5.0, and a corresponding mortality category level between 1 and 5 is assigned (level 1 [0.1 to 0.3]; level 2 [0.4 to 0.7]; level 3 [0.8 to 1.2]; level 4 [1.3 to 2.6] and level 5 [2.9 to 5.0]).[5]

In each scoring system, a higher score indicates a higher risk of mortality. For patients undergoing multiple procedures, the procedure with the highest level was scored. Every single cases risk scores were calculated using RACHS-1, ABC, ACC, and STSEACTS MC scoring systems.

In this study, our two primary outcomes were operative mortality and hospital morbidity. Operative mortality included all the deaths occurred during the hospital stay when the operation was performed and the deaths occurred after discharge within 30 days of the procedure. Intensive care unit stay longer than seven days was identified as hospital morbidity.

Statistical analysis
Data were analyzed using SPSS for Windows version 15.0 software (SPSS Inc., Chicago, IL, USA). Descriptive statistics were performed in relation to the distribution of variables and characteristics of the study population. Categorical variables were represented as frequencies; the numeric variables as mean or median with the respective measures of dispersion. Analysis of the discriminatory ability of the surgical risk stratification methods were performed using the C statistic comparison with receiver operating characteristic (ROC) curves of the four methods.[7]

Methods

Results

Of the study population, 336 (17.2%) were newborns and 1,213 (62.2%) were under one year of age. Of the cases, 220 were operated without using CPB. Median weight was 7.2 kg (range, 1.8 to 80 kg). A majority of the operations were performed in the year 2016 (n=475). Demographic characteristics were shown in Table 1.

Table 1: Demographic characteristics of patients undergoing cardiovascular surgery

Mortality and morbidity rates were 7.6% (n=149) and 27.7% (n=541), respectively. Mortality by years was 9.5% in 2012, 8.9% in 2013, 8.7% in 2014, 6% in 2015 and 5.9% in 2016. Seventeen cases could not be categorized in RACHS-1. In all scoring systems, as the category or the level increased, the mortality and the morbidity increased accordingly. The mortality and the morbidity rates were shown in Table 2.

Table 2: Distribution of scores over cohort

The results of the area under the ROC curve for mortality and morbidity are shown in Figure 1a, b. The RACHS-1 categories, ABC, ACC and STS-EACTS MC had satisfactory performance (above 0.650).

Figure 1: Receiver operating characteristic curves: (a) Mortality receiver operating characteristic curves for comparison. (b) Morbidity receiver operating characteristic curves for comparison.
ROC: Receiver operating characteristic; RACHS-1: Risk Adjustment in Congenital Heart Surgery; STS-EACTS MC: Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories; ACC: Aristotle Comprehensive Complexity score; ABC: Aristotle Basic Complexity score.

There was no statistical difference between the four forms of categorization and the areas under the ROC curve of the four methods for the discriminatory capacity for hospital mortality (Table 3).

Table 3: Results regarding area under receiver operating characteristic curve between methods concerning mortality and morbidity outcomes

Results

Discussion

In our study, we aimed to compare all four scoring systems according to their predictive values on mortality and morbidity in congenital heart surgery. Thus, we detected that STS-EACTS MC had a C-index of 0.803 (95% confidence interval [CI] 0.768-0.837), RACHS-1 of 0.729 (95% CI 0.689-0.770), ABC of 0.712 (95% CI 0.668-0.755), and finally ACC of 0.795 (95% CI 0.753- 0.837). All four scoring systems predicted mortality significantly similar to each other while STS-EACTS MC provided superior predictions. Since there are a limited number of studies that compare RACHS-1, ABC, ACC, and ST-EACTS MC scoring systems,[8-14] our study may be considered valuable and extensive.

There are different types of surgeries used in various complex congenital heart diseases. Therefore, scoring systems are needed to evaluate mortality and morbidity. It is difficult to establish a universally accepted risk stratification system of nomenclature as congenital heart surgery involves approximately 200 different diagnostic topics and procedures. Categorization of these into groups or relatively homogeneous strata is necessary to compare the outcomes within each category. In pediatric cardiac surgery, mortality analysis without stratification of complexity is a failure. This risk stratification has been identified essential for assessing and improving the quality.[15,16]

Risk Adjustment in Congenital Heart Surgery has been the first system used for this purpose. It is rather an easier model, because it needs less data. Although it does not contain all cardiac procedures and is based on personal foresight, it is widely used to determine mortality. RACHS-1 was first described by Jenkins et al.[3] They divided cardiac diseases into six categories. Expected mortalities in category 1, 2, 3, 4 and 6 are 0.4%, 3.8%, 8.5%, 19.4%, and 47.7%, respectively. For category 5, there is no estimation of mortality.

Cavalcante et al.[14] evaluated 3,071 cases and showed that mortality ratios according to RACHS-1 was 1.8% in category 1, 5.5% in category 2, 14.9% in category 3, 32.5% in category 4, and 68.8% in category 6. In our study, 1,931 of 1,950 cases were suitable for RACHS-1. Majority of the cases were in category 2 (n=838). Mortality rates were as follows: 0.5% in category 1, 3.4% in category 2, 10.5% in category 3, 13.8% in category 4, and 43.1% in category 6.

European Association for Cardiothoracic Surgery defined Aristotle scoring system as two separate scoring systems, namely, ABC and ACC, in 2014.[4] The first score was the Basic score, which adjusted only for the complexity of the procedure. The second score was the Comprehensive score, which took into account specific procedure-dependent and procedureindependent patient characteristics. It is divided into four levels according to difficulty. Lacour-Gayet et al.[4] indicated mortality and morbidity in ABC and ACC as follows; <1% in level 1, 1-5% in level 2, 5-10% in level 3, and 10-20% in level 4. In our study, mortality for ABC and ACC was 0.7%-0.8% for level 1, 5.4%-3.7% for level 2, 7.9%-5% for level 3, and 19.7-15.8% for level 4, respectively. Our results were similar to predicted values.

Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories is the most recent scoring system estimating mortality based on processing data from a database containing 148 different procedures. It divides into five subgroups according to difficulty level. Mortality rates are as follows; 0.8% in level 1, 2.6% in level 2, 5% in level 3, 9.9% in level 4, and 23.1% in level 5. In our study, mortality rates were 0.6% in level 1, 2% in level 2, 8.7% in level 3, 16% in level 4, and 55% in level 5. An evaluation of all the scoring systems showed that expected rates were obtained according to the results of RACHS-1, ABC, and ACC. Based on STS-EACTS MC, we may suggest that regulations should be developed to improve the results in levels 3-5.

Different results were achieved when the scoring systems were compared in different studies. Al-Radi et al.[15] evaluated t he m ortality p rediction of R ACHS-1 and ABC in 11,438 operated congenital heart disease patients. They both indicated strong predictive effect while the best scoring system was RACHS-1 (ROC curve 0.733&0.698).

Furthermore, Joshi et al.[8] retrospectively evaluated the mortality prediction using ROC analysis in 1,150 cases. The results were 0.677 (95% CI: 0.61-0.73) for ABC, 0.704 (95% CI: 0.64-0.76) for ACC, and 0.607 (95% CI: 0.55-0.66) for RACHS-1. They claimed that ACC had better results than both ABC and RACHS-1.

Moreover, Bojan et al.[12] compared ACC and RACHS-1 on 1,384 cases and suggested that ACC had superior results. They also stated that when a corrected model for age, prematurity and extracardiac anomalies was used, then RACHS-1 had an equal efficacy.

Cavalcanti et al.[16] compared mortality in three systems (STS-EACTS MC, ACC, and RACHS-1) on 360 cases who were under 18 years of age in Brazil. The three models had similar accuracy by calculating the area under the ROC curve: RACHS-1: 0.738, STS-EACTS MC: 0.739, and ACC: 0.766.

In addition, OBrien[5] compared mortality prediction of STS-EACTS MC to ACC and RACHS-1. The discriminatory capacity of STS-EACTS MC by area under the ROC curve (0.778) was higher than RACHS-1 (0.745) and ABC (0.687).

The association of mortality outcomes with annual hospital volume has been substantiated in the literature. Centers with <150 cases/year had an odds ratio of 1.59 for having higher operative mortality. Certainly, this reflects the learning curve for surgeons, perioperative cardiologists, and anesthesiologists.[17,18] Our study supports these opinions, with mortality being 9.5% in 2012 and decreasing to 5.9% in 2016.

Some studies have emphasized that scoring systems can be used as a predictor of morbidity. Larsen et al.[19] applied the RACHS-1 score to rank 957 procedures in Denmark and concluded that it is a good predictor of morbidity.

On the other hand, Kogon and Oster[11] evaluated 458 cases of congenital heart disease patients who were over 18 years old. For prolonged length of stay, areas under the ROC curve were 0.82, 0.76, and 0.61 for the Aristotle, STS-EACTS MC, and RACHS-1 scores, respectively. In our study, the morbidity predictability of the four scoring systems was high, with STS-EACTS MC having the best predictive score (0.732). This was followed by RACHS-1 (0.731) and ACC (0.730). The efficacy of ABC was lower than the others (0.685).

There were some limitations to our study. Primarily, it was a retrospective and single-centered study. Furthermore, the number of operations performed in the first year was low compared to the other years. In addition, there were three different surgical teams performing the surgeries.

In conclusion, all four scoring systems that assist in resolving the complexity of congenital heart disease and difficulty of surgical treatment were successful in determining hospital mortality and morbidity. However, although an evaluation of the comparative results showed that the scoring systems provided similar results, the Society of Thoracic Surgeons and European Association for Cardiothoracic Surgery mortality categories scoring system was superior than the others.

Declaration of conflicting interests
The authors declared no conflicts of interest with respect to the authorship and/or publication of this article.

Funding
The authors received no financial support for the research and/or authorship of this article.

Discussion