Abstract
Background and objectives: Different scores to predict acute kidney injury after cardiac surgery have been developed recently. The purpose of this study was to validate externally two clinical scores developed at Cleveland and Toronto.
Design, setting, participants, & measurements: A retrospective analysis was conducted of a prospectively maintained database of all cardiac surgeries performed during a 5-yr period (2002 to 2006) at a University Hospital in Madrid, Spain. Acute kidney injury was defined as the need for renal replacement therapy. For evaluation of the performance of both models, discrimination and calibration were measured.
Results: Frequency of acute kidney injury after cardiac surgery was 3.7% in the cohort used to validate the Cleveland score and 3.8% in the cohort used to validate the Toronto score. Discrimination of both models was excellent, with values for the areas under the receiving operator characteristics curves of 0.86 (95% confidence interval 0.81 to 0.9) and 0.82 (95% confidence interval 0.76 to 0.87), respectively. Calibration was poor, with underestimation of the risk for acute kidney injury except for patients within the very-low-risk category. The performance of both models clearly improved after recalibration.
Conclusions: Both models were found to be very useful to discriminate between patients who will and will not develop acute kidney injury after cardiac surgery; however, before using the scores to estimate risk probabilities at a specific center, recalibration may be needed.
Subtle acute renal dysfunction is almost universal after cardiac surgery. The incidence of acute kidney injury (AKI) in this setting varies depending on the definition used and the specific population studied. Even considering only its most severe form, defined by the need for renal replacement therapy (RRT), AKI rates after cardiac surgery range between 0.33% (1) and 9.5% (2). Besides patient- and procedure-related factors, local practice patterns may partially explain this nearly 30-fold difference (3).
The outcome of patients who have AKI and need RRT after cardiac surgery is poor, with high in-hospital mortality and resource utilization (4). Preventive measures have failed to show any benefit in a consistent way (5). Identifying patients who are at high risk for developing AKI after cardiac surgery before the procedure may help not only to provide a more detailed informed consent but also to focus on a specific cohort in which new preventive treatments can be studied.
Five predictive models of AKI after cardiac surgery in adult patients have been developed so far. In 1997, Chertow et al. (6) published a landmark study based on a large population database (>40,000 patients from 42 centers). This model has received external validation by Fortescue et al. (7) and by Eriksen et al. (8). In 2005, Thakar et al. (9) developed a clinical score based on a large cohort of patients (>30,000) from a single center. In contrast with Chertow's work, in that study, all surgical procedures were well represented and only recipients of a renal transplant and patients who were on preoperative dialysis were excluded. One year later, Mehta et al. (10), using data from the Society of Thoracic Surgeons database, published a bedside tool for predicting the risk for postoperative dialysis after cardiac surgery. During the past year, two new models were developed. Wijeysundera et al. (11) developed and validated a simplified renal index (SRI) based on patients who underwent cardiac surgery under cardiopulmonary bypass (CPB) at two Canadian centers. Finally, Palomba et al. (12) designed the Acute Kidney Injury after Cardiac Surgery (AKICS) score based on a cohort of patients who underwent elective surgery at a Brazilian center. In contrast with the other models, the AKICS score can predict less severe forms of AKI. The aim of this study was to validate externally the clinical scores developed by Thakar et al. (9) (Cleveland score) and Wijeysundera et al. (11) (SRI score, Toronto) for patients who underwent surgery at a University Hospital with a low-medium volume of activity and a different population surgical profile.
Materials and Methods
Study Population
Data from every cardiac surgical patient operated on at our institution and admitted to the postoperative cardiac surgical unit are routinely collected and entered in the Cardiac Anesthesia database. After institutional research ethics board approval for this study, data from all patients who received major cardiac surgical procedures between January 1, 2002, and December 31, 2006, were considered. During this period, there were no changes in either the CPB pump prime or the anesthesia protocol. The following categories of patients were excluded: Patients who had chronic renal insufficiency (CRI) and were receiving any form of dialysis therapy before surgery, recipients of a renal transplant before the cardiac procedure, intraoperative or early (<24 h) postoperative deaths, patients with postoperative course followed at another unit, and patients with missing data. Minor cardiac procedures (automated implantable cardioverter defibrillator, sternal work, cardiac tamponade) were not included. For patients who received more than one major surgical procedure during the same admission, only the first episode was considered.
Definitions
Preoperative renal function was assessed both by preoperative serum creatinine (sCr), defined as the nearest value to the surgical date, and by GFR estimated by the Cockcroft-Gault formula (13) (eGFR). AKI was defined as the need for RRT. Indications for RRT in patients with AKI were volume overload, uremia, or biochemical abnormalities. Before starting RRT, all patients were evaluated by one of two members of the nephrology department. Demographic variables and preoperative risk factors that were extracted from the database were those that were included in the original models (9,11): Age, gender, history of congestive heart failure, diabetes that required medical treatment or insulin therapy, chronic obstructive pulmonary disease that was treated with bronchodilators, previous cardiac surgery, preoperative sCr value (in mg/dl), preoperative eGFR, preoperative ventricular function (assessed either by angiography or by echocardiography), presence of an intra-aortic balloon pump before surgery, type of cardiac surgery (isolated coronary artery bypass graft [CABG], valve procedures, combined CABG and valve surgery, and other major surgical procedures), and operative status (elective, urgent, or emergent). For each one of these variables, the same definitions as in the original works were applied (9,11). Details of both scores including points ascribed to each variable appear in Table 1.
Risk factors and points in Cleveland and SRI (Toronto) scoresa
Statistical Analyses
Association between the development of AKI and preoperative factors was first explored by univariate analyses. χ2 test, unpaired t test, or Fisher exact test was used as appropriate. Second, additive scores from both models were calculated for every patient by adding points for each risk variable present as in the original studies. Discrimination of both scores in this cohort was evaluated by calculating the areas under the receiver operator characteristic curve (AUC) or C index. Calibration of both models was assessed first by calculating the individual risk for AKI for each patient in each model. Then, risks were ordered into deciles and observed cases were compared with predicted cases across deciles with the Hosmer and Lemeshow (HL) goodness-of-fit test (14). In addition, event rates in each risk category of the derivation cohorts were compared graphically with the observed event rates for the same categories. When recalibration was deemed necessary, it was accomplished through logistic regression, using the logit-transformed original predictions of both models as the independent variable, and the same outcome (AKI) as the dependent variable (logistic recalibration). Statistical analyses were performed with SPSS 12.0 (SPSS, Chicago, IL) and Epidat 3.1. (Pan American Health Organization, Washington, DC; http://www.paho.org). All analyses were evaluated at an α = 0.05 significance level.
Results
A total of 1892 major surgical procedures were performed during this 5-yr period in 1867 patients (25 patients receive two operations in different admissions). After discarding cases with CRI or kidney transplants (31 surgeries in 30 patients), intraoperative or early postoperative deaths (n = 52), patients who were followed at another unit (n = 10), second surgeries during the same admission period (n = 5), procedures with missing data (n = 13), and one patient with a left ventricular assist device, a total of 1780 cardiac surgeries were considered. This global cohort was used to validate the Cleveland score. For validation of the SRI score, 217 patients from this cohort were excluded, either because they were operated on without CPB (n = 213) or because their preoperative sCr value was >3.4 mg/dl (n = 4). These 1563 patients make the CPB cohort. Table 2 shows main characteristics and risk factors of the global cohort as well as their association with AKI.
Main characteristics and risk factors in the global cohort and their association with AKIa
Frequency of AKI was 3.7% (95% confidence interval [CI] 2.8 to 4.6) in the global cohort and 3.8% (95% CI 2.6 to 4.8) in the CPB cohort. Discrimination of both models was excellent. The AUC (95% CI) were 0.86 (0.81 to 0.9) and 0.82 (0.76 to 0.87) for the Cleveland and SRI scores, respectively. Because of the higher rate of AKI in our study sample, calibration of both models was poor (HL statistic values of 81.4 and 57.8 for the Cleveland and Toronto models, respectively; both P < 0.001). In our cohort, both models underestimated the risk for AKI except for patients within the very-low-risk category (Table 3, Figures 1 and 2). After logistic recalibration, calibration of both models improved (HL 12.9 [P = 0.17] for the Cleveland model and HL 10.4 [P = 0.32] for the SRI model).
Acute kidney injury (AKI) risk across risk categories in the Cleveland derivation cohort and in the global cohort. •, Cleveland derivation cohort; ○, the global cohort of the study; bars represent 95% confidence intervals (CI).
AKI risk across risk categories in the Toronto derivation cohort and in our cardiopulmonary bypass (CPB) cohort. ▪, Toronto derivation cohort; ○, CPB cohort of the study; bars represent 95% CI.
Frequency of AKI at different score levels and risk categories in the derivation cohorts of the original models and in the study cohortsa
Discussion
External validation of a prediction prognostic model becomes mandatory before applying it to new patients or outside its development context. Among the five prediction models available, we decided to validate the Cleveland and SRI scores because they include only variables that are known before surgery, are easy to calculate, and focus on a clinical relevant outcome (AKI that requires RRT). We have shown that both models have excellent discrimination and are able to distinguish between patients who will and will not develop AKI, with high confidence. This was already known for the Cleveland score (11,15), but, to our knowledge, this has been shown for the first time for the CRI; however, discrimination is only one aspect of the external validation process. Calibration, or agreement between the risk predicted and the risk observed, should also be considered. As expected, because of the higher incidence of AKI in our cohort, both scores generated inaccurate risk predictions, and their calibration in our sample of patients was poor, predicting lower risk than that actually observed. Although our study was not designed to find the reasons for this miscalibration, some considerations can be made.
First, miscalibration can be related to differences in the distribution of risk factors between the original derivation sample in which the model was developed and the sample in which it is tested—that is, variation in “case mix.” It is widely known that preoperative renal function is the main factor to consider when evaluating patients who are at risk for AKI after cardiac surgery. This is clearly accounted for in both models; although with different weights, both assign the highest score to patients with the worst preoperative renal function (Table 1). Patients in our cohort had lower preoperative eGFR than patients in the Toronto development sample (66 versus 82 ml/min). Similarly, other risk factors and their weight in a prediction model can vary among different studies. For instance, risk factors such as history of chronic heart failure or presence of chronic obstructive pulmonary disease or demographic factors such as gender, included in the Cleveland score, are not considered in the SRI score, reflecting, to some extent, the population from which they were developed. Most studies of risk factors for acute renal failure after cardiac surgery are based on populations of patients in which isolated CABG surgery represents >50% of the sample (16–19). This is also true for the scores that we tried to validate (52.5 and 65.0% in the derivation cohorts of the Cleveland and Toronto scores, respectively); however, in our cohort, only 35.7% of patients underwent isolated CABG surgery. In a similar way, patients in our cohort were older (mean age 66 yr versus 64 and 62 for the Cleveland and Toronto scores, respectively). In addition, the proportion of female patients was greater (40 versus 31 and 26%) and emergent cases were more frequent (6 versus 3 and 2%). On the contrary, other risk factors, such as previous cardiac surgery (12.2 versus 21.7% for the Cleveland score) or preoperative intra-aortic balloon pump (1.2 versus 1.5 and 2%), were less prevalent in our study sample. This different surgical population profile (worse preoperative renal function, less isolated CABG surgery, and more emergent cases in older patients with a higher proportion of women) may explain partially the underestimation of risk observed in our sample.
Second, even considering similar patients, miscalibration can be due to variation in local practices provided by anesthesiologists (intraoperative and postoperative care) and nephrologists (threshold to start RRT). These differences can influence the rate of AKI that requires RRT.
Another factor that could influence the calibration is the time frame considered for which the outcome was observed (20). Changes in renal function after cardiac surgery usually occur during the first 72 h (21–23), so any change in renal function that occurs during the first postoperative week, including the need for RRT, can be more directly related to surgery than changes that occur after this period, usually related to the development of postoperative complications (e.g., sepsis, bleeding, nephrotoxicity). For both models, the time frame considered was the “postoperative period,” usually defined as the period from admission to the intensive care unit after surgery until discharge home; however, no data about median time to start of RRT were provided for any of the models. In our cohort, only 50% of patients who had AKI that required RRT developed it during the first week (median time to start of RRT 7 d; data not shown), pointing to a relation between postoperative complications and the development of most severe forms of AKI. In fact, when the validation analysis was repeated without considering these “late” AKI cases, both models improved their calibration (HL 18.3 [P = 0.049] for the Cleveland model and HL 11.2 [P = 0.13] for the SRI model; data not shown), with no effect on discrimination (AUC values 0.88 and 0.81 respectively; data not shown). Indices that are based exclusively on preoperative data will fail to classify and assign risk to patients who develop unexpected intraoperative or postoperative complications. In the AKICS score developed by Palomba et al. (12), these facts have been taken into account. Those authors included some intraoperative (CPB time) and early postoperative variables (low cardiac output, central venous pressure at admission to the intensive care unit) and defined a clear-cut time frame of 7 d after surgery; however, this score cannot be used to provide a more detailed informed consent before surgery and will identify only patients who are at risk for AKI after elective surgery, a period when the process of injury may have started.
Finally, the surgical volume should also be considered. The influence of hospital volume on mortality is widely known (24). This effect has been proved for some major surgeries, including CABG (25,26), with centers and surgeons with higher volumes showing the best figures. Although this trend has not been studied, it can probably be applied also to other outcomes, such as AKI, which is a known independent risk factor for mortality.
Nonetheless, miscalibration does not make the scores useless. Because higher score suggests higher risk (Table 3), both models can still be used to detect patients who are at high risk for AKI. In this sense, we found both scores very easy to use. Calculation at the bedside or in the operating room before anesthesia induction can be done in just a few seconds. In choosing one of them instead of another, probably the main factor is the evaluation of the preoperative renal function provided by the local laboratory (sCr versus eGFR); however, great caution should be taken in applying the predicted risks in the informed consent process. Whenever this is needed, a calibration analysis should be done before relying on the risks predicted in the original models, because they may underestimate or overestimate the actual risk. Caution should also be taken when using the scores as inclusion or exclusion criteria in trials of preventive measures of AKI after cardiac surgery, because the sample size will vary depending on the incidence of each particular risk category at a specific center.
Some limitations of the study should be noted. The reduced sample of patients who were used to perform the external validation of the original models (10 to 15% of the original derivation cohorts) (27) provided wide CI in the higher risk categories (Figures 1 and 2); however, because of the low volume of cardiac surgery performed annually at our center, a period of >10 yr would have been needed to overcome this limitation. During such a long period, local protocols and practice patterns may have changed, making the results less reliable. We decided not to include in the analyses patients with missing data because they represented <1% of the global cohort. On the contrary, patients with preoperative mechanical ventilation or tracheostomy, originally excluded from the Cleveland derivation cohort, were included in our study, because these variables are not captured by our database. An estimation based on data recorded during 2007 showed that these categories represent <0.5% of the global cohort.
Conclusions
We have tried to validate externally two clinical scores that were developed to predict acute renal failure that needs RRT after cardiac surgery. Although both scores showed excellent discrimination, calibration was poor in our sample, with an underestimation of risk across risk categories except for patients in the very-low-risk stratum. Recalibration of the models improved their performance. Both scores can be used to detect patients who are at risk for AKI; however, depending on the incidence of this outcome at a specific center, a recalibration may be needed before using them to provide more detailed information.
Disclosures
None.
Acknowledgments
Drs. Thakar and Wijeysundera kindly provided the intercepts of their models for the calibration process. In addition, Dr. Wijeysundera made some useful suggestions. We also acknowledge all staff anesthesiologists at the cardiac anesthesia unit for constant contribution to the maintenance of the Cardiac Anesthesia Database.
A short version of this article was accepted as an abstract presentation at the 2008 European Anaesthesiology Congress (May 31 through June 3, 2008; Copenhagen, Denmark).
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
- Received February 1, 2008.
- Accepted April 10, 2008.
- Copyright © 2008 by the American Society of Nephrology
References
In this issue
Jump to section
More in this TOC Section
Cited By...
- Sex and the Risk of AKI Following Cardio-thoracic Surgery: A Meta-Analysis
- A new model to predict acute kidney injury requiring renal replacement therapy after cardiac surgery
- Dynamic Predictive Scores for Cardiac Surgery-Associated Acute Kidney Injury
- Risk of postoperative acute kidney injury in patients undergoing orthopaedic surgery--development and validation of a risk score and effect of acute kidney injury on survival: observational cohort study
- Predictors of Death and Dialysis in Severe AKI: The UPHS-AKI Cohort