Abstract
Background and objectives Acute kidney injury (AKI) requiring dialysis is associated with high mortality. Most prognostic tools used to describe case complexity and to project patient outcome lack predictive accuracy when applied in patients with AKI. In this study, we developed an AKI-specific predictive model for 60-day mortality and compared the model to the performance of two generic (Sequential Organ Failure Assessment [SOFA] and Acute Physiology and Chronic Health Evaluation II [APACHE II]) scores, and a disease specific (Cleveland Clinic [CCF]) score.
Design, setting, participants, & measurements Data from 1122 subjects enrolled in the Veterans Affairs/National Institutes of Health Acute Renal Failure Trial Network study; a multicenter randomized trial of intensive versus less intensive renal support in critically ill patients with AKI conducted between November 2003 and July 2007 at 27 VA- and university-affiliated centers.
Results The 60-day mortality was 53%. Twenty-one independent predictors of 60-day mortality were identified. The logistic regression model exhibited good discrimination, with an area under the receiver operating characteristic (ROC) curve of 0.85 (0.83 to 0.88), and a derived integer risk score yielded a value of 0.80 (0.77 to 0.83). Existing scoring systems, including APACHE II, SOFA, and CCF, when applied to our cohort, showed relatively poor discrimination, reflected by areas under the ROC curve of 0.68 (0.64 to 0.71), 0.69 (0.66 to 0.73), and 0.65 (0.62 to 0.69), respectively.
Conclusions Our new risk model outperformed existing generic and disease-specific scoring systems in predicting 60-day mortality in critically ill patients with AKI. The current model requires external validation before it can be applied to other patient populations.
Introduction
Acute kidney injury (AKI) requiring dialysis occurs in 1% to 4% of hospitalized patients and, in the setting of multisystem organ failure, is associated with exceptionally high morbidity and mortality (1–3). To date, there is no specific treatment for AKI. Moreover, neither differences in modality (intermittent hemodialysis versus continuous renal replacement therapy [CRRT]) nor intensity (higher Kt/Vurea and/or hemofiltration or hemodiafiltration volume versus lower) of therapy have been shown to favorably influence survival in randomized clinical trials (4–6).
Risk adjustment and mortality prediction are critically important for comparing outcomes across interventions and health systems. Over the past three decades, severity-scoring systems, which capture various combinations of physiologic derangement, organ dysfunction, and pre-existing comorbidity, have been developed and applied to predict outcomes in the intensive care unit (ICU) (7–9). Disease-specific scores have also been developed in critically ill patients with AKI. However, with changes in clinical characteristics of hospitalized patients with AKI, updated or new models are required to apply to contemporary cohorts (10–12).
The Veterans Affairs/National Institutes of Health (VA/NIH) Acute Renal Failure Trial Network (ATN) study (ClinicalTrials.gov, NCT00076219) was a multicenter randomized clinical trial of intensive versus less intensive renal support in critically ill patients with AKI conducted between November 2003 and July 2007 at 27 VA- and university-affiliated medical centers (5,13). Using data from all randomized patients, we aimed to develop a new AKI-specific predictive model and to examine the relative performance of commonly used generic and AKI-specific severity-of-illness scoring systems.
Study Population and Methods
Study Participants
The ATN study enrolled 1124 adults in critical care units who had AKI attributable to acute tubular necrosis, as well as sepsis or at least one additional nonrenal organ system failure. Specific inclusion and exclusion criteria and a detailed description of the study cohort are available elsewhere (5,13). Briefly, subjects with a clinical diagnosis of AKI requiring dialysis due to acute tubular necrosis were eligible for inclusion in the ATN study; patients with chronic kidney disease (defined as premorbid serum creatinine >2 mg/dl in men and >1.5 mg/dl in women) or prior kidney transplantation were excluded.
Explanatory Variables
In developing the new predictive model, we initially considered 117 clinically relevant variables, including demographic, clinical, and laboratory information at time of initiation of renal replacement therapy (RRT), corresponding to the time of entry into the ATN study. Variables that captured redundant physiologic parameters, had prevalence rates below 5%, or were treatment related were excluded; out of the remaining 53 candidate variables, 49 were missing data in fewer than 5% of subjects, and the remaining four variables were missing data in fewer than 15%. Two subjects were excluded due to missing laboratory data. All missing categorical data were related to medical history. Missing data were more common among patients consented by surrogate. Prior studies have demonstrated that when elements of medical history are unknown by surrogates, and there is no record of a condition being present, the condition is usually absent. For this reason, we viewed these missing data elements as informative and were imputed as “No,” with a probability of 0.9, and “Yes,” with a probability of 0.1. Missing data for continuous variables were imputed using regression-based maximum likelihood methods (14).
Model Building
We summarized demographic, clinical, and laboratory data as mean (SD) for continuous variables and frequency for discrete variables. We applied univariate logistic regression to each variable for the initial model selection with 60-day all-cause mortality as the dependent variable (589/1122), and subsequently evaluated for linear, quadratic, and cubic fit. Variables with P ≤ 0.1 on univariate analysis, or that were selected a priori based on clinical grounds, were included in the final multivariable logistic regression model. We retained variables in the multivariable model using backward selection with P ≤ 0.05. We a priori evaluated several multiplicative interactions based on clinical grounds. The coefficients from the fitted logistic regression model were used to develop an interactive Web-based tool to calculate the probability of 60-day mortality automatically.
Risk Score Construction
We also constructed a paper-based scoring system (integer risk score), derived from the above-mentioned risk model, to facilitate manual calculation. We chose a constant that, when multiplied by the parameter estimates of categorical variables in the model, would result in integers (score points) and minimize residuals. Subsequently, we multiplied coefficients of continuous variables by the constant, which then represents the increase in risk score for each unit increase of an individual continuous variable.
Model Performance
We assessed model discrimination using the area under the receiver operating characteristic (ROC) curve, the concordance index (C-statistic). We applied a bootstrap procedure (500 draws with replacement) to obtain parameter estimates for the C-statistic. Model optimism was subsequently assessed by comparing the bootstrap adjusted and the original model C-statistic (15). We assessed model calibration using the Hosmer–Lemeshow goodness-of-fit chi-squared test. To compare results of the new model to older generic and disease-specific scores, we compared model performance metrics for the Sequential Organ Failure Assessment (SOFA), Acute Physiology and Chronic Health Evaluation II (APACHE II), and Cleveland Clinic Foundation (CCF) scores applied to ATN study database (7–10). Model comparisons were carried out using nonparametric method proposed by DeLong (16).
Results
Adequate data were available on 1122 (99%) randomized subjects, with mean age 59.7 ± 15.3 years; 70.6% were male; 74.4% were Caucasian. The mean baseline serum creatinine concentration (sCr) before developing AKI was 1.1 ± 0.4 mg/dl. The main contributing etiologic factors for AKI were ischemia in 871 (81%), nephrotoxic agents in 286 (28%), and sepsis in 579 (55%); AKI was multifactorial in 626 (59%) subjects. Three hundred fifty-five (32%) subjects were treated with intermittent hemodialysis and 746 (66%) with either continuous veno-venous hemodiafiltration or sustained low-efficiency hemodialysis as their initial dialytic modality; 16 (1.4%) patients never received RRT. The overall ICU, in-hospital, and 60-day mortality rates were 44%, 50%, and 53%, respectively. Table 1 outlines the basic demographic and clinical characteristics at time of initiation of RRT.
Demographics and clinical characteristics on day of initial renal replacement therapy
Predictive Risk Model
We identified 21 nonredundant factors independently associated with 60-day all-cause mortality (Table 2). The bootstrap estimated C-statistics of the logistic regression model was 0.85 (0.83 to 0.88) and was not statistically different from the original model (P-value = 0.32). The model also had good calibration, as shown in Figure 1 (Hosmer–Lemeshow goodness-of-fit P = 0.76). From an a priori selection of multiplicative interactions, only the interaction of mechanical ventilation with high fraction of inspired oxygen requirement was statistically significant (P = 0.001) and, hence, included in the model. Patient location before index hospitalization and ICU length-of-stay before RRT initiation were independently associated with mortality; however, these were excluded from the final model due to concerns about the influence of local practice patterns and trial-related selection bias that might compromise external validity. Using the coefficients from the fitted model, we developed a computer-based calculator (http://rcc.simpal.com/RCEval.cgi?RCID=0iaLo3) for automated calculation of the probability of mortality within 60 days.
Predictive risk model for 60-day mortality in critically ill patients with AKI requiring dialysisa
Calibration of risk model for 60-day mortality. Horizontal axis represents actual observed 60-day mortality, whereas vertical axis is the predicted probability using the multivariable logistic regression model. Broken line indicates the line of identity (ideal prediction model). Circles represent subgroups of our cohort.
Integer Risk Score
To facilitate manual calculation of 60-day mortality risk, we mapped the coefficients from the final risk model to integer scores for the 21 included parameters (Table 3). The risk score is calculated as the arithmetic sum of the points for each of these parameters (range 0 to 59 points). Figure 2 illustrates the relation between the total sum of score points for an individual subject and probability of 60-day all-cause mortality. The integer risk score had slightly lower discriminative capacity compared with the nonintegerized risk model, with an area under the ROC of 0.80 compared with 0.85 (Figure 3). An elevated oxygen requirement (fraction of inspired oxygen [FiO2] ≥0.60) was present in 488 (43%) of patients at initiation of RRT and contributed the greatest weight to the point score.
Integer risk score for prediction of 60-day mortality in critically ill patients with AKI requiring dialysisa
Predicted versus observed 60-day all-cause mortality per integer risk score. Black, predicted mortality; gray, observed mortality.
Area under receiver operating characteristic (ROC) curves for the new risk score (0.85), integer risk score (0.80), APACHE II (0.68), total SOFA (0.69), and Cleveland Clinic score (0.65). APACHE, Acute Physiology and Chronic Health Evaluation; SOFA, Sequential Organ Failure Assessment. Green, risk model; black, integer risk score; red, SOFA score; yellow, APACHE II score; blue: Cleveland Clinic score.
Comparison with Previous Predictive Models
Figure 3 shows the ROC curves for our new risk-prediction model, as well as for APACHE II, SOFA, and CCF scores. The area under the ROC curve was 0.68 (0.64 to 0.71) for the APACHE II, 0.69 (0.66 to 0.73) for the SOFA, and 0.65 (0.62 to 0.69) for the CCF scores, as compared with values of 0.85 (0.83 to 0.88) for the nonintegerized risk model and 0.80 (0.77 to 0.83) for the integerized risk score. Both areas under the ROC curve values (0.80 and 0.85) were statistically better compared with the three previous models (P < 0.001). Higher values of both generic severity illness scores and the CCF score were associated with 60-day mortality (Table 1). Of note, the APACHE II score is validated for use at ICU admission (7), whereas the SOFA score is validated for sequential use each ICU day (9), and the CCF score (10), as the current model, was developed for use at initiation of RRT in the ICU.
Discussion
Using a large multicenter cohort of critically ill patients with AKI who required RRT we have developed a multivariable risk model and a scoring system for predicting 60-day mortality. Our new risk model and scoring system demonstrated good discrimination and calibration, and outperformed previously published generic and AKI-specific scoring systems.
We examined the performance of two widely used generic severity-of-illness scoring systems: the APACHE II score, which integrates components of both acute and chronic organ dysfunction, and the SOFA score, which assesses the presence and severity of organ dysfunction. The application of generic severity scores in specialized cohorts, such as patients with AKI, has been problematic not only because of relatively poor discrimination but also because of suboptimal calibration (11,12,17). The use of external benchmarks—score systems developed in a population with different clinical characteristics from the target population—may underperform for several reasons. For instance, databases used to build general severity scores consist of patients with ICU length of stay less than 3 days, whereas our cohort's average predialysis ICU stay was almost 1 week (18). Another pitfall is the choice of in-hospital mortality as the primary outcome; hospital or long-term care facility transfer practices and short follow-up times may underestimate the strength of the association with mortality (19). These limitations may be particularly relevant in patients with AKI, where RRT and/or recovery span over weeks rather than days (2). Moreover, patients with severe illness typically have an eventful ICU course, where the relation between a risk factor determined at baseline and outcome is likely to be nonlinear, may change over time, and/or reverse direction. For example, sCr and urine output are established components of various risk scores, as it is in AKI-specific models—including our new score; however, the correlation is positive (or U shaped) in generic severity scores (7,9) but reverses direction in AKI-specific scores (10,12). Rather than being a marker for the extent of impaired kidney function, in a population where all subjects have severe AKI, a lower sCr might reflect underlying inflammation and/or malnutrition, hypervolemia with dilution, and/or other complications in the setting of multisystem organ failure (12,20,21).
The performance of AKI-specific scoring systems have been comparable to generic severity of illness scores (11,12). Many of the AKI-specific scores are obsolete, or at least outdated, due to the inevitable change in patient characteristics, disease epidemiology, medical technology, and standards of care over time. Although these limitations could be partially remedied via recalibration or structural model revisions (adding new variables) (22), we took advantage of a large, heterogeneous, multicenter patient cohort, namely, patents with AKI requiring RRT enrolled in the ATN study, to develop a new model. The large sample size ensures adequate patient numbers in high-mortality risk groups and better accuracy. In addition, the timing of the score application at initiation of RRT has previously been shown to improve predictive capacity compared with time of onset of AKI or renal consultation (12,20). We also favored the use of physiologic parameters in our model, rather than treatment- or diagnosis-based variables. The former are likely to be more objective and less prone to variations in practice patterns in time and space, hence potentially improving external validity.
The new score contains factors reflecting chronic health, such as chronological age, underlying respiratory disease, cardiovascular disease, and malignancy. The absence of baseline kidney function as an independent predictor of outcome has been noted in previous reports; however, the present cohort excluded most patients with moderate to severe chronic kidney disease, limiting our ability to assess the impact of impaired baseline renal function on outcomes (12,20,23). Respiratory complications, reflected by a requirement for a high FiO2 and the need for mechanical ventilation, had the largest combined point allocation. This rather large impact attributed to pulmonary organ failure is likely partially confounded by volume-driven morbidity, which is highly prevalent in subjects with renal failure. Consistent with previous literature, postoperative status (i.e., “surgical” rather than “medical” AKI) was associated with a more favorable prognosis, most likely because patients with the most severe chronic conditions are generally not considered as candidates for elective surgery (selection bias) (20,24). The same selection bias also applies to the subgroup of subjects who used immunosuppressive medications, which included nonrenal solid organ transplant recipients and connective tissue disease, in contrast to subjects with compromised immune system due to malignancy.
This study has several important strengths. The sample was derived from many sites across the United States and was diverse in terms of age, gender, race/ethnicity, and cause of AKI. We carefully incorporated objective clinical variables that were tracked as part of a randomized clinical trial; data entry was checked for accuracy. We bootstrapped the sample to derive parameter estimates that would be less likely to be overfit to the derivation sample. The prediction accuracy achieved by both the risk model and the scoring system suggests strong internal validity of our results. The derivation of the paper-based scoring system from the new score model was accompanied by a notable loss in discriminative capacity. The latter compromise in predictive accuracy could be circumvented by using the Web-based calculator for the risk model. Finally, we compared the current risk model and scoring system to two generic and one disease-specific predictive models and found superior results with the current model.
There are several important limitations to the study presented here. Although the data were generated from 27 centers, recruitment was predominantly in tertiary-care academic medical centers. The participation of non-VA centers, on the other hand, provided better representation of non-Caucasian and female subjects. While we would anticipate excellent model performance if applied to nontertiary care hospitals, the distribution of health conditions in tertiary care hospitals is different than in smaller hospitals or in hospitals to which patients are less likely referred. However, it is noteworthy that none of the variables selected for inclusion in the multivariable model were unique to tertiary care hospitals (e.g., recipients of bone marrow transplantation or extracorporeal membrane oxygenation). The new scoring system was developed specifically for critically ill patients with AKI requiring RRT, and external validity is further limited by the inclusion/exclusion criteria of the ATN study. Patients with moderate to advanced baseline chronic kidney disease were excluded. However, only a small percentage (7.6%) of subjects were excluded based on that particular reason, and 39% had estimated GFR of less than 60 ml/min per 1.73 m2. AKI due to causes other than ATN were excluded; therefore, the model would not be expected to perform well among patients with other causes of AKI (e.g., prerenal azotemia, urinary tract obstruction, acute glomerulonephritis, or tubulointerstitial nephritis). Certain data elements were excluded due to a relatively high percentage of patients with missing measurements; one such variable is serum lactate, which has been shown to be a robust predictor of mortality in the critically ill (11). The combination of very low serum bicarbonate and acidemia included in our score is likely to capture most of the variation related to elevated serum lactate concentrations. In the ATN study, 724 patients had one hemodialysis treatment or up to 24 hours of CRRT before randomization. The majority of data used for developing our score were collected before any prerandomization RRT; however, few parameters used in our score, such as serum bicarbonate and phosphate, and urine output, may have been collected 24 hours into prerandomization therapy. It is unlikely, however, that any parameter that is rapidly modified by RRT will influence mortality at 60 days. This limitation is also inherent in many generic severity-of-illness scores, including the APACHE II and III scores, whose points are based strongly on physiologic (and to a lesser extent laboratory) variables that can often be remedied within the first ICU day.
In summary, we developed a novel risk model and a scoring system that accurately predicts 60-day all-cause mortality in critically ill patients with AKI requiring RRT. The model can be used for risk adjustment and possibly to stratify prospective subjects in upcoming clinical trials of AKI requiring RRT. Risk-prediction models always demonstrate their best calibration in the population in which they are generated; further validation of the model in other cohorts of patients with AKI will be required before it can be widely applied to clinical practice.
Disclosures
None.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
Access to UpToDate on-line is available for additional clinical information at www.cjasn.org.
- Received March 27, 2011.
- Accepted July 2, 2011.
- Copyright © 2011 by the American Society of Nephrology
References
In this issue
Jump to section
More in this TOC Section
Original Articles
Acute Kidney Injury/Acute Renal Failure
Cited By...
- AKI Treated with Renal Replacement Therapy in Critically Ill Patients with COVID-19
- Sepsis associated acute kidney injury
- Infrequent Provision of Palliative Care to Patients with Dialysis-Requiring AKI
- A Decision-Making Algorithm for Initiation and Discontinuation of RRT in Severe AKI
- Modality of RRT and Recovery of Kidney Function after AKI in Patients Surviving to Hospital Discharge
- Outcomes After Post-Traumatic AKI Requiring RRT in United States Military Service Members
- Biomarker Enhanced Risk Prediction for Adverse Outcomes in Critically Ill Patients Receiving RRT
- Associations between Intensity of RRT, Inflammatory Mediators, and Outcomes
- Staying on Target with Continuous Dialysis