Visual Abstract
Abstract
Background and objectives Critically ill patients with worsening AKI are at high risk for poor outcomes. Predicting which patients will experience progression of AKI remains elusive. We sought to develop and validate a risk model for predicting severe AKI within 72 hours after intensive care unit admission.
Design, setting, participants, & measurements We applied least absolute shrinkage and selection operator regression methodology to two prospectively enrolled, critically ill cohorts of patients who met criteria for the systemic inflammatory response syndrome, enrolled within 24–48 hours after hospital admission. The risk models were derived and internally validated in 1075 patients and externally validated in 262 patients. Demographics and laboratory and plasma biomarkers of inflammation or endothelial dysfunction were used in the prediction models. Severe AKI was defined as Kidney Disease Improving Global Outcomes (KDIGO) stage 2 or 3.
Results Severe AKI developed in 62 (8%) patients in the derivation, 26 (8%) patients in the internal validation, and 15 (6%) patients in the external validation cohorts. In the derivation cohort, a three-variable model (age, cirrhosis, and soluble TNF receptor-1 concentrations [ACT]) had a c-statistic of 0.95 (95% confidence interval [95% CI], 0.91 to 0.97). The ACT model performed well in the internal (c-statistic, 0.90; 95% CI, 0.82 to 0.96) and external (c-statistic, 0.93; 95% CI, 0.89 to 0.97) validation cohorts. The ACT model had moderate positive predictive values (0.50–0.95) and high negative predictive values (0.94–0.95) for severe AKI in all three cohorts.
Conclusions ACT is a simple, robust model that could be applied to improve risk prognostication and better target clinical trial enrollment in critically ill patients with AKI.
- inflammation
- acute kidney injury
- prediction
- Critical Illness
- Disease Progression
- Systemic Inflammatory Response Syndrome
- Biomarkers
- Liver Cirrhosis
- Demography
- Intensive Care Units
- Receptors
- Tumor Necrosis Factor
- Cohort Studies
Introduction
AKI is defined as an abrupt decline in kidney function and is a frequent occurrence among critically ill patients in the intensive care unit (ICU) (1,2). The development of kidney injury is highly associated with short- and long-term complications, including prolonged hospitalization, need for kidney replacement therapy, CKD, ESKD, and death (3,4). Sepsis remains the most common cause of AKI in critically ill patients, accounting for approximately 50% of all cases (4). Despite the substantial public health burden attributable to AKI, no effective pharmacopreventive or pharmacotherapeutic options exist for AKI (5).
The lack of successful interventional trials in AKI may be due to delayed clinical recognition, as the rise in serum creatinine lags several days behind initial kidney injury (6). Another limitation has been the focus on identifying minor changes in serum creatinine, which may have led to the misclassification of patients without true AKI (7). To address these challenges, we developed a predictive model using clinical variables and plasma biomarkers collected on study enrollment and focused on the identification of patients who progress to the development of severe AKI, defined as Kidney Disease Improving Global Outcomes (KDIGO) stage 2 or 3 (8), after study enrollment.
We hypothesized that clinical variables and plasma biomarker concentrations representative of two main pathways, endothelial dysfunction (9) and inflammation/apoptosis (10), would predict development of severe AKI within 72 hours after study enrollment. To develop and validate the predictive model, we conducted a prospective, multicenter three-step cohort study in 1337 critically ill patients. The prediction model was developed in the derivation cohort followed by validation in the internal and external validation cohorts.
Materials and Methods
Derivation and Internal Validation Cohort
Patients were recruited between 2006 and 2010 from ICUs at Harborview Medical Center (Seattle, WA). Adult patients admitted to an ICU who met two or more criteria for the systemic inflammatory response syndrome (11) were prospectively enrolled (12). Patients were excluded if they had major trauma, intracranial hemorrhage, HIV, immunosuppression, a current diagnosis of cancer, ESKD, or kidney transplant before ICU admission, as previously described (13). Plasma specimens were obtained at the time of study enrollment, which occurred within 24 hours of ICU admission. From the overall cohort, 70% of patients were randomly included in the derivation cohort and the remaining 30% in the internal validation cohort.
External Validation Cohort
Patients were enrolled from ICUs at Massachusetts General Hospital (Boston, MA) between 1999 and 2010 (14). Adult patients were enrolled in the ICU who had a defined risk factor for acute respiratory distress syndrome, including sepsis, trauma, multiple transfusions, or aspiration. All patients met two or more systemic inflammatory response syndrome criteria. Exclusion criteria were similar to the derivation cohort and also included patients with diffuse alveolar hemorrhage, chronic lung diseases, and directive to withhold intubation. Plasma specimens were obtained after study enrollment, which occurred within 48 hours of ICU admission (15) (Supplemental Figure 1).
AKI Definition
AKI was defined as an increase in serum creatinine of ≥0.3 mg/dl or 50% from the maximum day 0 serum creatinine value. We chose maximum day 0 serum creatinine as the reference value to ensure that the prediction model was identifying patients with an increase in serum creatinine after study enrollment. Patients with AKI were then staged on the basis of KDIGO guidelines (5). Severe AKI (KDIGO stage 2 or 3) was defined by ≥100% increase in serum creatinine from reference or a serum creatinine ≥4 mg/dl after study enrollment. Urine output criteria were not used to define AKI. Incident dialysis was not used to define severe AKI because dialysis may have been initiated for causes other than worsening kidney function (i.e., toxins or urgent fluid removal). Also, worsening kidney function would precede dialysis, and so these patients would be captured with the severe AKI definition.
Biomarker Measurements
Plasma biomarker concentrations were measured in pathways implicated in the development of AKI, specifically endothelial dysfunction (angiopoietin-1, angiopoietin-2, and soluble vascular cell adhesion molecule) and inflammation/apoptosis (soluble TNF receptor-1 [sTNFR-1], IL-6, IL-8, and soluble Fas) (10,16–19). Biomarkers were measured using an immunoassay-based method (Meso Scale Discovery, Rockville, MD) (20). Additional methods can be found in the Supplemental Material and Supplemental Table 1.
Model Selection and Development
The initial model building used least absolute shrinkage and selection operator (LASSO) (21) methodology, a penalized regression approach that shrinks regression coefficients toward zero, resulting in sparse, parsimonious models. Models were selected from covariates that included age, sex, race/ethnicity, body mass index, diabetes mellitus, cirrhosis, CKD, source of ICU admission (medical/surgical), smoking status, and log10-transformed biomarkers, which were transformed before analysis because of a highly right skewed distribution.
With the goal of balancing simplicity with predictive power, we selected the most parsimonious LASSO model that had near minimal crossvalidated predictive error (within 1 SD of the minimum) (22). Discriminatory ability was quantified using the c-statistic (area under the receiver operator characteristic curve); goodness of fit was evaluated via Cessie–van Houwelingen chi-squared statistics (23).
Our primary outcome was development of severe AKI within 72 hours after study enrollment. Patients with severe AKI were compared with patients without AKI and KDIGO stage 1 AKI. The 72-hour mark was chosen to be proximal to the measurement of plasma biomarkers and at a time period during which recovery or progression of AKI has been strongly associated with hospital outcomes (24–26). Issues of competing risks were minimal because death without preceding development of severe AKI was uncommon in the first 72 hours after study enrollment. For example, in the derivation cohort, only four patients died without developing severe AKI before death. Performance of the optimal prediction model was compared with the Acute Physiology, Age, Chronic Health Evaluate III (APACHE III) score and the reference serum creatinine (maximum day 0 value) with bootstrap P values that tested the difference in area under the receiver operator characteristic curves for each comparison.
We identified a threshold score for the prediction model that gave a positive predictive value (PPV), the probability that patients with a score above the threshold will develop severe AKI, of 0.95 in the derivation cohort. We likewise identified a threshold score that delivered a negative predictive value (NPV), the probability that patients with a score below the threshold will not develop severe AKI, of 0.95 in the derivation cohort.
All statistical analyses were performed in R v3.4.0 (R Core Team 2015).
Results
Characteristics of the Cohorts
There were 1337 medical and surgical critically ill patients included in the study with 749 in the derivation, 326 in the internal validation, and 262 in the external validation cohorts (Table 1). Mean age for participants in all cohorts fell within the sixth or seventh decades of life and they were predominantly men. In all three cohorts, most patients met the definition of sepsis (approximately 60% in derivation and internal validation cohort versus 90% in the external validation cohort). The prevalence of any AKI was similar across all three cohorts. The development of severe AKI was 8% in the derivation cohort (n=62), 8% in the internal validation cohort (n=26), and 6% in the external validation cohort (n=15). Severe AKI was associated with worse clinical outcomes. For example, in the derivation cohort, the 28-day mortality rate in the population with severe AKI was 27% compared with 10% in patients without severe AKI (Supplemental Table 2).
Characteristics, biomarkers, and clinical outcomes of derivation, internal validation, and external validation cohorts
Prediction of Severe AKI in the Derivation Cohort
We developed the prediction model in the derivation cohort. A LASSO model including 13 different variables most accurately predicted development of severe AKI (c-statistic, 0.95; 95% confidence interval [95% CI], 0.92 to 0.98) (Table 2). However, because of the complexity of this model and the need to measure seven biomarkers that are not currently clinically available, we developed a model that balanced parsimony with predictive accuracy. Using the “1 SD” (22,27) criterion, which selects the most parsimonious model whose crossvalidated predictive accuracy is within 1 SD of the minimum, we produced the ACT model. This three-variable model of age, cirrhosis, and sTNFR-1 [ACT] had a c-statistic of 0.95 (95% CI, 0.91 to 0.97) (Table 2). Plasma sTNFR-1 concentrations mostly accounted for the effectiveness of the ACT model (Supplemental Tables 3 and 4). In the derivation cohort, the ACT model performed better than APACHE III (c-statistic, 0.70; 95% CI, 0.62 to 0.78; P<0.001) and reference serum creatinine (c-statistic, 0.91; 95% CI, 0.85 to 0.96; P=0.05) to predict the development of severe AKI (Figure 1). The predicted risk of severe AKI using the ACT model compared with the observed risk was well calibrated in the derivation cohort (P=0.43) (Supplemental Figure 2). In a sensitivity analysis, we included all patients with incident hemodialysis in the severe AKI group. The number of patients with severe AKI within 72 hours increased by seven patients and the ACT model continued to perform well to predict the development of severe AKI (c-statistic, 0.95; 95% CI, 0.91 to 0.98).
Model performance in derivation, internal validation, and external validation cohorts for severe AKI within 72 hours after study enrollment
ACT model produces optimal area under the receiver operator characteristics curve for severe AKI. Area under the receiver operator characteristics curve for severe AKI. The c-statistic for APACHE III was 0.70 (95% CI, 0.63 to 0.78) in the derivation, 0.63 (95% CI, 0.50 to 0.74) in the internal validation, and 0.71 (95% CI, 0.58 to 0.84) in the external validation cohorts. The c-statistic for reference serum creatinine (SCr) was 0.91 (95% CI, 0.85 to 0.96) in the derivation, 0.85 (95% CI, 0.72 to 0.95) in the internal validation, and 0.93 (95% CI, 0.82 to 0.99) in the external validation cohorts. The c-statistic for the ACT model was 0.95 (95% CI, 0.91 to 0.98) in the derivation, 0.90 (95% CI, 0.81 to 0.96) in the internal validation, and 0.93 (95% CI, 0.89 to 0.97) in the external validation cohorts (P<0.05 compared with APACHE III in all three tests).
Prediction Model Performance in the Internal and External Validation Cohorts
We validated the ability of the ACT model to predict severe AKI. The c-statistic was 0.90 (95% CI, 0.82 to 0.96) in the internal validation cohort and 0.93 (95% CI, 0.89 to 0.97) in the external validation cohort. The ACT model performed significantly better than APACHE III (internal validation: c-statistic, 0.63; 95% CI, 0.50 to 0.74; and external validation: c-statistic, 0.71; 95% CI, 0.58 to 0.84) (Table 2). However, the ACT model did not perform significantly better than reference serum creatinine in the internal validation (c-statistic, 0.85; 95% CI, 0.72 to 0.95; P=0.24) and the external validation (c-statistic, 0.93; 95% CI, 0.82 to 0.99; P=0.91) cohorts. The ACT model was well calibrated in the internal validation and external validation cohorts (P=0.80 and P=0.17, respectively) (Supplemental Figure 2).
Because the maximum serum creatinine value at study enrollment was used as the reference to determine AKI status, many patients already had elevations in serum creatinine at the time of study enrollment. Thus, we tested the performance of the ACT model in a population with de novo severe AKI, serum creatinine ≤1.5 mg/dl on study enrollment. In this population the ACT model performed well to predict severe AKI, ranging from a c-statistic of 0.79 (95% CI, 0.52 to 0.97) in the internal validation cohort to 0.97 (95% CI, 0.92 to 1.00) in the external validation cohort. The ACT model was significantly better than the reference serum creatinine in the internal validation (P=0.002) and external validation (P<0.001) (Table 3). In additional sensitivity analyses, ACT performed well in predicting the development of severe AKI up until 7 days (c-statistics of ≥0.88 in all three cohorts) (Supplemental Table 5), and in patients with sepsis (Supplemental Table 6) (11).
Model performance in derivation, internal validation, and external validation cohorts for severe AKI within 72 hours after study enrollment in the population with enrollment serum creatinine ≤1.5 mg/dl
Clinical Utility of the Prediction Model
We derived cut-off thresholds for ACT providing PPV and NPV for severe AKI within 72 hours that might provide clinically useful information. An optimal ACT model threshold score gave an NPV of 0.95 and a PPV of 0.90 in the deviation cohort. These thresholds were carried forward and the PPV decreased to 0.78 in the internal validation cohort and 0.50 in the external validation cohort, whereas the NPV remained very high in the internal validation (0.95) and external validation (0.96) cohorts (Table 4). Similar, PPV and NPV values were found for the outcome of severe AKI within 7 days (Supplemental Table 7). Finally, the performance of the ACT model to predict severe AKI within 72 hours was evaluated for a range of predicted risk thresholds to potentially inform use in future studies (Supplemental Figure 2, Supplemental Table 8).
Negative and positive predictive values for the ACT model and outcome of severe AKI within 72 hoursa
Discussion
In this prospective study of critically ill patients, we demonstrated that a three-variable model, the ACT model, predicted the development of severe AKI. The ACT model was mostly driven by plasma sTNFR-1 concentrations. We also demonstrated that a threshold score for ACT can achieve high NPV (>0.9) for the identification of patients who will develop severe AKI either within 72 hours or 1 week after study enrollment. In AKI, multiple studies have shown associations with sTNFR-1 concentrations and kidney injury in specific patient populations, such as acute respiratory distress syndrome (17), septic shock (28), or diabetic nephropathy (29). We have extended these observations to demonstrate that sTNFR-1 in combination with age and cirrhosis is strongly predictive of development of severe AKI in two independent diverse populations of critically ill patients.
TNFR-1 is a 55-kD membrane receptor that in the kidney is predominantly expressed in the glomerular and peritubular capillary endothelium (30,31). TNFR-1 is essential for TNFα signaling via NF-κB, a signaling cascade that results in both inflammation and apoptosis (32). TNFR-1 can be shed into a soluble form (sTNFR-1) either through cleavage from the cell surface or through release of exosome-like vesicles with the full-length TNFR-1 (33). Elevated plasma sTNFR-1 concentrations are interpreted as a marker of TNFα pathway activation (34). Activation of the TNFα pathway is a key mediator in tubular cell injury and death, and contributes to interstitial inflammation and tubular cell loss in AKI (35).
The predictive performance of the ACT model and specifically sTNFR-1 raises the question of whether sTNFR-1 is similar to creatinine, that is, filtered by the kidney and elevated levels are simply reflections of decreased filtration by the kidneys. However, unlike creatinine (113 daltons), sTNFR-1 (55,000 daltons) is a large molecule that is unlikely to be regularly filtered at the glomerulus. Moreover, in studies in patients without kidney dysfunction, sTNFR-1 concentrations have been reported to predict subsequent long-term kidney function decline (29,36). Finally, sTNFR-1 concentrations in the urine increase rather than decrease with worsening kidney function, potentially suggesting that urinary sTNFR-1 is being locally released by the kidney (31). Thus, elevations in plasma sTNFR-1 concentrations are unlikely to be due to differences in filtration by the kidneys and may be involved in the pathophysiology of the development of severe AKI in critically ill patients.
Previous studies have examined plasma and urinary biomarkers to predict the development of severe AKI. In a two-stage (discovery and validation) study, Kashani et al. (37) determined that the combination of two urinary cell cycle arrest markers had a c-statistic of 0.80 for the prediction of stage 2 or 3 AKI within 12 hours of study enrollment. Parr et al. (38) extended the time period of observation and found that urinary L-type fatty acid binding protein added to a clinical model had a c-statistic of 0.82 for the prediction of severe AKI, dialysis, or death within 7 days. Alternative biomarker studies have shown variable potential to predict the development of severe AKI (39,40). This study adds to this literature and suggests that the ACT model may be another promising predictive tool for severe AKI.
To ensure that the predictive model was identifying subsequent severe AKI, the maximum serum creatinine at study enrollment was used as the reference value. In our cohorts, many patients already had an elevated serum creatinine on admission, and thus, in these patients, the ACT model identified progression of AKI. We performed a sensitivity analysis in patients with a serum creatinine ≤1.5 mg/dl at study enrollment to determine the performance of the ACT model in the prediction of incident severe AKI. In this group of patients, the ACT model continued to perform well to predict development of severe AKI and was significantly better than reference creatinine. This subgroup analysis suggests the independence of sTNFR-1, age, and cirrhosis versus serum creatinine to predict worsening kidney function, and indicates that the ACT model may be effective early after ICU admission during times of clinical uncertainty.
The clinical utility of the ACT model may lie in the high NPV. For example, if clinical trialists used a predicted probability threshold of 20%, then 88% of all patients screened with the ACT model could be excluded from study enrollment (Figure 2). Of these patients excluded, only 2% would develop severe AKI. Thus, the ACT model would be effective to exclude patients with low risk to develop severe AKI. In contrast, the moderate PPV (58%) would limit the identification of patients who are at high risk to develop severe AKI. Future application of the ACT model would need to balance the high NPV with the knowledge that only half of the patients identified would eventually develop severe AKI.
Performance of three variable prediction model (ACT) varies based on predicted risk of severe AKI. (A) ACT model and the model score threshold equation to determine predicted risk of severe AKI on the basis of age, cirrhosis, and sTNFR-1 concentrations. (B) Predicted risk of severe AKI for a person aged 55 years, with or without cirrhosis and increasing concentrations of sTNFR-1. (C) Performance of the ACT model to predict severe AKI within 72 hours for a range of predicted values. The ACT model score threshold is converted to the probability scale to determine a predicted risk of severe AKI. Excluded is the percentage of patients that had a model score below the respective ACT model score threshold. If this prediction model was used to improve the prognostic enrichment of a clinical trial, then these patients would be “excluded” as they are low risk to develop severe AKI. The positive predictive value is the probability of those with a predicted probability above the model score threshold truly having severe AKI. Data derived from the derivation cohort (n=749).
Our study has several strengths. First, the ACT model effectively discriminated low risk patients for severe AKI in all three cohorts. The high NPV of the ACT model would allow clinicians to confidently determine that patients with a low ACT score likely have a low risk of progressing to severe AKI. Second, our prediction model was effective in two large, well phenotyped, heterogeneous ICU populations demonstrating generalizability of our findings. Third, plasma samples were prospectively collected under standardized conditions in consecutive patients admitted to the ICU at two different medical centers. Fourth, unlike many previous biomarker-based prediction models, we focused on measuring circulating markers believed to be involved in the development of AKI rather than markers specific to kidney tubular damage.
The study also has limitations. First, urine output was not used in the determination of AKI status and severity. Although this may have reduced the number of patients identified with severe AKI, it has been suggested that current consensus criteria using urine output to diagnose AKI may be overly sensitive (41). Second, the overall number of cases of severe AKI in the cohorts was low. However, even with a low incidence of severe AKI, the ACT model was able to accurately predict the development of severe AKI in two independent ICU cohorts. Third, the lack of an outpatient baseline serum creatinine may have led to under recognition of patients with prevalent AKI on ICU admission. Fourth, the PPV of the ACT model was lower in the external validation cohort compared with the other cohorts. The higher severity of illness in the external validation cohort may have led to the ACT model threshold having a lower sensitivity to identify severe AKI. Fifth, the analyses were completed in a population with sepsis-associated AKI. Additional studies are warranted to validate the ACT model in populations with nonsepsis AKI risk factors, such as cardiac surgery or nephrotoxin exposure.
In summary, ACT is a novel, promising prediction model for the identification of patients at increased risk to develop severe AKI in the ICU.
Disclosures
None.
Supplemental Materials
This article contains the following supplemental material online at http://cjasn.asnjournals.org/lookup/suppl/doi:10.2215/CJN.04100318/-/DCSupplemental.
Supplemental Table 1. Number of patients above or below the limit of detection for each biomarker.
Supplemental Table 2. Model performance of individual variables in derivation, internal validation and external validation cohorts for severe AKI within 72 hours after study enrollment.
Supplemental Table 3. Univariate performance of each biomarker in the derivation cohort to predict severe AKI within 72 hours after study enrollment.
Supplemental Table 4. Model performance in derivation, internal validation and external validation cohorts for severe AKI within 7 days after study enrollment.
Supplemental Table 5. Model performance in derivation, internal validation and external validation cohorts for severe AKI within 72 hours after study enrollment in patients with sepsis.
Supplemental Table 6. Negative and positive predictive values for the ACT model and severe AKI within 7 days after study enrollment.
Supplemental Table 7. Risk of severe AKI within 72 hours after study enrollment (ACT model, derivation cohort).
Supplemental Table 8. Risk of 28-day mortality with severe AKI within 72 hours after study enrollment in the derivation cohort.
Supplemental Figure 1. Patient flow diagram for severe AKI prediction model.
Supplemental Figure 2. Calibration plots for the ACT model and APACHE III scores.
Supplemental Figure 3. Distribution of sTNFR-1 plasma concentrations by severe AKI status.
Acknowledgments
Study concept and design: P.K.B., L.R.Z., R.K., C.M., W.C.L., J.H., and M.M.W. Acquisition, analysis, or interpretation of data: all authors. Drafting of the manuscript: P.K.B. Critical revision of the manuscript for important intellectual content: all authors. Statistical analysis: P.K.B., L.R.Z., and R.K. Study supervision: D.C.C., J.H., W.C.L., and M.M.W.
This work was supported by grants from the National Heart, Lung, and Blood Institute (R01HL060710); the National Institute of Diabetes, Digestive and Kidney Diseases (F32DK112532); University of Washington Department of Medicine; and an unrestricted grant from the Northwest Kidney Centers to the Kidney Research Institute.
The funding sources had no role in design and conduct of the study, collection, management, analysis, and interpretation of the data, and preparation, review, or approval of the manuscript.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
See related editorial, “Prediction Models for AKI: Will They Result in Improved Outcomes for AKI?,” on pages 488–490.
- Received March 30, 2018.
- Accepted January 28, 2019.
- Copyright © 2019 by the American Society of Nephrology
References
In this issue
