Abstract
Background and objectives Billing codes are frequently used to identify AKI events in epidemiologic research. The goals of this study were to validate billing code–identified AKI against the current AKI consensus definition and to ascertain whether sensitivity and specificity vary by patient characteristic or over time.
Design, setting, participants, & measurements The study population included 10,056 Atherosclerosis Risk in Communities study participants hospitalized between 1996 and 2008. Billing code–identified AKI was compared with the 2012 Kidney Disease Improving Global Outcomes (KDIGO) creatinine-based criteria (AKIcr) and an approximation of the 2012 KDIGO creatinine- and urine output–based criteria (AKIcr_uop) in a subset with available outpatient data. Sensitivity and specificity of billing code–identified AKI were evaluated over time and according to patient age, race, sex, diabetes status, and CKD status in 546 charts selected for review, with estimates adjusted for sampling technique.
Results A total of 34,179 hospitalizations were identified; 1353 had a billing code for AKI. The sensitivity of billing code–identified AKI was 17.2% (95% confidence interval [95% CI], 13.2% to 21.2%) compared with AKIcr (n=1970 hospitalizations) and 11.7% (95% CI, 8.8% to 14.5%) compared with AKIcr_uop (n=1839 hospitalizations). Specificity was >98% in both cases. Sensitivity was significantly higher in the more recent time period (2002–2008) and among participants aged 65 years and older. Billing code–identified AKI captured a more severe spectrum of disease than did AKIcr and AKIcr_uop, with a larger proportion of patients with stage 3 AKI (34.9%, 19.7%, and 11.5%, respectively) and higher in-hospital mortality (41.2%, 18.7%, and 12.8%, respectively).
Conclusions The use of billing codes to identify AKI has low sensitivity compared with the current KDIGO consensus definition, especially when the urine output criterion is included, and results in the identification of a more severe phenotype. Epidemiologic studies using billing codes may benefit from a high specificity, but the variation in sensitivity may result in bias, particularly when trends over time are the outcome of interest.
Introduction
AKI is now recognized as a major public health concern; it is common, costly, and associated with exceedingly poor outcomes (1–4). Concomitant with the increase in physician awareness, the consensus definition of AKI has expanded, and milder cases (e.g., a 0.3-mg/dl increase in serum creatinine) are now considered clinically significant (5,6). AKI is often identified by hospital billing codes in epidemiologic studies (7–11), but the extent to which billing code practice has followed the consensus definition is unknown. Inconsistencies in billing codes may lead to biased estimates in clinical studies, particularly when accuracy varies with variables of interest, such as era.
Unfortunately, there is no gold standard method by which to evaluate AKI identification. No standardized procedure for event adjudication exists. Some epidemiologic studies use chart adjudication, some use automated abstraction of electronic medical records, and most do not adjudicate. The current Kidney Disease Improving Global Outcomes (KDIGO) AKI definition remains controversial, particularly with respect to the inclusion of KDIGO stage 1 (an increase in serum creatinine of 0.3 mg/dl over 48 hours, or a 50% increase in creatinine over 7 days) and the urine output criterion (urine output<0.5 ml/min per kg for >6 hours) (5,6,12). Few studies have collected urine output data; hence, the evidence supporting its incorporation in the AKI definition is substantially weaker than that for serum creatinine (6,12).
Billing code–identified AKI has poor sensitivity compared with older creatinine-based definitions, but this has not been updated with the KDIGO definition of AKI (13). Acknowledging the uncertainty regarding urine output and small changes in serum creatinine, we analyzed the performance of AKI billing codes compared with two KDIGO-based AKI definitions: AKIcr, based on the KDIGO creatinine criteria only, AKIcr_uop, based on both the KDIGO creatinine criteria and an approximation of the urine output–based criteria, and by AKI stage. In addition, we evaluated time period, patient age, race, sex, and comorbid conditions as potential modifiers of billing code sensitivity and specificity. Finally, we evaluated the case mix captured by AKI billing code, AKIcr, and AKIcr_uop, and their associations with subsequent in-hospital and 30-day mortality.
Materials and Methods
Study Population
The Atherosclerosis Risk in Communities (ARIC) study is an ongoing prospective cohort designed to study risk factors for cardiovascular disease and atherosclerosis (14). Original enrollment (1987–1989) included 15,792 individuals aged 45–64 years and recruited from four United States communities: Washington County, Maryland; Minneapolis, Minnesota; Jackson, Mississippi; and Forsyth County, North Carolina. Because of the difficulty in obtaining hospital charts from more historical periods, only participants with a hospitalization between 1996 and 2008 were included (n=10,056).
Data Source: Hospitalization Billing Codes
Vital status and intervening hospitalizations for ARIC participants are determined annually by telephone follow-up (response rate, 92% in year 20) as well as by active surveillance of community hospital discharge lists, local newspaper obituaries, state death lists, and death certificates from the Department of Vital Statistics. For each hospitalization, trained abstractors record all (or, if more than 26 codes were used, at least 26) discrete International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge billing codes, including both diagnostic and procedure codes; for deaths, abstractors additionally record all ICD-10-CM (ICD, Tenth Revision, CM) codes from the death certificate. For the current study, 34,179 hospitalized events occurring between study visit 4 (February 1, 1996, to January 30, 1999) and December 31, 2008, were evaluated.
Data Source: Electronic Medical Record Creatinine and Urine Output Data
Serum creatinine and urine output data were available in an electronic medical record for 3710 hospitalizations occurring between 2002 and 2008 in a single health system in Washington County, Maryland. Advantages of this system include a single laboratory measuring both inpatient and outpatient serum creatinine and an inpatient nursing protocol mandating routine capture of fluid intake and output. The majority of hospitalizations (67% over the study period) among ARIC study participants recruited from Washington County occurred in this health system. Hospitalizations for participants who began long-term dialysis (determined by linkage with the US Renal Data System [USRDS]) >30 days before hospital discharge date were not considered AKI, irrespective of electronic medical record serum creatinine values and urine output. Kidney transplant recipients were included. In total, 1970 hospitalizations had a preceding outpatient creatinine in the year before admission, and 1839 of these had available urine output data.
Data Source: Hospital Chart Review
To evaluate kidney events occurring in all study centers and across a greater time span, 665 hospital charts were requested. Charts were selected using a random-number generator within 10 strata of ICD-9-CM codes (Supplemental Table 1), oversampling for records containing codes for kidney disease and, among those with codes for dialysis, for a participant’s first dialysis hospitalization. For each requested hospitalization, the admission history and physical, nephrology consult (if applicable), discharge summary, and inpatient laboratory measures were abstracted, deidentified, and transferred to a central location for adjudication. Urine output from these hospitalizations was not available.
Kidney Event Definitions
For billing code data, an ICD-9-CM code of 584.x or an ICD-10-CM code of N17.x in any of the abstracted codes was considered AKI (Table 1). Hospitalizations with an AKI code and a billing code for dialysis (ICD-9-CM codes: V45.1, V56, 39.95, 54.98; ICD-10-CM codes: Z99.2, Z49, Z45.2) were considered to be AKI requiring renal replacement therapy (AKI-RRT, a subset of stage 3). For electronic medical record data, AKI was defined by the KDIGO creatinine-based criteria (AKIcr) and KDIGO combined creatinine- and urine output–based criteria (AKIcr_uop), with baseline creatinine defined as the average outpatient creatinine measured 10–365 days before admission (15). The KDIGO urine output criteria were approximated conservatively as <20 ml/hr×24 hours because admission weights were not available and the interval of urine collection varied. For hospital chart review, AKI was adjudicated according to the KDIGO creatinine-based criteria, with baseline creatinine defined as the prehospitalization creatinine, determined from the following (in order of preference): nephrology consult, admission or discharge note, or lowest inpatient serum creatinine (if judged adequate by adjudicating physician).
Definitions of AKI by data source
Chart Adjudication Procedure
Of the 665 charts requested, 546 (82%) were located. Charts were more likely to be available in the later years (71%, 87%, and 92% from 1996–2000, 2001–2004, and 2005–2008, respectively; P<0.001). Charts were adjudicated for the presence and stage of AKI by two independent physician reviewers (M.E.G./B.M. or M.E.G./S.W.) blinded to billing code. Initial agreement between reviewers was 91% (κ=0.82; P<0.001). After clarification and resolution of errors, consensus was reached on all charts.
Agreement between Gold Standard Sources
The agreement between chart-adjudicated AKIcr and electronic medical record AKIcr was 84% (κ=0.67; P<0.001) for the 36 charts with data from both sources. Rereview of discordant records revealed inconsistencies in the adjudicated baseline creatinine (preadmission creatinine was noted only through documentation from treating physician) and the baseline creatinine from the electronic medical record.
Statistical Analyses
Billing code–identified AKI was compared with AKIcr and AKIcr_uop. For comparison with chart review, analyses were adjusted for sampling technique, weighting by the inverse probability of being selected within the ICD-9-CM code strata (e.g., total number of hospitalizations within each stratum divided by number of charts reviewed in each stratum) (Table 2). For comparison with electronic medical record data, no weighting was performed because we did not sample these data. SEMs were calculated using a clustered sandwich estimator to account for potential within-participant correlation; validation was performed at a hospitalization level. Because the incidence of AKI is often compared by patient characteristics and over time, the sensitivity and specificity of AKI billing codes compared with AKIcr within the chart review sample were evaluated by the following factors: era (1996–2002, 2003–2008), participant age (<65, ≥65 years), sex, race (white, black), baseline diabetes (yes, no; assessed at last attended clinic visit), and baseline CKD (eGFR<60 ml/min per 1.73 m2 using the CKD-Epidemiology Collaboration creatinine equation [16]). Interactions were tested by logistic regression of AKI defined by billing code on the factor of interest among hospitalizations with (sensitivity) and without (specificity) AKIcr; P<0.05 was considered to represent a statistically significant difference.
Chart review: International Classification of Diseases, Ninth Revision, Clinical Modification nonoverlapping strata and weights within the Atherosclerosis Risk in Communities study population
To evaluate the implications of using different definitions of AKI, the proportions experiencing in-hospital and 30-day mortality following billing code–identified AKI, AKIcr, and AKIcr_uop were evaluated in the electronic medical record data sample. The odds of in-hospital and 30-day mortality associated with each AKI definition were compared using seemingly unrelated estimation (17). All analyses were performed using Stata/SE software, version 11.2 (Stata Corp., College Station, TX).
Results
Prevalence of AKI Billing Codes Overall
There were 34,179 hospitalizations among 10,056 ARIC study participants during the study period; 1354 (4.0%) hospitalizations among 980 participants had a billing code for AKI, and 117 (0.3%) hospitalizations among 108 participants had a billing code for both AKI and dialysis (AKI-RRT). Twenty billing code–identified AKI hospitalizations (10 of which were AKI-RRT) occurred more than 30 days after the USRDS-supplied ESRD date. Overall, the number of billing codes recorded increased over time, from a median of 6 in 1996 to 8 in 2008 (maximum in all years, 26).
Validity of AKI Billing Codes Using Chart Review
Nearly half (45.4%) of the 546 reviewed hospitalizations were from the earlier study era (1996–2002), 83.0% were in participants aged 65 and older, 35.0% were in African Americans, and 48.0% were in women. Physician adjudication classified 233 cases as AKIcr, of which 145 were stage 2–3 by KDIGO criteria (113 were AKI-RRT). Of the AKI-RRT events, 72.6% were among persons with CKD stage 3 or worse, and 69.0% of patients did not recover. In weighted analysis using adjudicated AKIcr as the gold standard, the sensitivity of ICD-9-CM codes for AKI was 17.4%; specificity was 99.6% (Table 3, first column). Positive and negative predictive values were 92.0% and 81.8%, respectively. ICD-9-CM sensitivity improved to 40.3% if only stage 2–3 AKI was considered (negative predictive value, 99.9%). Sensitivity and specificity of AKI-RRT codes were 36.5% and 99.9%, respectively.
Validation of AKI billing codes compared with AKI based on Kidney Disease Improving Global Outcomes (KDIGO) creatinine criteria and KDIGO combined creatinine and urine output criteria
Validity of AKI Billing Codes in Electronic Medical Record Data
There were 1970 hospitalizations with a preceding outpatient serum creatinine value in the electronic medical record. Slightly more hospitalizations occurred in the later years (9.3% during 2002 versus 16.9% during 2008). Given the more recent time period, most hospitalized patients (92.3%) were age 65 years or older; 48.9% were women. Because hospitalizations were solely in Washington County, Maryland, <1% of the patients were black. There were 361 AKI events (18.3%) by the KDIGO creatinine criteria (AKIcr) and 609 events (of 1839 with available urine output [33.1%]) by the KDIGO combined creatinine and urine output criteria (AKIcr_uop). Sensitivity of AKI billing codes was 17.2% compared with AKIcr and 11.7% compared with AKIcr_uop; specificity was 98.5% and 98.9%, respectively (Table 3, second and third columns).
Variation in Validity over Time and by Patient Factors
The validity of AKI billing codes compared with adjudicated chart review was assessed over time and by patient sex, age, race, diabetes, and CKD status (Figure 1). Sensitivity significantly improved between 1996–2002 and 2002–2008 (from 9.7% to 24.4%; P=0.01). The sensitivity of AKI billing codes was 7.3% in persons <65 years of age and 20.8% in those age≥65 years (P=0.03); this persisted after adjustment for era. Race was not associated with billing code sensitivity (18.0% among whites versus 15.7% among blacks; P=0.7). Specificity differed only by diabetes status (99.8% among patients without diabetes versus 98.8% among patients with diabetes; P=0.03). The sensitivity of AKI-RRT codes did not appear to vary over time, age, diabetes, or CKD status, but there was a difference in sensitivity by race (27.5% among whites versus 53.1% among blacks; P=0.02).
The sensitivity and specificity of AKI billing codes vary according to patient characteristics and hospitalization year. Validated against chart review. Figure depicts point estimates and 95% confidence intervals. *P<0.05.
Comparison of AKI Stage and Adverse Outcomes by Method of Identification
Among patients with AKI by billing code, 34.9% had stage 3 by the KDIGO creatinine criteria (Table 4). Far fewer had stage 3 among those with AKIcr (19.7%) and AKIcr_uop (11.5%). Similarly, in-hospital mortality was higher in billing-code AKI than AKIcr and AKIcr_uop (41.2%, 18.7%, and 12.8%, respectively) (Table 5). Trends in 30-day post-AKI mortality followed the same pattern (51.8%, 23.9%, and 18.6%), as did the odds ratios for in-hospital and 30-day post-discharge mortality (compared with non-AKI hospitalizations). In subgroup analysis, this difference was greater in the earlier era than the later era (odds ratio for in-hospital mortality for AKI billing codes, AKIcr, and AKIcr_uop, respectively, 30.2, 7.6, and 8.5 for 2002–2004 and 12.3, 9.3, and 5.9 for 2005–2008). There was no significant difference between odds ratios for AKIcr and AKIcr_uop. The cross-tabulation between AKI creatinine and urine output criteria showed the highest mortality among patients meeting both criteria, followed by those meeting creatinine criteria and then those with isolated oliguria (Supplemental Table 2).
AKI events identified by billing codes, serum creatinine, and combined creatinine and urine output criteria: stage (by creatinine criteria) and presence of oliguria
AKI events identified by billing codes, serum creatinine, and combined creatinine and urine output criteria: in-hospital and 30-day mortality
Discussion
This community-based study critically assessed current epidemiologic methods for identifying AKI. The sensitivity of AKI billing codes was limited (<20%), particularly with use of the combined criteria of change in creatinine or urine output to define AKI. Sensitivity estimates were slightly higher in more severe disease; yet even among patients with stage 2–3 AKI (>100% increase in serum creatinine), billing codes captured only 40% of cases. Billing codes indicating dialysis-requiring AKI, generally thought to be quite accurate (13), demonstrated high specificity (>99%) but low sensitivity (37%). Also notable was the variation in billing code sensitivity: by era and patient age for AKI in general and by race for dialysis-requiring AKI. This implies that studies investigating trends in AKI over time (or trends in AKI-RRT by race) may be biased.
Many studies—including some using serum creatinine to identify AKI—have reported that AKI incidence has increased over time (1,7,9,11,18–20), and several note that trends in AKI incidence might be affected by changes in billing practice. For example, the 2012 USRDS Annual Data Report points to discrepant trends in billing codes for AKI and AKI-RRT (the former increasing, the latter decreasing) as evidence for “code creep,” assuming that the threshold for dialysis in AKI has not changed (18). We confirm that billing practice has changed, with milder cases identified in recent years, a finding perhaps driven in part by the increased numbers of billing codes used. Applying our estimates of billing code validity to USRDS rates of billing code–identified AKI (0.2% in 1995 and 1.6% in 2007, or a 700% increase) would suggest an underestimation of rates and an overestimation of the relative increase (from 1.0% in 1995 to 4.0% in 2007, or a 300% increase) (18).
Our study expands upon previous validation studies (7,13,21–25). Waikar and colleagues reported that billing code–identified AKI had low sensitivity (35.4%) but high specificity (97.7%) compared with a 100% creatinine increase in inpatient creatinine (13). Using the current creatinine-based definition (which includes a 50% or 0.3-mg/dl increase) (6), an improved estimate of baseline creatinine, and a more geographically diverse population, we found that sensitivity was lower (17%), a difference attributable to the expanded AKI definition, since sensitivity compared with stage 2–3 AKI (>100% increase in serum creatinine) was similar (13). The sensitivity of AKI-RRT codes (36.5%) was much lower than the previous estimate (90.4%) (13), which may be due to differences in study design, such as sampling technique, or the uncertainty regarding AKI classification in patients with advanced CKD. Finally, we observed that billing codes capture a different case mix than that of KDIGO, with a higher proportion of stage 3 disease and a stronger association with short-term mortality, although these differences have attenuated over time. Trends in long-term outcomes after billing code–identified AKI should be interpreted cautiously.
To our knowledge, our study is the first to validate AKI billing codes against a definition incorporating both serum creatinine and urine output data (AKIcr_uop). Compared with AKIcr_uop, AKI billing codes performed poorly, but whether this stems from flawed billing practice or a flawed gold standard is uncertain. There remains debate over whether to include isolated oliguria in the AKI definition; this may translate to poor billing code capture. On the other hand, our data span the entire inpatient population and are subject to all the inaccuracies inherent in clinical data collection. We found that isolated oliguria was relatively common—even at our more conservative approximation of the KDIGO definition—similar to a study in intensive care patients (a setting where urine output measurement is thought more accurate) (26). In contrast, another intensive care study found very few cases of oliguria (defined as urine output<400 ml/d) without a concomitant rise in serum creatinine (27). The implication of isolated oliguria remains uncertain: in the intensive care study, it was associated with a higher rate of death than AKI without oliguria; in our study, the rate was lower (26). This difference may reflect error in output measurement outside the intensive care unit, or it may reflect differences in underlying rates of mortality, whereby patients in intensive care have a higher mortality before an observed change in serum creatinine. Similar to a previous study (28), however, we found that those with AKI by both creatinine and urine output criteria experienced the highest rate of in-hospital mortality, suggesting potentially useful risk implications.
Previous validation studies have not routinely adjusted for sampling frame. In our chart review study, unweighted estimates would grossly overestimate sensitivity and underestimate specificity, given the oversampling for hospitalizations with kidney events. The use of inverse probability weighting allows for generalization to the entire ARIC population. However, insofar as billing practices in the communities studied are dissimilar to those in a population of interest, our results may not be generalizable. Our electronic medical record data were also from a single center, although validation measures using these data were very similar to those obtained using chart review, which encompassed multiple hospitals and all four ARIC communities.
Our study highlights the uncertainty regarding the AKI “gold standard.” Event definitions and provider practice have evolved (5,6), and there are differences between chart review and electronic medical record abstraction. Even standardized chart review is highly dependent on provider recognition, documentation, and data ascertainment. When a diagnosis hinges on small changes in serum creatinine, the estimation of “baseline” creatinine is critical—a process, at least in the clinical setting, that is somewhat subjective (15,29–31). Using admission serum creatinine (sometimes the best available approach for a clinician) significantly underestimates AKI incidence (31), whereas using remote outpatient values may misclassify CKD progression as AKI. On the other hand, electronic medical records may have trouble distinguishing AKI events from laboratory errors or the receipt of dialysis; in addition, not all systems integrate inpatient and outpatient laboratory measures.
In summary, using a community-based sample of middle-to-older aged adults, we describe a validation study of AKI billing codes. Sensitivity was low, particularly when urine output was included in the gold standard definition, but it increased in recent years. Insofar as specificity is most important in the mitigation of study bias (32), the use of billing codes for studies of associations may be reasonable. On the other hand, the significant variability in sensitivity over time and by participant age warrants caution in other contexts.
Disclosures
M.E.G., B.M., S.W., and S.H.B. have no relevant financial relationships to disclose. Dr. Coresh has consulted for Amgen and Merck and has an investigator-initiated grant from Amgen. S.S.W. reports serving as a consultant to CVS Caremark, BioTrends Research Group, Harvard Clinical Research Institute, and Takeda; providing expert testimony for GE Heathcare and Salix; and receiving investigator-initiated grants from Otsuka, Merck, Genzyme, and Satellite Healthcare. The authors have no conflicts to report pertaining to the manuscript content.
Some of the data reported here have been supplied by the USRDS. The interpretation and reporting of these data are the responsibility of the authors and in no way should be seen as an official policy or interpretation of the US Government.
Acknowledgments
The authors thank the staff and participants of the ARIC study for their important contributions.
M.E.G. is supported by the National Institutes of Health National Institute of Diabetes and Digestive and Kidney Diseases grant K08DK092287. The ARIC study is carried out as a collaborative study supported by National Heart, Lung, and Blood Institute contracts (HHSN268201100005C, HHSN268201100006C, HHSN268201100007C, HHSN268201100008C, HHSN268201100009C, HHSN268201100010C, HHSN268201100011C, and HHSN268201100012C), as well as R01 DK076770.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
This article contains supplemental material online at http://cjasn.asnjournals.org/lookup/suppl/doi:10.2215/CJN.07650713/-/DCSupplemental.
- Received July 19, 2013.
- Accepted November 19, 2013.
- Copyright © 2014 by the American Society of Nephrology
References
In this issue
Jump to section
More in this TOC Section
Original Articles
Cited By...
- Renin-Angiotensin System Blockade after Acute Kidney Injury (AKI) and Risk of Recurrent AKI
- Acute Kidney Injury in Children Hospitalized With Diarrheal Illness in the United States
- Secular Trends in Incidence, Modality and Mortality with Dialysis Receiving AKI in Children in Ontario: A Population-Based Cohort Study
- Impact of AKI on Urinary Protein Excretion: Analysis of Two Prospective Cohorts
- Acute kidney injury after hip or knee replacement: Can we lower the risk?
- Sodium glucose cotransporter 2 inhibitors and risk of serious adverse events: nationwide register based cohort study
- Causes of Death after a Hospitalization with AKI
- Infrequent Provision of Palliative Care to Patients with Dialysis-Requiring AKI
- Epidemiology and Natural History of the Cardiorenal Syndromes in a Cohort with Echocardiography
- Prescription of renin-angiotensin system blockers and risk of acute kidney injury: a population-based cohort study
- National Trends and Impact of Acute Kidney Injury Requiring Hemodialysis in Hospitalizations With Atrial Fibrillation
- Race, APOL1 Risk, and eGFR Decline in the General Population
- Incidence and In-Hospital Mortality of Acute Kidney Injury (AKI) and Dialysis-Requiring AKI (AKI-D) After Cardiac Catheterization in the National Inpatient Sample
- Temporal Trends in AKI: Insights from Big Data
- Acute Kidney Injury Predicts Major Adverse Outcomes in Diabetes: Synergic Impact With Low Glomerular Filtration Rate and Albuminuria
- Plasma Urate and Risk of a Hospital Stay with AKI: The Atherosclerosis Risk in Communities Study