Original ArticleTransplantation

Machine Learning–Derived Integer-Based Score and Prediction of Tertiary Hyperparathyroidism among Kidney Transplant Recipients

An Integer-Based Score to Predict Tertiary Hyperparathyroidism

Namki Hong, Juhan Lee, Hyung Woo Kim, Jong Ju Jeong, Kyu Ha Huh and Yumie Rhee
CJASN July 2022, 17 (7) 1026-1035; DOI: https://doi.org/10.2215/CJN.15921221

Abstract

Background and objectives Tertiary hyperparathyroidism in kidney allograft recipients is associated with bone loss, allograft dysfunction, and cardiovascular mortality. Accurate pretransplant risk prediction of tertiary hyperparathyroidism may support individualized treatment decisions. We aimed to develop an integer score system that predicts the risk of tertiary hyperparathyroidism using machine learning algorithms.

Design, setting, participants, & measurements We used two separate cohorts: a derivation cohort with the data of kidney allograft recipients (n=669) who underwent kidney transplantation at Severance Hospital, Seoul, Korea between January 2009 and December 2015 and a multicenter registry dataset (the Korean Cohort Study for Outcome in Patients with Kidney Transplantation) as an external validation cohort (n=542). Tertiary hyperparathyroidism was defined as post-transplant parathyroidectomy. The derivation cohort was split into 75% training set (n=501) and 25% holdout test set (n=168) to develop prediction models and integer-based score.

Results Tertiary hyperparathyroidism requiring parathyroidectomy occurred in 5% and 2% of the derivation and validation cohorts, respectively. Three top predictors (dialysis duration, pretransplant intact parathyroid hormone, and serum calcium level measured at the time of admission for kidney transplantation) were identified to create an integer score system (dialysis duration, pretransplant serum parathyroid hormone level, and pretransplant calcium level [DPC] score; 0–15 points) to predict tertiary hyperparathyroidism. The median DPC score was higher in participants with post-transplant parathyroidectomy than in those without (13 versus three in derivation; 13 versus four in external validation; P<0.001 for all). Pretransplant dialysis duration, pretransplant serum parathyroid hormone level, and pretransplant calcium level score predicted post-transplant parathyroidectomy with comparable performance with the best-performing machine learning model in the test set (area under the receiver operating characteristic curve: 0.94 versus 0.92; area under the precision-recall curve: 0.52 versus 0.47). Serial measurement of DPC scores (≥13 at least two or more times, 3-month interval) during 12 months prior to kidney transplantation improved risk classification for post-transplant parathyroidectomy compared with single-time measurement (net reclassification improvement, 0.28; 95% confidence interval, 0.02 to 0.54; P=0.03).

Conclusions A simple integer-based score predicted the risk of tertiary hyperparathyroidism in kidney allograft recipients, with improved classification by serial measurement compared with single-time measurement.

Clinical Trial registry name and registration number: Korean Cohort Study for Outcome in Patients with Kidney Transplantation (KNOW-KT), NCT02042963

Introduction

Persistent hyperparathyroidism after kidney transplantation or tertiary hyperparathyroidism is associated with lower graft survival, higher cardiovascular morbidity, and mortality (1,2). Although successful recovery of kidney function after kidney transplantation results in progressive parathyroid hormone (PTH) reduction and regression of parathyroid gland hyperplasia in most cases, the resistance of the parathyroid gland to inhibitory feedback is reported to persist in 5%–30% of kidney transplant recipients (3⇓⇓⇓–7).

Parathyroidectomy is an effective treatment in patients with persistent hyperparathyroidism and hypercalcemia, which showed a favorable long-term effect on hypercalcemia control, bone density improvement, and all-cause or cardiovascular mortality reduction (8⇓–10). Although surgical indications in tertiary hyperparathyroidism have not reached a clear consensus yet, severe or persistent hypercalcemia has been proposed as a major indication for parathyroidectomy in kidney transplant recipients (11). Given the current uncertainty in indications and timing of parathyroidectomy for tertiary hyperparathyroidism, an accurate and clinically applicable tool to predict the risk of post-transplant parathyroidectomy during the pretransplant period may help guide individualized therapeutic approaches, including early surgical intervention.

In this study, we aimed to develop clinical models and a simple-to-use integer-based score using machine learning algorithms to predict the risk of tertiary hyperparathyroidism requiring parathyroidectomy in kidney allograft recipients.

Materials and Methods

Study Participants

We used a derivation cohort to develop machine learning–based models and integer-based, simple-to-use scores and predict the risk of tertiary hyperparathyroidism requiring parathyroidectomy after kidney transplantation and an external validation cohort to test the performance of the models in a multicenter dataset (Figure 1). The clinical and research activities being reported are consistent with the Principles of the Declaration of Istanbul as outlined in the Declaration of Istanbul on Organ Trafficking and Transplant Tourism. The time frame of the study is summarized in Supplemental Figure 1.

Figure 1.
Figure 1.

Construction of the study cohort. KNOW-KT, Korean Cohort Study for Outcome in Patients with Kidney Transplantation; PTH, parathyroid hormone.

Derivation Cohort.

To construct a derivation cohort, data on kidney allograft recipients who underwent kidney transplantation at Severance Hospital, Seoul, Korea between January 2009 and December 2015 (n=881) were retrieved from the Severance Hospital Clinical Data Warehouse. This study was approved by the institutional review board of Severance Hospital with a waiver of written permission acquisition for deidentified retrospective medical record review (institutional review board no. 4–2021–0749) with adherence to the Declaration of Helsinki. Patients who had graft failure within 2 years of post-transplant follow-up (return to dialysis or eGFR <30 ml/min per 1.73 m2) were excluded from the analysis (n=54). After further exclusion of participants with missing baseline PTH level (n=82), unspecified dialysis duration (n=16), prior history of kidney transplantation (n=46), previous parathyroidectomy (n=8), or previous thyroidectomy (n=6), a total of 669 participants remained in the final analysis cohort. The derivation cohort was randomly split into the train set (n=501; 75%) and the test set (n=168; 25%; a holdout set to test model performance) (Figure 1).

External Validation Cohort.

The Korean Cohort Study for Outcome in Patients with Kidney Transplantation (KNOW-KT) registry dataset was analyzed as an external validation cohort. KNOW-KT is a prospective, multicenter, observational cohort study encompassing eight transplant centers in South Korea. A total of 1034 participants were enrolled between 2012 and 2016 and followed up annually until December 2020 according to a prespecified protocol (12). Detailed medical history, medications, biochemical parameters, and surgical procedures were collected after obtaining written permission from study participants. Participants enrolled at Severance Hospital (n=311) were excluded from the KNOW-KT registry due to potential overlap with the derivation cohort. After further exclusion of individuals with previous kidney transplantation history (n=38), missing baseline biochemical values (n=33), unspecified dialysis duration (n=41), prior parathyroidectomy history (n=13), graft failure during follow-up after kidney transplantation (n=17), and missing values in follow-up calcium or PTH levels (n=39), a total of 542 participants remained for analysis (Figure 1).

Definition of Outcome

The outcome was defined as post-transplantation parathyroidectomy to control tertiary hyperparathyroidism in kidney transplant recipients in both cohorts (11,13). In the derivation cohort, medical records of participants from the inclusion date (2009–2015) to December 2020 were reviewed to ascertain the occurrence of post-transplantation parathyroidectomy during follow-up. In the external validation cohort, the record of post-transplantation parathyroidectomy was annually collected from the enrollment date (2012–2016) until December 2020 using a multicenter registry system.

Predictors

Predictor variables that were included in the model were age, sex, height, weight, pretransplant biochemical assessment (PTH, albumin-corrected serum calcium level, and phosphate) at the time of admission for kidney transplantation, dialysis duration, cause of kidney failure, and immunosuppressant types at the time of discharge. In a subset of the derivation cohort (n=264 of 669; 39%), serial measurements of biochemical parameters at two or more times (at least a 3-month interval between each measurement) during the 12 months prior to kidney transplantation were available (Supplemental Figure 1).

Machine Learning–Based Prediction Models

Model Development and Performance Test.

The development and validation process of the machine learning models and integer-based score system to predict post-transplant parathyroidectomy in kidney transplant recipients is presented in Supplemental Figure 2. Prediction models were built on the basis of four machine learning algorithms (random forest, extreme gradient boosting, light gradient-boosting machine, and regularized logistic regression) using scikit-learn Python library (version 1.0.2), xgboost (version 1.5.2), and lightgbm Python package (version 3.3.2) (14). Hyperparameters of each model were tuned using the grid search method with three-fold crossvalidation and repeated five times in the train set (Supplemental Table 1). Model performance was tested in the internal test set (a holdout set) and the external validation cohort using metrics, which included accuracy, precision, recall, F1 score, area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve (AUPRC).

Feature Importance and Independent Variable Analysis.

Feature importance and the effect of each predictor on tertiary hyperparathyroidism risk prediction were assessed using Shapley additive explanations (SHAP) values obtained from the best-performing machine learning model (15). The summary plot and the dependence scatterplot on the basis of SHAP values (shap Python library, version 0.40.0) were created (Supplemental Figure 2).

Development of an Integer-Based Risk Score (Dialysis Duration, Pretransplant Serum Parathyroid Hormone Level, and Pretransplant Calcium Level Score)

To develop an integer-based, simple-to-use score to predict the risk of tertiary hyperparathyroidism requiring parathyroidectomy after kidney transplantation, a Python library was used to create a risk scorecard (Supplemental Figure 2) (scorecardpy, version 0.1.9.2; https://pypi.org/project/scorecardpy/; G. Szepannek, unpublished observations). In the train set, three top-performing continuous predictors (dialysis duration, pretransplant serum parathyroid hormone level, and pretransplant calcium level [DPC] score; all measured at the time of admission for kidney transplantation) were automatically binned by a data-driven approach using the conditional inference tree method on the basis of weight of evidence (WOE; log([number of events in the bin/number of all events]/[number of nonevents in the bin/number of all nonevents]) (16). This enabled reliable transformation of continuous features into categorical features with a monotonic relationship with regard to the outcome event to create an integer-based score system. After transforming data of each bin into WOE values, a multivariable logistic regression model was fitted using the transformed data. The model was scaled into an integer-based score using regression coefficients and WOE (bin [corresponding score]: dialysis duration in months <55 [0], 55–84 [+3], ≥85 [+4]; serum intact PTH level in picograms per milliliter <160 [0], 160–299 [+2], 300–519 [+3], 520–719 [+4], ≥720 [+5]; serum calcium level in milligrams per deciliter <8.0 [0], 8.0–8.9 [+1], 9.0–9.9 [+4], ≥10 [+6]). The integer score to predict post-transplant tertiary hyperparathyroidism requiring parathyroidectomy (DPC score) was calculated as the sum of each component (dialysis duration in months+serum intact PTH level in picograms per milliliter+serum calcium level in milligrams per deciliter score; range, 0–15). The threshold for classifying a participant as high risk of post-transplant parathyroidectomy was determined at ≥13, which maximized the F1 score in the train set (Supplemental Figure 3).

Statistical Analyses

Group differences in continuous and categorical variables between the derivation and external validation cohorts were tested using the two-sample independent t test, the Wilcoxon rank-sum test, and the chi-squared test. AUROCs of models were compared using the methods of DeLong et al. (17). Differences in serial pretransplant DPC scores during the 12 months prior to kidney transplantation between event (post-transplant parathyroidectomy) and nonevent groups were analyzed using a linear mixed regression model treating individuals as the random effect variable. Net reclassification improvement (NRI) was calculated to test the additive prognostic value of serial pretransplant DPC score measurements (two or more times the high DPC score [≥13] with at least a 3-month interval during the 12 months prior to kidney transplantation) when compared with a single measurement of the DPC score at the time of admission for kidney transplantation in a subset of the derivation cohort (18,19). Statistical analyses were performed using Python 3.7.6 and Stata 16.1 (StataCorp LLC, College Station, TX). The statistical significance level was set as a two-sided P value =0.05.

Results

Characteristics of Study Participants

In the derivation cohort, post-transplant parathyroidectomy to control tertiary hyperparathyroidism occurred in 5% (n=32 of 669; train set, 4%; test set, 6%) during follow-up (median, 7 [6–9] years). The participants in external validation cohort (the KNOW-KT registry) had relatively shorter dialysis duration (median, 5 months [1–30] versus 8 [2–56]; P<0.001), with a lower event rate (n=11 of 542; 2%) during follow-up (median, 6 [5–7] years) compared with the derivation cohort (Table 1, Supplemental Table 2). The median duration for post-transplant parathyroidectomy was 3.0 years (1.7–3.9) in the derivation cohort and 1.5 years (0.5–2.0) in the external validation cohort. In both cohorts, individuals with post-transplant parathyroidectomy had higher pretransplant serum calcium, higher PTH level, and longer dialysis duration at baseline compared with individuals without (Table 1).

View this table:
Table 1.

Clinical characteristics of participants in two Korean cohort studies evaluated to derive and validate a score to predict tertiary hyperparathyroidism after kidney transplantation

Performance of Clinical Prediction Models

In the internal test set (a 25% holdout set of the derivation cohort), AUROC and AUPRC of tertiary hyperparathyroidism prediction models (random forest, extreme gradient boosting, light gradient-boosting machine, and regularized logistic regression models) ranged from 0.89 to 0.92 and from 0.34 to 0.47, respectively (Table 2). Precision and recall ranged from 0.47 to 0.62 and from 0.60 to 0.80, respectively, with the highest balance between precision and recall in the extreme gradient-boosting model (F1 score, 0.70; precision, 0.62; recall, 0.80). In the external validation set, AUROC and AUPRC ranged from 0.97 to 0.98 and from 0.31 to 0.51, respectively. Similar to the internal test set, the extreme gradient-boosting model showed a good F1 score (0.52), with precision of 0.50 and recall of 0.55 (AUROC, 0.98; AUPRC, 0.39) in the external validation cohort. Calibration of the extreme gradient-boosting model was acceptable in the external validation cohort (Hosmer–Lemeshow statistic chi square=7.93; P=0.44). In the extreme gradient-boosting model, the top three variables with the highest feature importance for predicting post-transplant parathyroidectomy were serum calcium level followed by dialysis duration and serum PTH level measured at the time of admission for kidney transplantation (Figure 2A). All three variables showed monotonic, nonlinear increasing association with the predicted risk of post-transplant parathyroidectomy; they showed relatively stable risk up to a certain value, and then a steeper increase in the risk at a range higher than that value (Figure 2, B–D).

View this table:
Table 2.

Performance of machine learning models and integer-based score in predicting post-transplant parathyroidectomy among kidney transplant recipients in the test set (holdout set) from the derivation cohort and the external validation cohort

Figure 2.
Figure 2.

Feature importance plot and dependence plots of the extreme gradient-boosting model to predict tertiary hyperparathyrodism. (A) Feature importance plot of the extreme gradient-boosting model using Shapley additive explanations (SHAP) values. Variables are presented in descending order according to their importance (serum calcium as the feature of top importance). Color indicates whether the variable is high (in red) or low (in blue) for that observation. Horizontal location (SHAP value) indicates whether the effect of the value is associated with higher (greater than or equal to zero) or lower risk (less than zero) of post-transplant parathyroidectomy. For example, longer dialysis duration was associated with higher risk of post-transplant parathyroidectomy in this plot. (B–D) Dependence scatterplots that show the effect of the top three predictors ([B] serum calcium level, [C] dialysis duration, and [D] parathyroid hormone level) on the predictions of risk of post-transplant parathyroidectomy made by the extreme gradient-boosting prediction model.

Integer Score System to Predict Tertiary Hyperparathyroidism

An integer-based score system (range, 0–15) on the basis of the three top-discriminating variables (dialysis duration, serum intact PTH level, and calcium level; DPC score) was developed using a train set in the derivation cohort (Figure 3). Individuals with tertiary hyperparathyroidism requiring post-transplant parathyroidectomy had higher mean DPC scores at baseline (at the time of admission for kidney transplantation) compared with those without in both the derivation (13 [11–13] versus 3 [1–6]; P<0.001) and external validation (13 [12–14] versus 4 [3–6]; P<0.001) cohorts. Precision, recall, and F1 score of the DPC score (threshold: high risk ≥13 versus low risk <13) were comparable with those of the machine learning–based ensemble model in the test set from the derivation cohort and the external validation cohort (Table 2). The risk of post-transplant parathyroidectomy was higher in the high–DPC score group (≥13) compared with the low–DPC score group in the train set (52% versus 2%), the test set (58% versus 2%), and the external validation cohort (44% versus 0.6%; P<0.001 for all) (Figure 3). AUROC and AUPRC of the DPC score for predicting post-transplant parathyroidectomy were 0.94 and 0.52 in the internal test set and 0.98 and 0.41 in the external validation cohort; these values were comparable with those of the extreme gradient-boosting model (Table 2, Supplemental Figure 4).

Figure 3.
Figure 3.

Calculation of the integer-based score (dialysis duration, pretransplant intact parathyroid hormone, and serum calcium level [DPC] score; upper panels) and the risk of post-transplant parathyroidectomy according to DPC score in the derivation and external validation cohorts (lower panels). Bars indicate observed risk of post-transplant parathyroidectomy across DPC scores. C, serum calcium level; D, dialysis duration; P, pretransplant intact parathyroid hormone.

Reclassification of Risk Using Serial Dialysis Duration, Pretransplant Intact Parathyroid Hormone, and Serum Calcium Level Score Measurements

A subset of participants in the derivation cohort with two or more serial pretransplant biochemical measurements with at least a 3-month interval during the 12 months prior to kidney transplantation (n=264 of 669; 39%) was analyzed to simulate the clinical benefit of serial measurements of the DPC score compared with a single-time measurement at the time of admission for kidney transplantation (Supplemental Figure 1). The post-transplant parathyroidectomy risk groups were reclassified using serial DPC score measurements (high risk: DPC score ≥13 at two or more time points with at least a 3-month interval) from the single-time measurement at the time of admission for kidney transplantation. During the pretransplant period, participants with post-transplant parathyroidectomy had constantly higher DPC scores across all time points compared with those without (Supplemental Figure 5). Compared with single-time DPC score measurements, serial measurements of DPC scores correctly reclassified four participants to high risk in the postparathyroidectomy event group (n=15) and six participants to low risk in the nonevent group (n=249; four of 15 and six of 249), whereas serial measurements of DPC scores incorrectly reclassified zero participant to low risk in the event group and two participants to high risk in the nonevent group (zero of 15 and two of 249), yielding a significant NRI (proportion of correctly classified minus incorrectly classified) by serial measurements of the DPC score (NRI, 0.28; 95% confidence interval, 0.02 to 0.54; P=0.03) (Table 3).

View this table:
Table 3.

Reclassification of the high-risk group for post-transplant parathyroidectomy by serial measurements of dialysis duration, pretransplant serum parathyroid hormone level, and pretransplant calcium level score compared with single-time measurements in a subset of the derivation cohort with at least two or serial measurements during 12 months prior to kidney transplantation

Discussion

In this study, we developed and validated an integer score system, the DPC score, to predict the risk of tertiary hyperparathyroidism requiring parathyroidectomy after kidney transplantation. Dialysis duration, serum PTH, and calcium level were identified as the top-discriminating features using machine learning–based algorithms. A simple-to-use DPC score yielded a good predictive performance that was comparable with the best-performing machine learning model. Serial measurements of DPC scores during the 12 months of the pretransplant period improved net classification of the post-transplant parathyroidectomy risk group compared with the single-time measurements of DPC scores at the time of admission for kidney transplantation.

Persistent hyperparathyroidism after kidney transplantation is associated with a higher risk of all-cause mortality, graft loss, and fracture (1⇓–3,5,7). Parathyroidectomy is proposed to be an effective treatment for persistent hyperparathyroidism (8,20,21). However, surgical indications for hyperparathyroidism in patients with CKD stage 5 during the pretransplant period have not reached a consensus yet. This remains a clinically unmet need for the successful implementation of early surgical intervention for persistent hyperparathyroidism after kidney transplantation (22,23). Creating a prediction model with key features to estimate the risk of persistent hyperparathyroidism requiring surgical intervention after kidney transplantation can be one way to provide guidance for clinical decisions by integrating several factors into the risk probability. For this, we developed an integer-based, simple-to-use score (DPC score) and machine learning–based models to predict the risk of tertiary hyperparathyroidism requiring surgical intervention and externally validated the performance of the models in a multicenter cohort. DPC score showed potential to improve risk stratification of tertiary hyperparathyroidism, especially when serial measurements were considered together during the pretransplant period. These findings suggest that the DPC score has the potential to tailor clinical decision making for managing hyperparathyroidism in kidney transplant recipients.

We used machine learning algorithms to find relevant key features and to create an integer score system using data-driven approach. The key value of developing an integer score system was to bin continuous variables into clinically relevant categories on the basis of the similarity of the WOE, which was utilized to develop a credit scorecard system in the financial domain (24,25). Binning continuous variables using the WOE has some advantages. By establishing a monotonic relationship with the outcome variable at the logistic scale, it might better reflect the nonlinear monotonic association with the risk of outcome that was observed in all three key variables in this study, which could be an advantage of the integer score approach over using logistic regression with continuous values with the assumption of linearity between independent variables and outcome (25). For sparsely distributed variables, such as PTH, missing observations or outliers could be handled more conveniently using the WOE binning. Although this approach would lead to loss of information due to binning to few categories, the integer score system could provide high clinical applicability without any need to access a computer system, with direct interpretability on the magnitude of the effect of each predictor on the outcome.

This study has several limitations. Individuals who had early graft failure were excluded from the analysis due to potential heterogeneity in defining outcomes. Although we did not observe significant differences in DPC scores between those with early graft failure and those without in the exploratory analysis (data not shown), whether DPC score is associated with early graft failure could be another research topic of interest. To enhance clinical applicability and simplicity, we developed models and DPC scores on the basis of single-time measured serum PTH, calcium levels measured at the time of admission for kidney transplantation, and dialysis duration at the corresponding time point. Recent Kidney Disease Improving Global Outcomes guidelines recommend making a treatment plan on the basis of serial assessments of biochemical parameters, including calcium and PTH levels, considered together (22). It should be noted that the application of DPC score or machine learning models developed in this study needs to follow this scheme, indicating the necessity to consider serial information regarding the progression of the disease status to guide a therapeutic decision. Of note, we observed that serial measurements of DPC scores in the pretransplant period improved classification of the high-risk group for tertiary hyperparathyroidism when compared with single-time measurements. Serial measurements of DPC scores incorrectly classified two participants without a post-transplant parathyroidectomy event from the low-risk group to the high-risk group. One of them had persistent hypercalcemia and elevated PTH levels during the 5 years of follow-up after kidney transplantation, despite the use of the maximum tolerated dose of cinacalcet; the patient refused to undergo surgery due to concerns regarding potential comorbidities related to surgery. Availability of surgical or medical treatment options for tertiary hyperparathyroidism might differ among countries or institutions. In this study, parathyroidectomy was readily available in the derivation cohort and all referral centers participating in the KNOW-KT registry (the external validation cohort) in this study. However, the use of cinacalcet to control hypercalcemia after kidney transplantation was not covered by the national health insurance system in South Korea. Whether the integer score system can be applied to environments with different treatment availability needs to be tested further.

In summary, a simple-to-use integer-based DPC score predicted the risk of tertiary hyperparathyroidism requiring parathyroidectomy in kidney allograft recipients, thus allowing for individualized therapeutic decisions; this merits further investigation.

Disclosures

All authors have nothing to disclose.

Funding

This research was supported by Korea Centers for Disease Control and Prevention grants 2012E3301100, 2013E3301600, 2013E3301601, 2013E3301602, 2016E3300200, 2016E3300201, 2016E3300202, 2019E320100, and 2019E320101 and Severance Hospital Research Fund for Clinical Excellency grant C-2019-0032.

Acknowledgments

We thank Jin Ryu, a medical student at Yonsei University College of Medicine, for supporting the machine learning modeling process. We thank the coordinating committee of KNOW-KT cohort team for supporting this study and the severance endocrinology data science platform (SENTINEL) team, Byunghwee Koh, Heewon Choi, and Heajong Park, for assistance in the data collection process (Sung-Kil Lim Research Award 2019 4-2018-1215).

Author Contributions

N. Hong conceptualized the study; N. Hong and H.W. Kim were responsible for data curation; N. Hong, J.J. Jeong, and H.W. Kim were responsible for investigation; N. Hong and J. Lee were responsible for formal analysis; N. Hong, J.J. Jeong, and J. Lee were responsible for methodology; K.H. Huh and Y. Rhee were responsible for project administration; N. Hong, K.H. Huh, J.J. Jeong, and J. Lee were responsible for resources; K.H. Huh, J.J. Jeong, H.W. Kim, and J. Lee were responsible for validation; N. Hong and J. Lee were responsible for visualization; K.H. Huh was responsible for funding acquisition; K.H. Huh, J.J. Jeong, and Y. Rhee provided supervision; N. Hong and J. Lee wrote the original draft; and N. Hong, K.H. Huh, J.J. Jeong, H.W. Kim, J. Lee, and Y. Rhee reviewed and edited the manuscript.

Data Sharing Statement

The datasets generated during and/or analyzed during this study are available from the corresponding author on reasonable request.

Supplemental Material

This article contains the following supplemental material online at http://cjasn.asnjournals.org/lookup/suppl/doi:10.2215/CJN.15921221/-/DCSupplemental.

Supplemental Table 1. Model hyperparameters.

Supplemental Table 2. Comparison of clinical characteristics of the derivation and external validation cohorts.

Supplemental Figure 1. Time frame of the study.

Supplemental Figure 2. Development and validation process of machine learning models and the integer-based score system (DPC score) to predict parathyroidectomy after kidney transplantation.

Supplemental Figure 3. The threshold for DPC scores to detect individuals at high risk of post-transplant parathyroidectomy was determined at the point that maximized the F1 score (balance between precision and recall) in the train set (high risk ≥13 versus low risk <13).

Supplemental Figure 4. Comparison of the discriminatory ability for post-transplant parathyroidectomy between the best-performing machine learning model (extreme gradient-boosting model) and the integer-based score (DPC score) in the internal test set and the external validation cohort.

Supplemental Figure 5. The time-dependent trajectory of the machine learning–derived integer-based score (DPC score) during the 12 months prior to kidney transplantation in a subset of the derivation cohort with measurements at two or more time points (at least a 3-month interval).

Footnotes

  • Published online ahead of print. Publication date available at www.cjasn.org.

  • Received December 8, 2021.
  • Accepted April 22, 2022.

References

  1. ↵
    1. Pihlstrøm H,
    2. Dahle DO,
    3. Mjøen G,
    4. Pilz S,
    5. März W,
    6. Abedini S,
    7. Holme I,
    8. Fellström B,
    9. Jardine AG,
    10. Holdaas H
    : Increased risk of all-cause mortality and renal graft loss in stable renal transplant recipients with hyperparathyroidism. Transplantation 99: 351–359, 2015
    OpenUrl
  2. ↵
    1. Bleskestad IH,
    2. Bergrem H,
    3. Leivestad T,
    4. Hartmann A,
    5. Gøransson LG
    : Parathyroid hormone and clinical outcome in kidney transplant patients with optimal transplant function. Clin Transplant 28: 479–486, 2014
    OpenUrlCrossRefPubMed
  3. ↵
    1. Araujo MJCLN,
    2. Ramalho JAM,
    3. Elias RM,
    4. Jorgetti V,
    5. Nahas W,
    6. Custodio M,
    7. Moysés RMA,
    8. David-Neto E
    : Persistent hyperparathyroidism as a risk factor for long-term graft failure: The need to discuss indication for parathyroidectomy. Surgery 163: 1144–1150, 2018
    OpenUrl
  4. ↵
    1. Evenepoel P,
    2. Claes K,
    3. Kuypers D,
    4. Maes B,
    5. Bammens B,
    6. Vanrenterghem Y
    : Natural history of parathyroid function and calcium metabolism after kidney transplantation: A single-centre study. Nephrol Dial Transplant 19: 1281–1287, 2004
    OpenUrlCrossRefPubMed
  5. ↵
    1. Perrin P,
    2. Caillard S,
    3. Javier RM,
    4. Braun L,
    5. Heibel F,
    6. Borni-Duval C,
    7. Muller C,
    8. Olagne J,
    9. Moulin B
    : Persistent hyperparathyroidism is a major risk factor for fractures in the five years after kidney transplantation. Am J Transplant 13: 2653–2663, 2013
    OpenUrlCrossRefPubMed
  6. ↵
    1. Roodnat JI,
    2. van Gurp EA,
    3. Mulder PG,
    4. van Gelder T,
    5. de Rijke YB,
    6. de Herder WW,
    7. Kal-van Gestel JA,
    8. Pols HA,
    9. Ijzermans JN,
    10. Weimar W
    : High pretransplant parathyroid hormone levels increase the risk for graft failure after renal transplantation. Transplantation 82: 362–367, 2006
    OpenUrlPubMed
  7. ↵
    1. Mazzaferro S,
    2. Pasquali M,
    3. Taggi F,
    4. Baldinelli M,
    5. Conte C,
    6. Muci ML,
    7. Pirozzi N,
    8. Carbone I,
    9. Francone M,
    10. Pugliese F
    : Progression of coronary artery calcification in renal transplantation and the role of secondary hyperparathyroidism and inflammation. Clin J Am Soc Nephrol 4: 685–690, 2009
    OpenUrlAbstract/FREE Full Text
  8. ↵
    1. Moreno P,
    2. Coloma A,
    3. Torregrosa JV,
    4. Montero N,
    5. Francos J,
    6. Codina S,
    7. Manonelles A,
    8. Bestard O,
    9. García-Barrasa A,
    10. Melilli E,
    11. Cruzado JM
    : Long-term results of a randomized study comparing parathyroidectomy with cinacalcet for treating tertiary hyperparathyroidism. Clin Transplant 34: e13988, 2020
    OpenUrl
  9. ↵
    1. Chen L,
    2. Wang K,
    3. Yu S,
    4. Lai L,
    5. Zhang X,
    6. Yuan J,
    7. Duan W
    : Long-term mortality after parathyroidectomy among chronic kidney disease patients with secondary hyperparathyroidism: A systematic review and meta-analysis. Ren Fail 38: 1050–1058, 2016
    OpenUrl
  10. ↵
    1. Callender GG,
    2. Malinowski J,
    3. Javid M,
    4. Zhang Y,
    5. Huang H,
    6. Quinn CE,
    7. Carling T,
    8. Tomlin R,
    9. Smith JD,
    10. Kulkarni S
    : Parathyroidectomy prior to kidney transplant decreases graft failure. Surgery 161: 44–50, 2017
    OpenUrlPubMed
  11. ↵
    1. Pitt SC,
    2. Sippel RS,
    3. Chen H
    : Secondary and tertiary hyperparathyroidism, state of the art surgical management. Surg Clin North Am 89: 1227–1239, 2009
    OpenUrlCrossRefPubMed
  12. ↵
    1. Yang J,
    2. Lee J,
    3. Huh KH,
    4. Park JB,
    5. Cho JH,
    6. Lee S,
    7. Ro H,
    8. Han SY,
    9. Kim YH,
    10. Jeong JC,
    11. Park BJ,
    12. Han DJ,
    13. Park SB,
    14. Chung W,
    15. Park SK,
    16. Kim CD,
    17. Kim SJ,
    18. Kim YS,
    19. Ahn C; KNOW-KT Study Group
    : KNOW-KT (KoreaN cohort study for outcome in patients with kidney transplantation: A 9-year longitudinal cohort study): Study rationale and methodology. BMC Nephrol 15: 77, 2014
    OpenUrlCrossRefPubMed
  13. ↵
    1. Torregrosa JV,
    2. Barros X
    : Management of hypercalcemia after renal transplantation. Nefrologia 33: 751–757, 2013
    OpenUrlPubMed
  14. ↵
    1. Pedregosa F,
    2. Varoquaux G,
    3. Gramfort A,
    4. Michel V,
    5. Thirion B,
    6. Grisel O,
    7. Blondel M,
    8. Müller A,
    9. Nothman J,
    10. Louppe G,
    11. Prettenhofer P,
    12. Weiss R,
    13. Dubourg V,
    14. Vanderplas J,
    15. Passos A,
    16. Cournapeau D,
    17. Brucher M,
    18. Perrot M,
    19. Duchesnay É
    : Scikit-learn: Machine learning in Python. J Machine Learn Res 12: 2825–2830, 2011
    OpenUrl
  15. ↵
    1. Lundberg SM,
    2. Lee S-I
    : A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, 2017
  16. ↵
    1. Hothorn T,
    2. Hornik K,
    3. Zeileis A
    : Unbiased recursive partitioning: A conditional inference framework. J Comput Graph Stat 15: 651–674, 2006
    OpenUrlCrossRef
  17. ↵
    1. DeLong ER,
    2. DeLong DM,
    3. Clarke-Pearson DL
    : Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 44: 837–845, 1988
    OpenUrlCrossRefPubMed
  18. ↵
    1. Lau WL,
    2. Obi Y,
    3. Kalantar-Zadeh K
    : Parathyroidectomy in the management of secondary hyperparathyroidism. Clin J Am Soc Nephrol 13: 952–961, 2018
    OpenUrlAbstract/FREE Full Text
  19. ↵
    1. Pencina MJ,
    2. D’Agostino Sr. RB,
    3. D’Agostino Jr. RB,
    4. Vasan RS
    : Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat Med 27: 157–172, 2008
    OpenUrlCrossRefPubMed
  20. ↵
    1. Dulfer RR,
    2. Koh EY,
    3. van der Plas WY,
    4. Engelsman AF,
    5. van Dijkum EJMN,
    6. Pol RA,
    7. Vogt L,
    8. de Borst MH,
    9. Kruijff S,
    10. Schepers A,
    11. Appelman-Dijkstra NM,
    12. Rotmans JI,
    13. Hesselink DA,
    14. van Eijck CHJ,
    15. Hoorn EJ,
    16. van Ginhoven TM; Dutch Hyperparathyroid Study Group
    : Parathyroidectomy versus cinacalcet for tertiary hyperparathyroidism: A retrospective analysis. Langenbecks Arch Surg 404: 71–79, 2019
    OpenUrl
  21. ↵
    1. Cruzado JM,
    2. Moreno P,
    3. Torregrosa JV,
    4. Taco O,
    5. Mast R,
    6. Gómez-Vaquero C,
    7. Polo C,
    8. Revuelta I,
    9. Francos J,
    10. Torras J,
    11. García-Barrasa A,
    12. Bestard O,
    13. Grinyó JM
    : A randomized study comparing parathyroidectomy with cinacalcet for treating hypercalcemia in kidney allograft recipients with hyperparathyroidism. J Am Soc Nephrol 27: 2487–2494, 2016
    OpenUrlAbstract/FREE Full Text
  22. ↵
    1. Kidney Disease: Improving Global Outcomes (KDIGO) CKD-MBDU Work Group
    : KDIGO 2017 clinical practice guideline update for the diagnosis, evaluation, prevention, and treatment of chronic kidney disease-mineral and bone disorder (CKD-MBD). Available at: https://kdigo.org/wp-content/uploads/2017/02/2017-KDIGO-CKD-MBD-GL-Update.pdf. Accessed December 4, 2021
  23. ↵
    1. National Kidney Foundation
    . KDOQI US commentary on the 2017 KDIGO clinical practice guideline update for the diagnosis, evaluation, prevention, and treatment of chronic kidney disease-mineral and bone disorder (CKD-MBD). Available at: https://www.ajkd.org/article/S0272-6386(17)30898-3/fulltext. Accessed December 4, 2021
  24. ↵
    1. Refaat M
    : Credit Risk Scorecards: Development and Implementation Using SAS, Durham, NC, Lulu Press Inc., 2011
  25. ↵
    1. Siddiqi N
    : Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring, Hoboken, NJ, John Wiley & Sons, 2012
Back to top

Podcast

Subscribe to podcast
Download MP3

In this issue

View Selected Citations (0)
Print
Download PDF
Email Article
Citation Tools
Request Permissions
Machine Learning–Derived Integer-Based Score and Prediction of Tertiary Hyperparathyroidism among Kidney Transplant Recipients
Namki Hong, Juhan Lee, Hyung Woo Kim, Jong Ju Jeong, Kyu Ha Huh, Yumie Rhee
CJASN Jul 2022, 17 (7) 1026-1035; DOI: 10.2215/CJN.15921221
del.icio.us logo Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo

Jump to section

Keywords