Establishment of predictive model for patients with kidney cancer bone metastasis: a study based on SEER database

Introduction

Kidney cancer is a malignant tumor originating from the renal tubule and collecting tubular epithelial system, and the incidence rate is 2% to 3% of adult malignant tumors (1). In 2018, the incidence of renal cancer in the United States had ranked 6th in male malignant tumors and 10th in female malignant tumors (2). According to the data, since 1950, the incidence and mortality of renal malignancies in the United States have been increasing year by year. By 2001, the incidence rate had increased by 126%, and the mortality rate had increased by 36.5%, while the 5-year survival rate had only increased by about 9% (3,4). In China, this upward trend is also very obvious. In 2015, the number of new and death cases was about 66,800 and 23,400, respectively (1).

In recent years, advances in imaging diagnostic techniques and surgical techniques have enabled earlier resection of early-stage kidney cancer, but there are still some patients with kidney cancer who have distant metastases at the initial diagnosis or after undergoing radical surgery (5). In addition to the lungs, bone is the second most common site of metastasis of kidney cancer (6). Bone metastases often occur in the mid-shaft bone, of which 71% are osteolytic lesions, 18% are osteogenic lesions, and 11% are mixed lesions. Kidney cancer bone metastasis (KCBM) is a catastrophic event that can lead to pain and pathology in patients (7,8). The incidence of skeletal-related events (SRE) after bone metastasis in patients with kidney cancer is higher (74%) than in breast cancer (64%), myeloma (51%), and prostate cancer (44%) (9). SRE such as fractures, spinal cord compression, and hypercalcemia seriously affect the quality of life.

Accurate prediction of the prognosis of patients with KCBM can help doctors and patients choose a further appropriate treatment. The Surveillance, Epidemiology, and End Results (SEER) database is the US’s leading cancer statistics database that records information on morbidity, mortality, and disease in millions of malignancies in some states and counties (10). We collected the data of patients with KCBM from this database for analysis and proposed to establish a clinical prediction model to provide a convenient and effective tool for predicting the prognosis and to evaluate its prediction accuracy.

Methods

Data collection

The National Cancer Institute’s SEER database covers about 28% of the population of the United States and collects data on cancer patients from 18 tumor registration centers (11). The latest data for the (1973–2016 varying) database released in November 2018 was obtained using SEER stat special software (version 8.3.6), and data acquisition was done in client-server mode. During the period from January 1, 2010 to December 31, 2015, screening patients with kidney cancer diagnosed with bone metastases. Exclusion criteria include: no/unknown kidney cancer patients with bone metastases, unknown survival time and vital status.

Inclusion codes and criteria

The main endpoints were overall survival (OS) and kidney cancer-special survival (KCSS). In this study, we classified patients according to the following factors, such as age (<50, 50–70, >70), gender (female, male), race (White, Black, others), marital status (Married, Unmarried), insurance status (Insured, Uninsured).

For the tumor pathology, the patients were classified according to histological type (clear cell carcinoma, other), grade (I, II, III, IV, unknown), T stage (T0, T1, T2, T3, T4, TX), N stage (N0, N1, NX).

For the number of other metastatic organs and treatment, the patients were classified according to number of extra-bone (brain, liver, and lung) metastatic organs (0, 1, 2, 3), surgery (yes, no), radiotherapy (RT) (yes, no) and chemotherapy (CT) (yes, no).

Patients grouping

In order to establish an effective prognostic prediction model, all patients were divided into a model establishment group and validation group according to a random assignment method (ratio 7:3). Among them, the model establishment group included a total of 2,966 patients, and the validation group included 1,268 patients.

Statistical analysis

Demographic information about KCBM patients using a method of descriptive statistics. The chi-square test was used to analyze the dead/live of categorical variables of prognostic factors in KCBM patients. The survival time of each prognostic factor was expressed as the median and interquartile ranges. Kaplan-Meier survival curves and log-rank tests were used to analyze the OS and KCSS for each prognostic factor. Multivariate cox regression analysis was used to analyze all-cause mortality (ACM) and kidney cancer-special mortality (KCSM) for each prognostic factor and categorical variable. Moreover, the hazard ratios (HR) and 95% confidence intervals (CIs) for all strata of each factor were also calculated. The P value <0.05 was considered statistically significant.

Kaplan-Meier survival curves and construction of nomograms

Kaplan-Meier survival curves were plotted for all prognostic factors. Based on the results of the multivariate Cox regression analysis, the prognostic predictors of P<0.001 in the log-rank test were included in the nomograms. The model was used to model establishment group data for internal verification of the nomograms, and the validation group data is used for external verification of the nomograms. The concordance index (C-index), receiver operating characteristic (ROC) curve and calibration curve were used to evaluate the accuracy of the model. The C-index was between 0.5 and 1, 0.5 was completely inconsistent, indicating that the model had no predictive effect, and 1 was completely consistent, indicating that the model’s prediction results were completely consistent with the actual. In general, the C-index was less accurate at 0.50–0.70, moderate accuracy between 0.71 and 0.90, and high accuracy above 0.90 (12,13). The area under the ROC curve (AUC) referred to the area around the ROC curve and the x-axis, (1,0)-(1,1). Similar to the C-index, the AUC was less accurate at 0.50–0.70, moderate accuracy between 0.71 and 0.90, and high accuracy above 0.90 (14,15). The predicted probability of the nomograms of the OS and KCSS for 1, 3 and 5 years were compared with the observed survival probability to obtain calibration plots (16,17). All statistical analysis, model establishment group and validation group generation and construction of nomograms were performed by the R project (Version 3.6.1).

Results

Demographic, pathological, number of other metastatic organs, and treatment features of KCBM patients

The screening process for patients included in the study was shown in Figure 1. The number and proportion of patients with various prognostic factors were shown in Table 1, and the median survival was shown in Table 2. The mean age and median age of 4,234 patients were 65.63 and 65 years, respectively. In entire group, the majority of the categorical variables were 50–70 years old (56.3%), male (68.9%), White (83.1%), married (57.9%), insured (80.0%), clear cell carcinoma (78.5%), grade unknown (66.7%), T3 (25.7%), N0 (54.8%), number of extra-bone metastatic organs was 0 (43.0%), no surgery (73.0%), radiotherapy (50.4%), and chemotherapy (50.0%).

Figure 1 Flowchart of patients identification and selection.

Table 1 Demographic information, pathology, number of other metastatic organs, and treatment information of KCBM patients
Full table

Table 2 Median survival and survival months of KCBM patients
Full table

In model establishment group, the majority of the categorical variables were 50–70 years old (56.7%), male (69.7%), White (82.5%), married (57.7%), insured (79.7%), clear cell carcinoma (78.2%), grade unknown (66.7%), T3 (25.6%), N0 (55.2%), number of extra-bone metastatic organs was 0 (43.2%), no surgery (72.5%), radiotherapy (50.4%), and no chemotherapy (50.1%).

In validation group, the majority of the categorical variables were 50–70 years old (55.4%), male (67.0%), White (84.5%), married (58.2%), insured (80.7%), clear cell carcinoma (79.3%), grade unknown (66.6%), T3 (25.9%), N0 (53.9%), number of extra-bone metastatic organs was 0 (42.7%), no surgery (74.1%), radiotherapy (50.2%), and chemotherapy (50.5%).

The impact of different variables on ACM and KCSM

There were 3,670 patients with ACM and 2,901 patients with KCSM (Figure 1, Table 3). In the demographic data, >70 years patients had the highest ACM (91.7%) and KCSM (88.7%). Gender differences had no significant effect on ACM (87.2% vs. 86.4%, P=0.468) and KCSM (84.3% vs. 83.4%, P=0.508). Black patients had the highest ACM (88.9%) and KCSM (86.6%). Unmarried patients had the highest ACM (89.1%) and KCSM (86.7%). Uninsured patients had the highest ACM (88.7%) and KCSM (86.7%).

Table 3 Univariate survival analyses of KCBM patients according to various clinicopathological variables
Full table

In tumour pathology data, patients with non-clear cell carcinoma had the highest ACM (91.5%) and KCSM (89.6%). Patients with grade II had the lowest ACM (68.2%) and KCSM (61.9%). Patients with T1 stage tumor had the lowest ACM (83.4%) and KCSM (78.6%). N0 stage tumor patients had the lowest ACM (81.5%) and KCSM (77.6%).

The number of extra-bone metastatic organs was 0, ACM and KCSM were lowest, 80.1% and 75.3% respectively. Among the treatment data, patients who did not undergo surgery had significantly higher ACM (92.7% vs. 70.5%, P<0.001) and KCSM (90.8% vs. 66.3%, P<0.001) than patients who underwent surgery. Radiotherapy had no significant effect on ACM (85.9% vs. 87.5%, P=0.127) and KCSM (83.3% vs. 84.1%, P=0.517) in patients. Receiving chemotherapy could significantly reduce ACM (84.8% vs. 88.6%, P<0.001) and KCSM (82.1% vs. 85.5%, P=0.007) in patients.

Kaplan-Meier survival curves of each prognostic factor

We plotted Kaplan-Meier survival curves for demographic factors (Figure 2), pathological factors (Figure 3), and the number of other metastatic organs and treatment (Figure 4). In addition, the log-rank test for all variables was shown in Table 4.

Figure 2 Survival curves in KCBM patients according to demographic factors. (A,B) Kaplan-Meier curves among patients stratified by age at diagnosis for OS (A) and KCSS (B); (C,D) Kaplan-Meier curves among patients stratified by gender for OS (C) and KCSS (D); (E,F) Kaplan-Meier curves among patients stratified by race for OS (E) and KCSS (F); (G,H) Kaplan-Meier curves among patients stratified by marital status for OS (G) and KCSS (H); (I,J) Kaplan-Meier curves among patients stratified by insurance status for OS (I) and KCSS (J). KCBM, kidney cancer bone metastasis; OS, overall survival; KCSS, kidney cancer-special survival.

Figure 3 Survival curves in KCBM patients according to pathological factors. (A,B) Kaplan-Meier curves among patients stratified by histological type for OS (A) and KCSS (B). (C,D) Kaplan-Meier curves among patients stratified by grade for OS (C) and KCSS (D). (E,F) Kaplan-Meier curves among patients stratified by T stage for OS (E) and KCSS (F). (G,H) Kaplan-Meier curves among patients stratified by N stage for OS (G) and KCSS (H). KCBM, kidney cancer bone metastasis; OS, overall survival; KCSS, kidney cancer-special survival.

Figure 4 Survival curves in KCBM patients according to number of other metastatic organs and treatment. (A,B) Kaplan-Meier curves among patients stratified by other metastases for OS (A) and KCSS (B); (C,D) Kaplan-Meier curves among patients stratified by surgery/no surgery for OS (C) and KCSS (D); (E,F) Kaplan-Meier curves among patients stratified by RT/no RT for OS (E) and KCSS (F); (G,H) Kaplan-Meier curves among patients stratified by CT/no CT for OS (G) and KCSS (H); KCBM, kidney cancer bone metastasis; OS, overall survival; KCSS, kidney cancer-special survival; CT, chemotherapy; RT, radiotherapy.

Table 4 Multivariate Cox regression analysis for ACM and KCSM in KCBM patients
Full table

It was observed that the increased in age was significantly related to the worsening prognosis (Figure 2A,B). There was no significant correlation between gender difference and prognosis survival (Figure 2C,D). Compared with other people, white and black were significantly associated with poor prognosis (Figure 2E,F). Unmarried patients were significantly associated with poor prognosis (Figure 2G,H). Uninsured patients were significantly associated with poor prognosis (Figure 2I,J).

Observing the survival curves of pathological factors, the histological type was clear cell carcinoma was clearly associated with a good prognosis (Figure 3A,B). Grade II tumors were significantly associated with a good prognosis (Figure 3C,D). T4 and TX tumors were significantly associated with poor prognosis (Figure 3E,F). Compared with N1 and NX tumors, N0 tumors clearly had a better prognosis (Figure 3G,H).

Observing the survival curves of the number of other metastatic organs and treatment. In addition to bone, the number of other metastatic organs was 0 significantly correlated with a good prognosis (Figure 4A,B). Surgical treatment could significantly improve the prognosis of patients (Figure 4C,D). Receiving RT or CT could improve the prognosis of patients to some extent (radiotherapy: Figure 4E,F; chemotherapy: Figure 4G,H).

Multivariate cox regression of prognostic factors in KCBM patients and the construction of nomograms

Multivariate cox regression analysis of all variables, and HR and 95% CIs were shown in Table 4. We have established their own nomogram for OS (Figure 5) and KCSS (Figure 6) respectively. These two nomograms included factors such as age, marital status, insurance status, histological type, grade, T stage, N stage, number of extra-bone metastatic organs, surgery, RT, and CT.

Figure 5 Nomogram of overall survival at 1, 3, and 5 years in patients with kidney cancer bone metastasis prediction.

Figure 6 Nomogram of kidney cancer-special survival at 1, 3, and 5 years in patients with kidney cancer bone metastasis prediction.

Interior and external verification of nomogram

The C-index of the nomogram of OS and KCSS was 0.733 and 0.752, respectively. The ROC curve results of the model establishment group and the validation group were shown in Figure 7 (ROC curve of OS) and Figure 8 (ROC curve of KCSS), respectively. In all ROC curves, all AUC values were greater than 0.7. The calibration plots of the model establishment group and the validation group showed good consistency between the predicted nomograms of OS (Figure 9) and KCSS (Figure 10).

Figure 7 ROC curve of overall survival (OS). (A,C,E) ROC curves for 1 year (A), 3 years (C), and 5 years (E), respectively, validated by the model establishment group; (B,D,F) ROC curves for 1 year (B), 3 years (D), and 5 years (F), respectively, validated by the validation group; AUC, area under the ROC curve.

Figure 8 ROC curve of kidney cancer-special survival (KCSS). (A,C,E) ROC curves for 1 year (A), 3 years (C), and 5 years (E), respectively, validated by the model establishment group; (B,D,F) ROC curves for 1 year (B), 3 years (D), and 5 years (F), respectively, validated by the validation group. AUC, area under the ROC curve.

Figure 9 Calibration plots of overall survival (OS). (A,C,E) Calibration plots for 1 year (A), 3 years (C), and 5 years (E), respectively, validated by the model establishment group; (B,D,F) calibration plots for 1 year (B), 3 years (D), and 5 years (F), respectively, validated by the validation group.

Figure 10 Calibration plots of kidney cancer-special survival (KCSS). (A,C,E) Calibration plots for 1 year (A), 3 year (C), and 5 years (E), respectively, validated by the model establishment group; (B,D,F) calibration plots for 1 year (B), 3 years (D), and 5 years (F), respectively, validated by the validation group.

Discussion

In the first visit to kidney cancer, 20–50% of patients have a local invasion or distant metastasis (18). Distant metastasis seriously affects the quality of life of patients and increases the difficulty of treatment (19). Especially bone metastasis is recognized as an important prognostic factor for patients with renal cancer. Bone metastasis, suggesting that the tumor enters the late stage, is generally considered to have a shorter survival period (18-20). Seaman et al. (20) found that the average survival time for patients with renal cell carcinoma and bone metastases was 13.8 months, compared with 25.3 months for patients without bone metastases. It was also believed that the prognosis of patients with cancer was related to the presence of bone metastases in the diagnosis of kidney cancer, and also to the patient’s own physical condition and treatment (21). Therefore, summarizing the clinical features and treatment methods of KCBM was conducive to improving the treatment level of such diseases. In addition, the prognostic prediction model established by using the currently collected data makes doctors had a more objective judgment on the prognosis of KCBM patients, and it was also convenient to promote and apply.

Demographic features of KCBM patients

The incidence rates varied from country to country or from region to region. Generally speaking, the incidence rate in developed countries was higher than that in developing countries. Urban areas were higher than in rural areas. There were approximately twice as many male patients as female patients. The age of high incidence was 50 to 70 years old (22). In our study, patients enrolled in the study were aged 50–70 years (56.3%), and the number of male patients was more than twice that of female patients (68.9% vs. 31.1%), which was similar to previous reports. The race was related to the prognosis of kidney cancer. Stafford et al. (23) analyzed the demographic factors and causes of death of 39,434 kidney cancer patients from the California Cancer Registry from 1988 to 2004 and found that black had higher mortality than whites and other races. This conclusion was also confirmed in our study. The ACM and KCSM were the highest in black patients with KCBM. In this study, it was found that people who were unmarried (separated, divorced, or single) had higher ACM and KCSM. Epidemiological investigations of not only kidney cancer but also a variety of cancers had all found an increase in mortality from unmarried status. A study found that patients who were unmarried were more likely not to undergo surgery. In the clear cell cancer patient population, the T stage of patents who had never been married was higher than those who were married, separated or divorced. Unmarried kidney cancer patients had higher ACM and cancer-specific mortality than those who were married (24). The status of insurance was also analyzed, and it was found that the ACM and KCSM of patients insured were significantly lower than those of patients not insured. This might be related to the patient’s ability to pay for the cost of treatment. The patient might be more actively faced with future treatments without worrying about the high cost of treatment, and clinicians will have fewer concerns when choosing treatment. A sound and comprehensive insurance system had a positive effect on the prognosis of KCBM patients.

Tumor pathological features of KCBM patients

The pathology of the tumor was also an important factor affecting the prognosis. Reports in the literature suggested that patients with a histological type of clear cell carcinoma had a better prognosis than patients with other tissue types (25-27). In our study, the probability of survival in patients with a histological type of clear cell carcinoma was significantly higher than in other types of patients, supporting the previous literature. Tumor grading and staging were closely related to prognosis. Nese et al. (28) found that according to different grades, the 5-year survival rate showed significant stratification in all types of renal cell carcinoma, with grade I being 77.8%, grade II being 69.6%, grade III being 48.8%, and grade IV being 35.5%. In our research, we also observed a very obvious stratification phenomenon. As the grading increases, the patient’s expected survival time decreases significantly. The same situation also occurred in the TNM stage of the tumor, the T or N stage increased, and the patient’s expected survival time also decreased significantly.

The number of other metastatic organs, and treatment features of KCBM patients

In addition to bone tissue, the lungs, brain, and liver were also organs that were prone to metastasis (29,30). Our study found that an increase in the number of metastatic organs indicates a poor prognosis. Therefore, it was recommended to conduct a comprehensive examination of patients with kidney cancer to determine the specific number of metastatic organs. The treatment of kidney cancer also affected the prognosis of patients with KCBM. There were reports that when kidney cancer was combined with multiple organs (especially the liver, brain, etc.), nephrectomy did not effectively increase the survival rate, which in turn led to an increase in death rate within 6 months after surgery (31). In addition, in a new CARMENA (Cancer du Rein Metastatique Nephrectomie et Anti angiogéniques) trial, the MSKCC (Memorial Sloan Kettering Cancer Center) prediction model was classified as an intermediate-risk or poor-risk patient with metastatic kidney cancer, the efficacy of the targeted drug sunitinib alone is not inferior to nephrectomy followed by sunitinib (32). This study changed our preference for surgery, especially in patients with intermediate-risk or poor-risk of metastatic kidney cancer. However, in this study, patients who underwent surgery had significantly lower ACM (70.5% vs. 92.7%, P<0.001) and KCSM (66.3% vs. 90.8%, P<0.001) than those who did not. It was reasonable to believe that the surgical treatment of kidney cancer was an effective method to improve prognosis. Although kidney cancer itself was not sensitive to radiotherapy, radiotherapy for bone metastases could alleviate bone pain, reduce the risk of pathological fractures, and relieve spinal cord compression (33,34). Our study also found that radiotherapy did not reduce ACM (85.9% vs. 87.5%, P=0.127) and KCSM (83.3% vs. 84.1%, P=0.517). Chemotherapy as important treatment, whether it was neoadjuvant chemotherapy or postoperative supplemental chemotherapy, was of great significance. In the present study, the risk of ACM and the risk of KCSM in patients who did not receive chemotherapy were 1.847 and 1.840 times higher than those who received chemotherapy, respectively. We insisted that active chemotherapy remained an effective way to improve prognosis.

Establishment and verification of nomograms

To make the results of multivariate Cox regression more visual and easy to use. We established nomograms for OS and KCSS, respectively, and verified the accuracy of the two prediction models. The C-index of both nomograms was greater than 0.7, achieving moderate accuracy. Secondly, the AUC values calculated by the ROC curve were also between 0.71 and 0.9, achieving moderate accuracy. Finally, we had separately drawn the calibration plots. In all the calibration plots, we could observe the better fitting degree between the predicted value and the actual value. Therefore, we believed that the predictive model as a whole has achieved moderate accuracy and could be used in actual clinical work.

Limitations

This study is based on the registration information of KCBM patients in the SEER database. Although the database summarizes the information of KCBM patients as detailed as possible, it still has its limitations. Firstly, we cannot obtain the performance status, comorbidities, time to metastasis, type of surgery/radiotherapy/systemic therapy performed and when during the natural history of the disease. Secondly, we cannot obtain the specific symptoms of bone metastases from individual patients and the bisphosphonate treatment of these patients from the database. The lack of these data makes the prediction accuracy of the model lower. Finally, the SEER database only describes whether patients receive chemotherapy, and does not show the toxic effects of chemotherapy, which also affects our judgment of the relationship between chemotherapy and prognosis. In addition to the limitations of the database itself, we also believe that the verification of the clinical prediction model requires more external data and requires multi-center, large sample data for repeated verification, which is a long-term and complicated work.

Conclusions

In this study, nomograms of OS and KCSS were established based on the published data of KCBM patients in the SEER database, and the model was validated internally and externally. These verifications confirmed the validity and accuracy of the model. At present, this model has the ability to predict the prognosis of KCBM patients and can be used in clinical work. However, in the future, more sophisticated external data is needed to repeatedly verify the model in order to achieve better clinical application capabilities.

Acknowledgments

I would like to express my special thanks to my partners for the encouragement and support they gave me during my study. Thanks to my girlfriend, Miss Sun, for her support for my life and research.

Funding: None.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/tau.2020.01.24). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Chen W, Zheng R, Baade PD, et al. Cancer statistics in China 2015. CA Cancer J Clin 2016;66:115-32. [Crossref] [PubMed]
2. Siegel RL, Miller KD, Jemal A. Cancer statistics 2018. CA Cancer J Clin 2018;68:7-30. [Crossref] [PubMed]
3. Pantuck AJ, Zisman A, Belldegrun AS. The changing natural history of renal cell carcinoma. J Urol 2001;166:1611-23. [Crossref] [PubMed]
4. Cronin KA, Ries LA, Edwards BK. The Surveillance, Epidemiology, and End Results (SEER) Program of the National Cancer Institute. Cancer 2014;120 Suppl 23:3755. [Crossref] [PubMed]
5. Lipton A. Skeletal Complications in Patients with Bone Metastases from Renal Cell Carcinoma and Therapeutic Benefits of Zoledronic Acid. Clin Cancer Res 2004;10:6397S-6403S. [Crossref] [PubMed]
6. Chandrasekar T, Klaassen Z, Goldberg H, et al. Metastatic renal cell carcinoma: Patterns and predictors of metastases—A contemporary population-based series. Urol Oncol 2017;35:661.e7-661.e14. [Crossref] [PubMed]
7. Barata PC. Treatment of Renal Cell Carcinoma: Current Status and Future Directions. CA Cancer J Clin 2017;67:507-24. [Crossref] [PubMed]
8. Chen SC, Kuo PL. Bone Metastasis from Renal Cell Carcinoma. Int J Mol Sci 2016. [Crossref] [PubMed]
9. Hoesl CE, Altwein JE. Biphosphonates in Advanced Prostate and Renal Cell Cancer – Current Status and Potential Applications. Urol Int 2006;76:97-105. [Crossref] [PubMed]
10. Curtis RE, Freedman DM, Ron E, et al. New malignancies among cancer survivors: SEER Cancer Registries, 1973-2000. Journal of Epidemiology & Community Health 2008;62:375-6. [Crossref]
11. Schrag D, Bach PB, Dahlman C, et al. Identifying and measuring hospital characteristics using the SEER-Medicare data and other claims-based sources. Med Care 2002;40:IV-96-103. [Crossref] [PubMed]
12. Gerds TA, Kattan MW, Schumacher M, et al. Estimating a time-dependent? concordance index for survival prediction models with covariate dependent censoring. Stat Med 2013;32:2173-84. [Crossref] [PubMed]
13. Figueira JR, Greco S, Roy B, et al. ELECTRE methods with interaction between criteria: An extension of the concordance index. Eur J Oper Res 2009;199:478-95. [Crossref]
14. Jones CM, Athanasiou T. Summary Receiver Operating Characteristic Curve Analysis Techniques in the Evaluation of Diagnostic Tests. Ann Thorac Surg 2005;79:16-20. [Crossref] [PubMed]
15. Cook NR. Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction. Circulation 2007;115:928-35. [Crossref] [PubMed]
16. Gerds TA, Andersen PK, Kattan MW. Calibration plots for risk prediction models in the presence of competing risks. Stat Med 2014;33:3191-203. [Crossref] [PubMed]
17. Stankov L. Calibration curves, scatterplots and the distinction between general knowledge and perceptual tasks. Learning & Individual Differences 1998;10:29-50. [Crossref]
18. Shinder BM, Kevin R, Douglas F, et al. Surgical Management of Advanced and Metastatic Renal Cell Carcinoma: A Multidisciplinary Approach. Front Oncol 2017;7:107. [Crossref] [PubMed]
19. Szendroi A, Dinya E, Kardos M, et al. Prognostic Factors and Survival of Renal Clear Cell Carcinoma Patients with Bone Metastases. Pathol Oncol Res 2010;16:29-38. [Crossref] [PubMed]
20. Seaman E, Goluboff ET, Ross S, et al. Association of radionuclide bone scan and serum alkaline phosphatase in patients with metastatic renal cell carcinoma. Urology 1996;48:692-5. [Crossref] [PubMed]
21. Watanabe M, Kitamura Y, Komatsubara S, et al. The evaluation of surgical management for metastatic lesions of renal cell carcinoma. Hinyokika Kiyo 1995;41:847-53. [PubMed]
22. Lindblad P. Epidemiology of renal cell carcinoma. Scand J Surg 2004;93:88-96. [Crossref] [PubMed]
23. Stafford HS, Saltzstein SL, Shimasaki S, et al. Racial/Ethnic and Gender Disparities in Renal Cell Carcinoma Incidence and Survival. J Urol 2008;179:1704-8. [Crossref] [PubMed]
24. Marchioni M, Martel T, Bandini M, et al. Marital status and gender affect stage, tumor grade, treatment type and cancer specific mortality in T1–2 N0 M0 renal cell carcinoma. World J Urol 2017;35:1899-905. [Crossref] [PubMed]
25. Amin MB, Amin MB, Tamboli P, et al. Prognostic Impact of Histologic Subtyping of Adult Renal Epithelial Neoplasms. Am J Surg Pathol 2002;26:281-91. [Crossref] [PubMed]
26. Motzer RJ. Treatment Outcome and Survival Associated With Metastatic Renal Cell Carcinoma of Non-Clear-Cell Histology. J Clin Oncol 2002;20:2376-81. [Crossref] [PubMed]
27. Cheville JC, Lohse CM, Zincke H, et al. Comparisons of Outcome and Prognostic Features Among Histologic Subtypes of Renal Cell Carcinoma. Am J Surg Pathol 2003;27:612-24. [Crossref] [PubMed]
28. Nese N, Paner GP, Mallin K, et al. Renal cell carcinoma: assessment of key pathologic prognostic parameters and patient characteristics in 47909 cases using the National Cancer Data Base. Ann Diagn Pathol 2009;13:1-8. [Crossref] [PubMed]
29. Chen XY, Lan M, Zhou Y, et al. Risk factors for bone metastasis from renal cell cancer. J Bone Oncol 2017;9:29-33. [Crossref] [PubMed]
30. Abdel-Rahman O. Clinical correlates and prognostic value of different metastatic sites in metastatic renal cell carcinoma. Future Oncol 2017. [Epub ahead of print]. [Crossref] [PubMed]
31. Culp SH, Tannir NM, Abel EJ, et al. Can we better select patients with metastatic renal cell carcinoma for cytoreductive nephrectomy? Cancer 2010;116:3378-88. [Crossref] [PubMed]
32. Méjean A, Ravaud A, Thezenas S, et al. Sunitinib Alone or after Nephrectomy in Metastatic Renal-Cell Carcinoma. N Engl J Med 2018;379:417-27. [Crossref] [PubMed]
33. Lutz S, Berk L, Chang E, et al. Palliative radiotherapy for bone metastases: an ASTRO evidence-based guideline. Int J Radiat Oncol Biol Phys 2011;79:965-76. [Crossref] [PubMed]
34. Zelefsky MJ, Greco C, Motzer R, et al. Tumor Control Outcomes After Hypofractionated and Single-Dose Stereotactic Image-Guided Intensity-Modulated Radiotherapy for Extracranial Metastases From Renal Cell Carcinoma. Int J Radiat Oncol Biol Phys 2012;82:1744-8. [Crossref] [PubMed]