Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1007/978-3-031-72086-4_19guideproceedingsArticle/Chapter ViewAbstractPublication PagesConference Proceedingsacm-pubtype
Article

Deep Learning for Cancer Prognosis Prediction Using Portrait Photos by StyleGAN Embedding

Published: 07 October 2024 Publication History

Abstract

Survival prediction for cancer patients is critical for optimal treatment selection and patient management. Current patient survival prediction methods typically extract survival information from patients’ clinical record data or biological and imaging data. In practice, experienced clinicians can have a preliminary assessment of patients’ health status based on patients’ observable physical appearances, which are mainly facial features. However, such assessment is highly subjective. In this work, the efficacy of objectively capturing and using prognostic information contained in conventional portrait photographs using deep learning for survival prediction purposes is investigated for the first time. A pre-trained StyleGAN2 model is fine-tuned on a custom dataset of our cancer patients’ photos to empower its generator with generative ability suitable for patients’ photos. The StyleGAN2 is then used to embed the photographs to its highly expressive latent space. Utilizing state-of-the-art survival analysis models and StyleGAN’s latent space embeddings, this approach predicts the overall survival for single as well as pan-cancer, achieving a C-index of 0.680 in a pan-cancer analysis, showcasing the prognostic value embedded in simple 2D facial images. In addition, thanks to StyleGAN’s interpretable latent space, our survival prediction model can be validated for relying on essential facial features, eliminating any biases from extraneous information like clothing or background. Moreover, our approach provides a novel health attribute obtained from StyleGAN’s extracted features, allowing the modification of face photographs to either a healthier or more severe illness appearance, which has significant prognostic value for patient care and societal perception, underscoring its potential important clinical value.

References

[1]
Bagnis A, Caffo E, Cipolli C, De Palma A, Farina G, and Mattarozzi K Judging health care priority in emergency situations: patient facial appearance matters Social Science & Medicine 2020 260
[2]
Berman J Modern classification of neoplasms: reconciling differences between morphologic and molecular approaches BMC cancer 2005 5 1-12
[3]
Capitanio U and Montorsi F Renal cancer The Lancet 2016 387 10021 894-906
[4]
Chen, J., Lu, S., Mao, Y., Tan, L., Li, G., Gao, Y., Tan, P., Huang, D., Zhang, X., Qiu, Y., et al.: An mri-based radiomics-clinical nomogram for the overall survival prediction in patients with hypopharyngeal squamous cell carcinoma: a multi-cohort study. European Radiology pp. 1–10 (2022)
[5]
Emura T, Chen YH, and Chen HY Survival prediction based on compound covariate under cox proportional hazard models PLoS One 2012 7 10
[6]
Gurovich Y, Hanani Y, Bar O, Nadav G, Fleischer N, Gelbman D, Basel-Salmon L, Krawitz PM, Kamphausen SB, Zenker M, et al. Identifying facial phenotypes of genetic disorders using deep learning Nature medicine 2019 25 1 60-64
[7]
Hui, D., Hess, K., Santos, R.d., Chisholm, G., Bruera, E.: A diagnostic model for impending death in cancer patients: preliminary report. Cancer 121(21), 3914–3921 (2015)
[8]
Karras T, Aittala M, Hellsten J, Laine S, Lehtinen J, and Aila T Training generative adversarial networks with limited data Advances in neural information processing systems 2020 33 12104-12114
[9]
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proc. CVPR. pp. 4401–4410 (2019)
[10]
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proc. CVPR. pp. 8110–8119 (2020)
[11]
Katzman JL, Shaham U, Cloninger A, Bates J, Jiang T, and Kluger Y Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network BMC medical research methodology 2018 18 1 1-12
[12]
Kidd AC, McGettrick M, Tsim S, Halligan DL, Bylesjo M, and Blyth KG Survival prediction in mesothelioma using a scalable lasso regression model: instructions for use and initial performance using clinical predictors BMJ open respiratory research 2018 5 1
[13]
Kim DW, Lee S, Kwon S, Nam W, Cha IH, and Kim HJ Deep learning-based survival prediction of oral cancer patients Scientific reports 2019 9 1 1-10
[14]
King DE Dlib-ml: A machine learning toolkit Journal of Machine Learning Research 2009 10 1755-1758
[15]
Kong X, Gong S, Su L, Howard N, and Kong Y Automatic detection of acromegaly from facial photographs using machine learning methods EBioMedicine 2018 27 94-102
[16]
Liang, B., Yang, N., He, G., Huang, P., Yang, Y.: Identification of the facial features of patients with cancer: a deep learning–based pilot study. Journal of Medical Internet Research 22(4), e17234 (2020)
[17]
Lin, H., Zelterman, D.: Modeling survival data: extending the cox model (2002)
[18]
Oken MM, Creech RH, Tormey DC, Horton J, Davis TE, McFadden ET, and Carbone PP Toxicity and response criteria of the eastern cooperative oncology group Am. J. Clin. Oncol. 1982 5 6 649-656
[19]
de Oliveira WA Quality of life, facial appearance and self-esteem in patients with orthodontic treatment Revista Mexicana de Ortodoncia 2017 5 3 138-139
[20]
Rankin M and Borah GL Perceived functional impact of abnormal facial appearance Plastic and reconstructive surgery 2003 111 7 2140-2146
[21]
Su Z, Liang B, Shi F, Gelfond J, Šegalo S, Wang J, Jia P, and Hao X Deep learning-based facial image analysis in medical research: a systematic review protocol BMJ open 2021 11 11
[22]
van Timmeren JE, Leijenaar RT, van Elmpt W, Reymen B, Oberije C, Monshouwer R, Bussink J, Brink C, Hansen O, and Lambin P Survival prediction of non-small cell lung cancer patients using radiomics analyses of cone-beam ct images Radiotherapy and Oncology 2017 123 3 363-369
[23]
Vale-Silva LA and Rohr K Long-term cancer survival prediction using multimodal deep learning Scientific Reports 2021 11 1 1-12
[24]
Wallis D and Buvat I Clever hans effect found in a widely used brain tumour mri dataset Medical Image Analysis 2022 77
[25]
Wang, D., Jing, Z., He, K., Garmire, L.X.: Cox-nnet v2. 0: improved neural-network-based survival prediction extended to large-scale emr data. Bioinformatics 37(17), 2772–2774 (2021)
[26]
Wankhede, D.S., Rangasamy, S.: Dynamic architecture based deep learning approach for glioblastoma brain tumor survival prediction. Neuroscience Informatics p. 100062 (2022)
[27]
Withington, E., Lonie, I., Chadwick, J., Mann, W.N., Lloyd, G., et al.: Hippocratic writings. Penguin UK (2005)
[28]
Yolcu, G., Oztel, I., Kazan, S., Oz, C., Palaniappan, K., Lever, T.E., Bunyak, F.: Deep learning-based facial expression recognition for monitoring neurological disorders. In: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). pp. 1652–1657. IEEE (2017)
[29]
Zhu, X., Yao, J., Huang, J.: Deep convolutional neural network for survival analysis with pathological images. In: 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). pp. 544–547. IEEE (2016)

Index Terms

  1. Deep Learning for Cancer Prognosis Prediction Using Portrait Photos by StyleGAN Embedding
                Index terms have been assigned to the content through auto-classification.

                Recommendations

                Comments

                Please enable JavaScript to view thecomments powered by Disqus.

                Information & Contributors

                Information

                Published In

                cover image Guide Proceedings
                Medical Image Computing and Computer Assisted Intervention – MICCAI 2024: 27th International Conference, Marrakesh, Morocco, October 6–10, 2024, Proceedings, Part V
                Oct 2024
                814 pages
                ISBN:978-3-031-72085-7
                DOI:10.1007/978-3-031-72086-4
                • Editors:
                • Marius George Linguraru,
                • Qi Dou,
                • Aasa Feragen,
                • Stamatia Giannarou,
                • Ben Glocker,
                • Karim Lekadir,
                • Julia A. Schnabel

                Publisher

                Springer-Verlag

                Berlin, Heidelberg

                Publication History

                Published: 07 October 2024

                Author Tags

                1. Survival prediction
                2. cancer
                3. StyleGAN
                4. deep learning
                5. latent space
                6. face prognosis
                7. explainable AI

                Qualifiers

                • Article

                Contributors

                Other Metrics

                Bibliometrics & Citations

                Bibliometrics

                Article Metrics

                • 0
                  Total Citations
                • 0
                  Total Downloads
                • Downloads (Last 12 months)0
                • Downloads (Last 6 weeks)0
                Reflects downloads up to 23 Feb 2025

                Other Metrics

                Citations

                View Options

                View options

                Figures

                Tables

                Media

                Share

                Share

                Share this Publication link

                Share on social media