Nothing Special   »   [go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

Automated Multimodal Machine Learning for Esophageal Variceal Bleeding Prediction Based on Endoscopy and Structured Data

  • Original Paper
  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Esophageal variceal (EV) bleeding is a severe medical emergency related to cirrhosis. Early identification of cirrhotic patients with at a high risk of EV bleeding is key to improving outcomes and optimizing medical resources. This study aimed to evaluate the feasibility of automated multimodal machine learning (MMML) for predicting EV bleeding by integrating endoscopic images and clinical structured data. This study mainly includes three steps: step 1, developing deep learning (DL) models using EV images by 12-month bleeding on TensorFlow (backbones include ResNet, Xception, EfficientNet, ViT and ConvMixer); step 2, training and internally validating MMML models integrating clinical structured data and DL model outputs to predict 12-month EV bleeding on an H2O-automated machine learning platform (algorithms include DL, XGBoost, GLM, GBM, RF, and stacking); and step 3, externally testing MMML models. Furthermore, existing clinical indices, e.g., the MELD score, Child‒Pugh score, APRI, and FIB-4, were also examined. Five DL models were transfer learning to the binary classification of EV endoscopic images at admission based on the occurrence or absence of bleeding events during the 12-month follow-up. An EfficientNet model achieved the highest accuracy of 0.868 in the validation set. Then, a series of MMML models, integrating clinical structured data and the output of the EfficientNet model, were automatedly trained to predict 12-month EV bleeding. A stacking model showed the highest accuracy (0.932), sensitivity (0.952), and F1-score (0.879) in the test dataset, which was also better than the existing indices. This study is the first to evaluate the feasibility of automated MMML in predicting 12-month EV bleeding based on endoscopic images and clinical variables.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The datasets and R codes in the study are available from the corresponding author on reasonable request. The code of computer vision models’ training is available to access: https://osf.io/ycxwr/?view_only=81b4f590605c472f9e979c854a573cce.

References

  1. Gines P, Krag A, Abraldes JG, Sola E, Fabrellas N, Kamath PS: Liver cirrhosis. Lancet 398:1359-1376, 2021

    Article  PubMed  Google Scholar 

  2. de Franchis R, Bosch J, Garcia-Tsao G, Reiberger T, Ripoll C, Baveno VIIF: Baveno VII - Renewing consensus in portal hypertension. J Hepatol, 2021

  3. de Franchis R, Baveno VIF: Expanding consensus in portal hypertension: Report of the Baveno VI Consensus Workshop: Stratifying risk and individualizing care for portal hypertension. J Hepatol 63:743-752, 2015

    Article  PubMed  Google Scholar 

  4. Collaborators GBDC: The global, regional, and national burden of cirrhosis by cause in 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet Gastroenterol Hepatol 5:245-266, 2020

    Article  Google Scholar 

  5. Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts H: Artificial intelligence in radiology. Nat Rev Cancer 18:500-510, 2018

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Choi J, et al.: Convolutional Neural Network Technology in Endoscopic Imaging: Artificial Intelligence for Endoscopy. Clin Endosc 53:117-126, 2020

    Article  PubMed  PubMed Central  Google Scholar 

  7. Mondal AK, Bhattacharjee A, Singla P, Prathosh AP: xViTCOS: Explainable Vision Transformer Based COVID-19 Screening Using Radiography. IEEE J Transl Eng Health Med 10:1100110, 2022

    Article  PubMed  Google Scholar 

  8. Ang TL, Carneiro G: Artificial intelligence in gastrointestinal endoscopy. J Gastroenterol Hepatol 36:5-6, 2021

    Article  PubMed  Google Scholar 

  9. Visaggi P, et al.: Artificial Intelligence in the Diagnosis of Upper Gastrointestinal Diseases. J Clin Gastroenterol 56:23-35, 2022

    Article  PubMed  Google Scholar 

  10. Bang CS, Lee JJ, Baik GH: Artificial Intelligence for the Prediction of Helicobacter Pylori Infection in Endoscopic Images: Systematic Review and Meta-Analysis Of Diagnostic Test Accuracy. J Med Internet Res 22:e21983, 2020

    Article  PubMed  PubMed Central  Google Scholar 

  11. Cho BJ, et al.: Automated classification of gastric neoplasms in endoscopic images using a convolutional neural network. Endoscopy 51:1121-1129, 2019

    Article  PubMed  Google Scholar 

  12. Mori Y, et al.: Real-Time Use of Artificial Intelligence in Identification of Diminutive Polyps During Colonoscopy: A Prospective Study. Ann Intern Med 169:357-366, 2018

    Article  PubMed  Google Scholar 

  13. Handelman GS, Kok HK, Chandra RV, Razavi AH, Lee MJ, Asadi H: eDoctor: machine learning and the future of medicine. J Intern Med 284:603-619, 2018

    Article  CAS  PubMed  Google Scholar 

  14. Obermeyer Z, Emanuel EJ: Predicting the Future - Big Data, Machine Learning, and Clinical Medicine. N Engl J Med 375:1216-1219, 2016

    Article  PubMed  PubMed Central  Google Scholar 

  15. Faes L, et al.: Automated deep learning design for medical image classification by health-care professionals with no coding experience: a feasibility study. Lancet Digit Health 1:e232-e242, 2019

    Article  PubMed  Google Scholar 

  16. Hung AJ, Chen J, Gill IS: Automated Performance Metrics and Machine Learning Algorithms to Measure Surgeon Performance and Anticipate Clinical Outcomes in Robotic Surgery. JAMA Surg 153:770-771, 2018

    Article  PubMed  PubMed Central  Google Scholar 

  17. Qi S, et al.: Multimodal Fusion With Reference: Searching for Joint Neuromarkers of Working Memory Deficits in Schizophrenia. IEEE Trans Med Imaging 37:93-105, 2018

    Article  PubMed  Google Scholar 

  18. Al'Aref SJ, et al.: Clinical applications of machine learning in cardiovascular disease and its relevance to cardiac imaging. Eur Heart J 40:1975-1986, 2019

    Article  CAS  PubMed  Google Scholar 

  19. North Italian Endoscopic Club for the S, Treatment of Esophageal V: Prediction of the first variceal hemorrhage in patients with cirrhosis of the liver and esophageal varices. A prospective multicenter study. N Engl J Med 319:983–989, 1988

  20. The general rules for recording endoscopic findings on esophageal varices. Jpn J Surg 10:84–87, 1980

  21. Borgli H, et al.: HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci Data 7:283, 2020

    Article  PubMed  PubMed Central  Google Scholar 

  22. Jiang H, et al.: A Multi-Label Deep Learning Model with Interpretable Grad-CAM for Diabetic Retinopathy Classification. Annu Int Conf IEEE Eng Med Biol Soc 2020:1560-1563, 2020

    Google Scholar 

  23. Roberts D, et al.: Treatment for bleeding oesophageal varices in people with decompensated liver cirrhosis: a network meta-analysis. Cochrane Database Syst Rev 4:CD013155, 2021

  24. Zhou Y, Zheng J, Li S, Zhou T, Zhang P, Li HB: Alcoholic Beverage Consumption and Chronic Diseases. Int J Environ Res Public Health 13, 2016

  25. Wang Y, Yu W, He M, Huang Y, Wang M, Zhu J: Serum cytoskeleton-associated protein 4 as a biomarker for the diagnosis of hepatocellular carcinoma. Onco Targets Ther 12:359-364, 2019

    Article  CAS  PubMed  Google Scholar 

  26. Sousa M, et al.: The Baveno VI criteria for predicting esophageal varices: validation in real life practice. Rev Esp Enferm Dig 109:704-707, 2017

    Article  PubMed  Google Scholar 

  27. Nawalerspanya S, Sripongpun P, Chamroonkul N, Kongkamol C, Piratvisuth T: Validation of original, expanded Baveno VI, and stepwise & platelet-MELD criteria to rule out varices needing treatment in compensated cirrhosis from various etiologies. Ann Hepatol 19:209-213, 2020

    Article  PubMed  Google Scholar 

  28. Colli A, et al.: Platelet count, spleen length, and platelet count-to-spleen length ratio for the diagnosis of oesophageal varices in people with chronic liver disease or portal vein thrombosis. Cochrane Database Syst Rev 4:CD008759, 2017

  29. Dong TS, et al.: Machine Learning-based Development and Validation of a Scoring System for Screening High-Risk Esophageal Varices. Clin Gastroenterol Hepatol 17:1894–1901 e1891, 2019

  30. Agarwal S, et al.: Development of a machine learning model to predict bleed in esophageal varices in compensated advanced chronic liver disease: A proof of concept. J Gastroenterol Hepatol 36:2935-2942, 2021

    Article  PubMed  Google Scholar 

  31. Dana J, et al.: Conventional and artificial intelligence-based imaging for biomarker discovery in chronic liver disease. Hepatol Int, 2022

  32. Abedalla A, Abdullah M, Al-Ayyoub M, Benkhelifa E: Chest X-ray pneumothorax segmentation using U-Net with EfficientNet and ResNet architectures. PeerJ Comput Sci 7:e607, 2021

    Article  PubMed  PubMed Central  Google Scholar 

  33. Deng H, Qi X, Guo X: Diagnostic Accuracy of APRI, AAR, FIB-4, FI, King, Lok, Forns, and FibroIndex Scores in Predicting the Presence of Esophageal Varices in Liver Cirrhosis: A Systematic Review and Meta-Analysis. Medicine (Baltimore) 94:e1795, 2015

  34. Ghasemian A, Hosseinmardi H, Galstyan A, Airoldi EM, Clauset A: Stacking models for nearly optimal link prediction in complex networks. Proc Natl Acad Sci U S A 117:23393-23400, 2020

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This study was funded by the National Natural Science Foundation of China [82000540]; Medical Education Collaborative Innovation Fund of Jiangsu University [JDY2022018]; Science and Technology Plan (Apply Basic Research) of Changzhou City [CJ20210006]; the Youth Program of Suzhou Health Committee [KJXW2021051 and KJXW2019001]. No funding body had any role in the design of the study and collection, analysis, interpretation of data, or in writing the manuscript.

Author information

Author notes

  1. Yu Wang, Yu Hong, Yue Wang, and Xin Zhou contribute equally.

Authors and Affiliations

  1. Department of General Surgery, Jintan Affiliated Hospital of Jiangsu University, Changzhou, China

    Yu Wang

  2. Department of Gastroenterology, The First Affiliated Hospital of Soochow University, Suzhou, 215000, China

    Yu Hong, Xin Gao, Chenyan Yu, Jiaxi Lin, Lu Liu, Jingwen Gao, Minyue Yin, Guoting Xu, Xiaolin Liu & Jinzhou Zhu

  3. Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China

    Yu Hong, Xin Gao, Chenyan Yu, Jiaxi Lin, Lu Liu, Jingwen Gao, Minyue Yin, Guoting Xu, Xiaolin Liu & Jinzhou Zhu

  4. Department of Hepatology, The Fifth People’s Hospital of Suzhou, Suzhou, 215000, China

    Yue Wang

  5. Department of Gastroenterology, Jintan Affiliated Hospital of Jiangsu University, Changzhou, China

    Xin Zhou

Authors
  1. Yu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xin Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chenyan Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jiaxi Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Lu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jingwen Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Minyue Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Guoting Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Xiaolin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Jinzhou Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Study concept and design (JZ, YH), acquisition of data (XZ, XG, CY), analysis and interpretation of data (JX, LL, JG, MY), drafting of the manuscript (YW), administrative, technical, or material support (XL, GX, JZ), and study supervision (YH). All authors have made a significant contribution to this study and have approved the final manuscript.

Corresponding author

Correspondence to Jinzhou Zhu.

Ethics declarations

Research Involving Human Participants and/or Animals

This study was approved by the Ethics Committee of The First Affiliated Hospital of Soochow University (the IRB approval number 2022098). All procedures performed in studies involving human participants were in accordance with the Helsinki Declaration of 1975 as revised in 2013.

Informed Consent

Informed consent was obtained from all individual participants included in the study. The authors affirm that human research participants provided informed consent for publication of the images in Fig. 3.

Conflict of Interest

The authors declare no competing interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 952 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Hong, Y., Wang, Y. et al. Automated Multimodal Machine Learning for Esophageal Variceal Bleeding Prediction Based on Endoscopy and Structured Data. J Digit Imaging 36, 326–338 (2023). https://doi.org/10.1007/s10278-022-00724-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-022-00724-6

Keywords

Search

Navigation