Nothing Special   »   [go: up one dir, main page]
Skip to main content

Advertisement

SpringerLink
Log in

Cardiac arrhythmia classification with rejection of ECG recordings based on uncertainty estimation from deep neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

With the development of deep learning-based methods, automated classification of electrocardiograms (ECGs) has recently gained much attention. Although the effectiveness of deep neural networks has been encouraging, the lack of information given by the outputs restricts clinicians’ reexamination. If the uncertainty estimation comes along with the classification results, cardiologists can pay more attention to “uncertain” cases. Our study aims to classify ECGs with rejection based on data uncertainty and model uncertainty. We perform experiments on a real-world 12-lead ECG dataset. First, we estimate uncertainties using the Monte Carlo dropout for each classification prediction, based on our deep neural network. Then, we accept predictions with uncertainty under a given threshold and provide “uncertain" cases for clinicians. Furthermore, we perform a simulation experiment using varying thresholds. Finally, with the help of a clinician, we conduct case studies to explain the results of large uncertainties and incorrect predictions with small uncertainties. The results show that correct predictions are more likely to have smaller uncertainties, and the performance on accepted predictions improves as the accepting ratio decreases (i.e., more rejections). The F1-score on accepted predictions improves by 1.3\(\sim\)30.9% when accepting ratio ranges from 97.3 to 45.3%. Case studies also help explain why rejection can improve the performance. Our study helps neural networks produce more accurate results and provide information on uncertainties to better assist clinicians in the diagnosis process. It can also enable deep learning-based ECG interpretation in clinical implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The dataset used in this study is publicly available in the CPSC 2018 repository at http://2018.icbeb.org/Challenge.html. Our code is publicly available at https://github.com/hsd1503/ecg_uncertainty.

Notes

  1. http://2018.icbeb.org/Challenge.html.

References

  1. Hong S, Zhou Y, Shang J, Xiao C, Sun J (2020) Opportunities and challenges of deep learning methods for electrocardiogram data: A systematic review. Comput Biol Med 122:103801

    Article  PubMed  Google Scholar 

  2. Elul Y, Rosenberg AA, Schuster A, Bronstein AM, Yaniv Y (2021) Meeting the unmet needs of clinicians from ai systems showcased for cardiology with deep-learning–based ecg analysis. In: Proceedings of the National Academy of Sciences 118(24)

  3. van de Leur RR, Blom LJ, Gavves E, Hof IE, van der Heijden JF, Clappers NC, Doevendans PA, Hassink RJ, van Es R (2020) Automatic triage of 12-lead ecgs using deep convolutional neural networks. J Am Heart Assoc 9(10):015138

    Google Scholar 

  4. Ribeiro AH, Ribeiro MH, Paixão GM, Oliveira DM, Gomes PR, Canazart JA, Ferreira MP, Andersson CR, Macfarlane PW, Wagner M Jr (2020) Automatic diagnosis of the 12-lead ecg using a deep neural network. Nat Commun 11(1):1–9

    Google Scholar 

  5. Parvaneh S, Rubin J, Babaeizadeh S, Xu-Wilson M (2019) Cardiac arrhythmia detection using deep learning: a review. J Electrocardiol 57:70–74. https://doi.org/10.1016/j.jelectrocard.2019.08.004

    Article  Google Scholar 

  6. Clifford GD, Liu C, Moody B, Li-wei HL, Silva I, Li Q, Johnson A, Mark RG (2017) Af classification from a short single lead ecg recording: the physionet/computing in cardiology challenge 2017. In: 2017 Computing in Cardiology (CinC), pp 1–4. IEEE

  7. Hong S, Fu Z, Zhou R, Yu J, Li Y, Wang K, Cheng G (2020) Cardiolearn: A cloud deep learning service for cardiac disease detection from electrocardiogram. In: Companion proceedings of the web conference 2020, pp 148–152

  8. Hong S, Zhou Y, Wu M, Shang J, Wang Q, Li H, Xie J (2019) Combining deep neural networks and engineered features for cardiac arrhythmia detection from ecg recordings. Physiol Measure 40(5):054009

    Article  ADS  Google Scholar 

  9. Hong S, Xiao C, Ma T, Li H, Sun J (2019) Mina: multilevel knowledge-guided attention for modeling electrocardiography signals. In: Proceedings of the 28th international joint conference on artificial intelligence, pp 5888–5894. AAAI Press

  10. Zhou Y, Hong S, Shang J, Wu M, Wang Q, Li H, Xie J (2019) K-margin-based residual-convolution-recurrent neural network for atrial fibrillation detection. In: IJCAI

  11. Attia ZI, Noseworthy PA, Lopez-Jimenez F, Asirvatham SJ, Deshmukh AJ, Gersh BJ, Carter RE, Yao X, Rabinstein AA, Erickson BJ (2019) An artificial intelligence-enabled ecg algorithm for the identification of patients with atrial fibrillation during sinus rhythm: a retrospective analysis of outcome prediction. Lancet 394(10201):861–867

    Article  PubMed  Google Scholar 

  12. Raghunath S, Cerna AEU, Jing L, Stough J, Hartzel DN, Leader JB, Kirchner HL, Stumpe MC, Hafez A, Nemani A, et al (2020) Prediction of mortality from 12-lead electrocardiogram voltage data using a deep neural network. Nat Med, pp 1–6

  13. Hong S, Xu Y, Khare A, Priambada S, Maher K, Aljiffry A, Sun J, Tumanov A (2020) Holmes: health online model ensemble serving for deep learning models in intensive care units. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery and data mining, pp 1614–1624

  14. Li K, Pan W, Li Y, Jiang Q, Liu G (2018) A method to detect sleep apnea based on deep neural network and hidden markov model using single-lead ecg signal. Neurocomputing 294:94–101

    Article  Google Scholar 

  15. Sun C, Hong S, Wang J, Dong X, Han F, Li H (2022) A systematic review of deep learning methods for modeling electrocardiograms during sleep. Physiol Measure

  16. Labati RD, Muñoz E, Piuri V, Sassi R, Scotti F (2019) Deep-ecg: convolutional neural networks for ecg biometric recognition. Pattern Recogn Lett 126:78–85

    Article  ADS  Google Scholar 

  17. Hong S, Wang C, Fu Z (2020) Cardioid: learning to identification from electrocardiogram data. Neurocomputing 412:11–18

    Article  Google Scholar 

  18. Siontis KC, Noseworthy PA, Attia ZI, Friedman PA (2021) Artificial intelligence-enhanced electrocardiography in cardiovascular disease management. Nat Rev Cardiol 18(7):465–478

    Article  PubMed  PubMed Central  Google Scholar 

  19. Fu Z, Hong S, Zhang R, Du S (2021) Artificial-intelligence-enhanced mobile system for cardiovascular health management. Sensors 21(3):773

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  20. Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, Cui C, Corrado G, Thrun S, Dean J (2019) A guide to deep learning in healthcare. Nat Med 25(1):24–29

    Article  CAS  PubMed  Google Scholar 

  21. Hannun AY, Rajpurkar P, Haghpanahi M, Tison GH, Bourn C, Turakhia MP, Ng AY (2019) Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat Med 25(1):65–69

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Smulyan H (2019) The computerized ecg: friend and foe. Am J Med 132(2):153–160

    Article  PubMed  Google Scholar 

  23. Musa N, Gital AY, Aljojo N, Chiroma H, Adewole KS, Mojeed HA, Faruk N, Abdulkarim A, Emmanuel I, Folawiyo YY, et al (2022) A systematic review and meta-data analysis on the applications of deep learning in electrocardiogram. J Ambient Intell Human Comput, pp 1–74

  24. Petmezas G, Stefanopoulos L, Kilintzis V, Tzavelis A, Rogers JA, Katsaggelos AK, Maglaveras N (2022) State-of-the-art deep learning methods on electrocardiogram data: systematic review. JMIR Med Inf 10(8):38454

    Article  Google Scholar 

  25. Chew HSJ, Achananuparp P (2022) Perceptions and needs of artificial intelligence in health care to increase adoption: scoping review. J Med Internet Res 24(1):32939

    Article  Google Scholar 

  26. Loftus TJ, Shickel B, Ruppert MM, Balch JA, Ozrazgat-Baslanti T, Tighe PJ, Efron PA, Hogan WR, Rashidi P, Upchurch GR Jr (2022) Uncertainty-aware deep learning in healthcare: a scoping review. PLOS Digital Health 1(8):0000085

    Article  Google Scholar 

  27. Jang J-H, Kim TY, Yoon D (2021) Effectiveness of transfer learning for deep learning-based electrocardiogram analysis. Healthcare Inf Res 27(1):19–28

    Article  Google Scholar 

  28. Bond RR, Novotny T, Andrsova I, Koc L, Sisakova M, Finlay D, Guldenring D, McLaughlin J, Peace A, McGilligan V (2018) Automation bias in medicine: the influence of automated diagnoses on interpreter accuracy and uncertainty when reading electrocardiograms. J Electrocardiol 51(6):6–11

    Article  Google Scholar 

  29. Charoenphakdee N, Cui Z, Zhang Y, Sugiyama M (2021) Classification with rejection based on cost-sensitive classification. In: Proceedings of machine learning research in international conference on machine learning, pp 1507–1517

  30. Geifman Y, El-Yaniv R (2019) Selectivenet: a deep neural network with an integrated reject option. In: Proceedings of machine learning research international conference on machine learning, pp 2151–2159

  31. Louizos C, Welling M (2017) Multiplicative normalizing flows for variational bayesian neural networks. In: Proceedings of machine learning research international conference on machine learning, pp 2218–2227

  32. Gal Y, Ghahramani Z (2016) Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: Proceedings of machine learning research international conference on machine learning, pp 1050–1059

  33. Bai X, Wang X, Liu X, Liu Q, Song J, Sebe N, Kim B (2021) Explainable deep learning for efficient and robust pattern recognition: a survey of recent developments. Pattern Recogn 120:108102

    Article  Google Scholar 

  34. Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Adv Neural Inf Process Syst 30

  35. Aseeri AO (2021) Uncertainty-aware deep learning-based cardiac arrhythmias classification model of electrocardiogram signals. Computers 10(6):82

    Article  Google Scholar 

  36. Malinin A, Gales M (2018) Predictive uncertainty estimation via prior networks. In: Proceedings of the 32nd international conference on neural information processing systems. NIPS’18, pp 7047–7058. Curran Associates Inc., Red Hook, NY, USA

  37. Liu F, Liu C, Zhao L, Zhang X, Wu X, Xu X, Liu Y, Ma C, Wei S, He Z (2018) An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. J Med Imag Health Inf 8(7):1368–1373

    Google Scholar 

  38. Radosavovic I, Kosaraju RP, Girshick R, He K, Dollár P (2020) Designing network design spaces. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10428–10436

  39. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  40. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision, pp 630–645. Springer

  41. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1492–1500

  42. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167

  43. Ramachandran P, Zoph B, Le QV (2017) Searching for activation functions. arXiv preprint arXiv:1710.05941

  44. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  Google Scholar 

  45. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  46. Damianou A, Lawrence ND (2013) Deep Gaussian processes. In: Carvalho CM, Ravikumar P (eds) Proceedings of the sixteenth international conference on artificial intelligence and statistics. Proceedings of machine learning research, vol 31, pp 207–215. Proceedings of machine learning research, Scottsdale, Arizona, USA

  47. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

  48. Murat F, Yildirim O, Talo M, Baloglu UB, Demir Y, Acharya UR (2020) Application of deep learning techniques for heartbeats detection using ecg signals-analysis and review. Comput Biol Med, 103726

  49. Mathews SM, Kambhamettu C, Barner KE (2018) A novel application of deep learning for single-lead ecg classification. Comput Biol Med 99:53–62

    Article  PubMed  Google Scholar 

  50. Yıldırım Ö, Pławiak P, Tan R-S, Acharya UR (2018) Arrhythmia detection using deep convolutional neural network with long duration ecg signals. Comput Biol Med 102:411–420

    Article  PubMed  Google Scholar 

  51. Moskalenko V, Zolotykh N, Osipov G (2019) Deep learning for ecg segmentation. In: International conference on neuroinformatics, pp 246–254. Springer

  52. Li Y, Qu Q, Wang M, Yu L, Wang J, Shen L, He K (2020) Deep learning for digitizing highly noisy paper-based ecg records. Comput Biol Med 127:104077

    Article  PubMed  Google Scholar 

  53. Zhou S, Sapp JL, AbdelWahab A, Trayanova N (2021) Deep learning applied to electrocardiogram interpretation. Can J Cardiol 37(1):17–18. https://doi.org/10.1016/j.cjca.2020.03.035

    Article  PubMed  Google Scholar 

  54. Cai W, Hu D (2020) ECG interpretation with deep learning, pp 143–156. https://doi.org/10.1007/978-981-15-3824-7_8

  55. Zhang W, Geng S, Hong S (2023) A simple self-supervised ecg representation learning method via manipulated temporal-spatial reverse detection. Biomed Signal Process Control 79:104194

    Article  Google Scholar 

  56. Hong S, Zhang W, Sun C, Zhou Y, Li H (2022) Practical lessons on 12-lead ecg classification: meta-analysis of methods from physionet/computing in cardiology challenge 2020. Front Physiol, 2505

  57. Bae MH, Lee JH, Yang DH, Park HS, Cho Y, Chae SC, Jun JE (2012) Erroneous computer electrocardiogram interpretation of atrial fibrillation and its clinical consequences. Clin Cardiol 35(6):48–353 https://arxiv.org/abs/https://onlinelibrary.wiley.com/doi/pdf/10.1002/clc.22000. https://doi.org/10.1002/clc.22000

  58. Schläpfer J, Wellens HJ (2017) Computer-interpreted electrocardiograms: benefits and limitations. J Am College Cardiol 70(9):1183–1192. https://doi.org/10.1016/j.jacc.2017.07.723

    Article  Google Scholar 

  59. Yang L, Zhang Z, Hong S, Xu R, Zhao Y, Shao Y, Zhang W, Yang MH, Cui B (2022) Diffusion models: a comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796

  60. Ge W, Jing J, An S, Herlopian A, Ng M, Struck AF, Appavu B, Johnson EL, Osman G, Haider HA (2021) Deep active learning for interictal ictal injury continuum eeg patterns. J Heurosci Methods 351:108966

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No.62102008).

Author information

Authors and Affiliations

  1. National Institute of Health Data Science, Peking University, Beijing, 100191, China

    Wenrui Zhang & Shenda Hong

  2. Institute of Medical Technology, Health Science Center of Peking University, Beijing, 100191, China

    Wenrui Zhang & Shenda Hong

  3. Department of Mathematics, National University of Singapore, Singapore, 119077, Singapore

    Wenrui Zhang

  4. Department of Electrocardiology, The First Affiliated Hospital of University of Science and Technology of China, Hefei, 230001, China

    Xinxin Di

  5. HeartVoice Medical Technology, Hefei, 230027, China

    Guodong Wei, Shijia Geng & Zhaoji Fu

  6. School of Management, University of Science and Technology of China, Hefei, 230026, China

    Zhaoji Fu

Authors
  1. Wenrui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinxin Di
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guodong Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shijia Geng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhaoji Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shenda Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shenda Hong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical standard

The authors state that this research complies with ethical standards. This research does not involve either human participants or animals.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, W., Di, X., Wei, G. et al. Cardiac arrhythmia classification with rejection of ECG recordings based on uncertainty estimation from deep neural networks. Neural Comput & Applic 36, 4047–4058 (2024). https://doi.org/10.1007/s00521-023-09267-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09267-5

Keywords

Search

Navigation