research-article

Open access

Model Confidence Calibration for Reliable COVID-19 Early Screening via Audio Signal Analysis

Authors:

Micky C. Nnamdi,

Junior Ben Tamo,

Sara Stackpole,

Benoit Marteau,

May Dongmei WangAuthors Info & Claims

BCB '23: Proceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics

Article No.: 39, Pages 1 - 6

https://doi.org/10.1145/3584371.3613015

Published: 04 October 2023 Publication History

Abstract

Advanced sensors in mobile devices have served as an effective screening tool for COVID-19 diagnosis, and an alternative to reverse transcription-polymerase chain reaction (rRT-PCR) tests, particularly in underdeveloped countries. In this study, we present a deep-learning approach to enable COVID-19 rapid diagnosis using cough signals. We then leverage spline calibration to enhance the reliability of predictions by calibrating model confidence. We conduct extensive experiments on the Coswara dataset to demonstrate the effectiveness of the proposed calibration approach in audio signal analysis. Our finding suggested that calibration could substantially enhance the reliability of COVID-19 early detection when compared to the uncalibrated model. Furthermore, our Spline calibration-based method outperformed other calibration methods, achieving an expected calibration error (ECE) of 0.148, an area under the receiver operating characteristic (AUROC) of 0.812, a Brier Loss of 0.189, and a logarithmic (Log) Loss of 0.584. The proposed confidence calibration framework on modern neural networks may enhance the reliability and trustworthiness of mobile healthcare for infectious respiratory disease screening in real-world applications.

References

[1]

Alican Akman, Harry Coppock, Alexander Gaskell, Panagiotis Tzirakis, Lyn Jones, et al. 2022. Evaluating the covid-19 identification resnet (cider) on the interspeech covid-19 from audio challenges. Frontiers in Digital Health 4 (2022).

[2]

Mahmoud Aly, Kamel H Rahouma, and Safwat M Ramzy. 2022. Pay attention to the speech: COVID-19 diagnosis using machine learning and crowdsourced respiratory and speech recordings. Alexandria Engineering Journal 61, 5 (2022), 3487--3500.

[3]

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321--357.

[4]

Asim H. Gazi, Patrick Lis, Arshawn Mohseni, Christopher Ompi, Felipe O. Giuste, et al. 2021. Respiratory Markers Significantly Enhance Anxiety Detection Using Multimodal Physiological Sensing. In 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). 1--4.

[5]

Felipe Giuste, Wenqi Shi, Yuanda Zhu, Tarun Naren, Monica Isgut, et al. 2022. Explainable artificial intelligence methods in combating pandemics: A systematic review. IEEE Reviews in Biomedical Engineering (2022).

[6]

Felipe O. Giuste, Lawrence L. He, Monica Isgut, Wenqi Shi, Blake J. Anderson, et al. 2021. Automated Risk Assessment of COVID-19 Patients at Diagnosis Using Electronic Healthcare Records. In 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). 1--4.

[7]

Aekansh Goel, Zachary Mudge, Sarah Bi, Charles Brenner, Nicholas Huffman, et al. 2022. Identification of covid-19 severity and associated genetic biomarkers based on scrna-SEQ data. In Proceedings of the 13th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics. 1--5.

Digital Library

[8]

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning. PMLR, 1321--1330.

[9]

Zolkepli Husein. 2020. Malaya-Speech. https://github.com/huseinzol05/malaya-speech. GitHub repository (2020).

[10]

Christian Infante, Daniel Chamberlain, R Fletcher, Y Thorat, and Rahul Kodgule. 2017. Use of cough sounds for diagnosis and screening of pulmonary disease. In 2017 IEEE global humanitarian technology conference (GHTC). IEEE, 1--10.

[11]

Monica Isgut, Logan Gloster, Katherine Choi, Janani Venugopalan, and May D. Wang. 2023. Systematic Review of Advanced AI Methods for Improving Healthcare Data Quality in Post COVID-19 Era. IEEE Reviews in Biomedical Engineering 16 (2023), 53--69.

[12]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[13]

Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, et al. 2020. Calibrated language model fine-tuning for in-and out-of-distribution data. arXiv preprint arXiv:2010.11506 (2020).

[14]

Ranganath Krishnan and Omesh Tickoo. 2020. Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems 33 (2020), 18237--18248.

[15]

Meelis Kull, Telmo Silva Filho, and Peter Flach. 2017. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics. PMLR, 623--631.

[16]

Hsuan-Tien Lin, Chih-Jen Lin, and Ruby C Weng. 2007. A note on Platt's probabilistic outputs for support vector machines. Machine learning 68 (2007), 267--276.

[17]

Brian Lucena. 2018. Spline-based probability calibration. arXiv preprint arXiv:1809.07751 (2018).

[18]

Saranga Kingkor Mahanta, Shubham Jain, and Darsh Kaushik. 2021. The brogrammers dicova 2021 challenge system report. Tech. Rep., Tech. rep. (2021).

[19]

Kellen Mermin-Bunnell, Yuanda Zhu, Andrew Hornback, Gregory Damhorst, Tiffany Walker, et al. 2023. Use of Natural Language Processing of Patient-Initiated Electronic Health Record Messages to Identify Patients With COVID-19 Infection. JAMA Network Open 6, 7 (07 2023), e2322299--e2322299.

[20]

Ananya Muguli, Lancelot Pinto, Neeraj Sharma, Prashant Krishnan, Prasanta Kumar Ghosh, et al. 2021. DiCOVA Challenge: Dataset, task, and baseline system for COVID-19 diagnosis using acoustics. arXiv preprint arXiv:2103.09148 (2021).

[21]

Tarun Naren, Yuanda Zhu, and May Dongmei Wang. 2021. COVID-19 Diagnosis Using Model Agnostic Meta-Learning on Limited Chest X-Ray Images. In Proceedings of the 12th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics (BCB '21). Association for Computing Machinery, New York, NY, USA, Article 42, 9 pages.

Digital Library

[22]

Lara Orlandic, Tomas Teijeiro, and David Atienza. 2021. The COUGHVID crowd-sourcing dataset, a corpus for the study of large-scale cough analysis algorithms. Scientific Data 8, 1 (2021), 156.

[23]

Paul Porter, Udantha Abeyratne, Vinayak Swarnkar, Jamie Tan, Ti-wan Ng, et al. 2019. A prospective multicentre study testing the diagnostic accuracy of an automated cough sound centred analytic system for the identification of common respiratory disorders in children. Respiratory research 20, 1 (2019), 1--10.

[24]

Sivaramakrishnan Rajaraman, Prasanth Ganesan, and Sameer Antani. 2022. Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks. Plos one 17, 1 (2022), e0262838.

[25]

Shivangi Sachdeva, Ronald W Davis, and Amit K Saha. 2021. Microfluidic point-of-care testing: commercial landscape and future directions. Frontiers in Bioengineering and Biotechnology 8 (2021), 602659.

[26]

KC Santosh, Nicholas Rasmussen, Muntasir Mamun, and Sunil Aryal. 2022. A systematic review on cough sound analysis for Covid-19 diagnosis and screening: is my cough sound COVID-19? PeerJ Computer Science 8 (2022), e958.

[27]

Garima Sharma, Karthikeyan Umapathy, and Sri Krishnan. 2022. Audio texture analysis of COVID-19 cough, breath, and speech sounds. Biomedical Signal Processing and Control 76 (2022), 103703.

[28]

Neeraj Sharma, Prashant Krishnan, Rohit Kumar, Shreyas Ramoji, Srikanth Raj Chetupalli, et al. 2020. Coswara-a database of breathing, cough, and voice sounds for COVID-19 diagnosis. arXiv preprint arXiv:2005.10548 (2020).

[29]

Wenqi Shi, Mitali S Gupte, and May D Wang. 2021. Learning from heterogeneous data via contrastive learning: An application in multi-source covid-19 radiography. In 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE, 1--4.

[30]

Wenqi Shi, Li Tong, Yuanda Zhu, and May D Wang. 2021. COVID-19 automatic diagnosis with radiographic imaging: Explainable attention transfer deep neural networks. IEEE Journal of Biomedical and Health Informatics 25, 7 (2021), 2376--2387.

[31]

Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).

[32]

Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. 694--699.

Digital Library

[33]

Alexandra Zamitalo, Qingtong Xie, Mayar Allam, Phinu Philip, Wenqi Shi, et al. 2022. Development of Machine Learning Regression Model for COVID-19 Drug Target Prediction. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2808--2815.

[34]

Yuanda Zhu, Aishwarya Mahale, Kourtney Peters, Lejy Mathew, Felipe Giuste, et al. 2022. Using Natural Language Processing on Free-Text Clinical Notes to Identify Patients with Long-Term COVID Effects. In Proceedings of the 13th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics (BCB '22). Association for Computing Machinery, New York, NY, USA, Article 46, 9 pages.

Digital Library

[35]

Yuchen Zhuang, Yue Yu, Lingkai Kong, Xiang Chen, and Chao Zhang. 2023. Dy-Gen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling. arXiv preprint arXiv:2305.19395 (2023).

Cited By

Tamo JNnamdi MLesbats LShi WZhong YWang M(2023)Uncertainty-Aware Ensemble Learning Models for Out-of-Distribution Medical Imaging Analysis2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)10.1109/BIBM58861.2023.10385418(4243-4250)Online publication date: 5-Dec-2023
https://doi.org/10.1109/BIBM58861.2023.10385418

Index Terms

Model Confidence Calibration for Reliable COVID-19 Early Screening via Audio Signal Analysis
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Rethinking Confidence Calibration for Failure Prediction
Computer Vision – ECCV 2022
Abstract
Reliable confidence estimation for the predictions is important in many safety-critical applications. However, modern deep neural networks are often overconfident for their incorrect predictions. Recently, many calibration methods have been ...
Integrating confidence calibration and adversarial robustness via adversarial calibration entropy
Abstract
The vulnerability of deep neural networks to adversarial samples poses significant security concerns. Previous empirical analyses have shown that increasing adversarial robustness through adversarial training leads to models making unconfident ...
Class-Similarity Based Label Smoothing for Confidence Calibration
Artificial Neural Networks and Machine Learning – ICANN 2021
Abstract
Generating confidence calibrated outputs is of utmost importance for the applications of deep neural networks in safety-critical decision-making systems. The output of a neural network is a probability distribution where the scores are estimated ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

BCB '23: Proceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics

September 2023

626 pages

ISBN:9798400701269

DOI:10.1145/3584371

General Chairs:
May D. Wang
SGeorgia Institute of Technology
,
Byung-Jun Yoon

Copyright © 2023 Owner/Author(s).

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGBio: ACM Special Interest Group on Bioinformatics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 October 2023

Check for updates

Author Tags

Qualifiers

Research-article

Conference

BCB '23

Sponsor:

SIGBio

BCB '23: 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics

September 3 - 6, 2023

TX, Houston, USA

Acceptance Rates

Overall Acceptance Rate 254 of 885 submissions, 29%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
195
Total Downloads

Downloads (Last 12 months)195
Downloads (Last 6 weeks)20

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tamo JNnamdi MLesbats LShi WZhong YWang M(2023)Uncertainty-Aware Ensemble Learning Models for Out-of-Distribution Medical Imaging Analysis2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)10.1109/BIBM58861.2023.10385418(4243-4250)Online publication date: 5-Dec-2023
https://doi.org/10.1109/BIBM58861.2023.10385418

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents