SoQal: Selective Oracle Questioning for Consistency Based Active Learning of Cardiac Signals

Dani Kiyasseh, Tingting Zhu, David A Clifton
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:11302-11340, 2022.

Abstract

Clinical settings are often characterized by abundant unlabelled data and limited labelled data. This is typically driven by the high burden placed on oracles (e.g., physicians) to provide annotations. One way to mitigate this burden is via active learning (AL) which involves the (a) acquisition and (b) annotation of informative unlabelled instances. Whereas previous work addresses either one of these elements independently, we propose an AL framework that addresses both. For acquisition, we propose Bayesian Active Learning by Consistency (BALC), a sub-framework which perturbs both instances and network parameters and quantifies changes in the network output probability distribution. For annotation, we propose SoQal, a sub-framework that dynamically determines whether, for each acquired unlabelled instance, to request a label from an oracle or to pseudo-label it instead. We show that BALC can outperform start-of-the-art acquisition functions such as BALD, and SoQal outperforms baseline methods even in the presence of a noisy oracle.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-kiyasseh22a, title = {{S}o{Q}al: Selective Oracle Questioning for Consistency Based Active Learning of Cardiac Signals}, author = {Kiyasseh, Dani and Zhu, Tingting and Clifton, David A}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {11302--11340}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/kiyasseh22a/kiyasseh22a.pdf}, url = {https://proceedings.mlr.press/v162/kiyasseh22a.html}, abstract = {Clinical settings are often characterized by abundant unlabelled data and limited labelled data. This is typically driven by the high burden placed on oracles (e.g., physicians) to provide annotations. One way to mitigate this burden is via active learning (AL) which involves the (a) acquisition and (b) annotation of informative unlabelled instances. Whereas previous work addresses either one of these elements independently, we propose an AL framework that addresses both. For acquisition, we propose Bayesian Active Learning by Consistency (BALC), a sub-framework which perturbs both instances and network parameters and quantifies changes in the network output probability distribution. For annotation, we propose SoQal, a sub-framework that dynamically determines whether, for each acquired unlabelled instance, to request a label from an oracle or to pseudo-label it instead. We show that BALC can outperform start-of-the-art acquisition functions such as BALD, and SoQal outperforms baseline methods even in the presence of a noisy oracle.} }
Endnote
%0 Conference Paper %T SoQal: Selective Oracle Questioning for Consistency Based Active Learning of Cardiac Signals %A Dani Kiyasseh %A Tingting Zhu %A David A Clifton %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-kiyasseh22a %I PMLR %P 11302--11340 %U https://proceedings.mlr.press/v162/kiyasseh22a.html %V 162 %X Clinical settings are often characterized by abundant unlabelled data and limited labelled data. This is typically driven by the high burden placed on oracles (e.g., physicians) to provide annotations. One way to mitigate this burden is via active learning (AL) which involves the (a) acquisition and (b) annotation of informative unlabelled instances. Whereas previous work addresses either one of these elements independently, we propose an AL framework that addresses both. For acquisition, we propose Bayesian Active Learning by Consistency (BALC), a sub-framework which perturbs both instances and network parameters and quantifies changes in the network output probability distribution. For annotation, we propose SoQal, a sub-framework that dynamically determines whether, for each acquired unlabelled instance, to request a label from an oracle or to pseudo-label it instead. We show that BALC can outperform start-of-the-art acquisition functions such as BALD, and SoQal outperforms baseline methods even in the presence of a noisy oracle.
APA
Kiyasseh, D., Zhu, T. & Clifton, D.A.. (2022). SoQal: Selective Oracle Questioning for Consistency Based Active Learning of Cardiac Signals. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:11302-11340 Available from https://proceedings.mlr.press/v162/kiyasseh22a.html.

Related Material