Active learning for semantic segmentation with multi-class label query
AUTHORs: 
Sehyun Hwang, Sohyun Lee, Hoyoung Kim, Minhyeon Oh, Jungseul Ok, Suha KwakAuthors Info & Claims
Article No.: 1175, Pages 27020 - 27039
Abstract
This paper proposes a new active learning method for semantic segmentation. The core of our method lies in a new annotation query design. It samples informative local image regions (e.g., superpixels), and for each of such regions, asks an oracle for a multi-hot vector indicating all classes existing in the region. This multi-class labeling strategy is substantially more efficient than existing ones like segmentation, polygon, and even dominant class labeling in terms of annotation time per click. However, it introduces the class ambiguity issue in training as it assigns partial labels (i.e., a set of candidate classes) to individual pixels. We thus propose a new algorithm for learning semantic segmentation while disambiguating the partial labels in two stages. In the first stage, it trains a segmentation model directly with the partial labels through two new loss functions motivated by partial label learning and multiple instance learning. In the second stage, it disambiguates the partial labels by generating pixel-wise pseudo labels, which are used for supervised learning of the model. Equipped with a new acquisition function dedicated to the multi-class labeling, our method outperforms previous work on Cityscapes and PASCAL VOC 2012 while spending less annotation cost. Our code and results are available at https://github.com/sehyun03/MulActSeg.
References
[1]
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11):2274-2282, 2012.
[2]
Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
[3]
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[4]
Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[5]
Nabiha Asghar, Pascal Poupart, Xin Jiang, and Hang Li. Deep active learning for dialogue generation. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM), 2017.
[6]
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In Proc. International Conference on Learning Representations (ICLR), 2020.
[7]
Clemens-Alexander Brust, Christoph Käding, and Joachim Denzler. Active learning for deep object detection. arXiv preprint arXiv:1809.09875, 2018.
[8]
Vivien Cabannnes, Alessandro Rudi, and Francis Bach. Structured prediction with partial labelling through the infimum loss. In Proc. International Conference on Machine Learning (ICML), 2020.
[9]
Lile Cai, Xun Xu, Jun Hao Liew, and Chuan Sheng Foo. Revisiting superpixels for active learning in semantic segmentation with realistic annotation costs. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[10]
Arantxa Casanova, Pedro O Pinheiro, Negar Rostamzadeh, and Christopher J Pal. Reinforced active learning for image segmentation. In Proc. International Conference on Learning Representations (ICLR),
[11]
Liyi Chen, Weiwei Wu, Chenchen Fu, Xiao Han, and Yuntao Zhang. Weakly supervised semantic segmentation with boundary exploration. In Proc. European Conference on Computer Vision (ECCV), 2020.
[12]
Liang-Chieh Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D. Collins, Ekin D. Cubuk, Barret Zoph, Hartwig Adam, and Jonathon Shlens. Naive-student: Leveraging semi-supervised learning in video sequences for urban scene segmentation. In Proc. European Conference on Computer Vision (ECCV), 2020.
[13]
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proc. European Conference on Computer Vision (ECCV), 2018.
[14]
Pascal Colling, Lutz Roese-Koerner, Hanno Gottschalk, and Matthias Rottmann. Metabox+: A new region based active learning method for semantic segmentation using priority maps. In International Conference on Pattern Recognition Applications and Methods (ICPRAM), 2021.
[15]
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[16]
Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. In Journal of Machine Learning Research (JMLR), 2011.
[17]
Chengliang Dai, Shuo Wang, Yuanhan Mo, Kaichen Zhou, Elsa Angelini, Yike Guo, and Wenjia Bai. Suggestive annotation of brain tumour images with gradient-guided sampling. In Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2020.
[18]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: a large-scale hierarchical image database. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[19]
Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 1997.
[20]
Zhang Dong, Zhang Hanwang, Tang Jinhui, Hua Xiansheng, and Sun Qianru. Causal intervention for weakly supervised semantic segmentation. In Proc. Neural Information Processing Systems (NeurIPS), 2020.
[21]
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision (IJCV), 2010.
[22]
Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. Proc. Neural Information Processing Systems (NeurIPS), 33:10948-10960, 2020.
[23]
S Alireza Golestaneh and Kris M Kitani. Importance of self-consistency in active learning for semantic segmentation. In Proc. British Machine Vision Conference (BMVC), 2020.
[24]
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In Proc. International Conference on Learning Representations (ICLR), 2022.
[25]
Ruifei He, Jihan Yang, and Xiaojuan Qi. Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[26]
Tao He, Xiaoming Jin, Guiguang Ding, Lan Yi, and Chenggang Yan. Towards better uncertainty sampling: Active learning with multiple views for deep convolutional neural network. In IEEE International Conference on Multimedia and Expo (ICME), 2019.
[27]
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 558-567, 2019.
[28]
Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang. Weakly-supervised semantic segmentation network with deep seeded region growing. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[29]
Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelligent Data Analysis (IDA), 2006.
[30]
Sehyun Hwang, Sohyun Lee, Sungyeon Kim, Jungseul Ok, and Suha Kwak. Combating label distribution shift for active domain adaptation. In Proc. European Conference on Computer Vision (ECCV), pages 549-566. Springer, 2022.
[31]
Suyog Dutt Jain and Kristen Grauman. Active image segmentation propagation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[32]
Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[33]
Tarun Kalluri, Girish Varma, Manmohan Chandraker, and C V Jawahar. Universal semi-supervised semantic segmentation. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[34]
Dahyun Kang, Piotr Koniusz, Minsu Cho, and Naila Murray. Distilling self-supervised vision transformers for weakly-supervised few-shot classification & segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19627-19638, 2023.
[35]
Tejaswi Kasarla, Gattigorla Nagendar, Guruprasad M Hegde, Vineeth Balasubramanian, and CV Jawahar. Region-based active learning for efficient labeling in semantic segmentation. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2019.
[36]
Rihuan Ke, Angelica Aviles-Rivero, Saurabh Pandey, Saikumar Reddy, and Carola-Bibiane Schönlieb. A three-stage self-training framework for semi-supervised semantic segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[37]
Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson W.H. Lau. Guided collaborative training for pixel-wise semi-supervised learning. In Proc. European Conference on Computer Vision (ECCV), 2020.
[38]
Hoyoung Kim, Minhyeon Oh, Sehyun Hwang, Suha Kwak, and Jungseul Ok. Adaptive superpixel for active learning in semantic segmentation. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[39]
Namyup Kim, Sehyun Hwang, and Suha Kwak. Learning to detect semantic boundaries with image-level class labels. International Journal of Computer Vision (IJCV), 130(9):2131-2148, 2022.
[40]
Suha Kwak, Seunghoon Hong, and Bohyung Han. Weakly supervised semantic segmentation using superpixel pooling network. In Proc. AAAI Conference on Artificial Intelligence (AAAI), 2017.
[41]
Donghyeon Kwon and Suha Kwak. Semi-supervised semantic segmentation with error localization network. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[42]
Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semi-supervised semantic segmentation with directional context-aware consistency. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[43]
Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[44]
Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[45]
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. International Conference on Learning Representations (ICLR), 2019.
[46]
Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proc. International Conference on Machine Learning (ICML), 2020.
[47]
Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. Cereals-cost-effective region-based active learning for semantic segmentation. In Proc. British Machine Vision Conference (BMVC), 2018.
[48]
Robert Mendel, Luis Antonio de Souza Jr, David Rauber, Joäo Paulo Papa, and Christoph Palm. Semi-supervised segmentation based on error-correcting supervision. In Proc. European Conference on Computer Vision (ECCV), 2020.
[49]
Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmentation with high- and low level consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.
[50]
Natalia Ostapuk, Jie Yang, and Philippe Cudré-Mauroux. Activelink: deep active learning for link prediction in knowledge graphs. In The World Wide Web Conference (WWW), 2019.
[51]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In AutoDiff, NIPS Workshop, 2017.
[52]
Congyu Qiao, Ning Xu, and Xin Geng. Decompositional generation process for instance-dependent partial label learning. In Proc. International Conference on Learning Representations (ICLR), 2023.
[53]
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proc. International Conference on Learning Representations (ICLR), 2018.
[54]
Gyungin Shin, Weidi Xie, and Samuel Albanie. All you need are a few pixels: semantic segmentation with pixelpick. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[55]
Yawar Siddiqui, Julien Valentin, and Matthias Nießner. Viewal: Active learning with viewpoint entropy for semantic segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[56]
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[57]
David Stutz, Alexander Hermans, and Bastian Leibe. Superpixels: An evaluation of the state-of-the-art. Computer Vision and Image Understanding, 166:1-27, 2018.
[58]
Guolei Sun, Wenguan Wang, Jifeng Dai, and Luc Van Gool. Mining cross-image semantics for weakly supervised semantic segmentation. In Proc. European Conference on Computer Vision (ECCV), 2020.
[59]
Michael Van den Bergh, Xavier Boix, Gemma Roig, Benjamin de Capitani, and Luc Van Gool. Seeds: Superpixels extracted via energy-driven sampling. In Proc. European Conference on Computer Vision (ECCV), 2012.
[60]
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[61]
Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. In Proc. Neural Information Processing Systems (NeurIPS), 2022.
[62]
Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2016.
[63]
Xiang Wang, Shaodi You, Xi Li, and Huimin Ma. Weakly-supervised semantic segmentation by iteratively mining common object features. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[64]
Zengmao Wang, Bo Du, Weiping Tu, Lefei Zhang, and Dacheng Tao. Incorporating distribution matching into uncertainty for multiple kernel active learning. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2019.
[65]
Zheng Wang and Jieping Ye. Querying discriminative and representative samples for batch mode active learning. ACM Transactions on Knowledge Discovery from Data (TKDD), 2015.
[66]
Tsung-Han Wu, Yi-Syuan Liou, Shao-Ji Yuan, Hsin-Ying Lee, Tung-I Chen, Kuan-Chih Huang, and Winston H Hsu. D2ada: Dynamic density-aware active domain adaptation for semantic segmentation. In Proc. European Conference on Computer Vision (ECCV), pages 449-467, 2022.
[67]
Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, and Xinjing Cheng. Towards fewer annotations: Active learning via region impurity and prediction uncertainty for domain adaptive semantic segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[68]
Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2017.
[69]
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Recommendations
Multi-label active learning by model guided distribution matching
Multi-label learning is an effective framework for learning with objects that have multiple semantic labels, and has been successfully applied into many real-world tasks. In contrast with traditional single-label learning, the cost of labeling a multi-...
Addressing class-imbalance in multi-label learning via two-stage multi-label hypernetwork
Multi-label learning is concerned with learning from data examples that are represented by a single feature vector while associated with multiple labels simultaneously. Existing multi-label learning approaches mainly focus on exploiting label ...
Weak Labeled Multi-Label Active Learning for Image Classification
MM '15: Proceedings of the 23rd ACM international conference on MultimediaIn order to achieve better classification performance with even fewer labeled images, active learning is suitable for these situations. Several active learning methods have been proposed for multi-label image classification, but all of them assume that ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2023 Neural Information Processing Systems Foundation, Inc.
Publisher
Curran Associates Inc.
Red Hook, NY, United States
Publication History
Published: 30 May 2024
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 18 Nov 2024