SimTxtSeg: Weakly-Supervised Medical Image Segmentation with Simple Text Cues
Authors: 
Yuxin Xie, Tao Zhou, Yi Zhou, Geng ChenAuthors Info & Claims
Pages 634 - 644
Abstract
Weakly-supervised medical image segmentation is a challenging task that aims to reduce the annotation cost while keep the segmentation performance. In this paper, we present a novel framework, SimTxtSeg, that leverages simple text cues to generate high-quality pseudo-labels and study the cross-modal fusion in training segmentation models, simultaneously. Our contribution consists of two key components: an effective Textual-to-Visual Cue Converter that produces visual prompts from text prompts on medical images, and a text-guided segmentation model with Text-Vision Hybrid Attention that fuses text and image features. We evaluate our framework on two medical image segmentation tasks: colonic polyp segmentation and MRI brain tumor segmentation, and achieve consistent state-of-the-art performance. Source code is available at: https://github.com/xyx1024/SimTxtSeg.
References
[1]
Ali S et al. A multi-centre polyp detection and segmentation dataset for generalisability assessment Sci. Data 2023 10 1 75
[2]
Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilariño, F.: WM-Dova maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians. Comput. Med. Imaging Graphics 43, 99–111 (2015)
[3]
Buda M, Saha A, and Mazurowski MA Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm Comput. Biol. Med. 2019 109 218-225
[4]
Deng G, Zou K, Ren K, Wang M, Yuan X, Ying S, Fu H, et al. Woo J et al. SAM-U: multi-box prompts triggered uncertainty estimation for reliable SAM in medical image MICCAI 2023 2023 Cham Springer 368-377
[5]
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
[6]
Diakogiannis FI, Waldner F, Caccetta P, and Wu C Resunet-a: a deep learning framework for semantic segmentation of remotely sensed data ISPRS J. Photogramm. Remote. Sens. 2020 162 94-114
[7]
Fan D-P, et al., et al. Martel AL, et al., et al. PraNet: parallel reverse attention network for polyp segmentation Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 2020 Cham Springer 263-273
[8]
Gama, P.H., Oliveira, H., dos Santos, J.A.: Learning to segment medical images from few-shot sparse labels. In: 2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 89–96. IEEE (2021)
[9]
Gröger, M., Borisov, V., Kasneci, G.: Boxshrink: From bounding boxes to segmentation masks. In: Workshop on Medical Image Learning with Limited and Noisy Data. pp. 65–75. Springer (2022)
[10]
Hu, X., Chen, Y.J., Ho, T.Y., Shi, Y.: Conditional diffusion models for weakly supervised medical image segmentation. arXiv preprint arXiv:2306.03878 (2023)
[11]
Jha D, et al., et al. Ro YM, et al., et al. Kvasir-SEG: a segmented polyp dataset MultiMedia Modeling 2020 Cham Springer 451-462
[12]
Kirillov, A., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
[13]
Lei, W., Wei, X., Zhang, X., Li, K., Zhang, S.: Medlsam: localize and segment anything model for 3d medical images. arXiv preprint arXiv:2306.14752 (2023)
[14]
Li, Z., Zheng, Y., Luo, X., Shan, D., Hong, Q.: Scribblevc: scribble-supervised medical image segmentation with vision-class embedding (2023)
[15]
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)
[16]
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
[17]
Ma J, He Y, Li F, Han L, You C, and Wang B Segment anything in medical images Nat. Commun. 2024 15 1 654
[18]
MMDetection Contributors: OpenMMLab Detection Toolbox and Benchmark (2018). https://github.com/open-mmlab/mmdetection
[19]
Roth HR, Yang D, Xu Z, Wang X, and Xu D Going to extremes: weakly supervised medical image segmentation Machine Learn. Knowl. Extraction 2021 3 2 507-524
[20]
Silva J, Histace A, Romain O, Dray X, and Granado B Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer Int. J. Comput. Assist. Radiol. Surg. 2014 9 283-293
[21]
Tajbakhsh N, Gurudu SR, and Liang J Automated polyp detection in colonoscopy videos using shape and context information IEEE Trans. Med. Imaging 2015 35 2 630-644
[22]
Wang, A., Xu, M., Zhang, Y., Islam, M., Ren, H.: Sme: spatial-spectral mutual teaching and ensemble learning for scribble-supervised polyp segmentation. arXiv preprint arXiv:2306.00451 (2023)
[23]
Wang, C., Zhang, D., Yan, R.: Boosting weakly-supervised image segmentation via representation, transform, and compensator. arXiv preprint arXiv:2309.00871 (2023)
[24]
Wang R, Lei T, Cui R, Zhang B, Meng H, and Nandi AK Medical image segmentation using deep learning: a survey IET Image Proc. 2022 16 5 1243-1267
[25]
Wang Z, Voiculescu I, et al. Bhattarai B et al. Weakly supervised medical image segmentation through dense combinations of dense pseudo-labels DEMI 2023 2023 Cham Springer 1-10
[26]
Wei J, Hu Y, Cui S, Zhou SK, Li Z, et al. Greenspan H et al. Weakpolyp: You only look bounding box for polyp segmentation MICCAI 2023 2023 Cham Springer 757-766
[27]
Wei J, Hu Y, Li G, Cui S, Kevin Zhou S, and Li Z Wang L, Dou Q, Fletcher PT, Speidel S, and Li S Boxpolyp: boost generalized polyp segmentation using extra coarse bounding box annotations MICCAI 2022 2022 Cham Springer 67-77
[28]
Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I.S., Xie, S.: Convnext v2: co-designing and scaling convnets with masked autoencoders. In: CVPR, pp. 16133–16142 (2023)
[29]
Xie X, Fan H, Yu Z, Bai H, Tang Y, et al. Liu H et al. Weakly-supervised medical image segmentation based on multi-task learning ICIRA 2022 2022 Cham Springer 395-404
[30]
Xu, Y., Gong, M., Xie, S., Batmanghelich, K.: Box-adapt: Domain-adaptive medical image segmentation using bounding boxsupervision (2021)
[31]
Ye, J., et al.: Sa-med2d-20m dataset: segment anything in 2D medical imaging with 20 million masks. arXiv preprint arXiv:2311.11969 (2023)
[32]
Zhong Y, Xu M, Liang K, Chen K, Wu M, et al. Greenspan H et al. Ariadne’s thread: using text prompts to improve segmentation of infected areas from chest x-ray images MICCAI 2023 2023 Cham Springer 724-733
Index Terms
- SimTxtSeg: Weakly-Supervised Medical Image Segmentation with Simple Text Cues
Index terms have been assigned to the content through auto-classification.
Recommendations
Semi-supervised Medical Image Segmentation via Learning Consistency Under Transformations
Medical Image Computing and Computer Assisted Intervention – MICCAI 2019AbstractThe scarcity of labeled data often limits the application of supervised deep learning techniques for medical image segmentation. This has motivated the development of semi-supervised techniques that learn from a mixture of labeled and unlabeled ...
Meta-learners for few-shot weakly-supervised medical image segmentation
AbstractMost uses of Meta-Learning in visual recognition are very often applied to image classification, with a relative lack of work in other tasks such as segmentation and detection. We propose a new generic Meta-Learning framework for few-shot weakly ...
Highlights- Weakly-supervised segmentation via meta-learning with no ImageNet pretraining.
- Proposal of pipelines to adapt classification learners to few-shot segmentation.
- Experiments on medical images for weakly-supervised few-shot ...
Boosted MIML method for weakly-supervised image semantic segmentation
Weakly-supervised image semantic segmentation aims to segment images into semantically consistent regions with only image-level labels are available, and is of great significance for fine-grained image analysis, retrieval and other possible ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Oct 2024
801 pages
ISBN:978-3-031-72110-6
DOI:10.1007/978-3-031-72111-3
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 07 October 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
View options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in