Unleashing the Power of Prompt-Driven Nucleus Instance Segmentation
Authors: 
Zhongyi Shui, Yunlong Zhang, Kai Yao, Chenglu Zhu, Sunyi Zheng, Jingxiong Li, Honglin Li, Yuxuan Sun, Ruizhe Guo, Lin YangAuthors Info & Claims
Pages 288 - 304
Abstract
Nucleus instance segmentation in histology images is crucial for a broad spectrum of clinical applications. Current dominant algorithms rely on regression of nuclear proxy maps. Distinguishing nucleus instances from the estimated maps requires carefully curated post-processing, which is error-prone and parameter-sensitive. Recently, the Segment Anything Model (SAM) has earned huge attention in medical image segmentation, owing to its impressive generalization ability and promptable property. Nevertheless, its potential on nucleus instance segmentation remains largely underexplored. In this paper, we present a novel prompt-driven framework that consists of a nucleus prompter and SAM for automatic nucleus instance segmentation. Specifically, the prompter is developed to generate a unique point prompt for each nucleus, while SAM is fine-tuned to produce its corresponding mask. Furthermore, we propose to integrate adjacent nuclei as negative prompts to enhance model’s capability to identify overlapping nuclei. Without complicated post-processing, our proposed method sets a new state-of-the-art performance on three challenging benchmarks. Code available at https://github.com/windygoo/PromptNucSeg.
References
[1]
Alberts, B., et al.: Essential cell biology. Garland Science (2015)
[2]
Chen, H., Qi, X., Yu, L., Heng, P.A.: Dcan: deep contour-aware networks for accurate gland segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2487–2496 (2016)
[3]
Chen, J., Huang, Q., Chen, Y., Qian, L., Yu, C.: Enhancing nucleus segmentation with haru-net: a hybrid attention based residual u-blocks network. arXiv preprint arXiv:2308.03382 (2023)
[4]
Chen S, Ding C, Liu M, Cheng J, and Tao D Cpp-net: context-aware polygon proposal network for nucleus segmentation IEEE Trans. Image Process. 2023 32 980-994
[5]
Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., et al.: Sam-med2d. arXiv preprint arXiv:2308.16184 (2023)
[6]
Deng, R., et al.: Segment anything model (sam) for digital pathology: assess zero-shot segmentation on whole slide imaging. arXiv preprint arXiv:2304.04155 (2023)
[7]
Deshmukh G, Susladkar O, Makwana D, Mittal S, et al. Feednet: a feature enhanced encoder-decoder lstm network for nuclei instance segmentation for histopathological diagnosis Phys. Med. Biol. 2022 67 19
[8]
Gamper J, Alemi Koohbanani N, Benet K, Khuram A, and Rajpoot N Reyes-Aldasoro CC, Janowczyk A, Veta M, Bankhead P, and Sirinukunwattana K PanNuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification Digital Pathology 2019 Cham Springer 11-19
[9]
Gamper, J., et al.: Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003.10778 (2020)
[10]
Graham S et al. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images Med. Image Anal. 2019 58
[11]
He, H., et al.: Cdnet: centripetal direction network for nuclear instance segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4026–4035 (2021)
[12]
He, H., et al.: Toposeg: topology-aware nuclear instance segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21307–21316 (2023)
[13]
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE International Conference on computer Vision, pp. 2961–2969 (2017)
[14]
Hörst, F., et al.: Cellvit: vision transformers for precise cell segmentation and classification. arXiv preprint arXiv:2306.15350 (2023)
[15]
Huang, Y., et al.: Segment anything model for medical images? Medical Image Analysis p. 103061 (2023)
[16]
Ilyas T, Mannan ZI, Khan A, Azam S, Kim H, and De Boer F Tsfd-net: tissue specific feature distillation network for nuclei segmentation and classification Neural Netw. 2022 151 1-15
[17]
Kirillov, A., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
[18]
Kumar N, Verma R, Sharma S, Bhargava S, Vahadane A, and Sethi A A dataset and a technique for generalized nuclear segmentation for computational pathology IEEE Trans. Med. Imaging 2017 36 7 1550-1560
[19]
Lei, W., Wei, X., Zhang, X., Li, K., Zhang, S.: Medlsam: localize and segment anything model for 3d medical images. arXiv preprint arXiv:2306.14752 (2023)
[20]
Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)
[21]
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
[22]
Lin, X., Xiang, Y., Zhang, L., Yang, X., Yan, Z., Yu, L.: Samus: adapting segment anything model for clinically-friendly and generalizable ultrasound image segmentation. arXiv preprint arXiv:2309.06824 (2023)
[23]
Lou, W., et al.: Structure embedded nucleus classification for histopathology images. arXiv preprint arXiv:2302.11416 (2023)
[24]
Ma, J., Wang, B.: Segment anything in medical images. arXiv preprint arXiv:2304.12306 (2023)
[25]
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. Ieee (2016)
[26]
Na, S., Guo, Y., Jiang, F., Ma, H., Huang, J.: Segment any cell: a sam-based auto-prompting fine-tuning framework for nuclei segmentation. arXiv preprint arXiv:2401.13220 (2024)
[27]
Naylor P, Laé M, Reyal F, and Walter T Segmentation of nuclei in histopathology images by deep regression of the distance map IEEE Trans. Med. Imaging 2018 38 2 448-459
[28]
Qu H, Yan Z, Riedlinger GM, De S, and Metaxas DN Shen D, Liu T, Peters TM, Staib LH, Essert C, Zhou S, Yap P-T, and Khan A Improving nuclei/gland instance segmentation in histopathology images by full resolution neural network and spatial constrained loss Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 2019 Cham Springer 378-386
[29]
Raza SEA, Cheung L, Shaban M, Graham S, Epstein D, Pelengaris S, Khan M, and Rajpoot NM Micro-net: a unified model for segmentation of various objects in microscopy images Med. Image Anal. 2019 52 160-173
[30]
Ronneberger O, Fischer P, and Brox T Navab N, Hornegger J, Wells WM, and Frangi AF U-Net: convolutional networks for biomedical image segmentation Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 2015 Cham Springer 234-241
[31]
Schmidt U, Weigert M, Broaddus C, and Myers G Frangi AF, Schnabel JA, Davatzikos C, Alberola-López C, and Fichtinger G Cell detection with star-convex polygons Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 2018 Cham Springer 265-273
[32]
Song, Q., et al.: Rethinking counting and localization in crowds: a purely point-based framework. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3365–3374 (2021)
[33]
Vu, Q.D., et al.: Methods for segmentation and classification of digital microscopy tissue images. Frontiers in bioengineering and biotechnology p. 53 (2019)
[34]
Wang, H., et al.: Sam-med3d. arXiv preprint arXiv:2310.15161 (2023)
[35]
Wu, J., et al.: Medical sam adapter: adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620 (2023)
[36]
Xu, Q., Kuang, W., Zhang, Z., Bao, X., Chen, H., Duan, W.: Sppnet: a single-point prompt network for nuclei image segmentation. In: International Workshop on Machine Learning in Medical Imaging, pp. 227–236. Springer (2023).
[37]
Yao, K., Huang, K., Sun, J., Hussain, A.: Pointnu-net: Keypoint-assisted convolutional neural network for simultaneous multi-tissue histology nuclei segmentation and classification. IEEE Trans. Emerging Topics Comput. Intell. (2023)
[38]
Zhang, C., et al.:Faster segment anything: towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289 (2023)
[39]
Zhang, K., Liu, D.: Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304.13785 (2023)
[40]
Zhao B, Chen X, Li Z, Yu Z, Yao S, Yan L, Wang Y, Liu Z, Liang C, and Han C Triple u-net: hematoxylin-aware nuclei segmentation with progressive dense feature aggregation Med. Image Anal. 2020 65
[41]
Zhou, Y., Onder, O.F., Dou, Q., Tsougenis, E., Chen, H., Heng, P.A.: Cia-net: robust nuclei instance segmentation with contour-aware information aggregation. In: Information Processing in Medical Imaging: 26th International Conference, IPMI 2019, Hong Kong, China, June 2–7, 2019, Proceedings 26, pp. 682–693. Springer (2019)
Index Terms
- Unleashing the Power of Prompt-Driven Nucleus Instance Segmentation
Index terms have been assigned to the content through auto-classification.
Recommendations
3D Nucleus Instance Segmentation for Whole-Brain Microscopy Images
Information Processing in Medical ImagingAbstractTissue clearing and light-sheet microscopy technologies offer new opportunities to quantify the three-dimensional (3D) neural structure at a cellular or even sub-cellular resolution. Although many efforts have been made to recognize nuclei in 3D ...
Template-driven segmentation of confocal microscopy images
High quality 3D visualization of anatomic structures is necessary for many applications. The anatomic structures first need to be segmented. A variety of segmentation algorithms have been developed for this purpose. For confocal microscopy images, the ...
Cytoplasm and nucleus segmentation in cervical smear images using Radiating GVF Snake
A Radiating Gradient Vector Flow (RGVF) Snake aiming at accurate extraction of both the nucleus and cytoplasm from a single-cell cervical smear image is proposed. After preprocessing, the areas in the image are roughly clustered into nucleus, cytoplasm ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Sep 2024
568 pages
ISBN:978-3-031-73382-6
DOI:10.1007/978-3-031-73383-3
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 03 November 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 30 Nov 2024
Other Metrics
Citations
View Options
View options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in