Abstract
The recently proposed Segment Anything Model (SAM) is a general tool for image segmentation, but it requires additional adaptation and careful fine-tuning for medical image segmentation, especially for small, irregularly-shaped, and boundary-ambiguous anatomical structures such as the knee cartilage that is of interest in this work. Repaired cartilage, after certain surgical procedures, exhibits imaging patterns unseen to pre-training, posing further challenges for using models like SAM with or without general-purpose fine-tuning. To address this, we propose a novel registration-based prompt engineering framework for medical image segmentation using SAM. This approach utilises established image registration algorithms to align the new image (to-be-segmented) and a small number of reference images, without requiring segmentation labels. The spatial transformations generated by registration align either the new image or pre-defined point-based prompts, before using them as input to SAM. This strategy, requiring as few as five reference images with defined point prompts, effectively prompts SAM for inference on new images, without needing any segmentation labels. Evaluation of MR images from patients who received cartilage stem cell therapy yielded Dice scores of 0.89, 0.87, 0.53, and 0.52 for segmenting femur, tibia, femoral- and tibial cartilages, respectively. This outperforms atlas-based label fusion and is comparable to supervised nnUNet, an upper-bound fair baseline in this application, both of which require full segmentation labels for reference samples. The codes are available at: https://github.com/chrissyinreallife/KneeSegmentWithSAM.git.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alirr, O.I., Rahni, A.A.A., Golkar, E.: An automated liver tumour segmentation from abdominal ct scans for hepatic surgical planning. IJCARS 13, 1169–1176 (2018)
Amyar, A., Modzelewski, R., Li, H., Ruan, S.: Multi-task deep learning based ct imaging analysis for covid-19 pneumonia: Classification and segmentation. COMPUT BIOL MED 126, 104037 (2020)
Capobianco, E., Dominietto, M.: Assessment of brain cancer atlas maps with multimodal imaging features. J. Transl. Med. 21(1), 1–11 (2023)
Chalcroft, L.F., Qu, J., Martin, S.A., Gayo, I.J., Minore, G.V., Singh, I.R., et al.: Development and evaluation of intraoperative ultrasound segmentation with negative image frames and multiple observer labels. In: MICCAI Workshop 2021. pp. 25–34. Springer (2021)
Chimutengwende-Gordon, M., Ahmad, M.A., Bentley, G., Brammah, J., Carrington, R., Miles, et al.: Stem cell transplantation for the treatment of osteochondral defects of the knee: Operative technique for a single-stage transplantation procedure using bone marrow-derived mesenchymal stem cells. The Knee 28, 400–409 (2021)
Costea, M., Zlate, A., Durand, M., Baudier, T., Grégoire, V., Sarrut, D., et al.: Comparison of atlas-based and deep learning methods for organs at risk delineation on head-and-neck ct images using an automated treatment planning system. Radiotherapy and Oncology 177, 61–70 (2022)
Czolbe, S., Arnavaz, K., Krause, O., Feragen, A.: Is segmentation uncertainty useful? In: IPMI 2021. pp. 715–726. Springer (2021)
Feyjie, A.R., Azad, R., Pedersoli, M., Kauffman, C., Ayed, I.B., Dolz, J.: Semi-supervised few-shot learning for medical image segmentation. arXiv preprint arXiv:2003.08462 (2020)
Haque, I.R.I., Neubert, J.: Deep learning approaches to biomedical image segmentation. IMU 18, 100297 (2020)
Heimann, T., Morrison, B.J., Styner, M.A., Niethammer, M., Warfield, S.: Segmentation of knee images: a grand challenge. In: Proc. MICCAI Workshop on Medical Image Analysis for the Clinic. vol. 1. Beijing, China (2010)
Huang, J., Jiang, K., Zhang, J., Qiu, H., Lu, L., Lu, S., et al.: Learning to prompt segment anything models. arXiv preprint arXiv:2401.04651 (2024)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
Li, Y., Fu, Y., Gayo, I.J., Yang, Q., Min, Z., Saeed, S.U., et al.: Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration. Med. Image Anal. 90, 102935 (2023)
Litjens, G., Toth, R., Van De Ven, W., Hoeks, C., Kerkstra, S., Van Ginneken, B., et al.: Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Med. Image Anal. 18(2), 359–373 (2014)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 15(1), 654 (2024)
Malhotra, P., Gupta, S., Koundal, D., Zaguia, A., Enbeyle, W., et al.: Deep neural networks for medical image segmentation. J. Healthc. Eng. 2022 (2022)
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3D vision 2016. pp. 565–571. Ieee (2016)
Modat, M., Ridgway, G.R., Taylor, Z.A., Lehmann, M., Barnes, J., Hawkes, D.J., et al.: Fast free-form deformation using graphics processing units. Comput Methods Programs Biomed 98(3), 278–284 (2010)
Montaña-Brown, N., Ramalhinho, J., Allam, M., Davidson, B., Hu, Y., Clarkson, M.J.: Vessel segmentation for automatic registration of untracked laparoscopic ultrasound to ct of the liver. IJCARS 16(7), 1151–1160 (2021)
Nguyen, N.T., Le, P.B.: Topological voting method for image segmentation. Journal of Imaging 8(2), 16 (2022)
Rohlfing, T., Brandt, R., Menzel, R., Russakoff, D.B., Maurer Jr, C.R.: Quo vadis, atlas-based segmentation? In: Handbook of Biomedical Image Analysis: Volume III: Registration Models, pp. 435–486. Springer (2005)
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI 2015. pp. 234–241. Springer (2015)
Rueckert, D., Sonoda, L.I., Hayes, C., Hill, D.L., Leach, M.O., Hawkes, D.J.: Nonrigid registration using free-form deformations: application to breast mr images. IEEE transactions on medical imaging 18(8), 712–721 (1999)
Saeed, S.U., Fu, Y., Stavrinides, V., Baum, Z.M., Yang, Q., Rusu, M., , et al.: Image quality assessment for machine learning tasks using meta-reinforcement learning. Med. Image Anal. 78, 102427 (2022)
Saeed, S.U., Yan, W., Fu, Y., Giganti, F., Yang, Q., Baum, Z., et al.: Image quality assessment by overlapping task-specific and task-agnostic measures: application to prostate multiparametric mr images for cancer segmentation. arXiv preprint arXiv:2202.09798 (2022)
Schreiner, M.M., Raudner, M., Marlovits, S., Bohndorf, K., Weber, M., Zalaudek, M., et al.: The mocart (magnetic resonance observation of cartilage repair tissue) 2.0 knee score and atlas. Cartilage 13(1_suppl), 571S–587S (2021)
Shan, L., Zach, C., Charles, C., Niethammer, M.: Automatic atlas-based three-label cartilage segmentation from mr knee images. Med. Image Anal. 18(7), 1233–1246 (2014)
Siddique, N., Paheding, S., Elkin, C.P., Devabhaktuni, V.: U-net and its variants for medical image segmentation: A review of theory and applications. Ieee Access 9, 82031–82057 (2021)
Tang, H., Liu, X., Sun, S., Yan, X., Xie, X.: Recurrent mask refinement for few-shot medical image segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3918–3928 (2021)
Thippeswamy, P.B., Nedunchelian, M., Rajasekaran, R.B., Riley, D., Khatkar, H., Rajasekaran, et al.: Updates in postoperative imaging modalities following musculoskeletal surgery. JCOT 22, 101616 (2021)
Vorontsov, E., Tang, A., Pal, C., Kadoury, S.: Liver lesion segmentation informed by joint liver segmentation. In: ISBI 2018. pp. 1332–1335. IEEE (2018)
Wang, S., Yang, D.M., Rong, R., Zhan, X., Xiao, G.: Pathology image analysis using segmentation deep learning algorithms. AJP 189(9), 1686–1698 (2019)
Wang, Y., Yao, Q., Kwok, J.T., Ni, L.M.: Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) 53(3), 1–34 (2020)
Acknowledgement
This work was supported in part by EPSRC [EP/T029404/1], Wellcome/EPSRC Centre for Interventional and Surgical Sciences [203145Z/16/Z] and the International Alliance for Cancer Early Detection, a partnership between Cancer Research UK [C73666/A31378], Canary Center at Stanford University, the University of Cambridge, OHSU Knight Cancer Institute, University College London and the University of Manchester.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, Y. et al. (2024). Segmentation by Registration-Enabled SAM Prompt Engineering Using Five Reference Images. In: Modat, M., Simpson, I., Špiclin, Ž., Bastiaansen, W., Hering, A., Mok, T.C.W. (eds) Biomedical Image Registration. WBIR 2024. Lecture Notes in Computer Science, vol 15249. Springer, Cham. https://doi.org/10.1007/978-3-031-73480-9_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-73480-9_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-73479-3
Online ISBN: 978-3-031-73480-9
eBook Packages: Computer ScienceComputer Science (R0)