Abstract
Recently, vision transformers have become outstanding segmentation structures for their remarkable global modeling capability. In current transformer-based models for medical image segmentation, convolutional layers are often replaced by transformers, or transformers are added to the deepest layer of the encoder to learn the global context. However, for the extracted multi-scale feature information, most existing methods tend to ignore the multi-scale dependencies, which leads to inadequate feature learning and fails to produce rich feature representations. In this paper, we propose ScaleNet from the perspective of feature interaction at different scales that can alleviate mentioned problems. Specifically, our approach consists of two multi-scale feature interaction modules: the spatial scale interaction (SSI) and the channel scale interaction (CSI). SSI uses a transformer to aggregate patches from different scale features to enhance the feature representations at the spatial scale. CSI uses a 1D convolutional layer and a fully connected layer to perform a global fusion of multi-level features at the channel scale. The combination of CSI and SSI enables ScaleNet to emphasize multi-scale dependencies and effectively resolve complex scale variations.
This work was supported in part by National Nature Science Foundation of China (62273150), Shanghai Natural Science Foundation (22ZR1421000), Shanghai Outstanding Academic Leaders Plan (21XD1430600), Science and Technology Commission of Shanghai Municipality (22DZ2229004).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J.: UNet++: a nested U-Net architecture for medical image segmentation. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS -2018. LNCS, vol. 11045, pp. 3–11. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00889-5_1
Hyunseok, S., et al.: Modified U-Net (mU-Net) with incorporation of object-dependent high level features for improved liver and liver-tumor segmentation in CT images. IEEE Trans. Med. Imaging 39(5), 1316–1325 (2019)
Myronenko, A.: 3D MRI brain tumor segmentation using autoencoder regularization. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 311–320. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_28
Oktay, O., Schlemper, J., Folgoc, L.L.: Attention U-Net: learning where to look for the pancreas. ArXiv preprint arXiv:1804.03999 (2018)
Luo, W., Li, Y., Urtasun, R., Zemel, R.: Understanding the effective receptive field in deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. ArXiv preprint arXiv:2010.11929 (2020)
Touvron, H., Cord, M., Douze, M., et al.: Training data-efficient image transformers distillation through attention. In: International Conference on Machine Learning, PMLR, pp. 10347–10357 (2021)
Huang, H., et al.: ScaleFormer: revisiting the transformer-based backbones from a scale-wise perspective for medical image segmentation. ArXiv preprint arXiv:2207.14552 (2022)
Chen, J., et al.: TransUNet: transformers make strong encoders for medical image segmentation. ArXiv preprint arXiv:2102.04306 (2021)
Xu, G., Wu, X., Zhang, X., He, X.: LeViT-UNet: make faster encoders with transformer for medical image segmentation. ArXiv preprint arXiv:2107.08623 (2021)
Shi, L., et al.: STM-UNet: an efficient U-shaped architecture based on Swin transformer and multi-scale MLP for medical image segmentation. ArXiv preprint arXiv:2304.12615 (2023)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012–10022 (2021)
Cao, H., et al.: Swin-UNet: UNet-like pure transformer for medical image segmentation. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds.) ECCV 2022. LNCS, vol. 13803, pp. 205–218. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-25066-8_9
Rahman, M.M., Marculescu, R.: Medical image segmentation via cascaded attention decoding. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6222–6231 (2023)
Huang, X., Gong, H., Zhang, J.: HST-MRF: heterogeneous Swin transformer with multi-receptive field for medical image segmentation. ArXiv preprint arXiv:2304.04614 (2023)
Huang, H., et al.: UNet3+: a full-scale connected UNet for medical image segmentation. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1055–1059 (2020)
Huang, X., Deng, Z., Li, D., Yuan, X: MISSFormer: an effective medical image segmentation transformer. ArXiv preprint arXiv:2109.07162 (2021)
You, C., Zhao, R., Liu, F.: Class-aware adversarial transformers for medical image segmentation. In: Advances in Neural Information Processing Systems, vol. 35, pp. 29582–29596 (2022)
Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)
Fu, S., et al.: Domain adaptive relational reasoning for 3D multi-organ segmentation. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12261, pp. 656–666. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_64
Schlemper, J., Oktay, O., Schaap, M.: Attention gated networks: learning to leverage salient regions in medical images. Med. Image Anal. 53, 197–207 (2019)
Wang, H., Xie, S., Lin, L.: Mixed transformer U-Net for medical image segmentation. In: Proceedings of the ICASSP, pp. 2390–2394 (2022)
Heidari, M., et al.: HiFormer: hierarchical multi-scale representations using transformers for medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6202–6212 (2023)
Ben, L., et al.: Segmentation outside the cranial vault challenge. In: MICCAI: Multi Atlas Labeling Beyond Cranial Vault-Workshop Challenge (2015)
Bernard, O., Lalande, A., Zotti, C.: Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved. IEEE Trans. Med. Imaging 37(11), 2514–2525 (2018)
Codella, N., et al.: Skin lesion analysis toward melanoma detection, a challenge hosted by the international skin imaging collaboration (ISIC). ArXiv preprint arXiv:1902.03368 (2019)
Azad, R., Heidari, M., Merhof, D.: TransNorm: transformer provides a strong spatial normalization mechanism for a deep segmentation model. IEEE Access 10, 108205–108215 (2022)
Azad, R., Bozorgpour, A., Asadi-Aghbolaghi, M., Merhof, D., Escalera, S.: Deep frequency re-calibration U-Net for medical image segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3274–3283 (2021)
Azad, R., Heidari, M., Wu, Y.: Contextual attention network: transformer meets U-Net. ArXiv preprint arXiv:2203.01932 (2022)
Azad, R., Jia, Y., Aghdam, E.K., Cohen-Adad, J., Merhof, D.: Enhancing medical image segmentation with TransCeption: a multi-scale feature fusion approach. ArXiv preprint arXiv:2301.10847 (2023)
Zongwei, W., Guillaume, A., Fabrice, M.: HiDAnet: RGB-D salient object detection via hierarchical depth awareness. IEEE Trans. Image Process. 32, 2160–2173 (2023)
Zhou, T., Fu, H., Chen, G.: Specificity-preserving RGB-D saliency detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4681–4691 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Feng, Y., Ma, T., Zeng, H., Xu, Z., Zhang, S., Wen, Y. (2024). ScaleNet: Rethinking Feature Interaction from a Scale-Wise Perspective for Medical Image Segmentation. In: Sheng, B., Bi, L., Kim, J., Magnenat-Thalmann, N., Thalmann, D. (eds) Advances in Computer Graphics. CGI 2023. Lecture Notes in Computer Science, vol 14498. Springer, Cham. https://doi.org/10.1007/978-3-031-50078-7_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-50078-7_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50077-0
Online ISBN: 978-3-031-50078-7
eBook Packages: Computer ScienceComputer Science (R0)