Abstract
New lesion segmentation is essential to estimate the disease progression and therapeutic effects during multiple sclerosis (MS) clinical treatments. However, the expensive data acquisition and expert annotation restrict the feasibility of applying large-scale deep learning models. Since single-time-point samples with all-lesion labels are relatively easy to collect, exploiting them to train deep models is highly desirable to improve new lesion segmentation. Therefore, we proposed a coaction segmentation (CoactSeg) framework to exploit the heterogeneous data (i.e., new-lesion annotated two-time-point data and all-lesion annotated single-time-point data) for new MS lesion segmentation. The CoactSeg model is designed as a unified model, with the same three inputs (the baseline, follow-up, and their longitudinal brain differences) and the same three outputs (the corresponding all-lesion and new-lesion predictions), no matter which type of heterogeneous data is being used. Moreover, a simple and effective relation regularization is proposed to ensure the longitudinal relations among the three outputs to improve the model learning. Extensive experiments demonstrate that utilizing the heterogeneous data and the proposed longitudinal relation constraint can significantly improve the performance for both new-lesion and all-lesion segmentation tasks. Meanwhile, we also introduce an in-house MS-23v1 dataset, including 38 Oceania single-time-point samples with all-lesion labels. Codes and the dataset are released at https://github.com/ycwu1997/CoactSeg.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Carass, A., et al.: Longitudinal multiple sclerosis lesion segmentation: resource and challenge. NeuroImage 148, 77–102 (2017)
Commowick, O., Cervenansky, F., Cotton, F., Dojat, M.: Msseg-2 challenge proceedings: multiple sclerosis new lesions segmentation challenge using a data management and processing infrastructure. In: MICCAI 2021, p. 126 (2021)
Commowick, O., et al.: Objective evaluation of multiple sclerosis lesion segmentation using a data management and processing infrastructure. Sci. Rep. 8(1), 13650 (2018)
Commowick, O., et al.: Multiple sclerosis lesions segmentation from multiple experts: the MICCAI 2016 challenge dataset. Neuroimage 244, 118589 (2021)
Gessert, N., et al.: 4d deep learning for multiple-sclerosis lesion activity segmentation. In: MIDL 2020 (2020)
Gessert, N., et al.: Multiple sclerosis lesion activity segmentation with attention-guided two-path CNNs. Computer. Med. Imaging Graph. 84, 101772 (2020)
Gold, R., et al.: Placebo-controlled phase 3 study of oral bg-12 for relapsing multiple sclerosis. N. Engl. J. Med. 367(12), 1098–1107 (2012)
He, T., et al.: MS or not MS: T2-weighted imaging (t2wi)-based radiomic findings distinguish MS from its mimics. Multip. Sclerosis Relat. Disord. 61, 103756 (2022)
Krüger, J., et al.: Fully automated longitudinal segmentation of new or enlarged multiple sclerosis lesions using 3d convolutional neural networks. NeuroImage: Clin. 28, 102445 (2020)
La Rosa, F., et al.: Multiple sclerosis cortical and WM lesion segmentation at 3t MRI: a deep learning method based on flair and mp2rage. NeuroImage: Clin. 27, 102335 (2020)
Ma, Y., et al.: Multiple sclerosis lesion analysis in brain magnetic resonance images: techniques and clinical applications. IEEE J. Biomed. Health Inf. 26(6), 2680–2692 (2022)
Macar, U., Karthik, E.N., Gros, C., Lemay, A., Cohen-Adad, J.: Team neuropoly: description of the pipelines for the MICCAI 2021 MS new lesions segmentation challenge. arXiv preprint arXiv:2109.05409 (2021)
Maier-Hein, L., et al.: Metrics reloaded: pitfalls and recommendations for image analysis validation. arXiv preprint arXiv:2206.01653 (2022)
Mariano, C., Yuling, L., Kain, K., Linda, L., Chenyu, W., Michael, B.: Estimating lesion activity through feature similarity: a dual path UNET approach for the msseg2 MICCAI challenge. https://github.com/marianocabezas/msseg2
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 3DV 2016, pp. 565–571 (2016)
Rakić, M., et al.: icobrain MS 5.1: combining unsupervised and supervised approaches for improving the detection of multiple sclerosis lesions. NeuroImage: Clin. 31, 102707 (2021)
Reuter, M., Schmansky, N.J., Rosas, H.D., Fischl, B.: Within-subject template estimation for unbiased longitudinal image analysis. Neuroimage 61(4), 1402–1418 (2012)
Schell, M., et al.: Automated brain extraction of multi-sequence MRI using artificial neural networks. In: Human Brain Mapping, pp. 1–13 (2019)
Sharmin, S., et al.: Confirmed disability progression as a marker of permanent disability in multiple sclerosis. Eur. J. Neurol. 29(8), 2321–2334 (2022)
Tang, Z., Cabezas, M., Liu, D., Barnett, M., Cai, W., Wang, C.: LG-net: lesion gate network for multiple sclerosis lesion inpainting. In: de Bruijne, M. et al. (eds.) MICCAI 2021, pp. 660–669. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87234-2_62
Wolleb, J., et al.: Learn to ignore: domain adaptation for multi-site MRI analysis. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022, pp. 725–735. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16449-1_69
Wu, Y., Ge, Z., Zhang, D., Xu, M., Zhang, L., Xia, Y., Cai, J.: Mutual consistency learning for semi-supervised medical image segmentation. Med. Image Anal. 81, 102530 (2022)
Wu, Y., Wu, Z., Wu, Q., Ge, Z., Cai, J.: Exploring smoothness and class-separation for semi-supervised medical image segmentation. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022, vol. 13435, pp. 34–43. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16443-9_4
Wu, Y., Xu, M., Ge, Z., Cai, J., Zhang, L.: Semi-supervised left atrium segmentation with mutual consistency training. In: de Bruijne, M., et al. (eds.) MICCAI 2021, pp. 297–306. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_28
Zeng, C., Gu, L., Liu, Z., Zhao, S.: Review of deep learning approaches for the segmentation of multiple sclerosis lesions on brain MRI. Front. Neuroinform. 14, 610967 (2020)
Zhang, H., et al.: Qsmrim-net: imbalance-aware learning for identification of chronic active multiple sclerosis lesions on quantitative susceptibility maps. NeuroImage: Clin. 34, 102979 (2022)
Zhang, H., Wang, R., Zhang, J., Liu, D., Li, C., Li, J.: Spatially covariant lesion segmentation. arXiv preprint arXiv:2301.07895 (2023)
Zhang, H., et al.: All-net: anatomical information lesion-wise loss function integrated into neural network for multiple sclerosis lesion segmentation. NeuroImage: Clin. 32, 102854 (2021)
Zhang, H., Yuan, X., Nguyen, Q.V.H., Pan, S.: On the interaction between node fairness and edge privacy in graph neural networks. arXiv preprint arXiv:2301.12951 (2023)
Zhang, J., Xie, Y., Xia, Y., Shen, C.: Dodnet: learning to segment multi-organ and tumors from multiple partially labeled datasets. In: CVPR 2021, pp. 1195–1204 (2021)
Acknowledgement
This work was supported in part by the Monash FIT Start-up Grant, in part by the Novartis (ID: 76765455), and in part by the Monash Institute of Medical Engineering (MIME) Project: 2022-13. We here appreciate the public repositories of SNAC [14] and Neuropoly [12], and also thanks for the efforts to collect and share the MS dataset [2] and the MS-23v1 dataset from Alfred Health, Australia.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wu, Y., Wu, Z., Shi, H., Picker, B., Chong, W., Cai, J. (2023). CoactSeg: Learning from Heterogeneous Data for New Multiple Sclerosis Lesion Segmentation. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14227. Springer, Cham. https://doi.org/10.1007/978-3-031-43993-3_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-43993-3_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43992-6
Online ISBN: 978-3-031-43993-3
eBook Packages: Computer ScienceComputer Science (R0)