Abstract
Longitudinal lesion or tumor tracking is an essential task in different clinical workflows, including treatment monitoring with follow-up imaging or planning of re-treatments for radiation therapy. Accurately establishing correspondence between lesions at different timepoints, recognizing new lesions or lesions that have disappeared is a tedious task that only grows in complexity as the number of lesions or timepoints increase. To address this task, we propose a generic approach based on multi-scale self-supervised learning. The multi-scale approach allows the efficient and robust learning of a similarity map between multi-timepoint image acquisitions to derive correspondence, while the self-supervised learning formulation enables the generic application to different types of lesions and image modalities. In addition, we impose optional supervision during training by leveraging tens of anatomical landmarks that can be extracted automatically. We train our approach at large scale with more than 50,000 computed tomography (CT) scans and validate it on two different applications: 1) Tracking of generic lesions based on the DeepLesion dataset, including liver tumors, lung nodules, enlarged lymph-nodes, for which we report highest matching accuracy of 92%, with localization accuracy that is nearly 10% higher than the state-of-the-art; and 2) Tracking of lung nodules based on the NLST dataset for which we achieve similarly high performance. In addition, we include an error analysis based on expert radiologist feedback, and discuss next steps as we plan to scale our system across more applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Eisenhauer, E.A., et al.: New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur. J. Cancer 45(2), 228–247 (2009)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR, pp. 4724–4733 (2017)
Yan, K., et al.: SAM: self-supervised learning of pixel-wise anatomical embeddings in radiological images. IEEE Trans. Med. Imaging 41, 2658–2669 (2020)
Ilya, L., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2017)
Ilya, L., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. ArXiv: Learning (2016)
Cai, J., et al.: Deep lesion tracker: monitoring lesions in 4D longitudinal imaging studies. In: CVPR, pp. 15154–15164 (2020)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations (2020)
Van Den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding (2018)
Yan, K., Wang, X., Lu, L., Summers, R.M.: DeepLesion: automated deep mining, categorization and detection of significant radiology image findings using large-scale clinical lesion annotations (2017)
National Lung Screening Trial Research Team: The National Lung Screening Trial: overview and study design. In: Radiology, pp. 243–253 (2011)
Perez-Garcia, F., Sparks, R., Ourselin, S.: TorchIO: a python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. Comput. Methods Prog. Biomed. 208, 106236 (2020)
Ghesu, F.C., et al.: Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans. IEEE Trans. Pattern Anal. Mach. Intell. 41, 176–189 (2019)
Marstal, K., Berendsen, F.F., Staring, M., Klein, S.: SimpleElastix: a user-friendly, multi-lingual library for medical image registration. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 574–582 (2016)
Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J.V., Dalca, A.V.: An unsupervised learning model for deformable medical image registration. In: CVPR, pp. 9252–9260 (2018)
Yan, K., et al.: Learning from multiple datasets with heterogeneous and partial labels for universal lesion detection in CT. IEEE Trans. Med. Imaging 40, 2759–2770 (2020)
Cai, J., et al.: Deep volumetric universal lesion detection using light-weight pseudo 3D convolution and surface point regression (2020)
Yan, K., et al.: Deep lesion graphs in the wild: relationship learning and organization of significant radiology image findings in a diverse large-scale lesion database. In: Conference on Computer Vision and Pattern Recognition, pp. 9261–9270 (2017)
Yan, K., Peng, Y., Sandfort, V., Bagheri, M., Lu, Z., Summers, R.M.: Holistic and comprehensive annotation of clinically significant findings on diverse CT images: learning from radiology reports and label ontology. In: CVPR, pp. 8515–8524 (2019)
Heinrich, M.P., Jenkinson, M., Brady, M., Schnabel, J.A.: MRF-based deformable registration and ventilation estimation of lung CT. IEEE Trans. Med. Imaging 32, 1239–1248 (2013)
Acknowledgements
The authors thank the National Cancer Institute for access to NCI’s data collected by the National Lung Screening Trial (NLST). The statements contained herein are solely those of the authors and do not represent or imply concurrence or endorsement by NCI.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Vizitiu, A., Mohaiu, A.T., Popdan, I.M., Balachandran, A., Ghesu, F.C., Comaniciu, D. (2023). Multi-scale Self-Supervised Learning for Longitudinal Lesion Tracking with Optional Supervision. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14220. Springer, Cham. https://doi.org/10.1007/978-3-031-43907-0_55
Download citation
DOI: https://doi.org/10.1007/978-3-031-43907-0_55
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43906-3
Online ISBN: 978-3-031-43907-0
eBook Packages: Computer ScienceComputer Science (R0)