Automatic Registration of SHG and H&E Images with Feature-Based Initial Alignment and Intensity-Based Instance Optimization: Contribution to the COMULIS Challenge
Pages 346 - 357
Abstract
The automatic registration of noninvasive second-harmonic generation microscopy to hematoxylin and eosin slides is a highly desired, yet still unsolved problem. The task is challenging because the second-harmonic images contain only partial information, in contrast to the stained H&E slides that provide more information about the tissue morphology. Moreover, both imaging methods have different intensity distributions. Therefore, the task can be formulated as a multi-modal registration problem with missing data. In this work, we propose a method based on automatic keypoint matching followed by deformable registration based on instance optimization. The method does not require any training and is evaluated using the dataset provided in the Learn2Reg challenge by the COMULIS organization. The method achieved relatively good generalizability resulting in 88% of success rate in the initial alignment and average target registration error equal to 2.48 on the external validation set. We openly release the source code and incorporate it in the DeeperHistReg image registration framework.
References
[1]
Alegro, M., Amaro-Jr, E., Loring, B., Heinsen, H., Alho, E., Zollei, L., Ushizima, D., Grinberg, L.T.: Multimodal whole brain registration: MRI and high resolution histology. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 194–202 (2016)
[2]
Alyami, W., Kyme, A., Bourne, R.: Histological validation of MRI: A review of challenges in registration of imaging and whole-mount histopathology. Journal of Magnetic Resonance Imaging 55(1), 11–22 (2022)
[3]
Awan, R., Raza, S.E.A., Lotz, J., Weiss, N., Rajpoot, N.: Deep feature based cross-slide registration. Computerized Medical Imaging and Graphics 104, 102162 (2023)
[4]
Borovec, J., Kybic, J., Arganda-Carreras, I., Sorokin, D.V., Bueno, G., Khvostikov, A.V., Bakas, S., Eric, I., Chang, C., Heldmann, S., et al.: ANHIR: automatic non-rigid histological image registration challenge. IEEE Transactions on Medical Imaging 39(10), 3042–3052 (2020)
[5]
DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: Self-supervised interest point detection and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 224–236 (2018)
[6]
Eliceiri, K., Li, B., Keikhosravi, A.: Multimodal biomedical dataset for evaluating registration methods (patches from tma cores). Zenodo, June (2020)
[7]
Gatenbee, C.D., Baker, A.M., Prabhakaran, S., Swinyard, O., Slebos, R.J., Mandal, G., Mulholland, E., Andor, N., Marusyk, A., Leedham, S., et al.: Virtual alignment of pathology image series for multi-gigapixel whole slide images. Nature Communications 14(1), 4502 (2023)
[8]
Ge, L., Wei, X., Hao, Y., Luo, J., Xu, Y.: Unsupervised histological image registration using structural feature guided convolutional neural network. IEEE Transactions on Medical Imaging 41(9), 2414–2431 (2022)
[9]
Hering, A., Hansen, L., Mok, T.C., Chung, A.C., Siebert, H., Häger, S., Lange, A., Kuckertz, S., Heldmann, S., Shao, W., et al.: Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning. IEEE Transactions on Medical Imaging 42(3), 697–712 (2022)
[10]
Jiang, H., Karpur, A., Cao, B., Huang, Q., Araujo, A.: OmniGlue: Generalizable Feature Matching with Foundation Model Guidance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19865–19875 (2024)
[11]
Keikhosravi, A., Li, B., Liu, Y., Eliceiri, K.W.: Intensity-based registration of bright-field and second-harmonic generation images of histopathology tissue sections. Biomedical Optics Express 11(1), 160–173 (2020)
[12]
Lin, Y., Liang, Z., He, Y., Huang, W., Guan, T.: End-to-end affine registration framework for histopathological images with weak annotations. Computer Methods and Programs in Biomedicine 241, 107763 (2023)
[13]
Lotz, J., Olesch, J., Müller, B., Polzin, T., Galuschka, P., Lotz, J., Heldmann, S., Laue, H., González-Vallinas, M., Warth, A., et al.: Patch-based nonlinear image registration for gigapixel whole slide images. IEEE Transactions on Biomedical Engineering 63(9), 1812–1819 (2015)
[14]
Lotz, J., Weiss, N., van der Laak, J., Heldmann, S.: Comparison of consecutive and restained sections for image registration in histopathology. Journal of Medical Imaging (2023)
[15]
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 91–110 (2004)
[16]
Marzahl, C., Wilm, F., Tharun, L., Perner, S., Bertram, C.A., Kröger, C., Voigt, J., Klopfleisch, R., Maier, A., Aubreville, M., et al.: Robust quad-tree based registration on whole slide images. In: MICCAI Workshop on Computational Pathology. pp. 181–190. PMLR (2021)
[17]
Pyatov, V., Sorokin, D.: Affine registration of histological images using transformer-based feature matching. Pattern Recognition and Image Analysis 32(3), 626–630 (2022)
[18]
Pyatov, V.A., Sorokin, D.V.: Tahir: Transformer-based affine histological image registration. In: International Conference on Pattern Recognition. pp. 541–552. Springer (2022)
[19]
Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: Learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4938–4947 (2020)
[20]
Shao, W., Banh, L., Kunder, C.A., Fan, R.E., Soerensen, S.J., Wang, J.B., Teslovich, N.C., Madhuripan, N., Jawahar, A., Ghanouni, P., et al.: ProsRegNet: A deep learning framework for registration of MRI and histopathology images of the prostate. Medical Image Analysis 68, 101919 (2021)
[21]
Weitz, P., Valkonen, M., Solorzano, L., Carr, C., Kartasalo, K., Boissin, C., Koivukoski, S., Kuusela, A., Rasic, D., Feng, Y., et al.: The ACROBAT 2022 Challenge: Automatic Registration Of Breast Cancer Tissue. arXiv 2305.18033 (2023)
[22]
Wodzinski, M., Marini, N., Atzori, M., Müller, H.: DeeperHistReg: Robust Whole Slide Images Registration Framework. arXiv:2404.14434 (2024)
[23]
Wodzinski, M., Marini, N., Atzori, M., Müller, H.: RegWSI: Whole slide image registration using combined deep feature-and intensity-based methods: Winner of the ACROBAT 2023 challenge. Computer Methods and Programs in Biomedicine 250, 108187 (2024)
[24]
Wodzinski, M., Müller, H.: DeepHistReg: Unsupervised deep learning registration framework for differently stained histology samples. Computer Methods and Programs in Biomedicine 198, 105799 (2021)
[25]
Wodzinski, M., Skalski, A.: Multistep, automatic and nonrigid image registration method for histology samples acquired using multiple stains. Physics in Medicine & Biology 66(2), 025006 (2021)
[26]
Zhao, S., Lau, T., Luo, J., Eric, I., Chang, C., Xu, Y.: Unsupervised 3d end-to-end medical image registration with volume tweening network. IEEE Journal of Biomedical and Health Informatics 24(5), 1394–1404 (2019)
Index Terms
- Automatic Registration of SHG and H&E Images with Feature-Based Initial Alignment and Intensity-Based Instance Optimization: Contribution to the COMULIS Challenge
Index terms have been assigned to the content through auto-classification.
Recommendations
Feature and Intensity Based Medical Image Registration Using Particle Swarm Optimization
Image registration is an important aspect in medical image analysis, and kinds use in a variety of medical applications. Examples include diagnosis, pre/post surgery guidance, comparing/merging/integrating images from multi-modal like Magnetic Resonance ...
Co-registration of in vivo human MRI brain images to postmortem histological microscopic images
Certain features such as small vascular lesions seen in human MRI (Magnetic Resonance Imaging) are detected reliably only in postmortem histological samples by microscopic imaging. Co-registration of these microscopically detected features to their ...
Multi-step, Learning-Based, Semi-supervised Image Registration Algorithm
Segmentation, Classification, and Registration of Multi-modality Medical Imaging DataAbstractThis paper presents a contribution to the Learn2Reg challenge organized jointly with the MICCAI 2020, more specifically, to the task related to inter-patient hippocampus registration in magnetic resonance images. The proposed algorithm is a multi-...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Oct 2024
377 pages
ISBN:978-3-031-73479-3
DOI:10.1007/978-3-031-73480-9
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 06 October 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 20 Dec 2024