Abstract
Accurate bronchoscope localization is essential for pulmonary interventions, by providing six degrees of freedom (DOF) in airway navigation. However, the robustness of current vision-based methods is often compromised in clinical practice, and they struggle to perform in real-time and to generalize across cases unseen during training. To overcome these challenges, we propose a novel Probabilistic Airway Navigation System (PANS), leveraging Monte-Carlo method with pose hypotheses and likelihoods to achieve robust and real-time bronchoscope localization. Specifically, our PANS incorporates diverse visual representations (e.g., odometry and landmarks) by leveraging two key modules, including the Depth-based Motion Inference (DMI) and the Bronchial Semantic Analysis (BSA). To generate the pose hypotheses of bronchoscope for PANS, we devise the DMI to accurately propagate the estimation of pose hypotheses over time. Moreover, to estimate the accurate pose likelihood, we devise the BSA module by effectively distinguishing between similar bronchial regions in endoscopic images, along with a novel metric to assess the congruence between estimated depth maps and the segmented airway structure. Under this probabilistic formulation, our PANS is capable of achieving the 6-DOF bronchoscope localization with superior accuracy and robustness. Extensive experiments on the collected pulmonary intervention dataset comprising 10 clinical cases confirm the advantage of our PANS over state-of-the-arts, in terms of both robustness and generalization in localizing deeper airway branches and the efficiency of real-time inference. The proposed PANS reveals its potential to be a reliable tool in the operating room, promising to enhance the quality and safety of pulmonary interventions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Andolfi, M., Potenza, R., Capozzi, R., Liparulo, V., Puma, F., Yasufuku, K.: The role of bronchoscopy in the diagnosis of early lung cancer: a review. Journal of thoracic disease 8(11), 3329 (2016)
Banach, A., King, F., Masaki, F., Tsukada, H., Hata, N.: Visually navigated bronchoscopy using three cycle-consistent generative adversarial network for depth estimation. Medical image analysis 73, 102164 (2021)
Borrego-Carazo, J., Sanchez, C., Castells-Rufas, D., Carrabina, J., Gil, D.: Bronchopose: an analysis of data and model configuration for vision-based bronchoscopy pose estimation. Computer Methods and Programs in Biomedicine 228, 107241 (2023)
Deguchi, D., Mori, K., Feuerstein, M., Kitasaka, T., Maurer Jr, C.R., Suenaga, Y., Takabatake, H., Mori, M., Natori, H.: Selective image similarity measure for bronchoscope tracking based on image registration. Medical Image Analysis 13(4), 621–633 (2009)
Deng, J., Li, P., Dhaliwal, K., Lu, C.X., Khadem, M.: Feature-based visual odometry for bronchoscopy: A dataset and benchmark. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 6557–6564. IEEE (2023)
Floris, I., Adam, J.M., Calderón, P.A., Sales, S.: Fiber optic shape sensors: A comprehensive review. Optics and Lasers in Engineering 139, 106508 (2021)
Folch, E.E., Labarca, G., Ospina-Delgado, D., Kheir, F., Majid, A., Khandhar, S.J., Mehta, H.J., Jantz, M.A., Fernandez-Bussy, S.: Sensitivity and safety of electromagnetic navigation bronchoscopy for lung cancer diagnosis: systematic review and meta-analysis. Chest 158(4), 1753–1769 (2020)
Gu, Y., Gu, C., Yang, J., Sun, J., Yang, G.Z.: Vision–kinematics interaction for robotic-assisted bronchoscopy navigation. IEEE Transactions on Medical Imaging 41(12), 3600–3610 (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: Evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2462–2470 (2017)
Karaoglu, M.A., Brasch, N., Stollenga, M., Wein, W., Navab, N., Tombari, F., Ladikos, A.: Adversarial domain feature adaptation for bronchoscopic depth estimation. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24. pp. 300–310. Springer (2021)
Luo, X., Mori, K.: A discriminative structural similarity measure and its application to video-volume registration for endoscope three-dimensional motion tracking. IEEE transactions on medical imaging 33(6), 1248–1261 (2014)
Luo, X., Xie, L., Zeng, H.Q., Wang, X., Li, S.: Monocular endoscope 6-dof tracking with constrained evolutionary stochastic filtering. Medical Image Analysis 89, 102928 (2023)
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2794–2802 (2017)
Mori, K., Deguchi, D., Sugiyama, J., Suenaga, Y., Toriwaki, J.i., Maurer Jr, C.R., Takabatake, H., Natori, H.: Tracking of a bronchoscope using epipolar geometry analysis and intensity-based image registration of real and virtual endoscopic images. Medical Image Analysis 6(3), 321–336 (2002)
Recasens, D., Lamarca, J., Fácil, J.M., Montiel, J., Civera, J.: Endo-depth-and-motion: Reconstruction and tracking in endoscopic videos using depth networks and photometric constraints. IEEE Robotics and Automation Letters 6(4), 7225–7232 (2021)
Sganga, J., Eng, D., Graetzel, C., Camarillo, D.: Offsetnet: Deep learning for localization in the lung using rendered images. In: 2019 international conference on robotics and automation (ICRA). pp. 5046–5052. IEEE (2019)
Sganga, J., Eng, D., Graetzel, C., Camarillo, D.B.: Autonomous driving in the lung using deep learning for localization. arXiv preprint arXiv:1907.08136 (2019)
Shen, M., Gu, Y., Liu, N., Yang, G.Z.: Context-aware depth and pose estimation for bronchoscopic navigation. IEEE Robotics and Automation Letters 4(2), 732–739 (2019)
Tan, Z., Feng, J., Zhou, J.: Sgnet: Structure-aware graph-based network for airway semantic segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 153–163. Springer (2021)
Thrun, S.: Probabilistic robotics. Communications of the ACM 45(3), 52–57 (2002)
Tian, Q., Liao, H., Huang, X., Chen, J., Zhang, Z., Yang, B., Ourselin, S., Liu, H.: Dd-vnb: A depth-based dual-loop framework for real-time visually navigated bronchoscopy. arXiv preprint arXiv:2403.01683 (2024)
Tian, Q., Liao, H., Huang, X., Yang, B., Wu, J., Chen, J., Li, L., Liu, H.: Bronchotrack: Airway lumen tracking for branch-level bronchoscopic localization. arXiv preprint arXiv:2402.12763 (2024)
Wang, C., Oda, M., Hayashi, Y., Villard, B., Kitasaka, T., Takabatake, H., Mori, M., Honma, H., Natori, H., Mori, K.: A visual slam-based bronchoscope tracking scheme for bronchoscopic navigation. International Journal of Computer Assisted Radiology and Surgery 15, 1619–1630 (2020)
Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7464–7475 (2023)
Xie, W., Jacobs, C., Charbonnier, J.P., van Ginneken, B.: Structure and position-aware graph neural network for airway labeling. arXiv preprint arXiv:2201.04532 (2022)
Zhao, C., Shen, M., Sun, L., Yang, G.Z.: Generative localization with uncertainty estimation through video-ct data for bronchoscopic biopsy. IEEE Robotics and Automation Letters 5(1), 258–265 (2019)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)
Acknowledgments
This work was supported by the Centre of AI and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, sponsored by InnoHK Funding, HKSAR, partially supported by Institute of Automation, Chinese Academy of Sciences, and partially supported by Sichuan Science and Technology Program (Grant number: 2023YFH0093).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tian, Q. et al. (2024). PANS: Probabilistic Airway Navigation System for Real-Time Robust Bronchoscope Localization. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15006. Springer, Cham. https://doi.org/10.1007/978-3-031-72089-5_44
Download citation
DOI: https://doi.org/10.1007/978-3-031-72089-5_44
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72088-8
Online ISBN: 978-3-031-72089-5
eBook Packages: Computer ScienceComputer Science (R0)