Abstract
Convolutional Neural Networks (CNN) have been found to have great potential in optical flow problems thanks to an abundance of data available for training a deep network. The displacement estimation step in UltraSound Elastography (USE) can be viewed as an optical flow problem. Despite the high performance of CNNs in optical flow, they have been rarely used for USE due to unique challenges that both input and output of USE networks impose. Ultrasound data has much higher high-frequency content compared to natural images. The outputs are also drastically different, where displacement values in USE are often smooth without sharp motions or discontinuities. The general trend is currently to use pre-trained networks and fine-tune them on a small simulation ultrasound database. However, realistic ultrasound simulation is computationally expensive. Also, the simulation techniques do not model complex motions, nonlinear and frequency-dependent acoustics, and many sources of artifact in ultrasound imaging. Herein, we propose an unsupervised fine-tuning technique which enables us to employ a large unlabeled dataset for fine-tuning of a CNN optical flow network. We show that the proposed unsupervised fine-tuning method substantially improves the performance of the network and reduces the artifacts generated by networks trained on computer vision databases.
Supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) RGPIN-2020-04612.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Azizi, S., et al.: Learning from noisy label statistics: detecting high grade prostate cancer in ultrasound guided biopsy. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 21–29. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_3
Zhou, S.K., Rueckert, D., Fichtinger, G.: Handbook of medical imagecomputing and computer assisted intervention. Academic Press, (2019)
Zhuang, B., Rohling, R., Abolmaesumi, P.: Region-of-interest-based closed-loop beamforming for spinal ultrasound imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 66(8), 1266–1280 (2019)
Ophir, J., et al.: Elastography: ultrasonic estimation and imaging of the elastic properties of tissues. Proc. Inst. Mech. Eng. Part H: J. Eng. Med. 213(3), 203–233 (1999)
Dosovitskiy, A., et al.: Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp. 2758–2766 (2015)
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2462–2470 (2017)
Sun, D., Yang, X., Liu, M.-Y., Kautz, J.: Pwc-net: cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8934–8943 (2018)
Hui, T.W., Tang, X., Change Loy, C.: Liteflownet: a lightweight convolutional neural network for optical flow estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8981–8989 (2018)
Kibria, M.G., Rivaz, H.: GLUENet: ultrasound elastography using convolutional neural network. In: Stoyanov, D., et al. (eds.) POCUS/BIVPCS/CuRIOUS/CPM -2018. LNCS, vol. 11042, pp. 21–28. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01045-4_3
Peng, B., Xian, Y., Jiang, J.: A convolution neural network-based speckletracking method for ultrasound elastography. In: 2018 IEEEInternational Ultrasonics Symposium (IUS), pp. 206–212. IEEE (2018)
Tehrani, A.K., Rivaz, H.: Displacement estimation in ultrasound elastography using pyramidal convolutional neural network. In: IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, (2020)
Wu, S., Gao, Z., Liu, Z., Luo, J., Zhang, H., Li, S.: Direct reconstruction of ultrasound elastography using an end-to-end deep neural network. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 374–382. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_43
Peng, B., Xian, Y., Zhang, Q., Jiang, J.: Neural network-based motion tracking for breast ultrasound strain elastography: an initial assessment of performance and feasibility. Ultrason. Imaging. 42(2), 74–91 (2020)
Gao, Z., et al.: Learning the implicit strain reconstruction in ultrasound elastography using privileged information. Med. Image Anal. 58, 11–18 (2019)
Evain, E., Faraz, K., Grenier, T., Garcia, D., De Craene, M., Bernard, O.: A pilot study on convolutional neural networks for motion estimation from ultrasound images. Ferroelectrics, and Frequency Control, IEEE Transactions on Ultrasonics, (2020)
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)
Sandrin, L., et al.: Transient elastography: a new noninvasive method for assessment of hepatic fibrosis. Ultrasound Med. Biol. 29(12), 1705–1713 (2003)
Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 270–279 (2017)
Meister, S., Hur, J., Roth, S.: Unflow: unsupervised learning of optical flow with a bidirectional census loss. In: Thirty-Second AAAI Conference on Artificial Intelligence, (2018)
Ren, Z., Yan, J., Yang, X., Yuille, A., Zha, H.: Unsupervised learning of optical flow with patch consistency and occlusion estimation. Pattern Recogn. 103, 107191 (2020)
Wang, Y., Yang, Y., Yang, Z., Zhao, L., Wang, P., Xu, W.: Occlusion aware unsupervised learning of optical flow. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4884–4893 (2018)
Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vision 106(2), 115–137 (2014)
Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 801, pp. 151–158. Springer, Heidelberg (1994). https://doi.org/10.1007/BFb0028345
Hashemi, H.S., Rivaz, H.: Global time-delay estimation in ultrasound elastography. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 64(10), 1625–1636 (2017)
Mirzaei, M., Asif, A., Rivaz, H.: Combining total variation regularization with window-based time delay estimation in ultrasound elastography. IEEE Trans. Med. Imaging 38(12), 2744–2754 (2019)
Chen, T., Xu, B., Zhang, C., Guestrin, C.: Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 (2016)
Rivaz, H., Boctor, E.M., Choti, M.A., Hager, G.D.: Real-time regularized ultrasound elastography. IEEE Trans. Med. Imaging 30(4), 928–945 (2011)
Acknowledgement
We thank NVIDIA for the donation of the GPU. The in vivo data was collected at Johns Hopkins Hospital. We thank E. Boctor, M. Choti and G. Hager for giving us access to this data.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
K. Z. Tehrani, A., Mirzaei, M., Rivaz, H. (2020). Semi-supervised Training of Optical Flow Convolutional Neural Networks in Ultrasound Elastography. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12263. Springer, Cham. https://doi.org/10.1007/978-3-030-59716-0_48
Download citation
DOI: https://doi.org/10.1007/978-3-030-59716-0_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59715-3
Online ISBN: 978-3-030-59716-0
eBook Packages: Computer ScienceComputer Science (R0)