Abstract
Convolutional Neural Networks (CNN) have been found to have great potential in optical flow problems thanks to an abundance of data available for training a deep network. The displacement estimation step in UltraSound Elastography (USE) can be viewed as an optical flow problem. Despite the high performance of CNNs in optical flow, they have been rarely used for USE due to unique challenges that both input and output of USE networks impose. Ultrasound data has much higher high-frequency content compared to natural images. The outputs are also drastically different, where displacement values in USE are often smooth without sharp motions or discontinuities. The general trend is currently to use pre-trained networks and fine-tune them on a small simulation ultrasound database. However, realistic ultrasound simulation is computationally expensive. Also, the simulation techniques do not model complex motions, nonlinear and frequency-dependent acoustics, and many sources of artifact in ultrasound imaging. Herein, we propose an unsupervised fine-tuning technique which enables us to employ a large unlabeled dataset for fine-tuning of a CNN optical flow network. We show that the proposed unsupervised fine-tuning method substantially improves the performance of the network and reduces the artifacts generated by networks trained on computer vision databases.
Supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) RGPIN-2020-04612.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Azizi, S., et al.: Learning from noisy label statistics: detecting high grade prostate cancer in ultrasound guided biopsy. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-Lรณpez, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 21โ29. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_3
Zhou, S.K., Rueckert, D., Fichtinger, G.: Handbook of medical imagecomputing and computer assisted intervention. Academic Press, (2019)
Zhuang, B., Rohling, R., Abolmaesumi, P.: Region-of-interest-based closed-loop beamforming for spinal ultrasound imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 66(8), 1266โ1280 (2019)
Ophir, J., et al.: Elastography: ultrasonic estimation and imaging of the elastic properties of tissues. Proc. Inst. Mech. Eng. Part H: J. Eng. Med. 213(3), 203โ233 (1999)
Dosovitskiy, A., et al.: Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp. 2758โ2766 (2015)
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2462โ2470 (2017)
Sun, D., Yang, X., Liu, M.-Y., Kautz, J.: Pwc-net: cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8934โ8943 (2018)
Hui, T.W., Tang, X., Change Loy, C.: Liteflownet: a lightweight convolutional neural network for optical flow estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8981โ8989 (2018)
Kibria, M.G., Rivaz, H.: GLUENet: ultrasound elastography using convolutional neural network. In: Stoyanov, D., et al. (eds.) POCUS/BIVPCS/CuRIOUS/CPM -2018. LNCS, vol. 11042, pp. 21โ28. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01045-4_3
Peng, B., Xian, Y., Jiang, J.: A convolution neural network-based speckletracking method for ultrasound elastography. In: 2018 IEEEInternational Ultrasonics Symposium (IUS), pp. 206โ212. IEEE (2018)
Tehrani, A.K., Rivaz, H.: Displacement estimation in ultrasound elastography using pyramidal convolutional neural network. In: IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, (2020)
Wu, S., Gao, Z., Liu, Z., Luo, J., Zhang, H., Li, S.: Direct reconstruction of ultrasound elastography using an end-to-end deep neural network. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-Lรณpez, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 374โ382. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_43
Peng, B., Xian, Y., Zhang, Q., Jiang, J.: Neural network-based motion tracking for breast ultrasound strain elastography: an initial assessment of performance and feasibility. Ultrason. Imaging. 42(2), 74โ91 (2020)
Gao, Z., et al.: Learning the implicit strain reconstruction in ultrasound elastography using privileged information. Med. Image Anal. 58, 11โ18 (2019)
Evain, E., Faraz, K., Grenier, T., Garcia, D., De Craene, M., Bernard, O.: A pilot study on convolutional neural networks for motion estimation from ultrasound images. Ferroelectrics, and Frequency Control, IEEE Transactions on Ultrasonics, (2020)
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935โ2947 (2017)
Sandrin, L., et al.: Transient elastography: a new noninvasive method for assessment of hepatic fibrosis. Ultrasound Med. Biol. 29(12), 1705โ1713 (2003)
Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 270โ279 (2017)
Meister, S., Hur, J., Roth, S.: Unflow: unsupervised learning of optical flow with a bidirectional census loss. In: Thirty-Second AAAI Conference on Artificial Intelligence, (2018)
Ren, Z., Yan, J., Yang, X., Yuille, A., Zha, H.: Unsupervised learning of optical flow with patch consistency and occlusion estimation. Pattern Recogn. 103, 107191 (2020)
Wang, Y., Yang, Y., Yang, Z., Zhao, L., Wang, P., Xu, W.: Occlusion aware unsupervised learning of optical flow. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4884โ4893 (2018)
Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vision 106(2), 115โ137 (2014)
Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 801, pp. 151โ158. Springer, Heidelberg (1994). https://doi.org/10.1007/BFb0028345
Hashemi, H.S., Rivaz, H.: Global time-delay estimation in ultrasound elastography. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 64(10), 1625โ1636 (2017)
Mirzaei, M., Asif, A., Rivaz, H.: Combining total variation regularization with window-based time delay estimation in ultrasound elastography. IEEE Trans. Med. Imaging 38(12), 2744โ2754 (2019)
Chen, T., Xu, B., Zhang, C., Guestrin, C.: Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 (2016)
Rivaz, H., Boctor, E.M., Choti, M.A., Hager, G.D.: Real-time regularized ultrasound elastography. IEEE Trans. Med. Imaging 30(4), 928โ945 (2011)
Acknowledgement
We thank NVIDIA for the donation of the GPU. The in vivo data was collected at Johns Hopkins Hospital. We thank E. Boctor, M. Choti and G. Hager for giving us access to this data.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
ยฉ 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
K. Z. Tehrani, A., Mirzaei, M., Rivaz, H. (2020). Semi-supervised Training of Optical Flow Convolutional Neural Networks in Ultrasound Elastography. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention โ MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12263. Springer, Cham. https://doi.org/10.1007/978-3-030-59716-0_48
Download citation
DOI: https://doi.org/10.1007/978-3-030-59716-0_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59715-3
Online ISBN: 978-3-030-59716-0
eBook Packages: Computer ScienceComputer Science (R0)