Abstract
Bone age assessment is a pediatric examination that determines the difference between skeletal age and chronological age. The discrepancy between the two ages will often trigger the likelihood of genetic disorders, hormonal complications and abnormalities of maturity in the skeletal system. Recently, although some automated bone age assessment methods by analyzing radiographs have been researched, the available text data from radiological reports are not used. Texts and radiographs are two different modals, the fusion of them can give us much more information for bone age assessment. In this paper, we present a novel multi-modal data fusion-learning network, called RT-FuseNet, for bone age assessment utilizing radiographs and texts. Specifically, we develop a convolutional neural network with spatial pyramid pooling layer and attention mechanism module to ensure the integrity of the image space information and enhance the subtle difference of features among radiographs respectively. In addition, texts are incorporated into the learning model to jointly learn non-linear correlations between various heterogeneous data. To evaluate the proposed approach, two datasets are used and several neural network structures are compared. Experimental results show that the proposed approach performs well.
Similar content being viewed by others
References
Albanese A, Stanhope R (2010) Investigation of delayed puberty. Clin. Endocrinol. 43(1):105–110
Athanasios C, Maria B, George K, Miranda A (2007) Bone age estimation and prediction of final height in patients with beta-thalassaemia major: A comparison between the two most common methods. In: Pediatric radiology, vol 37, pp 1241–1246
Chen T, Ma X, Ying X, Wang W, Yuan C, Lu W, Chen D, Wu J (2019) Multi-modal fusion learning for cervical dysplasia diagnosis. In: IEEE 16th International Symposium on Biomedical Imaging (ISBI)
Dai J, Qi H, Xiong Y, Li Y, Zhang G, Hu H, Wei Y (2017) Deformable convolutional networks. In: International Conference on Computer Vision
Ferdinand H, Inga B, Heike R, Thomas D, Hauke S (2013) Bone age assessment using the classifying generalized hough transform. In: Pattern recognition, pp 313–322
Gao F, Ge Y, Liu Y (2018) Remember and forget: Video and text fusion for video question answering. Multimed Tools Appl 77:29269–29282
Greulich W, Pyle S (1959) Radiographic atlas of skeletal development of the hand and wrist. The American Journal of the Medical Sciences 238(3)
Guo C, Chai Y, Wang C (2016) Multi-source heterogeneous data recognition based on linguistic labels. In: International conference on cyber-enabled distributed computing and knowledge discovery, pp. 278–285
Hao P, Chokuwa S, Xie X, Wu F, Wu J, Bai C (2019) Skeletal bone age assessments for young children based on regression convolutional neural networks. Math. Biosci. Eng. 16(6):6454–6466
Hao P, You K, Feng H, Xu X, Zhang F, Wu F, Zhang P, Chen W (2019) Lung adenocarcinoma diagnosis in one stage. Neurocomputing
He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37 (9):1904–1916
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition
Huang G, Liu Z, Maaten LVD, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition
Iglovikov V, Rakhlin A, Kalinin A, Shvets A (2018) Paediatric bone age assessment using deep convolutional neural networks. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 300–308
Lee H, Tajmir S, Lee J, Zissen M, Yeshiwas B, Alkasab T, Choy G, Do S (2017) Fully automated deep learning system for bone age assessment. J. Digit. Imaging 30(4):427–441
Mutasa S, Chang PD, Ruzal-Shapiro C, Ayyala R (2018) Mabal: a novel deep-learning architecture for machine-assisted bone age labeling. J. Digit. Imaging 31:513–519
Ren X, Li T, Yang X, Wang S, Ahmad S, Xiang L, Stone S, Li L, Zhan Y, Shen D, Wang Q (2018) Regression convolutional neural network for automated pediatric bone age assessment from hand radiograph. IEEE Journal of Biomedical and Health Informatics, pp 2168–2194
Rijn RV, Thodberg HH (2013) Bone age assessment: automated techniques coming of age?. In: Acta Radiologica, vol 54, pp 1024–9
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2019) Grad-cam: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis., pp 1–24
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations
Somkantha K, Nipon T, Auephanwiriyakul S (2011) Bone age assessment in young children using automatic carpal bone feature extraction and support vector regression. J. Digit. Imaging 24(6):1044–1058
Souza D, Oliveira MM (2018) End-to-end bone age assessment with residual learning. In: 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), pp 197–203
Spampinato C, Palazzo S, Giordamo D, Aldinucci M, Leonardi R (2017) Deep learning for automated skeletal bone age assessment in x-ray images. Med. Image Anal. 36:41–51
Steenkiste TV, Ruyssinck J, Janssens OV (2018) et.al., V.F.V. Automated assessment of bone age using deep learning and gaussian process regression. In: 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 674–677
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE Conference on computer vision and pattern recognition, pp 1–9
Tanner J, Whitehouse R, Cameron N, Marshall W, Healy M, Goldstein H (1975) Assessment of skeletal maturity and prediction of adult height (TW2 method)
Thodberg H, Kreiborg S, Juul A, Pedersen K (2009) The bonexpert method for automated determination of skeletal maturity. IEEE Trans. Med. Imaging 28(1):52–66
Woo S, Park J, Lee J, Kweon IS (2018) Cbam: Convolutional block attention module. In: European Conference on Computer Vision (ECCV), pp 3–19
Wu E, Kong B, Bai J, Lu Y, Gao F, Zhang S, et al. (2019) Residual attention based network for hand bone age assessment. In: IEEE 16th International Symposium on Biomedical Imaging (ISBI), pp 1158–1161
Zhang L, Xie Y, Xidao L, Zhang X (2018) Multi-source heterogeneous data fusion. In: International conference on artificial intelligence and big data, pp 47–51
Acknowledgements
This project was supported by Zhejiang Provincial Natural Science Foundation of China (No. LY18F020034, LY18C130012), National Natural Science Foundation of China (No.61801428, 61672543), the Zhejiang University Education Foundation (No. K18-511120-004 and No. K17-518051-021), and the Major Scientific Project of Zhejiang Lab (No. 2018DG0ZX01).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hao, P., Ye, T., Xie, X. et al. Radiographs and texts fusion learning based deep networks for skeletal bone age assessment. Multimed Tools Appl 80, 16347–16366 (2021). https://doi.org/10.1007/s11042-020-08943-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-08943-1