Abstract
Nowadays, thyroid ultrasound examination faces some problems such as weak effective feature information, plentiful noise, and small samples. Our research aims at helping doctors making decision more accurately and quickly to identify the characteristics of patients’ thyroid nodules based on ultrasound images. Firstly, after pre-processing ultrasound images of thyroid nodules, a noise reduction method is proposed by using weighted adaptive gamma correction which can effectively suppress the generation of noise and improve the global information contrast ratio. Secondly, fine-tuning transfer learning to pre-train ResNet-18 convolutional neural network is used to solve over-fitting under small samples. Thirdly, an adaptive threshold Local Ternary Pattern algorithm is proposed to extract local texture features of the ultrasound images in order to enhance the classification performance. Finally, a multi-scale feature fusion approach, which combines the local texture features and the deep features (the global texture features) automatically extracted by convolutional layers, is carried out by following a second fine-tuning training in ResNet-18 convolutional neural network based on the multi-scale joint features. The test results show: (1) the improved Adaptive Threshold Local Ternary Pattern algorithm demonstrates better performance than other algorithms in extracting texture features on the experimental thyroid nodule dataset, which has fewer misclassified samples and can better describe the texture information of the ultrasound image nodules. (2) The classification accuracy is significantly promoted in the given real test set based on the improving ResNet-18 convolutional neural network by using the proposed multi-scale feature fusion approach.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
T. Ghosh, M-H.-Z. Abedin, H. Al Banna, N. Mumenin, and M. Abu Yousuf, “Performance analysis of state of the art convolutional neural network architectures in Bangla handwritten character recognition,” Pattern Recognit. Image Anal. 31, 60–71 (2021). https://doi.org/10.1134/S1054661821010089
S. Khan, T. R. Soomro, and M. M. Alam, “Application of image processing in detection of bone diseases using X‑rays,” Pattern Recognit. Image Anal. 30, 97–107 (2020). https://doi.org/10.1134/S1054661820010071
U. R. Acharya, S. V. Sree, G. Swapna, S. Gupta, F. Molinari, R. Garberoglio, A. Witkowska, and J. S. Suri, “Effect of complex wavelet transform filter on thyroid tumor classification in three-dimensional ultrasound,” J. Eng. Med. 227, 284–292 (2013). https://doi.org/10.1177/0954411912472422
J. Chi, E. Walia, P. Babyn, J. Wang, G. Groot, and M. Eramian, “Thyroid nodule classification in ultrasound images by Fine-Tuning deep convolutional neural network,” J. Digital Imaging 30, 477–486 (2017). https://doi.org/10.1007/s10278-017-9997-y
B. Migda, M. Migda, M. S. Migda, and R. Z. Slapa, “Use of the Kwak thyroid image reporting and data system (K-TIRADS) in differential diagnosis of thyroid nodules: systematic review and meta-analysis,” Europ. Radiol. 28, 2380–2388 (2018). https://doi.org/10.1007/s00330-017-5230-0
B. Sonawane and P. Sharma, “Deep learning based approach of emotion detection and grading system,” Pattern Recognit. Image Anal. 30, 726–740 (2020). https://doi.org/10.1134/S1054661820040239
Q. Fu, M. Celenk, and A. Wu, “An improved algorithm based on CLAHE for ultrasonic well logging image enhancement,” Cluster Comput. 22, 12609–12618 (2018). https://doi.org/10.1007/s10586-017-1692-8
S.-J. Chen, C.-Y. Chang, K.-Y. Chang, J.-E. Tzeng, Y.‑T. Chen, C.-W. Lin, W.-C. Hsu, and C.-K. Wei, “Classification of the thyroid nodules based on characteristic sonographic textural feature and correlated histopathology using hierarchical support vector machines,” Ultrasound Med. Biol. 36, 2018–2026 (2010). https://doi.org/10.1016/j.ultrasmedbio.2010.08.019
W. Li, P. Cao, D. Zhao, and J. Wang, “Pulmonary nodule classification with deep convolutional neural networks on computed tomography images,” Comput. Math. Methods Med. 2016, 6215085 (2016). https://doi.org/10.1155/2016/6215085
H.-C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, and R. M. Summers, “Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning,” IEEE Trans. Med. Imaging 35, 1285–1298 (2016). https://doi.org/10.1109/TMI.2016.2528162
F. N. Tessler, W. D. Middleton, E. G. Grant, J. K. Hoang, L. L. Berland, S. A. Teefey, J. J. Cronan, M. D. Beland, T. S. Desser, M. C. Frates, L. W. Hammers, U. M. Hamper, J. E. Langer, C. C. Reading, L. M. Scoutt, and A. T. Stavros, “ACR thyroid imaging, reporting and data system (TI-RADS): White paper of the ACR TI-RADS committee,” J. Am. Coll. Radiol. 14, 587–595 (2017). https://doi.org/10.1016/j.jacr.2017.01.046
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, 2016 (IEEE, 2016), pp. 770–778. https://doi.org/10.1109/CVPR.2016.90
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, 2016 (IEEE, 2016), pp. 2818–2825. https://doi.org/10.1109/CVPR.2016.308
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, Inception-ResNet and the impact of residual connections on learning,” in Proc. 31st AAAI Conf. on Artificial Intelligence, San Francisco, Calif., 2017 (AAAI Press, 2017), pp. 4278–4284.
K. Karampidis, E. Kavallieratou, and G. Papadourakis, “A dilated convolutional neural network as feature selector for spatial image steganalysis – A hybrid classification scheme,” Pattern Recognit. Image Anal. 30, 342–358 (2020). https://doi.org/10.1134/S1054661820030098
S. Katsigiannis, E. G. Keramidas, and D. Maroulis, “A contourlet transform feature extraction scheme for ultrasound thyroid texture classification,” Eng. Intell. Syst. 18, 138–145 (2010).
N. C. Y. Koh, K. S. Sim, and C. P. Tso, “CT brain lesion detection through combination of recursive sub-image histogram equalization in wavelet domain and adaptive gamma correction with weighting distribution,” in Int. Conf. on Robotics, Automation, and Sciences (ICORAS), Melaka, Malaysia, 2016 (IEEE, 2016), pp. 1–6. https://doi.org/10.1109/ICORAS.2016.7872603
W. Jin, B. Li, and M. Yu, “Feature extraction based on equalized ULBP for face recognition,” in Int. Conf. Computer Science Electronics Engineering, Hangzhou, Zhejiang, China, 2012 (2012), pp. 532–536. https://doi.org/10.1109/ICCSEE.2012.233
W. S. Li, L. D. Wang, and L. F. Zhou, “A face recognition method based on LTP adaptive threshold,” Small Microcomput. Syst. 35, 2099–2103 (2014).
K. Li, Y. Wei, Z. Yang, and W. Wei, “Image inpainting algorithm based on TV model and evolutionary algorithm,” Soft Comput. 20, 885–893 (2016). https://doi.org/10.1007/s00500-014-1547-7
X. T. Liu, L. B. Liu, and X. P. Ma, “Thyroid nodule detection based on median filtering and residual net,” Computer Eng. Appl. 55, 254–259 (2019).
F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte, “Breast cancer histopathological image classification using convolutional neural networks,” in Int. Joint Conf. on Neural Networks (IJCNN), Vancouver, 2016 (IEEE, 2016), pp. 2560–2567. https://doi.org/10.1109/IJCNN.2016.7727519
S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng. 22, 1345–1359 (2010). https://doi.org/10.1109/TKDE.2009.191
ACKNOWLEDGMENTS
This work was supported by the Key R&D Program of JiangXi Province of China (project no. 20181BBG70031) and National Natural Science Foundation of China (project no. 62066027).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
COMPLIANCE WITH ETHICAL STANDARDS
This article is a completely original work of its authors; it has not been published before and will not be sent to other publications until the PRIA Editorial Board decides not to accept it for publication.
Conflicts of Interest
The process of writing and the content of the article does not give grounds for raising the issue of a conflict of interest.
Additional information
Xinyi Wei was born in Yushan, Jiangxi, China, in 2001. She is an undergraduate in Nanchang University and is now in her junior year. Her research interests include image processing, machine learning, and artificial intelligence.
Siwei Zhang was born in Yiyang Hunan, China, in 2001. He is an undergraduate in Nanchang University and is now in his junior year. His research interests include image processing, machine learning, and artificial intelligence.
Qi Qi was born in 1993, Shandong, China. She is currently pursuing the PhD degree with the School of Medicine, Nanchang University, China. Her research interests include superficial ultrasound and medical image recognition in the field of ultrasound medicine.
Hao Fu was born in Jiangxi, China, in 1997. He is a graduate student in Nanchang University. He received a Bachelor degree in Software Engineering from Donghua University in 2017. His main research interests include machine learning and artificial intelligence.
Taorong Qiu (corresponding author) was born in Sanming, Fujian, China, in 1964. He is a Professor in the School of Computer Science, Nanchang University, Jiangxi, China. He received PhD degree in Computer Application Technology from Beijing Jiaotong University in 2009. His research interests include image processing, pattern recognition, artificial intelligence, and machine learning.
Aiyun Zhou born in Nanchang, Jiangxi Province in 1960. She was graduated from Jiangxi Medical College in 1983 with a Bachelor’s degree. She is now the director, chief physician, Professor, master and doctoral supervisor of the Department of Ultrasound Diagnosis in the First Affiliated Hospital of Nanchang University. Her main research interests include medical imaging and ultrasound diagnosis.
Rights and permissions
About this article
Cite this article
Xinyi Wei, Zhang, S., Qi, Q. et al. Predicting Malignancy and Benign Thyroid Nodule Using Multi-Scale Feature Fusion and Deep Learning. Pattern Recognit. Image Anal. 31, 830–841 (2021). https://doi.org/10.1134/S1054661821040283
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S1054661821040283