Nothing Special   »   [go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

Hybrid models for classifying histological images: An association of deep features by transfer learning with ensemble classifier

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The use of a convolutional neural network with transfer learning is a strategy that defines high-level features, commonly explored to study patterns in medical images. These features can be analyzed via different methods in order to design hybrid models with more useful and accurate solutions for clinical practice. In this paper, a computational scheme is presented to define hybrid models through deep features by transfer learning, selection by ranking and a robust ensemble classifier with five algorithms. The obtained models were applied to classify histological images from breast, colorectal and liver tissue. The strategy developed here allows knowing important results and conditions to improve models of computer-aided diagnosis, even exploring classic CNN models. The features were defined using layers from the AlexNet and ResNet-50 architectures. The attributes were organized into subsets of the most relevant features and submitted to a k-fold cross-validation process. The best hybrid models were obtained with deep features from the ResNet-50 network, using distinct layers (activation_48_relu and avg_pool) and a maximum of 35 descriptors. These hybrid models provided 98.00% and 99.32% of accuracy values, with emphasis on histological images of breast cancer, indicating the best solution among those available in the specialized Literature. Also, these models provided more relevant results for classifying UCSB and LG datasets than regularized techniques and CNN architectures, exploring data augmentation or not. The computational scheme with detailed information regarding the main hybrid models is a relevant contribution to the community interested in the study of machine learning techniques for pattern recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

Not applicable.

Code availability

Not applicable.

References

  1. Yang M, Kpalma K, Ronsin J (2008) A survey of shape feature extraction techniques

  2. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press

  3. Gad AF (2018) Convolutional neural networks. In: Practical Computer Vision Applications Using Deep Learning with CNNs, Springer, pp 183–227

  4. Hinton GE (2007) Learning multiple layers of representation. Trends Cogn Sci 11(10):428–434

    Article  PubMed  Google Scholar 

  5. Le QV (2013) Building high-level features using large scale unsupervised learning. In: 2013 IEEE international conference on acoustics, speech and signal processing, IEEE, pp 8595–8598

  6. BenTaieb A, Hamarneh G (2017) Adversarial stain transfer for histopathology image analysis. IEEE Trans Med Imaging 37(3):792–802

    Article  Google Scholar 

  7. Sethy PK, Behera SK (2022) Automatic classification with concatenation of deep and handcrafted features of histological images for breast carcinoma diagnosis. Multimedia Tools and Applications 81(7):9631–9643

    Article  Google Scholar 

  8. Saxena S, Shukla S, Gyanchandani M (2020) Breast cancer histopathology image classification using kernelized weighted extreme learning machine. International Journal of Imaging Systems and Technology

  9. Zhang R, Zhu J, Yang S, Hosseini MS, Genovese A, Chen L, Rowsell C, Damaskinos S, Varma S, Plataniotis KN (2022) Histokt: Cross knowledge transfer in computational pathology. ICASSP 2022–2022 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), IEEE, pp 1276–1280

    Google Scholar 

  10. Roberto GF, Lumini A, Neves LA, do Nascimento MZ, (2021) Fractal neural network: A new ensemble of fractal geometry and convolutional neural networks for the classification of histology images. Expert Syst Appl 166:114103

  11. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  12. Kim YJ, Bae JP, Chung JW, Park DK, Kim KG, Kim YJ (2021) New polyp image classification technique using transfer learning of network-in-network structure in endoscopic images. Sci Rep 11(1):3605

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  13. Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C (2018) A survey on deep transfer learning. In: International conference on artificial neural networks, Springer, pp 270–279

  14. Longo LHDC, Martins AS, Do Nascimento MZ, Dos Santos LFS, Roberto GF, Neves LA (2022) Ensembles of fractal descriptors with multiple deep learned features for classification of histological images. 2022 29th International Conference on Systems. Signals and Image Processing (IWSSIP), IEEE, pp 1–4

    Google Scholar 

  15. Ghandour C, El-Shafai W, El-Rabaie S (2023) Medical image enhancement algorithms using deep learning-based convolutional neural network. Journal of Optics pp 1–11

  16. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, Ieee, pp 248–255

  17. Kumar S, Sharma S (2021) Sub-classification of invasive and non-invasive cancer from magnification independent histopathological images using hybrid neural networks. Evolutionary Intelligence pp 1–13

  18. dos Santos FP, Ponti MA (2019) Alignment of local and global features from multiple layers of convolutional neural network for image classification. 2019 32nd SIBGRAPI Conference on Graphics. Patterns and Images (SIBGRAPI), IEEE, pp 241–248

    Chapter  Google Scholar 

  19. Coccia M (2020) Deep learning technology for improving cancer care in society: New directions in cancer imaging driven by artificial intelligence. Technol Soc 60:101198

    Article  Google Scholar 

  20. dos Santos FP, Ponti MA (2018) Robust feature spaces from pre-trained deep network layers for skin lesion classification. 2018 31st SIBGRAPI Conference on Graphics. Patterns and Images (SIBGRAPI), IEEE, pp 189–196

    Chapter  Google Scholar 

  21. Younas F, Usman M, Yan WQ (2022) An ensemble framework of deep neural networks for colorectal polyp classification. Multimedia Tools and Applications pp 1–22

  22. Tenguam JJ, Longo LHDC, Silva AB, De Faria PR, Do Nascimento MZ, Neves LA (2022) Classification of h &e images exploring ensemble learning with two-stage feature selection. 2022 29th International Conference on Systems. Signals and Image Processing (IWSSIP), IEEE, pp 1–4

    Google Scholar 

  23. Abraham B, Nair MS (2020) Computer-aided detection of covid-19 from x-ray images using multi-cnn and bayesnet classifier. Biocybernetics and biomedical engineering 40(4):1436–1445

    Article  PubMed  PubMed Central  Google Scholar 

  24. Novitasari DCR, Hendradi R, Caraka RE, Rachmawati Y, Fanani NZ, Syarifudin A, Toharudin T, Chen RC (2020) Detection of covid-19 chest x-ray using support vector machine and convolutional neural network. Commun Math Biol Neurosci 2020:Article–ID

  25. Urbanowicz RJ, Meeker M, La Cava W, Olson RS, Moore JH (2018) Relief-based feature selection: Introduction and review. J Biomed Inform 85:189–203

    Article  PubMed  PubMed Central  Google Scholar 

  26. Manhrawy II, Qaraad M, El-Kafrawy P (2021) Hybrid feature selection model based on relief-based algorithms and regulizer algorithms for cancer classification. Concurrency and Computation: Practice and Experience 33(17):e6200

    Article  Google Scholar 

  27. Ghosh P, Azam S, Jonkman M, Karim A, Shamrat FJM, Ignatious E, Shultana S, Beeravolu AR, De Boer F (2021) Efficient prediction of cardiovascular disease using machine learning algorithms with relief and lasso feature selection techniques. IEEE Access 9:19304–19326

    Article  Google Scholar 

  28. Zebari R, Abdulazeez A, Zeebaree D, Zebari D, Saeed J (2020) A comprehensive review of dimensionality reduction techniques for feature selection and feature extraction. Journal of Applied Science and Technology Trends 1(2):56–70

    Article  Google Scholar 

  29. Bolón-Canedo V, Sánchez-Marono N, Alonso-Betanzos A, Benítez JM, Herrera F (2014) A review of microarray datasets and applied feature selection methods. Inf Sci 282:111–135

    Article  Google Scholar 

  30. Li M, Ma X, Chen C, Yuan Y, Zhang S, Yan Z, Chen C, Chen F, Bai Y, Zhou P et al (2021) Research on the auxiliary classification and diagnosis of lung cancer subtypes based on histopathological images. Ieee Access 9:53687–53707

    Article  Google Scholar 

  31. Burçak KC, Uğuz H (2022) A new hybrid breast cancer diagnosis model using deep learning model and relieff. Traitement du Signal 39(2):521–529

    Article  Google Scholar 

  32. Silva AB, De Oliveira CI, Pereira DC, Tosta TA, Martins AS, Loyola AM, Cardoso SV, De Faria PR, Neves LA, Do Nascimento MZ (2022) Assessment of the association of deep features with a polynomial algorithm for automated oral epithelial dysplasia grading. In: 2022 35th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, vol 1, pp 264–269

  33. Watanabe K, Kobayashi T, Wada T (2016) Semi-supervised feature transformation for tissue image classification. PLoS ONE 11(12):e0166413

    Article  PubMed  PubMed Central  Google Scholar 

  34. Dos Santos LFS, Neves LA, Rozendo GB, Ribeiro MG, do Nascimento MZ, Tosta TAA, (2018) Multidimensional and fuzzy sample entropy (sampenmf) for quantifying h &e histological images of colorectal cancer. Comput Biol Med 103:148–160

  35. Roberto GF, Nascimento MZ, Martins AS, Tosta TA, Faria PR, Neves LA (2019) Classification of breast and colorectal tumors based on percolation of color normalized images. Computers & Graphics 84:134–143

    Article  Google Scholar 

  36. Bouziane A, Boumali S, Berkane N, Guendouz FS (2020) A hybrid approach for automatic breast cancer detection. In: 2020 International Conference on e-Health and Bioengineering (EHB), IEEE, pp 1–4

  37. Araújo T, Aresta G, Castro E, Rouco J, Aguiar P, Eloy C, Polónia A, Campilho A (2017) Classification of breast cancer histology images using convolutional neural networks. PLoS ONE 12(6):e0177544

    Article  PubMed  PubMed Central  Google Scholar 

  38. Papastergiou T, Zacharaki EI, Megalooikonomou V (2018) Tensor decomposition for multiple-instance classification of high-order medical data. Complexity 2018

  39. Kausar T, Wang M, Idrees M, Lu Y (2019) Hwdcnn: Multi-class recognition in breast histopathology with haar wavelet decomposed image based convolution neural network. Biocybernetics and Biomedical Engineering 39(4):967–982

    Article  Google Scholar 

  40. Feng Y, Zhang L, Yi Z (2018) Breast cancer cell nuclei classification in histopathology images using deep neural networks. Int J Comput Assist Radiol Surg 13(2):179–191

    Article  PubMed  Google Scholar 

  41. Li Y, Xie X, Shen L, Liu S (2019) Reverse active learning based atrous densenet for pathological image classification. BMC Bioinformatics 20(1):1–15

    Article  Google Scholar 

  42. Tavolara TE, Niazi MKK, Arole V, Chen W, Frankel W, Gurcan MN (2019) A modular cgan classification framework: Application to colorectal tumor detection. Sci Rep 9(1):1–8

    Article  Google Scholar 

  43. Lee JS, Wu WK (2022) Breast tumor tissue image classification using diu-net. Sensors 22(24):9838

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  44. Sena P, Fioresi R, Faglioni F, Losi L, Faglioni G, Roncucci L (2019) Deep learning techniques for detecting preneoplastic and neoplastic lesions in human colorectal histological images. Oncol Lett 18(6):6101–6107

    PubMed  PubMed Central  Google Scholar 

  45. Awan R, Al-Maadeed S, Al-Saady R, Bouridane A (2020) Glandular structure-guided classification of microscopic colorectal images using deep learning. Computers & Electrical Engineering 85:106450

    Article  Google Scholar 

  46. Dabass M, Vig R, Vashisth S (2018) Five-grade cancer classification of colon histology images via deep learning. In: CRC Press, p 18

  47. Dabass M, Vashisth S, Vig R (2022) A convolution neural network with multi-level convolutional and attention learning for classification of cancer grades and tissue structures in colon histopathological images. Comput Biol Med 147:105680

    Article  PubMed  Google Scholar 

  48. Bianconi F, Kather JN, Reyes-Aldasoro CC (2020) Experimental assessment of color deconvolution and color normalization for automated classification of histology images stained with hematoxylin and eosin. Cancers 12(11):3337

    Article  PubMed  PubMed Central  Google Scholar 

  49. MATLAB (2019) 9.6.0.1072779 (R2019a). The MathWorks Inc., Natick, Massachusetts

  50. Witten IH, Frank E (2002) Data mining: practical machine learning tools and techniques with java implementations. ACM SIGMOD Rec 31(1):76–77

    Article  Google Scholar 

  51. Gelasca ED, Byun J, Obara B, Manjunath B (2008) Evaluation and benchmark for biological image segmentation. In: 2008 15th IEEE International Conference on Image Processing, IEEE, pp 1816–1819

  52. Sirinukunwattana K, Pluim JP, Chen H, Qi X, Heng PA, Guo YB, Wang LY, Matuszewski BJ, Bruni E, Sanchez U et al (2017) Gland segmentation in colon histology images: The glas challenge contest. Med Image Anal 35:489–502

    Article  PubMed  Google Scholar 

  53. AGEMAP NIoA (2020) The atlas of gene expression in mouse aging project (agemap). https://ome.grc.nia.nih.gov/iicbu2008/agemap/index.html, acesso em: 04/05/2020

  54. Rajesh G, Anirudh V, Archana R, Kumar PP, Manoj K (2023) An improved skin cancer classification method using deep convolutional neural networks and transfer learning models. Journal of Engineering Sciences 14(05)

  55. Viet-Linh T (2023) Deep convolutional neural network-based transfer learning method for health condition identification of cable in cable-stayed bridge. Journal of Materials and Engineering Structures 10(1):5–18

    Google Scholar 

  56. Lu S, Lu Z, Zhang YD (2019) Pathological brain detection based on alexnet and transfer learning. Journal of computational science 30:41–47

    Article  Google Scholar 

  57. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  58. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 807–814

  59. Al Rahhal MM, Bazi Y, Abdullah T, Mekhalfi ML, AlHichri H, Zuair M (2018) Learning a multi-branch neural network from multiple sources for knowledge adaptation in remote sensing imagery. Remote Sensing 10(12):1890

  60. Ribeiro MG, Neves LA, do Nascimento MZ, Roberto GF, Martins AS, Tosta TAA, (2019) Classification of colorectal cancer based on the association of multidimensional and multiresolution features. Expert Syst Appl 120:262–278

  61. Kononenko I, Robnik-Sikonja M, Pompe U (1996) Relieff for estimation and discretization of attributes in classification, regression, and ilp problems. Artificial intelligence: methodology, systems, applications pp 31–40

  62. Kononenko I, Šimec E, Robnik-Šikonja M (1997) Overcoming the myopia of inductive learning algorithms with relieff. Appl Intell 7(1):39–55

    Article  Google Scholar 

  63. Robnik-Šikonja M, Kononenko I (2003) Theoretical and empirical analysis of relieff and rrelieff. Mach Learn 53(1):23–69

    Article  Google Scholar 

  64. Cui X, Li Y, Fan J, Wang T (2022) A novel filter feature selection algorithm based on relief. Appl Intell 52(5):5063–5081

    Article  Google Scholar 

  65. Sagi O, Rokach L (2018) Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8(4):e1249

    Google Scholar 

  66. Kassani SH, Kassani PH (2019) A comparative study of deep learning architectures on melanoma detection. Tissue Cell 58:76–83

    Article  Google Scholar 

  67. Cleary JG, Trigg LE (1995) K*: An instance-based learner using an entropic distance measure. In: Machine Learning Proceedings 1995, Elsevier, pp 108–114

  68. Le Cessie S, Van Houwelingen JC (1992) Ridge estimators in logistic regression. J Roy Stat Soc: Ser C (Appl Stat) 41(1):191–201

    Google Scholar 

  69. Lewis DD (1998) Naive (bayes) at forty: The independence assumption in information retrieval. In: Machine Learning: ECML-98: 10th European Conference on Machine Learning Chemnitz, Germany, April 21–23, 1998 Proceedings 10, Springer, pp 4–15

  70. Breiman L (2001) Random forests. Machine learning 45(1):5–32

    Article  Google Scholar 

  71. Alpaydin E (2009) Introduction to machine learning. MIT press

  72. Kittler J, Hatef M, Duin RP, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239

    Article  Google Scholar 

  73. King AP, Eckersley RJ (2019) Chapter 6 - inferential statistics iii: Nonparametric hypothesis testing. In: Eckersley RJ (ed) King AP. Statistics for Biomedical Engineers and Scientists, Academic Press, pp 119–145

    Google Scholar 

  74. Majtner T, Yildirim-Yayilgan S, Hardeberg JY (2016) Combining deep learning and hand-crafted features for skin lesion classification. 2016 Sixth International Conference on Image Processing Theory. Tools and Applications (IPTA), IEEE, pp 1–6

    Google Scholar 

  75. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. nature 542(7639):115–118

  76. dos Santos FP, Ribeiro LS, Ponti MA (2019) Generalization of feature embeddings transferred from different video anomaly detection domains. J Vis Commun Image Represent 60:407–416

    Article  Google Scholar 

  77. Shi Z, Hao H, Zhao M, Feng Y, He L, Wang Y, Suzuki K (2019) A deep cnn based transfer learning method for false positive reduction. Multimedia Tools and Applications 78(1):1017–1033

    Article  Google Scholar 

  78. Ng AY (2004) Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In: Proceedings of the twenty-first international conference on Machine learning, p 78

  79. Kolter JZ, Ng AY (2009) Regularization and feature selection in least-squares temporal difference learning. In: Proceedings of the 26th annual international conference on machine learning, pp 521–528

  80. Schölkopf B, Smola AJ, Bach F, et al (2002) Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press

  81. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, et al (2011) Scikit-learn: Machine learning in python. Journal of machine learning research 12(Oct):2825–2830

  82. Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ (2008) Liblinear: A library for large linear classification. Journal of machine learning research 9(Aug):1871–1874

  83. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research 7(Jan):1–30

  84. Nanni L, Brahnam S, Ghidoni S, Maguolo G (2019) General purpose (genp) bioimage ensemble of handcrafted and learned features with data augmentation. arXiv preprint arXiv:1904.08084

  85. Dong X, Yu Z, Cao W, Shi Y, Ma Q (2020) A survey on ensemble learning. Front Comp Sci 14:241–258

    Article  Google Scholar 

Download references

Funding

This study was financed in part by the: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001; National Council for Scientific and Technological Development CNPq (Grants #132940/2019-1, #313643/2021-0 and #311404/2021-9); the State of Minas Gerais Research Foundation - FAPEMIG (Grant #APQ-00578-18); the State of São Paulo Research Foundation - FAPESP (Grant #2022/03020-1).

Author information

Authors and Affiliations

  1. Department of Computer Science and Statistics (DCCE), São Paulo State University (UNESP), Rua Cristóvão Colombo, 2265, 15054-000, São José do Rio Preto, São Paulo, Brazil

    Cléber I. de Oliveira & Leandro A. Neves

  2. Faculty of Computer Science (FACOM) - Federal University of Uberlândia (UFU), Avenida João Neves de Ávila 2121, Bl.B, 38400-902, Uberlândia, Minas Gerais, Brazil

    Marcelo Z. do Nascimento & Guilherme F. Roberto

  3. Science and Technology Institute (ICT), Federal University of São Paulo (UNIFESP), Avenida Cesare Mansueto Giulio Lattes, 1201, 12247-014, São José dos Campos, São Paulo, Brazil

    Thaína A. A. Tosta

  4. Federal Institute of Triângulo Mineiro (IFTM), Rua Belarmino Vilela Junqueira sn, 38305-200, Ituiutaba, Minas Gerais, Brazil

    Alessandro S. Martins

Authors
  1. Cléber I. de Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcelo Z. do Nascimento
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guilherme F. Roberto
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thaína A. A. Tosta
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alessandro S. Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Leandro A. Neves
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cléber I. de Oliveira.

Ethics declarations

Conflicts of interest / Competing interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Oliveira, C.I., do Nascimento, M.Z., Roberto, G.F. et al. Hybrid models for classifying histological images: An association of deep features by transfer learning with ensemble classifier. Multimed Tools Appl 83, 21929–21952 (2024). https://doi.org/10.1007/s11042-023-16351-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16351-4

Keywords

Search

Navigation