Abstract
In this paper, we propose a hybrid deep learning-based method for brain tumor classification using whole slide images (WSIs) and multimodal magnetic resonance image (mMRI). It comprises two methods: a WSI-based method and a mMRI-based method. For the WSI-based method, many patches are sampled from the WSI for each category as the training dataset. However, not all the sampling patches are representative of the category to which their corresponding WSI belongs without the annotations by pathologists. Therefore, some error tolerance schemes were applied when training the classification model to achieve better generalization. For the mMRI-based method, we firstly apply a 3D convolutional neural network (3DCNN) on the multimodal magnetic resonance image (mMRI) for brain tumor segmentation, which distinguishes brain tumors from healthy tissues, then the segmented tumors are used for tumor subtype classification using 3DCNN. Lastly, an ensemble scheme using the two methods was performed to reach a consensus as the final predictions. We evaluate the proposed method with the patient dataset from Computational Precision Medicine: Radiology-Pathology Challenge (CPM: Rad-Path) on Brain Tumor Classification 2020. The performance of the prediction results on the validation set reached 0.886 in f1_micro, 0.801 in kappa, 0.8 in balance_acc, and 0.829 in the overall average. The experimental results show that the performance with the consideration of both MRI and WSI outperforms the performance using single type of image dataset. Accordingly, the fusion from two image datasets can provide more sufficient information in diagnosis for the system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ostrom, Q.T., Gittleman, H., Truitt, G., Boscia, A., Kruchko, C., Barnholtz-Sloan, J.S.: CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2011–2015. Neuro Oncol. 20(Suppl_4), iv1–iv86 (2018)
Louis, D.N., et al.: The 2016 World Health Organization classification of tumors of the central nervous system: a summary. Acta Neuropathol. 131(6), 803–820 (2016)
Chen, J., McKay, R.M., Parada, L.F.: Malignant glioma: lessons from genomics, mouse models, and stem cells. Cell 149(1), 36–47 (2012)
Louis, D.N., et al.: The 2007 WHO classification of tumours of the central nervous system. Acta Neuropathol. 114(2), 97–109 (2007)
Ma, X., Jia, F.: Brain tumor classification with multimodal MR and pathology images. In: Crimi, A., Bakas, S. (eds.) BrainLes 2019. LNCS, vol. 11993, pp. 343–352. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46643-5_34
Chan, H.-W., Weng, Y.-T., Huang, T.-Y.: Automatic classification of brain tumor types with the MRI scans and histopathology images. In: Crimi, A., Bakas, S. (eds.) BrainLes 2019. LNCS, vol. 11993, pp. 353–359. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46643-5_35
Xue, Y., et al.: Brain tumor classification with tumor segmentations and a dual path residual convolutional neural network from MRI and pathology images. In: Crimi, A., Bakas, S. (eds.) BrainLes 2019. LNCS, vol. 11993, pp. 360–367. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46643-5_36
Kothari, S., Phan, J.H., Young, A.N., Wang, M.D.: Histological image classification using biologically interpretable shape-based features. BMC Med. Imaging 13(1), 9 (2013)
Chang, H., Zhou, Y., Spellman, P., Parvin, B.: Stacked predictive sparse coding for classification of distinct regions in tumor histopathology. In: 2013 Proceedings of the IEEE International Conference on Computer Vision, pp. 169–176 (2013)
Zacharaki, E.I., et al.: Classification of brain tumor type and grade using MRI texture and shape in a machine learning scheme. Magn. Reson. Med. Off. J. Int. Soc. Magn. Reson. Med. 62(6), 1609–1618 (2009)
Machhale, K., Nandpuru, H.B., Kapur, V., Kosta, L.: MRI brain cancer classification using hybrid classifier (SVM-KNN). In: 2015 International Conference on Industrial Instrumentation and Control (ICIC), pp. 60–65. IEEE (2015)
Zulpe, N., Pawar, V.: GLCM textural features for brain tumor classification. Int. J. Comput. Sci. Issues (IJCSI) 9(3), 354 (2012)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: Sequential deep learning for human action recognition. In: Salah, A.A., Lepri, B. (eds.) HBU 2011. LNCS, vol. 7065, pp. 29–39. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25446-8_4
Young, T., Hazarika, D., Poria, S., Cambria, E.: Recent trends in deep learning based natural language processing. IEEE Comput. Intell. Mag. 13(3), 55–75 (2018)
Havaei, M., et al.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)
Sajjad, M., Khan, S., Muhammad, K., Wu, W., Ullah, A., Baik, S.W.: Multi-grade brain tumor classification using deep CNN with extensive data augmentation. J. Comput. Sci. 30, 174–182 (2019)
Pei, L., Vidyaratne, L., Hsu, W.-W., Rahman, M.M., Iftekharuddin, K.M.: Brain tumor classification using 3D convolutional neural network. In: Crimi, A., Bakas, S. (eds.) BrainLes 2019. LNCS, vol. 11993, pp. 335–342. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46643-5_33
Barker, J., Hoogi, A., Depeursinge, A., Rubin, D.L.: Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles. Med. Image Anal. 30, 60–71 (2016)
Sultan, H.H., Salem, N.M., Al-Atabany, W.: Multi-classification of brain tumor images using deep neural network. IEEE Access 7, 69215–69225 (2019)
Bakas, S., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629 (2018)
Bakas, S., et al.: Segmentation labels and radiomic features for the pre-operative scans of the TCGA-LGG collection. The Cancer Imaging Archive, vol. 286 (2017)
Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2014)
Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. Data 4, 170117 (2017)
Northcutt, C.G., Jiang, L., Chuang, I.L.: Confident learning: estimating uncertainty in dataset labels. arXiv preprint arXiv:1911.00068 (2019)
Xie, Q., Luong, M.-T., Hovy, E., Le, Q.V.: Self-training with noisy student improves imagenet classification. In: 2020 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687–10698 (2020)
Kurc, T., et al.: Segmentation and classification in digital pathology for glioma research: challenges and deep learning approaches. Front. Neurosci. 14(27) (2020). Original Research
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Zhou, Z.-H.: A brief introduction to weakly supervised learning. Nat. Sci. Rev. 5(1), 44–53 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Pei, L., Hsu, WW., Chiang, LA., Guo, JM., Iftekharuddin, K.M., Colen, R. (2021). A Hybrid Convolutional Neural Network Based-Method for Brain Tumor Classification Using mMRI and WSI. In: Crimi, A., Bakas, S. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2020. Lecture Notes in Computer Science(), vol 12659. Springer, Cham. https://doi.org/10.1007/978-3-030-72087-2_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-72087-2_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72086-5
Online ISBN: 978-3-030-72087-2
eBook Packages: Computer ScienceComputer Science (R0)