Abstract
Whole-Slide Images (WSI) are emerging as a promising resource for studying biological tissues, demonstrating a great potential in aiding cancer diagnosis and improving patient treatment. However, the manual pixel-level annotation of WSIs is extremely time-consuming and practically unfeasible in real-world scenarios. Multi-Instance Learning (MIL) have gained attention as a weakly supervised approach able to address lack of annotation tasks. MIL models aggregate patches (e.g., cropping of a WSI) into bag-level representations (e.g., WSI label), but neglect spatial information of the WSIs, crucial for histological analysis. In the High-Grade Serous Ovarian Cancer (HGSOC) context, spatial information is essential to predict a prognosis indicator (the Platinum-Free Interval, PFI) from WSIs. Such a prediction would bring highly valuable insights both for patient treatment and prognosis of chemotherapy resistance. Indeed, NeoAdjuvant ChemoTherapy (NACT) induces changes in tumor tissue morphology and composition, making the prediction of PFI from WSIs extremely challenging. In this paper, we propose GDS-MIL, a method that integrates a state-of-the-art MIL model with a Graph ATtention layer (GAT in short) to inject a local context into each instance before MIL aggregation. Our approach achieves a significant improvement in accuracy on the “Ome18” PFI dataset. In summary, this paper presents a novel solution for enhancing PFI prediction in HGSOC, with the potential of significantly improving treatment decisions and patient outcomes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
\(x^{ins}_i\) represents a patch extracted from the \(X^{bag}\), i.e., the entire WSI.
- 2.
A fixed threshold is applied on the HSV (Hue Saturation Brightness) color space of the WSI thumbnail and later propagated to \(5\times \) and \(20\times \) resolutions.
References
Allegretti, S., Bolelli, F., Cancilla, M., Pollastri, F., Canalini, L., Grana, C.: How does connected components labeling with decision trees perform on GPUs? In: Vento, M., Percannella, G. (eds.) CAIP 2019. LNCS, vol. 11678, pp. 39–51. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29888-3_4
Bontempo, G., Porrello, A., Bolelli, F., Calderara, S., Ficarra, E.: DAS-MIL: Distilling Across Scales for MIL classification of histological WSIs. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2023 (2023)
Brody, S., Alon, U., Yahav, E.: How attentive are graph attention networks? In: International Conference on Learning Representations (2022)
Caron, M., et al.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650–9660 (2021)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2021)
Evans, A.J., et al.: US food and drug administration approval of whole slide imaging for primary diagnosis: a key milestone is reached and new questions are raised. Arch. Pathol. Lab. Med. 142(11), 1383–1387 (2018)
Fatemi, M., et al.: Inferring spatial transcriptomics markers from whole slide images to characterize metastasis-related spatial heterogeneity of colorectal tumors: a pilot study. J. Pathol. Inf. 14, 100308 (2023)
Feng, J., Zhou, Z.H.: Deep MIML network. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 1884–1890. AAAI Press (2017)
Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: Proceedings of the 35th International Conference on Machine Learning, pp. 2127–2136. PMLR (2018)
Kehoe, S., et al.: Primary chemotherapy versus primary surgery for newly diagnosed advanced ovarian cancer (CHORUS): an open-label, randomised, controlled, non-inferiority trial. Lanchet 386(9990), 249–257 (2015)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations. ICLR (2017)
Kumar, N., Gupta, R., Gupta, S.: Whole slide imaging (WSI) in pathology: current perspectives and future directions. J. Digit. Imaging 33(4), 1034–1040 (2020)
Landi, F., Baraldi, L., Corsini, M., Cucchiara, R.: Embodied vision-and-language navigation with dynamic convolutional filters. In: Proceedings of the British Machine Vision Conference (2019)
Laury, A.R., Blom, S., Ropponen, T., Virtanen, A., Carpén, O.M.: Artificial intelligence-based image analysis can predict outcome in high-grade serous carcinoma via histology alone. Sci. Rep. 11(1), 19165 (2021)
Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14318–14328 (2021)
Lovino, M., Ciaburri, M.S., Urgese, G., Di Cataldo, S., Ficarra, E.: DEEPrior: a deep learning tool for the prioritization of gene fusions. Bioinformatics 36(10), 3248–3250 (2020)
Lu, M.Y., Chen, R.J., Wang, J., Dillon, D., Mahmood, F.: Semi-supervised histology classification using deep multiple instance learning and contrastive predictive coding. arXiv preprint: arXiv:1910.10825 (2019)
Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5(6), 555–570 (2021)
Martina, J.D., Simmons, C., Jukic, D.M.: High-definition hematoxylin and eosin staining in a transition to digital pathology. J. Pathol. Inf. 2(1), 45 (2011)
Meza-Perez, S., Randall, T.D.: Immunological functions of the Omentum. Trends Immunol. 38(7), 526–536 (2017)
Morelli, D., Fincato, M., Cornia, M., Landi, F., Cesari, F., Cucchiara, R.: Dress code: high-resolution multi-category virtual try-on. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2022)
Nikolaidi, A., Fountzilas, E., Fostira, F., Psyrri, A., Gogas, H., Papadimitriou, C.: Neoadjuvant treatment in ovarian cancer: new perspectives, new challenges. Front. Oncol., 3758 (2022)
Panariello, A., Porrello, A., Calderara, S., Cucchiara, R.: Consistency-based self-supervised learning for temporal anomaly localization. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds.) ECCV 2022. Lecture Notes in Computer Science, vol. 13805, pp. 338–349. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-25072-9_22
Pinheiro, P.O., Collobert, R.: From image-level to pixel-level labeling with convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1713–1721 (2015)
Ponzio, F., Urgese, G., Ficarra, E., Di Cataldo, S.: Dealing with lack of training data for convolutional neural networks: the case of digital pathology. Electronics 8(3), 256 (2019)
Porrello, A., Abati, D., Calderara, S., Cucchiara, R.: Classifying signals on irregular domains via convolutional cluster pooling. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1388–1397 (2019)
Pujade-Lauraine, E., Combe, P.: Recurrent ovarian cancer. Ann. Oncol. 27, i63–i65 (2016)
Shao, Z., et al.: TransMIL: transformer based correlated multiple instance learning for whole slide image classification. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 2136–2147 (2021)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations (ICLR 2015) (2015)
Sokolova, M., Japkowicz, N., Szpakowicz, S.: Beyond accuracy, F-Score and ROC: a family of discriminant measures for performance evaluation. In: Sattar, A., Kang, B. (eds.) AI 2006. LNCS (LNAI), vol. 4304, pp. 1015–1021. Springer, Heidelberg (2006). https://doi.org/10.1007/11941439_114
Tomei, M., Baraldi, L., Calderara, S., Bronzin, S., Cucchiara, R.: RMS-Net: regression and masking for soccer event spotting. In: 25th International Conference on Pattern Recognition (ICPR), pp. 7699–7706. IEEE (2021)
Tourniaire, P., Ilie, M., Hofman, P., Ayache, N., Delingette, H.: Attention-based multiple instance learning with mixed supervision on the camelyon16 dataset. In: Proceedings of the MICCAI Workshop on Computational Pathology, pp. 216–226. PMLR (2021)
Velickovic, P., et al.: Graph attention networks. Stat 1050(20), 10–48550 (2017)
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Philip, S.Y.: A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32(1), 4–24 (2020)
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020)
Yu, K.H., et al.: Deciphering serous ovarian carcinoma histopathology and platinum response by convolutional neural networks. BMC Med. 18(1), 1–14 (2020)
Zhang, H., et al.: DTFD-MIL: double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18802–18812 (2022)
Zhang, J., Hu, J.: Image segmentation based on 2D Otsu method with histogram analysis. In: International Conference on Computer Science and Software Engineering, vol. 6, pp. 105–108. IEEE (2008)
Acknowledgements
This project has received funding from DECIDER, the European Union’s Horizon 2020 research and innovation programme under GA No. 965193, and from the Department of Engineering “Enzo Ferrari” of the University of Modena through the FARD-2022 (Fondo di Ateneo per la Ricerca 2022).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bontempo, G., Bartolini, N., Lovino, M., Bolelli, F., Virtanen, A., Ficarra, E. (2023). Enhancing PFI Prediction with GDS-MIL: A Graph-Based Dual Stream MIL Approach. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing – ICIAP 2023. ICIAP 2023. Lecture Notes in Computer Science, vol 14233. Springer, Cham. https://doi.org/10.1007/978-3-031-43148-7_46
Download citation
DOI: https://doi.org/10.1007/978-3-031-43148-7_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43147-0
Online ISBN: 978-3-031-43148-7
eBook Packages: Computer ScienceComputer Science (R0)