Abstract
Synthesizing multimodality medical data provides complementary knowledge and helps doctors make precise clinical decisions. Although promising, existing multimodal brain graph synthesis frameworks have several limitations. First, they mainly tackle only one problem (intra- or inter-modality), limiting their generalizability to synthesizing inter- and intra-modality simultaneously. Second, while few techniques work on super-resolving low-resolution brain graphs within a single modality (i.e., intra), inter-modality graph super-resolution remains unexplored though this would avoid the need for costly data collection and processing. More importantly, both target and source domains might have different distributions, which causes a domain fracture between them. To fill these gaps, we propose a multi-resolution StairwayGraphNet (SG-Net) framework to jointly infer a target graph modality based on a given modality and super-resolve brain graphs in both inter and intra domains. Our SG-Net is grounded in three main contributions: (i) predicting a target graph from a source one based on a novel graph generative adversarial network in both inter (e.g., morphological-functional) and intra (e.g., functional-functional) domains, (ii) generating high-resolution brain graphs without resorting to the time consuming and expensive MRI processing steps, and (iii) enforcing the source distribution to match that of the ground truth graphs using an inter-modality aligner to relax the loss function to optimize. Moreover, we design a new Ground Truth-Preserving loss function to guide both generators in learning the topological structure of ground truth brain graphs more accurately. Our comprehensive experiments on predicting target brain graphs from source graphs using a multi-resolution stairway showed the outperformance of our method in comparison with its variants and state-of-the-art method. SG-Net presents the first work for graph alignment and synthesis across varying modalities and resolutions, which handles graph size, distribution, and structure variations. Our Python TIS-Net code is available on BASIRA GitHub at https://github.com/basiralab/SG-Net.
This project has been funded by the TUBITAK 2232 Fellowship (Project No: 118C288).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Cao, B., Zhang, H., Wang, N., Gao, X., Shen, D.: Auto-gan: self-supervised collaborative learning for medical image synthesis. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10486–10493 (2020)
Yu, B., Wang, Y., Wang, L., Shen, D., Zhou, L.: Medical image synthesis via deep learning. In: Lee, G., Fujita, H. (eds.) Deep Learning in Medical Image Analysis. AEMB, vol. 1213, pp. 23–44. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-33128-3_2
Singh, N.K., Raza, K.: Medical image generation using generative adversarial networks: A review, pp. 77–96. A Computational Perspective in Healthcare, Health Informatics (2021)
Wang, C.: Dicyc: gan-based deformation invariant cross-domain information fusion for medical image synthesis. Inf. Fus. 67, 147–160 (2021)
Liu, Y., Pan, Y., Yang, W., Ning, Z., Yue, L., Liu, M., Shen, D.: Joint neuroimage synthesis and representation learning for conversion prediction of subjective cognitive decline. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 583–592 (2020)
Zhan, B., et al.: LR-CGAN: latent representation based conditional generative adversarial network for multi-modality MRI synthesis. Biomedical Signal Processing and Control 66, 102457 (2021)
van den Heuvel, M.P., Sporns, O.: A cross-disorder connectome landscape of brain dysconnectivity. Nature Rev. Neurosci. 20, 435–446 (2019)
Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond euclidean data. IEEE Sig. Process. Mag. 34, 18–42 (2017)
Bessadok, A., Mahjoub, M.A., Rekik, I.: Graph neural networks in network neuroscience. arXiv preprint arXiv:2106.03535 (2021)
Bessadok, A., Mahjoub, M.A., Rekik, I.: Topology-aware generative adversarial network for joint prediction of multiple brain graphs from a single brain graph. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 551–561 (2020)
Zhang, L., Wang, L., Zhu, D.: Recovering brain structural connectivity from functional connectivity via multi-gcn based generative adversarial network. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 53–61 (2020)
Isallari, M., Rekik, I.: GSR-Net: Graph super-resolution network for predicting high-resolution from low-resolution functional brain connectomes. In: International Workshop on Machine Learning in Medical Imaging, pp. 139–149 (2020)
Pan, S., Hu, R., Long, G., Jiang, J., Yao, L., Zhang, C.: Adversarially regularized graph autoencoder for graph embedding. arXiv preprint arXiv:1802.04407 (2018)
Simonovsky, M., Komodakis, N.: Dynamic edge-conditioned filters in convolutional neural networks on graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3693–3702 (2017)
Zemouri, R.: Semi-supervised adversarial variational autoencoder. Mach. Learn. Know. Extr. 2, 361–378 (2020)
Goodfellow, I.J., et al.: Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014)
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241 (2015)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Gürler, Z., Nebli, A., Rekik, I.: Foreseeing brain graph evolution over time using deep adversarial network normalizer. In: International Workshop on Predictive Intelligence In Medicine, pp. 111–122 (2020)
Fey, M., Lenssen, J.E.: Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428 (2019)
Fischl, B.: Neuroimage. Freesurfer 62, 774–781 (2012)
Dosenbach, N.U., et al.: Prediction of individual brain maturity using FMRI. Science 329, 1358–1361 (2010)
Shen, X., Tokoglu, F., Papademetris, X., Constable, R.T.: Groupwise whole-brain parcellation from resting-state FMRI data for network node identification. Neuroimage 82, 403–415 (2013)
Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam (2018)
Glasser, M.F., et al.: A multi-modal parcellation of human cerebral cortex. Nature 536, 171 (2016)
Acknowledgments
This work was funded by generous grants from the European H2020 Marie Sklodowska-Curie action (grant no. 101003403, http://basira-lab.com/normnets/) to I.R. and the Scientific and Technological Research Council of Turkey to I.R. under the TUBITAK 2232 Fellowship for Outstanding Researchers (no. 118C288, http://basira-lab.com/reprime/). However, all scientific contributions made in this project are owned and approved solely by the authors.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Mhiri, I., Mahjoub, M.A., Rekik, I. (2021). StairwayGraphNet for Inter- and Intra-modality Multi-resolution Brain Graph Alignment and Synthesis. In: Lian, C., Cao, X., Rekik, I., Xu, X., Yan, P. (eds) Machine Learning in Medical Imaging. MLMI 2021. Lecture Notes in Computer Science(), vol 12966. Springer, Cham. https://doi.org/10.1007/978-3-030-87589-3_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-87589-3_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87588-6
Online ISBN: 978-3-030-87589-3
eBook Packages: Computer ScienceComputer Science (R0)