Nothing Special   »   [go: up one dir, main page]
Skip to main content
SpringerLink
Log in

Hybrid generative adversarial network based on a mixed attention fusion module for multi-modal MR image synthesis algorithm

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Recently, medical image synthesis has attracted the attention of an increasing number of researchers. However, most of current approaches suffer from the loss of multi-modal complementary information and thus fail to preserve the property of each modality, resulting in image distortion and texture detail loss. To alleviate this issue, a multi-modal magnetic resonance (MR) image synthesis algorithm based on a mixed attention fusion module in hybrid generative adversarial network is proposed. Firstly, a novel mixed attention fusion (MAF) module aggregating an adaptive fusion strategy (AFS) and a soft attention module is proposed to fuse the high-level semantic information and the low-level fine-grained feature at different scales between different layers to exploit rich representative complementary information adaptively. Subsequently, Resnet-bottlenect attention mechanism (Res-BAM) is designed to perform adaptive optimization and exploit mutual information while preserving the original property of each modality. Thereafter, the attention weight is inferred by a 1D channel feature map and a 2D spatial feature map, and multiplied with the original feature map in order to get the adaptive feature map, which is integrated with the original feature map in a residual connection to preserve the original property of each modality and prevent network degradation. Finally, the structural similarity (SSIM) and \({\text{L}}_{1}\)-norm are point-wise combined by an optimal weighting impact factor to preserve the high frequency information, brightness, color and SSIM, which are viewed as the original property of each modality. The experimental results demonstrate the superiority of our model on the state of the art in quantitative measures, reasonable visual quality and clinic significance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig.12
Fig. 13

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Chartsias A, Joyce T, Dharmakumar R, Tsaftaris SA (2017) Adversarial image synthesis for unpaired multi-modal cardiac data. International workshop on simulation and synthesis in medical imaging. Springer, pp 3–13

    Chapter  Google Scholar 

  2. Lee D, Kim J, Moon W-J, Ye JC (2019) CollaGAN: collaborative GAN for missing image data imputation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2487–2496

  3. Gaur L, Bhatia U, Jhanjhi NZ, Muhammad G, Masud M (2021) Medical image-based detection of COVID-19 using Deep Convolution Neural Networks. Springer, Berlin Heidelberg

    Google Scholar 

  4. Biswas M, Gaur L, Alenezi F, Santosh KC, Mahbub MK (2022) Deep features to detect pulmonary abnormalities in chest X-rays due to infectious diseaseX: Covid-19, pneumonia, and tuberculosis. Inform Sci Int J 592:592

    Google Scholar 

  5. Yang H, Wang L, Xu Y, Liu X (2023) CovidViT: a novel neural network with self-attention mechanism to detect Covid-19 through X-ray images. Int J Mach Learn Cybern 14(3):973–987

    Article  Google Scholar 

  6. Degen J, Heinrich MP (2016) Multi-atlas based pseudo-ct synthesis using multimodal image registration and local atlas fusion strategies. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 160–168

  7. Burgos N et al (2014) Attenuation correction synthesis for hybrid PET-MR scanners: application to brain studies. IEEE Trans Med Imaging 33(12):2332–2341

    Article  Google Scholar 

  8. Chen M, Carass A, Jog A, Lee J, Roy S, Prince JLJ (2017) Cross contrast multi-channel image registration using image synthesis for MR brain images. Med Image Anal 36:2–14

    Article  Google Scholar 

  9. Jog A, Carass A, Roy S, Pham DL, Prince JL (2017) Random forest regression for magnetic resonance image synthesis. Med Image Anal 35:475–488

    Article  Google Scholar 

  10. Wang Y et al (2016) Semisupervised tripled dictionary learning for standard-dose PET image prediction using low-dose PET and multimodal MRI. IEEE Trans Biomed Eng 64(3):569–579

    Article  Google Scholar 

  11. Qu X, Hou Y, Lam F, Guo D, Zhong J, Chen ZJ (2014) Magnetic resonance image reconstruction from undersampled measurements using a patch-based nonlocal operator. Med Image Anal 18(6):843–856

    Article  Google Scholar 

  12. Ye DH, Zikic D, Glocker B, Criminisi A, Konukoglu E (2013) Modality propagation: coherent synthesis of subject-specific scans with data-driven regularization. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 606-613

  13. Huynh T et al (2015) Estimating CT image from MRI data using structured random forest and auto-context model. IEEE Trans Med Imaging 35(1):174–183

    Article  Google Scholar 

  14. LeCun Y, Bottou L, Bengio Y, Haffner PJ (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  15. Sharma DK, Gaur L, Okunbor D (2007) Image compression and feature extraction using Kohonen's self-organizing map neural network. J Strategic E-commerce 5(1/2):25

    Google Scholar 

  16. Li R et al (2014) Deep learning based imaging data completion for improved brain disease diagnosis. International conference on medical image computing and computer-assisted intervention. Springer, pp 305–312

    Google Scholar 

  17. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  18. Gao F, Wu T, Chu X, Yoon H, Xu Y, Patel B (2019) Deep residual inception encoder–decoder network for medical imaging synthesis. IEEE J Biomed Health Inform 24(1):39–49

    Article  Google Scholar 

  19. Creswell A, White T, Dumoulin V, Arulkumaran K, Sengupta B, Bharath AA (2018) Generative adversarial networks: An overview. IEEE Signal Process Mag 35(1):53–65

    Article  Google Scholar 

  20. Yi X, Walia E, Babyn P (2019) Generative adversarial network in medical imaging: a review. Med Image Anal 58:101552

    Article  Google Scholar 

  21. Ben-Cohen A et al (2019) Cross-modality synthesis from CT to PET using FCN and GAN networks for improved automated lesion detection. Eng Appl Artif Intell 78:186–194

    Article  Google Scholar 

  22. Armanious K et al (2020) MedGAN: Medical image translation using GANs. Comput Med Imaging Graph 79:101684

    Article  Google Scholar 

  23. Dar SU, Yurt M, Karacan L, Erdem A, Erdem E, Çukur T (2019) Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Trans Med Imaging 38(10):2375–2388

    Article  Google Scholar 

  24. Nie D et al (2017) Medical image synthesis with context-aware generative adversarial networks. International conference on medical image computing and computer-assisted intervention. Springer, pp 417–425

    Google Scholar 

  25. Wang Y et al (2018) 3D auto-context-based locality adaptive multi-modality GANs for PET synthesis. IEEE Trans Med Imaging 38(6):1328–1339

    Article  Google Scholar 

  26. Huang Y, Shao L, Frangi AF (2017) Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6070–6079

  27. Du G, Zhou L, Li Z, Wang L, Lü K (2023) Neighbor-aware deep multi-view clustering via graph convolutional network. Inf Fusion 93:330–343

    Article  Google Scholar 

  28. Du G, Zhou L, Yang Y, Lü K, Wang L (2021) Deep multiple auto-encoder-based multi-view clustering. Data Sci Eng 6(3):323–338

    Article  Google Scholar 

  29. Olut S, Sahin YH, Demir U, Unal G (2018) Generative adversarial training for MRA image synthesis using multi-contrast MRI. In: International workshop on predictive intelligence in medicine, Springer, pp. 147-154

  30. A. Sharma and G. J. I. t. o. m. i. Hamarneh, "Missing MRI pulse sequence synthesis using multi-modal generative adversarial network," vol. 39, no. 4, pp. 1170–1183, 2019.

  31. Reaungamornrat S, Sari H, Catana C, Kamen A (2022) Multimodal image synthesis based on disentanglement representations of anatomical and modality specific features, learned using uncooperative relativistic GAN. Med Image Anal 80:102514

    Article  Google Scholar 

  32. Yi D, Lei Z, Li SZ (2015) Shared representation learning for heterogenous face recognition. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), vol. 1, IEEE, pp. 1–7

  33. Chen G, Srihari SN (2015) Generalized K-fan multimodal deep model with shared representations. arXiv preprint arXiv:1503.07906

  34. Ngiam J, Khosla A, Kim M, Nam J, Lee H, Ng AY (2011) Multimodal deep learning. In: Proceedings of the 28th international conference on machine learning (ICML-11), pp 689–696

  35. Jog A, Roy S, Carass A, Prince JL (2013) Magnetic resonance image synthesis through patch regression. In: 2013 IEEE 10th International Symposium on Biomedical Imaging, IEEE, pp. 350–353

  36. Wen C, Huai T, Zhang Q, Song Z, Cao F (2022) A new rotation forest ensemble algorithm. Int J Mach Learn Cybern 13(11):3569–3576

    Article  Google Scholar 

  37. Huang Y, Shao L, Frangi AF (2017) Cross-modality image synthesis via weakly coupled and geometry co-regularized joint dictionary learning. IEEE Trans Med Imaging 37(3):815–827

    Article  Google Scholar 

  38. Oksuz I (2022) Dictionary learning for medical image synthesis. Biomedical Image Synthesis and Simulation. Elsevier, pp. 79–89

  39. Lee J, Carass A, Jog A, Zhao C, Prince JL (2017) Multi-atlas-based CT synthesis from conventional MRI with patch-based refinement for MRI-based radiotherapy planning. In: Medical Imaging 2017: Image Processing, vol. 10133: SPIE, pp. 434–439

  40. Jog A, Carass A, Roy S, Pham DL, Prince JL (2015) MR image synthesis by contrast learning on neighborhood ensembles. Med Image Anal 24(1):63–76

    Article  Google Scholar 

  41. Zhao C, Carass A, Lee J, He Y, Prince JL (2017) Whole brain segmentation and labeling from CT using synthetic MR images. In: International Workshop on Machine Learning in Medical Imaging, Springer, pp. 291-298

  42. Miller MI, Christensen GE, Amit Y, Grenander U (1993) Mathematical textbook of deformable neuroanatomies. Proc Natl Acad Sci 90(24):11944–11948

    Article  Google Scholar 

  43. Guibas JT, Virdi TS, Li PS (2017) Synthetic medical images from dual generative adversarial networks. arXiv preprint arXiv:1709.01872

  44. Zhang T et al (2019) SkrGAN: sketching-rendering unconditional generative adversarial networks for medical image synthesis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 777-785

  45. You C et al (2019) CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE). IEEE Trans Med Imaging 39(1):188–203

    Article  Google Scholar 

  46. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125–1134

  47. Sangkloy P, Lu J, Fang C, Yu F, Hays J (2017) Scribbler: controlling deep image synthesis with sketch and color. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400–5409

  48. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536–2544

  49. Zhang H; Xu T; Li H; Zhang S; Wang X; Huang X; Metaxas DN (2017) StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 5907–5915

  50. Mathesul S, Bhutkar G, Rambhad A (2022) AttnGAN: realistic text-to-image synthesis with attentional generative adversarial networks. In: IFIP Conference on Human-Computer Interaction

  51. Yin G, Liu B, Sheng L, Yu N, Wang X, Shao J (2019) Semantics disentangling for text-to-image generation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  52. Wang F et al (2017) Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156–3164

  53. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  54. Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp. 3–19

  55. Hardoon DR, Szedmak S, Shawe-Taylor JJ (2004) Canonical correlation analysis: An overview with application to learning methods. Neural Comput 16(12):2639–2664

    Article  Google Scholar 

  56. Nazarpour A, Adibi PJ (2015) Two-stage multiple kernel learning for supervised dimensionality reduction. Pattern Recogn 48(5):1854–1862

    Article  Google Scholar 

  57. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Springer, pp 234–241

    Google Scholar 

  58. Zhang H, Goodfellow I, Metaxas D, Odena A (2019) Self-attention generative adversarial networks. In: International conference on machine learning PMLR, pp. 7354–7363

  59. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, Glocker B, Rueckert D (2018) Attention U-Net: learning where to look for the pancreas. arXiv preprint. arXiv:1804.03999

  60. Li S, Liu J, Song Z (2022) Brain tumor segmentation based on region of interest-aided localization and segmentation U-Net. Int J Mach Learn Cybern 13(9):2435–2445

  61. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp. 2223–2232

  62. Menze BH et al (2014) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 34(10):1993–2024

    Article  Google Scholar 

  63. Zhou T, Fu H, Chen G, Shen J, Shao LJ (2020) Hi-net: hybrid-fusion network for multi-modal MR image synthesis. IEEE Transact Med Imaging 39(9):2772–2781

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by “Famous teacher of teaching” of Yunnan 10000 Talents Program, the National Nature Science Foundation of China under Grants 62266049 and 62066047, the Program of Yunnan Key Laboratory of Intelligent Systems and Computing under grants 202205AG070003, the Postgraduate Research and Innovation Foundation of Yunnan University 2021Z075 and 2021Y256.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Yunnan University, Kunming, 650504, China

    Haiyan Li, Yongqiang Han & Jun Chang

  2. Journal of Yunnan University Natural Science Edition, Yunnan University, Kunming, 650504, China

    Liping Zhou

Authors
  1. Haiyan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongqiang Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liping Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

HL: conceptualization, investigation, writing—review and editing. YH: validation, formal analysis, visualization, software, writing—original draft. JC: writing—review and editing. LZ: resources, writing—review & editing.

Corresponding author

Correspondence to Jun Chang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Han, Y., Chang, J. et al. Hybrid generative adversarial network based on a mixed attention fusion module for multi-modal MR image synthesis algorithm. Int. J. Mach. Learn. & Cyber. 15, 2111–2130 (2024). https://doi.org/10.1007/s13042-023-02019-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-02019-w

Keywords

Search

Navigation