Nothing Special   »   [go: up one dir, main page]
Skip to main content
SpringerLink
Log in

Multi-scale fusion and adaptively attentive generative adversarial network for image de-raining

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The quality of images taken on rainy days is decreased due to rain streaks. These degraded images affect the performance of vision applications (e.g., face detection and verification). Besides, because of multi-directions, multi-sizes and multi-densities of rain streaks, the existing rain removal methods lack deeper insight into the rain image and are not ideal to remove rain. To address the problem, we propose a novel rain removal model named Multi-scale Fusion and Adaptively Attentive Generative Adversarial Network (MFAA-GAN) to efficiently remove rain streaks. First, in MFAA-GAN, to extract multiscale features and the correlation information between cross-scale features, we design a multiscale feature fusion module that takes two parallel residual dense blocks with different sizes of convolution kernels. Secondly, we add an adaptive attention algorithm including spatial attention and channel attention in generator to capture local and global position information respectively. Third, in the training procedure, we propose a new multi-scale perceptual loss function to reduce artifacts introduced by GAN and ensure better visual quality. The experiments on synthetic and real datasets prove that MFAA-GAN is superior to other rain removal models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The datasets generated and analyzed during the current study are available from the corresponding author upon reasonable request.

References

  1. Yang Y, Cai J, Yang H, Zhang J, Zhao X (2020) Expert Syst Appl 139:112846. https://doi.org/10.1016/j.eswa.2019.112846

    Article  Google Scholar 

  2. Yang H, Zhou L, Cai J, Shi C, Yang Y, Zhao X, Duan J, Yin X (2022) Mon Not R Astron Soc 518(4):5904. https://doi.org/10.1093/mnras/stac3292

    Article  Google Scholar 

  3. Yang H, Shi C, Cai J, Zhou L, Yang Y, Zhao X, He Y, Hao J (2022) Monthly Notices of the Royal Astronomical Society 517(4):5496. https://doi.org/10.1093/mnras/stac2975

    Article  Google Scholar 

  4. Cai J, Hao J, Yang H, Zhao X, Yang Y (2023) Information Sciences 632:164. https://doi.org/10.1016/j.ins.2023.02.088, https://www.sciencedirect.com/science/article/pii/S0020025523002840

  5. Qu CX, Yang H, Cai JH, Xun Y, Guang Pu Xue Yu Guang Pu Fen Xi (2020) Spectrosc Spectr Anal 40:1304. https://doi.org/10.3964/j.issn.1000-0593(2020)04-1304-05

  6. Wu W, Zhou K, Chen XD, Yong JH (2022) Comput Vis Image Underst 216:103341. https://doi.org/10.1016/j.cviu.2021.103341, https://www.sciencedirect.com/science/article/pii/S1077314221001752

  7. Wu W, Chen XD, Yang W, Yong JH (2023) Knowledge-Based Systems 273:110614. https://doi.org/10.1016/j.knosys.2023.110614, https://doi.org/www.sciencedirect.com/science/article/pii/S0950705123003647

  8. Yang W, Wu W, Chen XD, Tao X, Mao X (2023) Appl Intell 1–15. https://api.semanticscholar.org/CorpusID:258162096

  9. Wu W, Yang W, Ma W, Chen XD (2023) IEEE Transactions on Circuits and Systems for Video Technology pp. 1. https://doi.org/10.1109/TCSVT.2023.3263903

  10. Zhang H, Patel VM (2018) in Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 695–704

  11. Fu X, Liang B, Huang Y, Ding X, Paisley J (2020) IEEE Transactions on Neural Networks and Learning Systems 31(6):1794. https://doi.org/10.1109/TNNLS.2019.2926481

    Article  Google Scholar 

  12. Jiang K, Wang Z, Yi P, Chen C, Huang B, Luo Y, Ma J, Jiang J (2020) in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8346–8355

  13. Wang C, Xing X, Wu Y, Su Z, Chen J (2020) in Proceedings of the 28th ACM international conference on multimedia, pp. 1643– 1651

  14. Park Y, Jeon M, Lee J, Kang M (2022) Signal Process: Image Commun 105:116701

  15. Qian R, Tan RT, Yang W, Su J, Liu J (2018) in Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2482–2491

  16. Zhang H, Sindagi V, Patel VM (2019) IEEE transactions on circuits and systems for video technology 30(11):3943

    Article  Google Scholar 

  17. Li R, Cheong LF, Tan RT (2019) in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1633–1642

  18. Li G, He X, Zhang W, Chang H, Dong L, Lin L (2018) in Proceedings of the 26th ACM International Conference on Multimedia, p. 1056–1064. https://doi.org/10.1145/3240508.3240636

  19. Wang G, Sun C, Sowmya A (2019) in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5644– 5652

  20. Qin Q, Yan J, Wang Q, Wang X, Li M, Wang Y (2021) IEEE Access 9:119881

    Article  Google Scholar 

  21. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) IEEE Transactions on Image Processing 26(6):2944. https://doi.org/10.1109/TIP.2017.2691802

    Article  MathSciNet  Google Scholar 

  22. Yu C, Chang Y, Li Y, Zhao X, Yan L (2021) in Proceedings of the 29th ACM International Conference on Multimedia, pp. 2634–2642

  23. Li X, Wu J, Lin Z, Liu H (2018) Zha H (2018). In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds) Computer Vision - ECCV 2018. Springer International Publishing, Cham, pp 262–277

    Chapter  Google Scholar 

  24. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RW (2019) in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12,270–12,279

  25. Wei Y, Zhang Z, Wang Y, Xu M, Yang Y, Yan S, Wang M (2021) IEEE Trans Image Process 30:4788

    Article  Google Scholar 

  26. Wang H, Xie Q, Zhao Q, Li Y, Liang Y, Zheng Y, Meng D (2023) IEEE Transactions on Neural Networks and Learning Systems

  27. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3855–3863

  28. Mu P, Chen J, Liu R, Fan X, Luo Z (2019) IEEE Signal Processing Letters 26(2):307. https://doi.org/10.1109/LSP.2018.2889277

    Article  Google Scholar 

  29. Yi Q, Li J, Dai Q, Fang F, Zhang G, Zeng T (2021) in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4218–4227. https://doi.org/10.1109/ICCV48922.2021.00420

  30. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1357–1366

  31. Guo Q, Sun J, Juefei-Xu F, Ma L, Xie X, Feng W, Liu Y, Zhao J (2021) in Proceedings of the 35th AAAI Conference on Artificial Intelligence, pp. 1487–1495

  32. Guo Q, Sun J, Juefei-Xu F, Ma L, Lin D, Feng W, Wang S (2022) CoRR abs/2201.02366. https://arxiv.org/abs/2201.02366

  33. Yang Y, Lu H (2019) in 2019 IEEE International Conference on Multimedia and Expo (ICME) (IEEE, 2019), pp. 1378–1383

  34. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2472–2481

  35. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3937–3946

  36. Wang C, Zhang M, Su Z, Yao G, Wang Y, Sun X, Luo X (2019) IEEE Access 7:84420

    Article  Google Scholar 

  37. Wang Y, Song Y, Ma C, Zeng B (2020) in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16 (Springer), pp. 367–382

  38. Zhu H, Wang C, Zhang Y, Su Z, Zhao G (2020) in 2020 IEEE International Conference on Multimedia and Expo (ICME) (IEEE), pp. 1–6

  39. Chen H, Wang Y, Guo T, Xu C, Deng Y, Liu Z, Ma S, Xu C, Xu C, Gao W (2021) in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12,299–12,310

  40. Wang H, Xie Q, Zhao Q, Meng D (2020) in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3103–3112

  41. Luo Y, Xu Y, Ji H (2015) in 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3397–3405. https://doi.org/10.1109/ICCV.2015.388

  42. Gu S, Meng D, Zuo W, Zhang L (2017) in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1708–1716

  43. Yang W, Tan RT, Wang S, Fang Y, Liu J (2020) IEEE Transactions on pattern analysis and machine intelligence 43(11):4059

    Article  Google Scholar 

  44. Deng LJ, Huang TZ, Zhao XL, Jiang TX (2018) Appl Math Model 59:662. https://doi.org/10.1016/j.apm.2018.03.001, https://www.sciencedirect.com/science/article/pii/S0307904X18301069

  45. Kang LW, Lin CW, Fu YH (2012) IEEE Transactions on Image Processing 21(4):1742. https://doi.org/10.1109/TIP.2011.2179057

    Article  MathSciNet  Google Scholar 

  46. Chen YL, Hsu CT (2013) in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1968–1975. https://doi.org/10.1109/ICCV.2013.247

  47. Zhang H, Patel VM (2017), in 2017 IEEE Winter conference on applications of computer vision (WACV) (IEEE, 2017), pp. 1259–1267

  48. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2736–2744. https://doi.org/10.1109/CVPR.2016.299

  49. Bossu J, Hautiere N, Tarel JP (2011) Int J Comput Vis 93(3):348 . https://doi.org/10.1007/s11263-011-0421-7, https://hal.science/hal-00903543

  50. Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2019) in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3877–3886

  51. Wang X, Girshick R, Gupta A, He K (2018) in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794–7803

  52. Zhang H, Goodfellow I, Metaxas D, Odena A (2019) in International conference on machine learning (PMLR), pp. 7354–7363

  53. Hu J, Shen L, Sun G (2018) in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141

  54. Park J, Woo S, Lee JY, Kweon IS (2020) Int J Comput Vision 128(4):783–798. https://doi.org/10.1007/s11263-019-01283-0

    Article  Google Scholar 

  55. Woo S, Park J, Lee JY, Kweon IS (2018) in Proceedings of the European conference on computer vision (ECCV), pp. 3–19

  56. Ioffe S, Szegedy C (2015) in Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 (JMLR.org), ICML’15, p. 448–456

  57. Yu F, Koltun V (2015) arXiv preprint arXiv:1511.07122

  58. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9

  59. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) IEEE Trans Image Process 13(4):600

    Article  Google Scholar 

  60. Chen J, Tan CH, Hou J, Chau LP, Li H (2018) Robust video content alignment and compensation for rain removal in a cnn framework

  61. Kingma DP, Ba J (2015) in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, ed. by Y. Bengio, Y. LeCun. http://arxiv.org/abs/1412.6980

  62. He K, Sun J, Tang X (2013) IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409. https://doi.org/10.1109/TPAMI.2012.213

    Article  Google Scholar 

  63. Johnson J, Alahi A, Fei-Fei L (2016) in Computer Vision – ECCV 2016, ed. by B. Leibe, J. Matas, N. Sebe, M. Welling (Springer International Publishing, Cham), pp. 694–711

  64. Srinivas A, Lin TY, Parmar N, Shlens J, Abbeel P, Vaswani A (2021) in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition16:519–16,529

  65. Hao Z, Gai S, Li P (2023) IEEE Trans Circ Syst Video Technol 33(6):2642. https://doi.org/10.1109/TCSVT.2022.3229730

    Article  Google Scholar 

Download references

Acknowledgements

The work is supported by the National Natural Science Foundation of China (Grant Nos. U1931209), Projects of Science and Technology Cooperation and Exchange of Shanxi Province (Grant Nos. 202204041101037, 202204041101033). Fundamental Research Program of Shanxi Province(Grant No. 20210302123223, 202103021224275). Guanghe Fund (No. ghfund202302032024).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Taiyuan University of Science and Technology, 030024, Taiyuan, China

    Haifeng Yang, Jiajia Guo, Yongjie Xin, Jianghui Cai, Min Zhang, Xujun Zhao & Yanting He

  2. Shanxi Key Laboratory of Advanced Control and Equipment Intelligence, Taiyuan, China

    Haifeng Yang & Jianghui Cai

  3. School of Computer Science and Technology, North University of China, 030051, Taiyuan, China

    Jianghui Cai

  4. School of Civil Engineering, Central South University, 410083, Changsha, China

    Yingyue Zhao

Authors
  1. Haifeng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiajia Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yongjie Xin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianghui Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Min Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xujun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yingyue Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yanting He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianghui Cai.

Ethics declarations

Competing interests

The authors have no relevant financial or non-financial interests to disclose other than the aforementioned funding.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, H., Guo, J., Xin, Y. et al. Multi-scale fusion and adaptively attentive generative adversarial network for image de-raining. Appl Intell 53, 30954–30970 (2023). https://doi.org/10.1007/s10489-023-05114-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05114-1

Keywords

Search

Navigation