Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

MBMR-Net: multi-branches multi-resolution cross-projection network for single image super-resolution

Published: 01 September 2022 Publication History

Abstract

Deep convolutional neural networks (CNNs) have achieved significant developments in the field of single image super resolution (SISR) due to their nonlinear expression ability. However, existing architectures either rely on the representations learned from a single scale or extract deep features by cascading multiple resolutions, which unuse or underutilize the interdependence between low-resolution (LR) images and high-resolution (HR) images. In view of this trait, we propose a deep network called the multi-branches multi-resolution cross-projection network (MBMR-Net), which has multiple parallel branches, and cross-projection is performed between multiple branches to exchange information. Then, we introduce a novel attention unit that integrates second-order channel attention with spatial attention to better fuse information from multiple resolutions. Moreover, in terms of the characteristics of the model, we devise a loss function for enhancing the restoration of high-frequency details while ensuring the content information. Extensive quantitative and qualitative evaluations on benchmark datasets illustrate the effectiveness of our method and its competitive performance over state-of-the-art methods.

References

[1]
Bulat A and Tzimiropoulos G Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs IEEE Conference on Computer Vision and Pattern Recognition 2018 109-117
[2]
Chen Y, Tai Y, Liu X, Shen C, and Yang J FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors IEEE Conference on Computer Vision and Pattern Recognition 2018 2492-2501
[3]
Pham CH, Ducournau A, Fablet R, and Rousseau F Brain MRI Super-Resolution using Deep 3D Convolutional Networks IEEE International Symposium on Biomedical Imaging 2017 197-200
[4]
Lyu Q, Shan H, Steber C, Helis C, and Wang G Multi-Contrast Super-Resolution MRI Through a Progressive Network IEEE Transactions on Medical Imaging 2020 39 9 2738-2749
[5]
Rasti P, Uiboupin T, Escalera S, and Anbarjafari G Convolutional Neural Network Super Resolution for Face Recognition in Surveillance Monitoring International Conference on Articulated Motion and Deformable Objects 2016 175-184
[6]
Xun Y, Zhou P, and Wang M Person Reidentification via Structural Deep Metric Learning IEEE Transactions on Neural Networks and Learning System 2018 30 10 2987-2998
[7]
Chao D, Chen CL, He K, and Tang X Learning a Deep Convolutional Network for Image Super-Resolution European Conference on Computer Vision (ECCV) 2014 184-199
[8]
Chao D, Chen CL, and Tang X Accelerating the Super-Resolution Convolutional Neural Network European Conference on Computer Vision (ECCV) 2016 391-407
[9]
Kim J, Lee JK, and Lee KM Accurate image super-resolution using very deep convolutional networks IEEE conference on computer vision and pattern recognition 2016 1646-1654
[10]
Mao XJ, Shen C, and Yang YB Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections Advances in neural information processing systems (NIPS) 2016
[11]
Ying T, Jian Y, and Liu X Image Super-Resolution via Deep Recursive Residual Network IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017 2790-2798
[12]
Tai Y, Yang J, Liu X, and Xu C MemNet: A Persistent Memory Network for Image Restoration IEEE International Conference on Computer Vision (ICCV) 2017 4549-4557
[13]
Lim B, Son S, Kim H, Nah S, and Lee KM Enhanced Deep Residual Networks for Single Image Super-Resolution IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2017 1132-1140
[14]
Lai WS, Huang JB, Ahuja N, and Yang MH Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017 5835-5843
[15]
Haris M, Shakhnarovich G, and Ukita NJA Deep Back-Projection Networks For Super-Resolution IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018 1664-1673
[16]
Zhu F and Zhao Q Efficient Single Image Super-Resolution via Hybrid Residual Feature Learning with Compact Back-Projection Network IEEE International Conference on Computer Vision Workshop (ICCVW) 2019 2453-2460
[17]
Jie H, Li S, Gang S, Albanie S Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(8, 2017):2011–2023
[18]
Zhang Y, Li K, Li K, Wang L, Zhong B, and Fu Y Image Super-Resolution Using Very Deep Residual Channel Attention Networks European Conference on Computer Vision 2018 294-310
[19]
Hu Y, Li J, Huang Y, and Gao X Channel-Wise and Spatial Feature Modulation Network for Single Image Super-Resolution IEEE Transactions on Circuits and Systems for Video Technology 2019 3911-3927
[20]
Dai T, Cai J, Zhang Y, Xia ST, and Zhang L Second-order Attention Network for Single Image Super-Resolution IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2019 11057-11066
[21]
Gao Z, Xie J, Wang Q, and Li P Global Second-Order Pooling Convolutional Networks IEEEConference on Computer Vision and Pattern Recognition (CVPR) 2020 3019-3028
[22]
Fe W, Jiang IM, Chen Q, Yang S, and Tang X Residual Attention Network for Image Classification IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017 6450-6458
[23]
Li K, Wu Z, Peng KC, Ernst J, and Fu Y Tell Me Where to Look: Guided Attention Inference Network IEEE Conference on Computer Vision and Pattern Recognition 2018 9215-9223
[24]
Zhou W, Bovik AC, Sheikh HR, and Simoncelli EP Image quality assessment: from error visibility to structural similarity IEEE Trans Image Process 2004 13 4 600-612
[25]
Kim J, Lee JK, and Lee KM Deeply-Recursive Convolutional Network for Image Super-Resolution IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 5016-5023
[26]
He K, Zhang X, Ren S, and Sun J Deep Residual Learning for Image Recognition IEEE conference on computer vision and pattern recognition (CVPR) 2016 770-778
[27]
Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, and Wang Z Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network IEEE Conference on Computer Vision and Pattern Recognition 2016 105-114
[28]
Huang G, Liu Z, Pleiss G, Van DML, and Weinberger K Convolutional networks with dense connectivity IEEE transactions on pattern analysis machine intelligence 2019
[29]
Tong T, Li G, Liu X, and Gao Q Image Super-Resolution Using Dense Skip Connections IEEE International Conference on Computer Vision (ICCV) 2017 4809-4817
[30]
Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, and Bengio Y Generative Adversarial Networks Advances in Neural Information Processing Systems (NIPS) 2014 2672-2680
[31]
Haris M, Shakhnarovich G, and Ukita N Deep Back-ProjectiNetworks for Single Image Super-Resolution IEEE Transactions on Pattern Analysis Machine Intelligence 2021 43 12 4323-4337
[32]
Wang Z, Chen J, and Hoi S Deep Learning for Image Super-resolution: A Survey IEEE Transactions on Pattern Analysis and Machine Intelligence 2020 43 10 3365-3387
[33]
Qin D and Gu X Deep Residual-Dense Attention Network for Image Super-Resolution International Conference on Neural Information Processing 2019 3-10
[34]
Zhang Y, Li K, Li K, Zhong B, and Fu Y Residual non-local attention networks for image restoration 2019 https://arxiv.org/pdf/1903.10082v1.pdf
[35]
Wang J, Sun K, Cheng T, Jiang B, and Xiao B Deep High-Resolution Representation Learning for Visual Recognition IEEE Transactions on Pattern Analysis and Machine Intelligence 2019 43 10 3349-3364
[36]
Wang Q, Li P, Zuo W, and Lei Z RAID-G: Robust Estimation of Approximate Infinite Dimensional Gaussian with Application to Material Recognition IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 4433-4441
[37]
Koniusz P, Fei Y, Gosselin PH, and Mikolajczyk K Higher-order Occurrence Pooling for Bags-of-Words: Visual Concept Detection IEEE Transactions on Pattern Analysis Machine Intelligence 2017 39 2 313-326
[38]
Li P, Xie J, Wang Q, and Zuo W Is second-order information helpful for large-scale visual recognition? IEEE international conference on computer vision (ICCV) 2017 2089-2097
[39]
Zhao H, Gallo O, Frosio L, and Kautz J Loss Functions for Image Restoration With Neural Networks IEEE Transactions on Computational Imaging 2017 3 1 47-57
[40]
Yuan Y, Liu S, Zhang J, Zhang Y, Dong C, and Lin L Unsupervised Image Super-Resolution using Cycle-in-Cycle Generative Adversarial Networks IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2018 814-823
[41]
Zhu JY, Park T, Isola P, and Efros AA Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks IEEE International Conference on Computer Vision (ICCV) 2017 2242-2251
[42]
Agustsson E, Timofte R, NTIRE (2017) Challenge on single image super-resolution: dataset and study. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), vol 2017, pp 1122–1131
[43]
Bevilacqua M, Roumy A, Guillemot C, and Morel A Low-Complexity Single Image Super-Resolution Based on Nonnegative Neighbor Embedding British Machine Vision Conference 2012 1-10
[44]
Zey De R, Elad M, and Protter M On Single Image Scale-Up Using Sparse-Representations 2010 Springer 711-730
[45]
Martin D, Fowlkes C, Tal D, and Malik J A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics IEEE International Conference on Computer Vision (ICCV) 2002 416-423
[46]
Huang JB, Singh A, and Ahuja N Single image super-resolution from transformed self-exemplars IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015 5197-5206
[47]
Matsui Y, Ito K, Aramaki Y, Fujimoto A, Ogawa T, Yamasaki T, and Aizawa K Sketch-based Manga Retrieval using Manga109 Dataset Multimedia Tools and Applications 2017 76 20 21811-21838
[48]
Kingma D and Ba JJCS Adam: a method for stochastic optimization International conference on learning representations (ICLR) 2014
[49]
Shi W, Caballero J, Huszár F, Totz J, and Wang Z Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network IEEE conference on computer vision and pattern recognition (CVPR) 2016 1874-1883
[50]
Fang F, Li J, Zeng T (2020) Soft-edge Assisted Network for Single Image Super-Resolution. IEEE Transactions on Image Processing:4656–4668
[51]
Li J, Fang F, Mei K, and Zhang G Multi-scale Residual Network for Image Super-Resolution European Conference on Computer Vision (ECCV) 2018 527-542
[52]
Li F, Bai H, and Zhao Y FilterNet: Adaptive Information Filtering Network for Accurate and Fast Image Super-Resolution IEEE Transactions on Circuits Systems for Video Technology 2020 30 6 1511-1523
[53]
He Z, Cao Y, Du L, Xu B, and Zhuang Y MRFN: Multi-Receptive-Field Network for Fast and Accurate Single Image Super-Resolution IEEE Transactions on Multimedia 2019 22 4 1042-1054
[54]
R. Timofte, E. Agustsson, L.V. Gool, e. al., NTIRE 2017 Challenge on single image super-resolution: methods and results, in: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017, pp. 1110–1121

Index Terms

  1. MBMR-Net: multi-branches multi-resolution cross-projection network for single image super-resolution
    Index terms have been assigned to the content through auto-classification.

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image Applied Intelligence
    Applied Intelligence  Volume 52, Issue 11
    Sep 2022
    1249 pages

    Publisher

    Kluwer Academic Publishers

    United States

    Publication History

    Published: 01 September 2022
    Accepted: 28 January 2022

    Author Tags

    1. Multi-branches
    2. Cross-projection
    3. Attention mechanism
    4. Single image super-resolution

    Qualifiers

    • Research-article

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 0
      Total Downloads
    • Downloads (Last 12 months)0
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 22 Nov 2024

    Other Metrics

    Citations

    View Options

    View options

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media