Scale adaptive and lightweight super-resolution with a selective hierarchical residual network
Authors: 
Jiawang Dan, Zhaowei Qu, Xiaoru Wang, Fu Li, Jiahang Gu, Bing MaAuthors Info & Claims
Pages 8 - 14
Abstract
Deep convolutional neural networks have made remarkable achievements in single-image super-resolution tasks in recent years. However, current methods do not consider the characteristics of super-resolution that the adjacent areas carry similar information. In this paper, we propose a scale adaptive and lightweight super-resolution with a selective hierarchical residual network (SHRN), which utilizes the repeated texture features. Specifically, SHRN is stacked by several selective hierarchical residual blocks (SHRB). The SHRB mainly contains a hierarchical feature fusion structure (HFFS) and a selective feature fusion structure (SFFS). The HFFS uses multiple branches to obtain multiscale features due to the varying texture size of objects. The SFFS fuses features of adjacent branches to select effective information. Plenty of experiments demonstrate that our lightweight model achieves better performance against other methods by extracting scale adaptive features and utilizing the repeated texture structure.
References
[1]
Chao Dong, Chen Change Loy, Kaiming He, and X. Tang. 2014. Learning a Deep Convolutional Network for Image Super-Resolution. In ECCV.
[2]
Jiwon Kim, J. Lee, and Kyoung Mu Lee. 2016. Accurate Image Super-Resolution Using Very Deep Convolutional Networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 1646–1654.
[3]
J. Kim, J. Lee, and Kyoung Mu Lee. 2016. Deeply-Recursive Convolutional Network for Image Super-Resolution. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 1637–1645.
[4]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced Deep Residual Networks for Single Image Super-Resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017), 1132–1140.
[5]
Yulun Zhang, Kunpeng Li, K. Li, L. Wang, B. Zhong, and Yun Fu. 2018. Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In ECCV.
[6]
Ying Tai, Jian Yang, and X. Liu. 2017. Image Super-Resolution via Deep Recursive Residual Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 2790–2798.
[7]
Ying Tai, Jian Yang, X. Liu, and Chunyan Xu. 2017. MemNet: A Persistent Memory Network for Image Restoration. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 4549–4557.
[8]
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. 2018. Fast, Accurate, and, Lightweight Super-Resolution with Cascading Residual Network. ArXiv abs/1803.08664 (2018).
[9]
Xiangxiang Chu, Bo Zhang, Hailong Ma, R. Xu, Jixiang Li, and Qingyuan Li. 2019. Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search. ArXiv abs/1901.07261 (2019).
[10]
Juncheng Li, F. Fang, Kangfu Mei, and Guixu Zhang. 2018. Multi-scale Residual Network for Image Super-Resolution. In ECCV.
[11]
A. Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. Commun. ACM 60 (2012), 84 – 90.
[12]
T. Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. 2017. Interleaved Group Convolutions. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 4383–4392.
[13]
Guotian Xie, Jingdong Wang, Ting Zhang, Jianhuang Lai, Richang Hong, and G. Qi. 2018. IGCV2: Interleaved Structured Sparse Convolutional Neural Networks. arXiv: Computer Vision and Pattern Recognition (2018).
[14]
K. Sun, Mingjie Li, Dong Liu, and Jingdong Wang. 2018. IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks. In BMVC.
[15]
J. Carreira, H. Madeira, and J. Silva. 1998. Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers. IEEE Trans. Software Eng. 24 (1998), 125–136.
[16]
A. Howard, Menglong Zhu, Bo Chen, D. Kalenichenko, W. Wang, Tobias Weyand, M. Andreetto, and H. Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. ArXiv abs/1704.04861 (2017).
[17]
Mark Sandler, A. Howard, Menglong Zhu, A. Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 4510–4520.
[18]
X. Zhang, X. Zhou, Mengxiao Lin, and Jian Sun. 2018. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 6848–6856.
[19]
Ningning Ma, X. Zhang, Hai-Tao Zheng, and Jian Sun. 2018. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. In ECCV.
[20]
F. Yu and V. Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. CoRR abs/1511.07122 (2016).
[21]
F. Yu, V. Koltun, and T. Funkhouser. 2017. Dilated Residual Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 636–644.10 The Name of the Title is Hope Woodstock ’18, June 03–05, 2018, Woodstock, NY
[22]
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. 2019. Selective Kernel Networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 510–519.
[23]
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 770–778.
[24]
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity Mappings in Deep Residual Networks. ArXiv abs/1603.05027 (2016).
[25]
Felix Heide, Douglas Lanman, D. Reddy, J. Kautz, K. Pulli, and D. Luebke. 2014. Cascaded displays: spatiotemporal superresolution using offset pixel layers. ACM Trans. Graph. 33 (2014), 60:1–60:11.
[26]
Zhong Zhen-jie, Huai Li-bo, and W. Qi. 2020. Application Research of Overexposure Image Restoration Algorithm Based on Dynamic Convolution Template. Proceedings of the 2020 the 4th International Conference on Innovation in Artificial Intelligence (2020).
[27]
Pourya Shamsolmoali, Masoumeh Zareapoor, Junhao Zhang, and J. Yang. 2019. Image super resolution by dilated dense progressive network. Image Vis. Comput. 88 (2019), 9–18.
[28]
F. Yu, Dequan Wang, and Trevor Darrell. 2018. Deep Layer Aggregation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 2403–2412.
[29]
Gustav Larsson, M. Maire, and Gregory Shakhnarovich. 2017. FractalNet: Ultra-Deep Neural Networks without Residuals. ArXiv abs/1605.07648 (2017).
[30]
Christian Szegedy,W. Liu, Y. Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, D. Erhan, V. Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015), 1–9.
[31]
S. Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ArXiv abs/1502.03167 (2015).
[32]
Christian Szegedy, V. Vanhoucke, S. Ioffe, Jon Shlens, and Z. Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 2818–2826.
[33]
Christian Szegedy, S. Ioffe, V. Vanhoucke, and Alexander Amir Alemi. 2017. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. ArXiv abs/1602.07261 (2017).
[34]
L. Itti and C. Koch. 2001. Computational modelling of visual attention. Nature Reviews Neuroscience 2 (2001), 194–203.
[35]
L. Itti, C. Koch, and E. Niebur. 2009. A Model of Saliency-Based Visual Attention for Rapid Scene Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20 (2009), 1254–1259.
[36]
H. Larochelle and Geoffrey E. Hinton. 2010. Learning to combine foveal glimpses with a third-order Boltzmann machine. In NIPS.
[37]
V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu. 2014. Recurrent Models of Visual Attention. In NIPS.
[38]
B. Olshausen, C. Anderson, and D. V. Van Essen. 1993. A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. In The Journal of neuroscience : the official journal of the Society for Neuroscience.
[39]
Fei Wang, Mengqing Jiang, Chen Qian, S. Yang, Cheng Li, H. Zhang, Xiaogang Wang, and X. Tang. 2017. Residual Attention Network for Image Classification. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 6450–6458.
[40]
Jie Hu, L. Shen, Samuel Albanie, Gang Sun, and Enhua Wu. 2020. Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (2020), 2011–2023.
[41]
Jongchan Park, S. Woo, Joon-Young Lee, and In-So Kweon. 2018. BAM: Bottleneck Attention Module. In BMVC.
[42]
S. Woo, Jongchan Park, Joon-Young Lee, and In-So Kweon. 2018. CBAM: Convolutional Block Attention Module. In ECCV.
[43]
C. Ledig, L. Theis, Ferenc Huszár, J. Caballero, Andrew Aitken, Alykhan Tejani, J. Totz, Zehan Wang, and W. Shi. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 105–114.
[44]
Chao Dong, Chen Change Loy, and X. Tang. 2016. Accelerating the Super-Resolution Convolutional Neural Network. In ECCV.
[45]
W. Shi, J. Caballero, Ferenc Huszár, J. Totz, A. Aitken, R. Bishop, D. Rueckert, and Zehan Wang. 2016. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 1874–1883.
[46]
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. 2018. Deep back-projection networks for superresolution. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1664–1673.
[47]
Yulun Zhang, Yapeng Tian, Y. Kong, B. Zhong, and Yun Fu. 2018. Residual Dense Network for Image Super-Resolution. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 2472–2481.
[48]
Wei-Sheng Lai, Jia-Bin Huang, N. Ahuja, and Ming-Hsuan Yang. 2017. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 5835–5843.
[49]
Wei-Sheng Lai, Jia-Bin Huang, N. Ahuja, and Ming-Hsuan Yang. 2019. Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (2019), 2599–2613.
[50]
Sachin Mehta, M. Rastegari, Anat Caspi, L. Shapiro, and Hannaneh Hajishirzi. 2018. ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation. ArXiv abs/1803.06815 (2018).
[51]
Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, and T. Huang. 2015. Deep Networks for Image Super-Resolution with Sparse Prior. 2015 IEEE International Conference on Computer Vision (ICCV) (2015), 370–378.
[52]
Mehdi S. M. Sajjadi, B. Schölkopf, and M. Hirsch. 2017. EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 4501–4510.
[53]
Kai Zhang, W. Zuo, and Lei Zhang. 2018. Learning a Single Convolutional Super-Resolution Network for Multiple Degradations. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 3262–3271.
[54]
V. Nair and Geoffrey E. Hinton. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines. In ICML. 9 Woodstock ’18, June 03–05, 2018, Woodstock, NY Trovato and Tobin,
[55]
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated Residual Transformations for Deep Neural Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 5987–5995.
[56]
Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xinyu Zhang, Ming-Hsuan Yang, and P. Torr. 2021. Res2Net: A New Multi-Scale Backbone Architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (2021), 652–662. 8 The Name of the Title is Hope Woodstock ’18, June 03–05, 2018, Woodstock, NY.
[57]
Liang-Chieh Chen, G. Papandreou, I. Kokkinos, Kevin Murphy, and A. Yuille. 2018. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (2018), 834–848.
[58]
R. Timofte, Eirikur Agustsson, L. Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, Xintao Wang, Yapeng Tian, K. Yu, Yulun Zhang, Shixiang Wu, Chao Dong, L. Lin, Y. Qiao, Chen Change Loy, W. Bae, Jae Jun Yoo, Yoseob Han, J. C. Ye, Jae-Seok Choi, M. Kim, Yuchen Fan, J. Yu, Wei Han, Ding Liu, Haichao Yu, Zhangyang Wang, Humphrey Shi, X. Wang, T. Huang, Yunjin Chen, Kai Zhang, W. Zuo, Z. Tang, Linkai Luo, S. Li, M. Fu, L. Cao, Wen Heng, G. Bui, Truc Le, Ye Duan, D. Tao, Ruxin Wang, Xu Lin, Jianxin Pang, Jinchang Xu, Y. Zhao, Xiangyu Xu, Jin shan Pan, Deqing Sun, Y. Zhang, X. Song, Yuchao Dai, X. Qin, X. Huynh, Tiantong Guo, H. Mousavi, T. Vu, V. Monga, C. Cruz, K. Egiazarian, V. Katkovnik, Rakesh Mehta, A. Jain, Abhinav Agarwalla, C. Praveen, R. Zhou, Hongdiao Wen, C. Zhu, Zhiqiang Xia, Z. Wang, and Q. Guo. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017), 1110–1121.
[59]
Marco Bevilacqua, A. Roumy, C. Guillemot, and M. Alberi-Morel. 2012. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. In BMVC.
[60]
Roman Zeyde, Michael Elad, and M. Protter. 2010. On Single Image Scale-Up Using Sparse-Representations. In Curves and Surfaces.
[61]
D. Martin, Charless C. Fowlkes, D. Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 2 (2001), 416–423 vol.2.
[62]
Jia-Bin Huang, Abhishek Singh, and N. Ahuja. 2015. Single image super-resolution from transformed self-exemplars. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015), 5197–5206.
[63]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2015).
[64]
Zhou Wang, A. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (2004), 600–612.
[65]
Jae-Seok Choi and M. Kim. 2017. A Deep Convolutional Neural Network with Selection Units for Super-Resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017), 1150–1156.
[66]
T. Tong, Gen Li, Xiejie Liu, and Qinquan Gao. 2017. Image Super-Resolution Using Dense Skip Connections. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 4809–4817.
[67]
Yifan Wang, L. Wang, Hongyu Wang, and P. Li. 2019. End-to-End Image Super-Resolution via Deep and Shallow Convolutional Networks. IEEE Access 7 (2019), 31959–31970.
[68]
T. Dai, Jianrui Cai, Yong-Bing Zhang, S. Xia, and Lei Zhang. 2019. Second-Order Attention Network for Single Image Super-Resolution. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 11057–11066.
[69]
Xiangxiang Chu, Bo Zhang, R. Xu, and Hailong Ma. 2020. Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search. ArXiv abs/1901.01074 (2020).
Index Terms
- Scale adaptive and lightweight super-resolution with a selective hierarchical residual network
Index terms have been assigned to the content through auto-classification.
Recommendations
Deep Residual Attention Network for Spectral Image Super-Resolution
Computer Vision – ECCV 2018 WorkshopsAbstractSpectral imaging sensors often suffer from low spatial resolution, as there exists an essential tradeoff between the spectral and spatial resolutions that can be simultaneously achieved, especially when the temporal resolution needs to be ...
Spatial-Spectral Deep Residual Network for Hyperspectral Image Super-Resolution
AbstractRecently, single hyperspectral image super-resolution (SR) methods based on deep learning have been extensively studied. However, there has been limited technical development focusing on single hyperspectral image super-resolution due to the high-...
Attention hierarchical network for super-resolution
AbstractDeep neural networks with attention mechanism for super-resolution (SR) have achieved good SR performance by focusing on the high-frequency components of images. However, during the SR process, it is difficult for these networks to obtain multi-...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2021
246 pages
ISBN:9781450388634
DOI:10.1145/3461353
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 September 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
ICIAI 2021
ICIAI 2021: 2021 the 5th International Conference on Innovation in Artificial Intelligence
March 5 - 8, 2021
Xia men, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 72Total Downloads
- Downloads (Last 12 months)9
- Downloads (Last 6 weeks)1
Reflects downloads up to 22 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format