Activating More Information in Arbitrary-Scale Image Super-Resolution
Authors: 
Yaoqian Zhao, Qizhi Teng, Honggang Chen, Shujiang Zhang, Xiaohai He, Yi Li, Ray E. SheriffAuthors Info & Claims
Abstract
Single-image super-resolution (SISR) has experienced vigorous growth with the rapid development of deep learning. However, handling arbitrary scales (e.g., integers, non-integers, or asymmetric) using a single model remains a challenging task. Existing super-resolution (SR) networks commonly employ static convolutions during feature extraction, which cannot effectively perceive changes in scales. Moreover, these continuous-scale upsampling modules only utilize the scale factors, without considering the diversity of local features. To activate more information for better reconstruction, two plug-in and compatible modules for fixed-scale networks are designed to perform arbitrary-scale SR tasks. Firstly, we design a Scale-aware Local Feature Adaptation Module (SLFAM), which adaptively adjusts the attention weights of dynamic filters based on the local features and scales. It enables the network to possess stronger representation capabilities. Then we propose a Local Feature Adaptation Upsampling Module (LFAUM), which combines scales and local features to perform arbitrary-scale reconstruction. It allows the upsampling to adapt to local structures. Besides, deformable convolution is utilized letting more information to be activated in the reconstruction, enabling the network to better adapt to the texture features. Extensive experiments on various benchmark datasets demonstrate that integrating the proposed modules into a fixed-scale SR network enables it to achieve satisfactory results with non-integer or asymmetric scales while maintaining advanced performance with integer scales.
References
[1]
C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295–307, Feb. 2016.
[2]
H. Chen et al., “Real-world single image super-resolution: A brief review,” Inf. Fusion, vol. 79, pp. 124–145, 2022.
[3]
S. Anwar, S. Khan, and N. Barnes, “A deep journey into super-resolution: A survey,” ACM Comput. Surv., vol. 53, no. 3, pp. 1–34, 2020.
[4]
B. B. Moser et al., “Hitchhiker's guide to super-resolution: Introduction and recent advances,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 8, pp. 9862–9882, Aug. 2023.
[5]
F. Li et al., “Learning detail-structure alternative optimization for blind super-resolution,” IEEE Trans. Multimedia, vol. 25, pp. 2825–2838, 2023.
[6]
H. Wang, X. Chen, B. Ni, Y. Liu, and J. Liu, “Omni aggregation networks for lightweight image super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, pp. 22378–22387.
[7]
Y. Liu et al., “Iterative network for image super-resolution,” IEEE Trans. Multimedia, vol. 24, pp. 2259–2272, 2022.
[8]
L. Wang et al., “Learning a single network for scale-arbitrary super-resolution,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 4801–4810.
[9]
M.-I. Georgescu et al., “Multimodal multi-head convolutional attention with various kernel sizes for medical image super-resolution,” in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis., 2023, pp. 2195–2205.
[10]
H. Yang et al., “Deep learning in medical image super resolution: A review,” Appl. Intell., vol. 53, pp. 20891–20916, 2023.
[11]
K. Chen et al., “Continuous remote sensing image super-resolution based on context interaction in implicit function space,” IEEE Trans. Geosci. Remote Sens., vol. 61, 2023, Art. no.
[12]
H. Wu, N. Ni, and L. Zhang, “Learning dynamic scale awareness and global implicit functions for continuous-scale super-resolution of remote sensing images,” IEEE Trans. Geosci. Remote Sens., vol. 61, 2023, Art. no.
[13]
G. Gao, L. Tang, F. Wu, H. Lu, and J. Yang, “JDSR-GAN: Constructing an efficient joint learning network for masked face super-resolution,” IEEE Trans. Multimedia, vol. 25, pp. 1505–1512, 2023.
[14]
A. Agarwal, N. Ratha, M. Vatsa, and R. Singh, “Impact of super-resolution and human identification in drone surveillance,” in Proc. IEEE/CVF Int. Workshop Inf. Forensics Secur., 2021, pp. 1–6.
[15]
B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep residual networks for single image super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2017, pp. 136–144.
[16]
X. Hu et al., “Meta-SR: A magnification-arbitrary network for super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 1575–1584.
[17]
C. Lemke, M. Budka, and B. Gabrys, “Metalearning: A survey of trends and technologies,” Artif. Intell. Rev., vol. 44, pp. 117–130, 2015.
[18]
Y. Chen et al., “Dynamic convolution: Attention over convolution kernels,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 11030–11039.
[19]
J. Dai et al., “Deformable convolutional networks,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2017, pp. 764–773.
[20]
X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable convnets v2: More deformable, better results,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 9308–9316.
[21]
Y. Romano, M. Protter, and M. Elad, “Single image interpolation via adaptive nonlocal sparsity-based modeling,” IEEE Trans. Image Process., vol. 23, no. 7, pp. 3085–3098, Jul. 2014.
[22]
S. Zhu, Z. He, S. Liu, and B. Zeng, “MMSE-directed linear image interpolation based on nonlocal geometric similarity,” IEEE Signal Process. Lett., vol. 24, no. 8, pp. 1178–1182, Aug. 2017.
[23]
W. Dong, L. Zhang, R. Lukac, and G. Shi, “Sparse representation based image interpolation with nonlocal autoregressive modeling,” IEEE Trans. Image Process., vol. 22, no. 4, pp. 1382–1394, Apr. 2013.
[24]
J. Jiang et al., “Single image super-resolution via locally regularized anchored neighborhood regression and nonlocal means,” IEEE Trans. Multimedia, vol. 19, no. 1, pp. 15–26, Jan. 2017.
[25]
H. Chen, X. He, L. Qing, and Q. Teng, “Single image super-resolution via adaptive transform-based nonlocal self-similarity modeling and learning-based gradient regularization,” IEEE Trans. Multimedia, vol. 19, no. 8, pp. 1702–1717, Aug. 2017.
[26]
G. Chantas, S. N. Nikolopoulos, and I. Kompatsiaris, “Heavy-tailed self-similarity modeling for single image super resolution,” IEEE Trans. Image Process., vol. 30, pp. 838–852, 2021.
[27]
J. Sun, Z. Xu, and H.-Y. Shum, “Image super-resolution using gradient profile prior,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2008, pp. 1–8.
[28]
J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang, “Coupled dictionary training for image super-resolution,” IEEE Trans. Image Process., vol. 21, no. 8, pp. 3467–3478, Aug. 2012.
[29]
K. Zhang, X. Gao, D. Tao, and X. Li, “Single image super-resolution with non-local means and steering kernel regression,” IEEE Trans. Image Process., vol. 21, no. 11, pp. 4544–4556, Nov. 2012.
[30]
W. Li et al., “Cross-receptive focused inference network for lightweight image super-resolution,” IEEE Trans. Multimedia, vol. 26, pp. 864–877, 2024.
[31]
Y. Zhang et al., “Weakly-supervised contrastive learning-based implicit degradation modeling for blind image super-resolution,” Knowl.-Based Syst., vol. 249, 2022, Art. no.
[32]
C. Tian et al., “Coarse-to-fine CNN for image super-resolution,” IEEE Trans. Multimedia, vol. 23, pp. 1489–1502, 2021.
[33]
D. Zhang, J. Shao, Z. Liang, L. Gao, and H. T. Shen, “Large factor image super-resolution with cascaded convolutional neural networks,” IEEE Trans. Multimedia, vol. 23, pp. 2172–2184, 2021.
[34]
J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1646–1654.
[35]
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in Proc. IEEE Ronference Comput. Vis. Pattern Recognit., 2018, pp. 2472–2481.
[36]
Y. Zhang et al., “Image super-resolution using very deep residual channel attention networks,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 286–301.
[37]
T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 11065–11074.
[38]
X. Chen, X. Wang, J. Zhou, Y. Qiao, and C. Dong, “Activating more pixels in image super-resolution transformer,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, pp. 22367–22377.
[39]
J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 7132–7141.
[40]
W. Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1874–1883.
[41]
M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 818–833.
[42]
Y. Fu, J. Chen, T. Zhang, and Y. Lin, “Residual scale attention network for arbitrary scale image super-resolution,” Neurocomputing, vol. 427, pp. 201–211, 2021.
[43]
Y. Chen, S. Liu, and X. Wang, “Learning continuous image representation with local implicit image function,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 8628–8638.
[44]
J. Lee and K. H. Jin, “Local texture estimator for implicit representation function,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 1929–1938.
[45]
F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2017, pp. 1251–1258.
[46]
H.-W. Chen et al., “Cascaded local implicit transformer for arbitrary-scale super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, pp. 18257–18267.
[47]
E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image super-resolution: Dataset and study,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2017, pp. 126–135.
[48]
M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” in Proc. Brit. Mach. Vis. Conf., 2012, pp. 135.1–135.10.
[49]
R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in Proc. Curves Surfaces: 7th Int. Conf., 2012, pp. 711–730.
[50]
D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2001, pp. 416–423.
[51]
J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2015, pp. 5197–5206.
[52]
Y. Matsui et al., “Sketch-based manga retrieval using manga109 dataset,” Multimedia Tools Appl., vol. 76, pp. 21811–21838, 2017.
[53]
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
[54]
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. Int. Conf. Learn. Representations, 2015, pp. 1–15.
[55]
J. Cai, H. Zeng, H. Yong, Z. Cao, and L. Zhang, “Toward real-world single image super-resolution: A new benchmark and a new model,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 3086–3095.
[56]
J. Gu and C. Dong, “Interpreting super-resolution networks with local attribution maps,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 9199–9208.
Recommendations
Image super-resolution: use of self-learning and gabor prior
ACCV'12: Proceedings of the 11th Asian conference on Computer Vision - Volume Part IIIRecent approaches on single image super-resolution (SR) have attempted to exploit self-similarity to avoid the use of multiple images. In this paper, we propose an SR method based on self-learning and Gabor prior. Given a low resolution (LR) test image ...
Bidirectional scale-aware upsampling network for arbitrary-scale video super-resolution
AbstractThe performance of video super-resolution (VSR) has significantly improved. However, the current methods only focus on a single scale factor, treating the VSR of different scale factors independently and disregarding video super-resolution of ...
Highlights- Using a trained single model to get HR at arbitrary scales in a bidirectional way.
- A module captures the feature of all the different scale factors explicitly.
- Computing the losses of all the scale factors in a unify way.
- The ...
Arbitrary-scale Super-resolution via Deep Learning: A Comprehensive Survey
AbstractSuper-resolution (SR) is an essential class of low-level vision tasks, which aims to improve the resolution of images or videos in computer vision. In recent years, significant progress has been made in image and video super-resolution techniques ...
Highlights- This work is the first systematic review on arbitrary scale super-resolution (SR).
- Two novel taxonomies for arbitrary scale SR methods are proposed.
- The advantages and limitations of each class of methods are analyzed.
- The ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
1520-9210 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 08 March 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 19 Nov 2024
Other Metrics
Citations
View Options
View options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in