Cited By
View all- Sun YChen QXu WHuang AYan CZheng B(2024)Enhanced local distribution learning for real image super-resolutionComputer Vision and Image Understanding10.1016/j.cviu.2024.104092247(104092)Online publication date: Oct-2024
An Image Arbitrary-Scale Super-Resolution Network Using Frequency-domain Information
Article No.: 81, Pages 1 - 23
Abstract
Image super-resolution (SR) is a technique to recover lost high-frequency information in low-resolution (LR) images. Since spatial-domain information has been widely exploited, there is a new trend to involve frequency-domain information in SR tasks. Besides, image SR is typically application-oriented and various computer vision tasks call for image arbitrary magnification. Therefore, in this article, we study image features in the frequency domain to design a novel image arbitrary-scale SR network. First, we statistically analyze LR-HR image pairs of several datasets under different scale factors and find that the high-frequency spectra of different images under different scale factors suffer from different degrees of degradation, but the valid low-frequency spectra tend to be retained within a certain distribution range. Then, based on this finding, we devise an adaptive scale-aware feature division mechanism using deep reinforcement learning, which can accurately and adaptively divide the frequency spectrum into the low-frequency part to be retained and the high-frequency one to be recovered. Finally, we design a scale-aware feature recovery module to capture and fuse multi-level features for reconstructing the high-frequency spectrum at arbitrary scale factors. Extensive experiments on public datasets show the superiority of our method compared with state-of-the-art methods.
References
[1]
Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 126–135.
[2]
Parichehr Behjati, Pau Rodriguez, Armin Mehri, Isabelle Hupont, Carles Fernandez Tena, and Jordi Gonzalez. 2021. Overnet: Lightweight multi-scale super-resolution with overscaling network. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2694–2703.
[3]
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. BMVA Press 135 (2012), 1–10.
[4]
Runyuan Cai, Yue Ding, and Hongtao Lu. 2021. FreqNet: A frequency-domain image super-resolution network with dicrete cosine transform. arXiv:2111.10800. Retrieved from https://arxiv.org/abs/2111.10800
[5]
Rui Chen and Yan Zhang. 2022. Learning dynamic generative attention for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology (2022), 1–1.
[6]
Yinbo Chen, Sifei Liu, and Xiaolong Wang. 2021. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8628–8638.
[7]
Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision. 391–407.
[8]
Alireza Esmaeilzehi, M. Omair Ahmad, and M. N. Shanmukha Swamy. 2021. SRNSSI: A deep light-weight network for single image super resolution using spatial and spectral information. IEEE Transactions on Computational Imaging 7 (2021), 409–421.
[9]
Jinsheng Fang, Hanjiang Lin, Xinyu Chen, and Kun Zeng. 2022. A hybrid network of CNN and transformer for lightweight image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1103–1112.
[10]
Jing Fang, Jing Xiao, Xu Wang, Dan Chen, and Ruimin Hu. 2022. Arbitrary scale super resolution network for satellite imagery. China Communications 19, 8 (2022), 234–246.
[11]
Dario Fuoli, Luc Van Gool, and Radu Timofte. 2021. Fourier space losses for efficient perceptual image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2360–2369.
[12]
T. Guo, H. Seyed Mousavi, and V. Monga. 2019. Adaptive transform domain image super-resolution via orthogonally regularized deep networks. IEEE Transactions on Image Processing 28, 9 (2019), 4685–4700.
[13]
Tiantong Guo, Hojjat Seyed Mousavi, Tiep Huu Vu, and Vishal Monga. 2017. Deep wavelet prediction for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 104–113.
[14]
Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. 2021. Task-driven super resolution: Object detection in low-resolution images. In Proceedings of the International Conference on Neural Information Processing. 387–395.
[15]
Cheeun Hong, Heewon Kim, Sungyong Baik, Junghun Oh, and Kyoung Mu Lee. 2022. DAQ: Channel-wise distribution-aware quantization for deep image super-resolution networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2675–2684.
[16]
Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. 2019. Meta-SR: A magnification-arbitrary network for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1575–1584.
[17]
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition. 5197–5206.
[18]
Syed Ali Khayam. 2003. The discrete cosine transform (DCT): Theory and application. Michigan State University 114 (2003), 1–31.
[19]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646–1654.
[20]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1637–1645.
[21]
Jaewon Lee and Kyong Hwan Jin. 2022. Local texture estimator for implicit representation function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1929–1938.
[22]
Dawa Chyophel Lepcha, Bhawna Goyal, Ayush Dogra, and Vishal Goyal. 2023. Image super-resolution: A comprehensive review, recent trends, challenges and applications. Information Fusion 91 (2023), 230–260.
[23]
Junxuan Li, Shaodi You, and Antonio Robles-Kelly. 2018. A frequency domain neural network for fast image super-resolution. In Proceedings of the 2018 International Joint Conference on Neural Networks. 1–8.
[24]
Yanchun Li, Jianglian Cao, Zhetao Li, Sangyoon Oh, and Nobuyoshi Komuro. 2021. Lightweight single image super-resolution with dense connection distillation network. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 1s (2021), 1–17.
[25]
Jie Liang, Hui Zeng, and Lei Zhang. 2022. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5657–5666.
[26]
Jing Liu, Yuan Xie, Haichuan Song, Wang Yuan, and Lizhuang Ma. 2020. Residual attention network for wavelet domain super-resolution. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 2033–2037.
[27]
Lixiong Liu, Bao Liu, Hua Huang, and Alan Conrad Bovik. 2014. No-reference image quality assessment based on spatial and spectral entropies. Signal Processing: Image Communication 29, 8 (2014), 856–863.
[28]
Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. 2018. Multi-level wavelet-CNN for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 773–782.
[29]
Yuqing Liu, Qi Jia, Xin Fan, Shanshe Wang, Siwei Ma, and Wen Gao. 2022. Cross-SRN: Structure-preserving super-resolution network with cross convolution. IEEE Transactions on Circuits and Systems for Video Technology 32, 8 (2022), 4927–4939.
[30]
Yuqing Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. 2022. Sequential hierarchical learning with distribution transformation for image super-resolution. ACM Transactions on Multimedia Computing, Communications, and Applications(2022).
[31]
Haoyu Ma, Bingchen Gong, and Yizhou Yu. 2022. Structure-aware meta-fusion for image super-resolution. ACM Transactions on Multimedia Computing, Communications, and Applications 18, 2 (2022), 1–25.
[32]
Siwei Ma, Xinfeng Zhang, Chuanmin Jia, Zhenghui Zhao, Shiqi Wang, and Shanshe Wang. 2019. Image and video compression with neural networks: A review. IEEE Transactions on Circuits and Systems for Video Technology 30, 6 (2019), 1683–1698.
[33]
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 416–423.
[34]
Yiqun Mei, Yuchen Fan, and Yuqian Zhou. 2021. Image super-resolution with non-local sparse attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3517–3526.
[35]
Sourav Mishra, Toshihiko Yamasaki, and Hideaki Imaizumi. 2019. Improving image classifiers for small datasets by learning rate adaptations. In Proceedings of the 2019 16th International Conference on Machine Vision Applications. IEEE, 1–6.
[36]
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. 2012. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters 20, 3 (2012), 209–212.
[37]
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1928–1937.
[38]
Zhihong Pan, Baopu Li, Dongliang He, Mingde Yao, Wenhao Wu, Tianwei Lin, Xin Li, and Errui Ding. 2022. Towards bidirectional arbitrary image rescaling: Joint optimization and cycle idempotence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17389–17398.
[39]
Samuel Schulter, Christian Leistner, and Horst Bischof. 2015. Fast and accurate image upscaling with super-resolution forests. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3791–3799.
[40]
Sanghyun Son and Kyoung Mu Lee. 2021. SRWarp: Generalized image super-resolution under arbitrary transformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7782–7791.
[41]
Le Sun, Chenyang Ma, Yunjie Chen, Yuhui Zheng, Hiuk Jae Shim, Zebin Wu, and Byeungwoo Jeon. 2020. Low rank component induced spatial-spectral kernel method for hyperspectral image classification. IEEE Transactions on Circuits and Systems for Video Technology 30, 10 (2020), 3829–3842.
[42]
Longguang Wang, Yingqian Wang, Zaiping Lin, Jungang Yang, Wei An, and Yulan Guo. 2021. Learning a single network for scale-arbitrary super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4801–4810.
[43]
Yifan Wang, Lijun Wang, Hongyu Wang, and Peihua Li. 2018. Resolution-aware network for image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 29, 5 (2018), 1259–1269.
[44]
Huapeng Wu, Zhengxia Zou, Jie Gui, Wen-Jun Zeng, Jieping Ye, Jun Zhang, Hongyi Liu, and Zhihui Wei. 2020. Multi-grained attention networks for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 31, 2 (2020), 512–522.
[45]
Jie Xie, Leyuan Fang, Bob Zhang, Jocelyn Chanussot, and Shutao Li. 2021. Super resolution guided deep network for land cover classification from remote sensing images. IEEE Transactions on Geoscience and Remote Sensing 60 (2021), 1–12.
[46]
Jingwei Xin, Jie Li, Xinrui Jiang, Nannan Wang, Heng Huang, and Xinbo Gao. 2022. Wavelet-based dual recursive network for image super-resolution. IEEE Transactions on Neural Networks and Learning Systems 33, 2 (2022), 707–720.
[47]
Ruyu Xu, Xuejing Kang, Chunxiao Li, Hong Chen, and Anlong Ming. 2022. DCT-FANet: DCT based frequency attention network for single image super-resolution. Displays 74 (2022), 102220.
[48]
Shengke Xue, Wenyuan Qiu, Fan Liu, and Xinyu Jin. 2020. Faster image super-resolution by improved frequency-domain neural networks. Signal, Image and Video Processing 14, 2 (2020), 257–265.
[49]
Bin-Cheng Yang and Gangshan Wu. 2022. Efficient single image super-resolution using dual path connections with multiple scale learning. ACM Transactions on Multimedia Computing, Communications and Applications (2022).
[50]
Jiayu Yang, Chunhui Yang, Fei Xiong, Feng Wang, and Ronggang Wang. 2022. Learned low bitrate video compression with space-time super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1786–1790.
[51]
Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang, Jing-Hao Xue, and Qingmin Liao. 2019. Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia 21, 12 (2019), 3106–3121.
[52]
Xin Yang, Hengrui Li, Xiaochuan Li, and Tao Li. 2023. HIFGAN: A high-frequency information based generative adversarial network for image super-resolution. ACM Transactions on Multimedia Computing, Communications and Applications (2023).
[53]
Senrong You, Baiying Lei, Shuqiang Wang, Charles K. Chui, Albert C. Cheung, Yong Liu, Min Gan, Guocheng Wu, and Yanyan Shen. 2022. Fine perceptive gans for brain mr image super-resolution in wavelet domain. IEEE Transactions on Neural Networks and Learning Systems (2022), 1–13.
[54]
Jiahui Yu, Yuchen Fan, Jianchao Yang, Ning Xu, Zhaowen Wang, Xinchao Wang, and Thomas Huang. 2018. Wide activation for efficient and accurate image super-resolution. arXiv:1808.08718. Retrieved from https://arxiv.org/abs/1808.08718
[55]
Jun-Seok Yun and Seok-Bong Yoo. 2022. Single image super-resolution with arbitrary magnification based on high-frequency attention network. Mathematics 10, 2 (2022), 275.
[56]
Dongyang Zhang, Jie Shao, and Heng Tao Shen. 2020. Kernel attention network for single image super-resolution. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 3 (2020), 1–15.
[57]
Yi Zhang, Ming e Jing, Yibo Fan, and Xiaoyang Zeng. 2020. Single image super-resolution neural network using frequency-domain information. In Proceedings of the 2020 IEEE 15th International Conference on Solid-State Integrated Circuit Technology. 1–3.
[58]
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2472–2481.
[59]
Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, and Hanspeter Pfister. 2022. Discrete cosine transform network for guided depth map super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5697–5707.
[60]
Yong Zhou, Silin Chen, Jiaqi Zhao, Rui Yao, Yong Xue, and Abdulmotaleb El Saddik. 2022. CLT-Det: Correlation learning based on transformer for detecting dense objects in remote sensing images. IEEE Transactions on Geoscience and Remote Sensing 60 (2022), 1–15.
[61]
Jin Zhu, Chuan Tan, Junwei Yang, Guang Yang, and Pietro Lio’. 2021. Arbitrary scale super-resolution for medical images. International Journal of Neural Systems 31, 10 (2021), 2150037.
[62]
Fuhao Zou, Wei Xiao, Wanting Ji, Kunkun He, Zhixiang Yang, Jingkuan Song, Helen Zhou, and Kai Li. 2020. Arbitrary-oriented object detection via dense feature fusion and attention model for remote sensing super-resolution image. Neural Computing and Applications 32, 18 (2020), 14549–14562.
Index Terms
- An Image Arbitrary-Scale Super-Resolution Network Using Frequency-domain Information
Recommendations
Deep Residual Attention Network for Spectral Image Super-Resolution
Computer Vision – ECCV 2018 WorkshopsAbstractSpectral imaging sensors often suffer from low spatial resolution, as there exists an essential tradeoff between the spectral and spatial resolutions that can be simultaneously achieved, especially when the temporal resolution needs to be ...
Context-aware single image super-resolution using sparse representation and cross-scale similarity
Methods for single image super-resolution (SR) are broadly divided into two categories: (i) edge-focused SR and (ii) example-based SR. Edge-focused methods are focused on producing sharp edges with minimal artifacts; however, these methods cannot ...
Spatial-Spectral Deep Residual Network for Hyperspectral Image Super-Resolution
AbstractRecently, single hyperspectral image super-resolution (SR) methods based on deep learning have been extensively studied. However, there has been limited technical development focusing on single hyperspectral image super-resolution due to the high-...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2024
665 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 November 2023
Online AM: 16 August 2023
Accepted: 10 August 2023
Revised: 07 August 2023
Received: 06 February 2023
Published in TOMM Volume 20, Issue 3
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China (NSFC)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 444Total Downloads
- Downloads (Last 12 months)331
- Downloads (Last 6 weeks)15
Reflects downloads up to 23 Nov 2024
Other Metrics
Citations
Cited By
View all- Sun YChen QXu WHuang AYan CZheng B(2024)Enhanced local distribution learning for real image super-resolutionComputer Vision and Image Understanding10.1016/j.cviu.2024.104092247(104092)Online publication date: Oct-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in