Abstract
The deep neural network (DNN) has made significant progre-ss in the single remote sensing image super-resolution (SRSISR). The success of DNN-based SRSISR methods mainly stems from the use of the global information and the fusion of shallow features and the deep features, which fits the non-local self-similarity characteristic of the remote sensing image very well. However, for the fusion of different depth (level) features, most DNN-based SRSISR methods always use the simple skip-connection, e.g. the element-wise addition or concatenation, to transform the feature coming from preceding layers to later layers directly. To achieve sufficient complementation between different levels and capture more informative features, in this paper, we propose a stage-mutual-affine network (SMAN) for high-quality SRSISR. First, for the use of the global information, we construct a convolution-transformer dual-branch module (CTDM), in which we propose an adaptive multi-head attention (AMHA) strategy to dynamically rescale the head-wise features of the transformer for more effective global information extraction. Then, the global information is fused with the local structure information extracted by the convolution branch for more accurate recurrence information reconstruction. Second, a novel hierarchical feature aggregation module (HFAM) is proposed to effectively fuse shallow features and deep features by using a mutual affine convolution operation. The superiority of the proposed HFAM is that it achieves sufficient complementation and enhances the representational capacity of the network by extracting the global information and exploiting the interdependencies between different levels of features, effectively. Extensive experiments demonstrate the superior performance of our SMAN over the state-of-the-art methods in terms of both qualitative evaluation and quantitative metrics.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anwar, S., Barnes, N.: Densely residual Laplacian super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 44(3), 1192–1204 (2022)
Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2004. CVPR 2004, vol. 1, p. I (2004)
Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11057–11066 (2019)
Dong, X., Sun, X., Jia, X., Xi, Z., Gao, L., Zhang, B.: Remote sensing image super-resolution using novel dense-sampling networks. IEEE Trans. Geosci. Remote Sens. 59(2), 1618–1633 (2021)
Dong, X., Wang, L., Sun, X., Jia, X., Gao, L., Zhang, B.: Remote sensing image super-resolution using second-order multi-scale networks. IEEE Trans. Geosci. Remote Sens. 59(4), 3473–3485 (2021)
Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1664–1673 (2018)
Haut, J.M., Paoletti, M.E., Fernández-Beltran, R., Plaza, J., Plaza, A., Li, J.: Remote sensing single-image superresolution based on a deep compendium model. IEEE Geosci. Remote Sens. Lett. 16(9), 1432–1436 (2019)
Hou, B., Zhou, K., Jiao, L.: Adaptive super-resolution for remote sensing images based on sparse representation with global joint dictionary model. IEEE Trans. Geosci. Remote Sens. 56(4), 2312–2327 (2017)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Ji, X., Lu, Y., Guo, L.: Image super-resolution with deep convolutional neural network. In: 2016 IEEE First International Conference on Data Science in Cyberspace (DSC), pp. 626–630 (2016)
Jo, Y., Oh, S.W., Vajda, P., Kim, S.J.: Tackling the ill-posedness of super-resolution through adaptive target generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16236–16245 (2021)
Kong, X., Zhao, H., Qiao, Y., Dong, C.: ClassSR: a general framework to accelerate super-resolution networks by data characteristic. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12016–12025 (2021)
Lei, S., Shi, Z.: Hybrid-scale self-similarity exploitation for remote sensing image super-resolution. IEEE Trans. Geosci. Remote Sens. 60, 1–10 (2022)
Lei, S., Shi, Z., Zou, Z.: Super-resolution for remote sensing images via local-global combined network. IEEE Geosci. Remote Sens. Lett. 14(8), 1243–1247 (2017)
Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140 (2017)
Mei, Y., Fan, Y., Zhou, Y.: Image super-resolution with non-local sparse attention. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3516–3525 (2021)
Mei, Y., Fan, Y., Zhou, Y., Huang, L., Huang, T.S., Shi, H.: Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5689–5698 (2020)
Niu, B., et al.: Single image super-resolution via a holistic attention network. In: European Conference on Computer Vision, pp. 191–207. Springer (2020). https://doi.org/10.1007/978-3-030-58610-2_12
Pan, Z., Ma, W., Guo, J., Lei, B.: Super-resolution of single remote sensing image based on residual dense backprojection networks. IEEE Trans. Geosci. Remote Sens. 57(10), 7918–7933 (2019)
Pan, Z., Yu, J., Huang, H., Hu, S., Zhang, A., Ma, H.: Super-resolution based on compressive sensing and structural self-similarity for remote sensing images. IEEE Trans. Geosci. Remote Sens. 51(9), 4864–4876 (2013)
Qiu, Y., Wang, R., Tao, D., Cheng, J.: Embedded block residual network: a recursive restoration model for single-image super-resolution. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4179–4188 (2019)
Shao, Z., Wang, L., Wang, Z., Deng, J.: Remote sensing image super-resolution using sparse representation and coupled sparse autoencoder. IEEE J. Sel. Top. Appl. Earth Obser. Remote Sens. 12(8), 2663–2674 (2019)
Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874–1883 (2016)
Song, D., Wang, Y., Chen, H., Xu, C., Xu, C., Tao, D.: AdderSR: towards energy efficient image super-resolution. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15643–15652 (2021)
Wang, L., et al.: Exploring sparsity in image super-resolution for efficient inference. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4917–4926 (2021)
Wang, T., Sun, W., Qi, H., Ren, P.: Aerial image super resolution via wavelet multiscale convolutional neural networks. IEEE Geosci. Remote Sens. Lett. 15(5), 769–773 (2018)
Wen, W., Ren, W., Shi, Y., Nie, Y., Zhang, J., Cao, X.: Video super-resolution via a spatio-temporal alignment network. IEEE Trans. Image Process. 31, 1761–1773 (2022)
Yan, Y., Ren, W., Hu, X., Li, K., Shen, H., Cao, X.: SRGAT: Single image super-resolution with graph attention network. IEEE Trans. Image Process. 30, 4905–4918 (2021)
Zhang, D., Shao, J., Li, X., Shen, H.T.: Remote sensing image super-resolution via mixed high-order attention network. IEEE Trans. Geosci. Remote Sens. 59(6), 5183–5196 (2021)
Zhang, S., Yuan, Q., Li, J., Sun, J., Zhang, X.: Scene-adaptive remote sensing image super-resolution using a multiscale attention network. IEEE Trans. Geosci. Remote Sens. 58(7), 4764–4779 (2020)
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018)
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tang, S., Liu, J., Xie, X., Yang, S., Zeng, W., Wang, X. (2022). A Stage-Mutual-Affine Network for Single Remote Sensing Image Super-Resolution. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13537. Springer, Cham. https://doi.org/10.1007/978-3-031-18916-6_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-18916-6_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18915-9
Online ISBN: 978-3-031-18916-6
eBook Packages: Computer ScienceComputer Science (R0)