Abstract
Light field (LF) cameras record multiple perspectives by a sparse sampling of real scenes, and these perspectives provide complementary information. This information is beneficial to LF super-resolution (LFSR). Compared with traditional single-image super-resolution, LF can exploit parallax structure and perspective correlation among different LF views. Furthermore, the performance of existing methods are limited as they fail to deeply explore the complementary information across LF views. In this paper, we propose a novel network, called the light field complementary-view feature attention network (LF-CFANet), to improve LFSR by dynamically learning the complementary information in LF views. Specifically, we design a residual complementary-view spatial and channel attention module (RCSCAM) to effectively interact with complementary information between complementary views. Moreover, RCSCAM captures the relationships between different channels, and it is able to generate informative features for reconstructing LF images while ignoring redundant information. Then, a maximum-difference information supplementary branch (MDISB) is used to supplement information from the maximum-difference angular positions based on the geometric structure of LF images. This branch also can guide the process of reconstruction. Experimental results on both synthetic and real-world datasets demonstrate the superiority of our method. The proposed LF-CFANet has a more advanced reconstruction performance that displays faithful details with higher SR accuracy than state-of-the-art methods.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Huang, F.-C.; Luebke, D.; Wetzstein, G. The light field stereoscope. In: Proceedings of the ACM SIGGRAPH Emerging Technologies, Article No. 24, 2015.
Yu, J. A light-field journey to virtual reality. IEEE MultiMedia Vol. 24, No. 2, 104–112, 2017.
Sheng, H.; Wang, S.; Zhang, Y.; Yu, D. X.; Cheng, X. Z.; Lyu, W. F.; Xiong, Z. Near-online tracking with co-occurrence constraints in blockchain-based edge computing. IEEE Internet of Things Journal Vol. 8, No. 4, 2193–2207, 2021.
Sheng, H.; Zhang, Y.; Wu, Y. B.; Wang, S.; Lyu, W. F.; Ke, W.; Xiong, Z. Hypothesis testing based tracking with spatio-temporal joint interaction modeling. IEEE Transactions on Circuits and Systems for Video Technology Vol. 30, No. 9, 2971–2983, 2020.
Wang, S.; Sheng, H.; Zhang, Y.; Wu, Y. B.; Xiong, Z. A general recurrent tracking framework without real data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 13199–13208, 2021.
Wang, S.; Sheng, H.; Zhang, Y.; Yang, D.; Shen, J.; Chen, R. Blockchain-empowered distributed multi-camera multi-target tracking in edge computing. IEEE Transactions on Industrial Informatics, doi: https://doi.org/10.1109/TII.2023.3261890, 2023.
Zhu, H.; Wang, Q.; Yu, J. Y. Occlusion-model guided antiocclusion depth estimation in light field. IEEE Journal of Selected Topics in Signal Processing Vol. 11, No. 7, 965–978, 2017.
Piao, Y. R.; Li, X.; Zhang, M.; Yu, J. Y.; Lu, H. C. Saliency detection via depth-induced cellular automata on light field. IEEE Transactions on Image Processing Vol. 29, 1879–1889, 2020.
Zhang, M.; Li, J.; Ji, W.; Piao, Y.; Lu, H. Memory-oriented decoder for light field salient object detection. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Article No. 81, 898–908, 2019.
Bishop, T. E.; Favaro, P. The light field camera: Extended depth of field, aliasing, and superresolution. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 5, 972–986, 2012.
Wanner, S.; Goldluecke, B. Spatial and angular variational super-resolution of 4D light fields. In: Computer Vision - ECCV 2012. Lecture Notes in Computer Science, Vol. 7576. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin, Heidelberg, 608–621, 2012.
Wanner, S.; Goldluecke, B. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 3, 606–619, 2014.
Mitra, K.; Veeraraghavan, A. Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 22–28, 2012.
Yuan, Y.; Cao, Z. Q.; Su, L. J. Light-field image superresolution using a combined deep CNN based on EPI. IEEE Signal Processing Letters Vol. 25, No. 9, 1359–1363, 2018.
Yoon, Y.; Jeon, H. G.; Yoo, D.; Lee, J. Y.; Kweon, I. S. Learning a deep convolutional network for light-field image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision Workshop, 57–65, 2015.
Yoon, Y.; Jeon, H. G.; Yoo, D.; Lee, J. Y.; Kweon, I. S. Light-field image super-resolution using convolutional neural network. IEEE Signal Processing Letters Vol. 24, No. 6, 848–852, 2017.
Li, D. L.; Yang, D.; Wang, S. Z.; Sheng, H. Light field super-resolution based on spatial and angular attention. In: Wireless Algorithms, Systems, and Applications. Lecture Notes in Computer Science, Vol. 12937. Liu, Z.; Wu, F.; Das, S. K. Eds. Springer Cham, 314–325, 2021.
Zhang, S.; Lin, Y. F.; Sheng, H. Residual networks for light field image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11038–11047, 2019.
Yeung, H. W. F.; Hou, J. H.; Chen, X. M.; Chen, J.; Chen, Z. B.; Chung, Y. Y. Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Transactions on Image Processing Vol. 28, No. 5, 2319–2330, 2019.
Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7132–7141, 2018.
Wang, L. G.; Wang, Y. Q.; Liang, Z. F.; Lin, Z. P.; Yang, J. G.; An, W.; Guo, Y. L. Learning parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12242–12251, 2019.
Wang, Y. Q.; Ying, X. Y.; Wang, L. G.; Yang, J. G.; An, W.; Guo, Y. L. Symmetric parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 766–775, 2021.
Yang, W. M.; Zhang, X. C.; Tian, Y. P.; Wang, W.; Xue, J. H.; Liao, Q. M. Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia Vol. 21, No. 12, 3106–3121, 2019.
Wang, Z. H.; Chen, J.; Hoi, S. C. H. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 10, 3365–3387, 2021.
Dong, C.; Loy, C. C.; He, K. M.; Tang, X. O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, No. 2, 295–307, 2015.
Dong, C.; Loy, C. C.; He, K. M.; Tang, X. O. Learning a deep convolutional network for image super-resolution. In: Computer Vision - ECCV 2014. Lecture Notes in Computer Science, Vol. 8692. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 184–199, 2014.
Kim, J.; Lee, J. K.; Lee, K. M. Accurate image superresolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1646–1654, 2016.
Lim, B.; Son, S.; Kim, H.; Nah, S.; Lee, K. M. Enhanced deep residual networks for single image superresolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1132–1140, 2017.
Timofte, R.; Agustsson, E.; Van Gool, L.; Yang, M.-H.; Zhang, L. NTIRE 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1110–1121, 2017.
Zhang, Y. L.; Tian, Y. P.; Kong, Y.; Zhong, B. N.; Fu, Y. Residual dense network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2472–2481, 2018.
Zhang, Y. L.; Tian, Y. P.; Kong, Y.; Zhong, B. N.; Fu, Y. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 7, 2480–2495, 2021.
Zhang, Y. L.; Li, K. P.; Li, K.; Wang, L. C.; Zhong, B. N.; Fu, Y. Image super-resolution using very deep residual channel attention networks. In: Computer Vision - ECCV 2018. Lecture Notes in Computer Science, Vol. 11211. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 294–310, 2018.
Dai, T.; Cai, J. R.; Zhang, Y. B.; Xia, S. T.; Zhang, L. Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11057–11066, 2019.
Zhang, F. L.; Wang, J.; Shechtman, E.; Zhou, Z. Y.; Shi, J. X.; Hu, S. M. PlenoPatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 5, 1561–1573, 2017.
Rossi, M.; Frossard, P. Geometry-consistent light field super-resolution via graph-based regularization. IEEE Transactions on Image Processing Vol. 27, No. 9, 4207–4218, 2018.
Alain, M.; Smolic, A. Light field super-resolution via LFBM5D sparse coding. In: Proceedings of the 25th IEEE International Conference on Image Processing, 2501–2505, 2018.
Huang, Y.; Wang, W.; Wang, L. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, 235–243, 2015.
Wang, Y. L.; Liu, F.; Zhang, K. B.; Hou, G. Q.; Sun, Z. N.; Tan, T. N. LFNet: A novel bidirectional recurrent convolutional neural network for light-field image superresolution. IEEE Transactions on Image Processing Vol. 27, No. 9, 4274–4286, 2018.
Jin, J.; Hou, J. H.; Chen, J.; Kwong, S. Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2257–2266, 2020.
Wang, Y. Q.; Wang, L. G.; Yang, J. G.; An, W.; Yu, J. Y.; Guo, Y. L. Spatial-angular interaction for light field image super-resolution. In: Computer Vision - ECCV 2020. Lecture Notes in Computer Science, Vol. 12368. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 290–308, 2020.
Mo, Y.; Wang, Y.; Xiao, C.; Yang, J.; An, W. Dense dual-attention network for light field image superresolution. IEEE Transactions on Circuits and Systems for Video Technology Vol. 32, No. 7, 4431–4443, 2022.
Rerabek, M.; Ebrahimi, T. New light field image dataset. In: Proceedings of the 8th International Conference on Quality of Multimedia Experience, 2016.
Honauer, K.; Johannsen, O.; Kondermann, D.; Goldluecke, B. A dataset and evaluation methodology for depth estimation on 4D light fields. In: Computer Vision - ACCV 2016. Lecture Notes in Computer Science, Vol. 10113. Lai, S. H.; Lepetit, V.; Nishino, K.; Sato, Y. Eds. Springer Cham, 19–34, 2017.
Wanner, S.; Meister, S.; Goldluecke, B. Datasets and benchmarks for densely sampled 4D light fields. In: Proceedings of the Vision, Modeling & Visualization, 225–226, 2013.
Le Pendu, M.; Jiang, X. R.; Guillemot, C. Light field inpainting propagation via low rank matrix completion. IEEE Transactions on Image Processing Vol. 27, No. 4, 1981–1993, 2018.
Vaish, V.; Adams, A. The (new) Stanford light field archive. Computer Graphics Laboratory, Stanford University, 2008. Available at http://lightfield.stanford.edu/index.html.
Raj, A. S.; Lowney, M.; Shah, R.; Wetzstein, G. Stanford Lytro light field archive. 2016. Available at http://lightfields.stanford.edu/LF2016.html.
Chen, L. C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A. L. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 40, No. 4, 834–848, 2018.
Ying, X. Y.; Wang, Y. Q.; Wang, L. G.; Sheng, W. D.; An, W.; Guo, Y. L. A stereo attention module for stereo image super-resolution. IEEE Signal Processing Letters Vol. 27, 496–500, 2020.
Kim, J.; Lee, J. K.; Lee, K. M. Accurate image superresolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1646–1654, 2016.
Acknowledgements
This study was partially supported by the National Key R&D Program of China (2018YFB2100500), the National Natural Science Foundation of China (61872025), the Science and Technology Development Fund, Macau SAR (0001/2018/AFJ), and the Open Fund of the State Key Laboratory of Software Development Environment (SKLSDE-2021ZX-03). We thank the HAWKEYE Group for their support.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Wei Zhang received his M.S. degree in control engineering from Beijing University of Chemical Technology in 2019. He is currently a Ph.D. student in the Faculty of Applied Sciences, Macao Polytechnic University, China. His research interests include image processing and computer vision.
Wei Ke received his Ph.D. degree from the School of Computer Science and Engineering, Beihang University, China. He is an associate professor of computing, Macao Polytechnic University. His research interests include programming languages, image processing, computer graphics, and component-based engineering and systems. His recent research focuses on the design and implementation of open platforms for applications of computer graphics and pattern recognition.
Da Yang received his B.S. degree from the School of Computer Science and Engineering, Beihang University in 2012. He is currently pursuing a Ph.D. degree with the School of Computer Science and Engineering, Beihang University.
Hao Sheng received his B.S. and Ph.D. degrees from the School of Computer Science and Engineering of Beihang University in 2003 and 2009, respectively. Now he is a professor and Ph.D. supervisor in the School of Computer Science and Engineering, Beihang University. He is working on computer vision, pattern recognition, and machine learning.
Zhang Xiong received his B.S degree from Harbin Engineering University in 1982, and his M.S. degree from Beihang University in 1985. He is a professor and Ph.D. supervisor in the School of Computer Science and Engineering, Beihang University. He is working on computer vision, information security, and data visualization.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Zhang, W., Ke, W., Yang, D. et al. Light field super-resolution using complementary-view feature attention. Comp. Visual Media 9, 843–858 (2023). https://doi.org/10.1007/s41095-022-0297-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-022-0297-1