Article

Learning with Privileged Information for Efficient Image Super-Resolution

Authors:

Bumsub HamAuthors Info & Claims

Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIV

Pages 465 - 482

https://doi.org/10.1007/978-3-030-58586-0_28

Published: 23 August 2020 Publication History

Abstract

Convolutional neural networks (CNNs) have allowed remarkable advances in single image super-resolution (SISR) over the last decade. Most SR methods based on CNNs have focused on achieving performance gains in terms of quality metrics, such as PSNR and SSIM, over classical approaches. They typically require a large amount of memory and computational units. FSRCNN, consisting of few numbers of convolutional layers, has shown promising results, while using an extremely small number of network parameters. We introduce in this paper a novel distillation framework, consisting of teacher and student networks, that allows to boost the performance of FSRCNN drastically. To this end, we propose to use ground-truth high-resolution (HR) images as privileged information. The encoder in the teacher learns the degradation process, subsampling of HR images, using an imitation loss. The student and the decoder in the teacher, having the same network architecture as FSRCNN, try to reconstruct HR images. Intermediate features in the decoder, affordable for the student to learn, are transferred to the student through feature distillation. Experimental results on standard benchmarks demonstrate the effectiveness and the generalization ability of our framework, which significantly boosts the performance of FSRCNN as well as other SR methods. Our code and model are available online: https://cvlab.yonsei.ac.kr/projects/PISR.

References

[1]

Ahn N, Kang B, and Sohn K-A Ferrari V, Hebert M, Sminchisescu C, and Weiss Y Fast, accurate, and lightweight super-resolution with cascading residual network Computer Vision – ECCV 2018 2018 Cham Springer 256-272

Digital Library

[2]

Ahn, S., Hu, S.X., Damianou, A., Lawrence, N.D., Dai, Z.: Variational information distillation for knowledge transfer. In: CVPR (2019)

[3]

Bai Y, Zhang Y, Ding M, and Ghanem B Ferrari V, Hebert M, Sminchisescu C, and Weiss Y SOD-MTGAN: small object detection via multi-task generative adversarial network Computer Vision – ECCV 2018 2018 Cham Springer 210-226

Digital Library

[4]

Barber, D., Agakov, F.V.: The IM algorithm: a variational approach to information maximization. In: NIPS (2003)

[5]

Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: BMVC (2012)

[6]

Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: CVPR (2004)

[7]

Cho, J.H., Hariharan, B.: On the efficacy of knowledge distillation. In: ICCV (2019)

[8]

Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: CVPR (2017)

[9]

Dai S, Han M, Xu W, Wu Y, Gong Y, and Katsaggelos AK SoftCuts: a soft edge smoothness prior for color image super-resolution IEEE TIP 2009 18 5 969-981

[10]

Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE TPAMI 38(2), (2015)

[11]

Dong C, Loy CC, and Tang X Leibe B, Matas J, Sebe N, and Welling M Accelerating the super-resolution convolutional neural network Computer Vision – ECCV 2016 2016 Cham Springer 391-407

[12]

Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., Garcia, R.: Incorporating second-order functional knowledge for better option pricing. In: NIPS (2001)

[13]

Freeman WT, Jones TR, and Pasztor EC Example-based super-resolution IEEE CG&A 2002 22 2 56-65

[14]

Gao, Q., Zhao, Y., Li, G., Tong, T.: Image super-resolution using knowledge distillation. In: ACCV (2018)

[15]

Garcia NC, Morerio P, and Murino V Ferrari V, Hebert M, Sminchisescu C, and Weiss Y Modality distillation with multiple stream networks for action recognition Computer Vision – ECCV 2018 2018 Cham Springer 106-121

Digital Library

[16]

Greenspan H Super-resolution in medical imaging Comput. J. 2008 52 1 43-63

Digital Library

[17]

Gunturk BK, Batur AU, Altunbasak Y, Hayes MH, and Mersereau RM Eigenface-domain super-resolution for face recognition IEEE TIP 2003 12 5 597-606

[18]

Han, S., Mao, H., Dally, W.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: ICLR (2016)

[19]

Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: NIPS (2015)

[20]

Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: CVPR (2018)

[21]

He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: ICCV (2015)

[22]

Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., Choi, J.Y.: A comprehensive overhaul of feature distillation. In: ICCV (2019)

[23]

Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Workshop (2014)

[24]

Hoffman, J., Gupta, S., Darrell, T.: Learning with side information through modality hallucination. In: CVPR (2016)

[25]

Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: CVPR (2015)

[26]

Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: ACMMM (2019)

[27]

Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: CVPR (2018)

[28]

Sun, J., Xu, Z., Shum, H.-Y.: Image super-resolution using gradient profile prior. In: CVPR (2008)

[29]

Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)

[30]

Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: CVPR (2016)

[31]

Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

[32]

Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR (2017)

[33]

Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., Wu, W.: Feedback network for image super-resolution. In: CVPR (2019)

[34]

Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: CVPR Workshop (2017)

[35]

Lin WS, Tjoa SK, Zhao HV, and Liu KR Digital image source coder forensics via intrinsic fingerprints IEEE TIFS 2009 4 3 460-475

[36]

Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., Wang, J.: Structured knowledge distillation for semantic segmentation. In: CVPR (2019)

[37]

Lopez-Paz, D., Bottou, L., Schölkopf, B., Vapnik, V.: Unifying distillation and privileged information. In: ICLR (2016)

[38]

Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. In: ICLR (2017)

[39]

Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: NIPS (2016)

[40]

Martin, D., Fowlkes, C., Tal, D., Malik, J., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001)

[41]

Mirzadeh, S.I., Farajtabar, M., Li, A., Ghasemzadeh, H.: Improved knowledge distillation via teacher assistant: bridging the gap between student and teacher. In: AAAI (2020)

[42]

Paszke, A., et al.: Automatic differentiation in PyTorch (2017)

[43]

Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: FitNets: hints for thin deep nets. In: ICLR (2015)

[44]

Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: CVPR (2015)

[45]

Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: CVPR (2017)

[46]

Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: a persistent memory network for image restoration. In: ICCV (2017)

[47]

Timofte, R., De, V., Gool, L.V.: Anchored neighborhood regression for fast example-based super-resolution. In: ICCV (2013)

[48]

Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L.: NTIRE 2017 challenge on single image super-resolution: methods and results. In: CVPR Workshop (2017)

[49]

Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: ICCV (2017)

[50]

Vapnik V and Izmailov R Learning using privileged information: similarity control and knowledge transfer JMLR 2015 16 2023-2049

Digital Library

[51]

Vapnik V and Vashist A A new learning paradigm: learning using privileged information Neural Netw. 2009 22 5–6 544-557

Digital Library

[52]

Wang Z et al. Image quality assessment: from error visibility to structural similarity IEEE TIP 2004 13 4 600-612

Digital Library

[53]

Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: NIPS (2016)

[54]

Xie, S., Girshick, R., Dollar, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: CVPR (2017)

[55]

Yan Q, Xu Y, Yang X, and Nguyen TQ Single image super-resolution based on gradient profile sharpness IEEE TIP 2015 24 10 3187-3202

[56]

Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution as sparse representation of raw image patches. In: CVPR (2008)

[57]

Yim, J., Joo, D., Bae, J., Kim, J.: A gift from knowledge distillation: fast optimization, network minimization and transfer learning. In: CVPR (2017)

[58]

Zagoruyko, S., Komodakis, N.: Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: ICLR (2017)

[59]

Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces (2010)

[60]

Zhang K, Zuo W, Chen Y, Meng D, and Zhang L Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising IEEE TIP 2017 26 3142-3155

[61]

Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: CVPR (2017)

[62]

Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: CVPR (2018)

[63]

Zou, W.W., Yuen, P.C.: Very low resolution face recognition problem. IEEE TIP 21(1), (2011)

Cited By

Pan LLi GXu KLv YZhang WLi LLei L(2024)Dual residual and large receptive field network for lightweight image super-resolutionNeurocomputing10.1016/j.neucom.2024.128158600:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.neucom.2024.128158
Jiang YFeng CZhang FBull D(2024)MTKD: Multi-Teacher Knowledge Distillation for Image Super-ResolutionComputer Vision – ECCV 202410.1007/978-3-031-72933-1_21(364-382)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72933-1_21
Qiu SChen JLi XWan RXue XPu J(2024)Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic SegmentationComputer Vision – ECCV 202410.1007/978-3-031-72907-2_22(371-388)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72907-2_22
Show More Cited By

Recommendations

Edge-directed single image super-resolution via cross-resolution sharpening function learning

Edge-directed single image super-resolution methods have been paid more attentions due to their sharp edge preserving in the recovered high-resolution image. Their core is the high-resolution gradient estimation. In this paper, we propose a novel cross-...
Efficient single image super-resolution via graph-constrained least squares regression

We explore in this paper an efficient algorithmic solution to single image super-resolution (SR). We propose the gCLSR, namely graph-Constrained Least Squares Regression, to super-resolve a high-resolution (HR) image from a single low-resolution (LR) ...
Efficient Single Image Super-Resolution via Graph Embedding
ICME '12: Proceedings of the 2012 IEEE International Conference on Multimedia and Expo

We explore in this paper efficient algorithmic solutions to single image super-resolution (SR). We propose the GESR, namely Graph Embedding Super-Resolution, to super-resolve a high-resolution (HR) image from a single low-resolution (LR) observation. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIV

Aug 2020

844 pages

ISBN:978-3-030-58585-3

DOI:10.1007/978-3-030-58586-0

Editors:
Andrea Vedaldi
University of Oxford, Oxford, UK
,
Horst Bischof
Graz University of Technology, Graz, Austria
,
Thomas Brox
University of Freiburg, Freiburg im Breisgau, Germany
,
Jan-Michael Frahm
University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

© Springer Nature Switzerland AG 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 23 August 2020

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pan LLi GXu KLv YZhang WLi LLei L(2024)Dual residual and large receptive field network for lightweight image super-resolutionNeurocomputing10.1016/j.neucom.2024.128158600:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.neucom.2024.128158
Jiang YFeng CZhang FBull D(2024)MTKD: Multi-Teacher Knowledge Distillation for Image Super-ResolutionComputer Vision – ECCV 202410.1007/978-3-031-72933-1_21(364-382)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72933-1_21
Qiu SChen JLi XWan RXue XPu J(2024)Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic SegmentationComputer Vision – ECCV 202410.1007/978-3-031-72907-2_22(371-388)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72907-2_22
Qin HZhang YDing YLiu YLiu XDanelljan MYu FOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)QuantSRProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668605(56838-56848)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668605
Li ZWang YZhang J(2023)Low-light image enhancement with knowledge distillationNeurocomputing10.1016/j.neucom.2022.10.083518:C(332-343)Online publication date: 21-Jan-2023
https://dl.acm.org/doi/10.1016/j.neucom.2022.10.083
Yang SSanghavi SRahmanian HBakus JVishwanathan SKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Toward understanding privileged features distillation in learning-to-rankProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3602203(26658-26670)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3602203
Chen BLin MSheng KZhang MChen PLi KCao LJi R(2022)ARM: Any-Time Super-Resolution MethodComputer Vision – ECCV 202210.1007/978-3-031-19800-7_15(254-270)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-19800-7_15
Ma CZhang JZhou JLu J(2022)Learning Series-Parallel Lookup Tables for Efficient Image Super-ResolutionComputer Vision – ECCV 202210.1007/978-3-031-19790-1_19(305-321)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-19790-1_19
Zhang YWang HQin CFu YRanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Aligned structured sparsity learning for efficient image super-resolutionProceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3540467(2695-2706)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3540467
Lu HLu YLi GSun YWang SLi Y(2021)Scale-Aware Distillation Network for Lightweight Image Super-ResolutionPattern Recognition and Computer Vision10.1007/978-3-030-88010-1_11(128-139)Online publication date: 29-Oct-2021
https://dl.acm.org/doi/10.1007/978-3-030-88010-1_11

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents