Abstract
Image restoration is the problem of restoring a real degraded image. Previous studies mostly focused on single distortion. However, most of the real images experience multiple distortions, and single distortion image restoration algorithms can not effectively improve the image quality. Moreover, few existing hybrid distortion image restoration algorithms can not deal with single distortion. Therefore, an end-to-end pipeline network based on stagewise training is proposed in this paper. Specifically, the network selects three typical image restoration tasks: denoising, inpainting, and super resolution. The whole training process is divided into single distortion training, hybrid distortion training of two types, and hybrid distortion training of three types. The design of loss function draws on the idea of deep supervision. Experimental results prove that the proposed method is not only superior to other methods in hybrid-distorted image restoration, but also suitable for single distortion image restoration.
摘要
图像复原是将退化图像恢复至接近理想图像的过程。以前的研究大多集中在单一失真图像上,然而大多数真实图像都经历了多种失真,单一失真图像复原算法无法有效提高图像质量。此外,现有的几种混合失真图像复原算法不具备处理单一失真的兼容性。因此,本文提出了一种基于分阶段训练的端到端神经网络。具体来说,该网络选择了三个典型的图像复原任务:图像去噪、图像修复和图像超分辨率。整个训练过程分为单一失真训练、两种类型的混合失真训练和三种类型的混合失真训练。损失函数的设计是基于深度监督的思想。实验结果表明,该方法不仅在混合失真图像复原方面优于其他方法,而且适用于单一失真图像复原。
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
FU X Y, HUANG JB,ZENG D L, et al. Removing rain from single images via a deep detail network [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 1715–1723.
YANG W H, TAN R T, FENG J S, et al. Deep joint rain detection and removal from a single image [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 1685–1694.
QIAN R, TAN R T, YANG W H, et al. Attentive generative adversarial network for raindrop removal from A single image [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2482–2491.
JIN X, CHEN Z B, LIN J X, et al. Unsupervised single image deraining with self-supervised constraints [C]//2019 IEEE International Conference on Image Processing. Taipei, China: IEEE, 2019: 2761–2765.
ZHANG K, ZUO W, ZHANG L. FFDNet: toward a fast and flexible solution for CNN based image denoising [J]. IEEE Transactions on Image Processing, 2018: 27(9): 4608–4622.
GUO S, YAN Z F, ZHANG K, et al. Toward convolutional blind denoising of real photographs [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 1712–1722.
TIAN C W, XU Y, ZUO W M. Image denoising using deep CNN with batch renormalization [J]. Neural Networks, 2020, 121: 461–473.
DONG L F, GAN Y Z, MAO X L, et al. Learning deep representations using convolutional auto-encoders with symmetric skip connections [C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing. Calgary, AB, Canada: IEEE, 2018: 3006–3010.
WANG N, LI J Y, ZHANG L F, et al. MUSICAL: Multi-scale image contextual attention learning for inpainting [C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao, China: International Joint Conferences on Artificial Intelligence Organization, 2019: 3748–3754.
LIU H Y, JIANG B, XIAO Y, et al. Coherent semantic attention for image inpainting [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea: IEEE, 2019: 4169–4178.
ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2472–2481.
DAI T, CAI J R, ZHANG Y B, et al. Second-order attention network for single image super-resolution [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 11057–11066.
ZHANG K, VAN GOOL L, TIMOFTE R. Deep unfolding network for image super-resolution [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 3214–3223.
YU K, DONG C, LIN L, et al. Crafting a toolchain for image restoration by deep reinforcement learning [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2443–2452.
SUGANUMA M, LIU X, OKATANI T. Attention-based adaptive selection of operations for image restoration in the presence of unknown combined distortions [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 9031–9040.
QIAN G C, WANG Y H, DONG C, et al. Rethinking the pipeline of demosaicing, denoising and super-resolution [EB/OL]. (2019-05-07). https://arxiv.org/abs/1905.02538.
WANG L, KIM T K, YOON K J. EventSR: from asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 8312–8322.
YANG T B, YAN Y, YUAN Z N, et al. Why does stagewise training accelerate convergence of testing error over SGD? [EB/OL]. (2018-12-10). https://arxiv.org/abs/1812.03934.
ZHOU Y Y, XIE L X, FISHMAN E K, et al. Deep supervision for pancreatic cyst segmentation in abdominal CT scans [M]//Medical image computing and computer assisted intervention — MICCAI 2017. Cham: Springer, 2017: 222–230.
LEMPITSKY V, VEDALDI A, ULYANOV D. Deep image prior [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 9446–9454.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hou, S., Zhu, W. & Li, H. Stagewise Training for Hybrid-Distorted Image Restoration. J. Shanghai Jiaotong Univ. (Sci.) 28, 793–801 (2023). https://doi.org/10.1007/s12204-022-2453-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12204-022-2453-2