Abstract
After a long period of research and development, 2D image object detection technology has been greatly improved in terms of efficiency and accuracy, but it is still a great challenge in dealing with motion blurred images. Most of the current deblurring algorithms are too computationally intensive to meet the demands of real-time tasks. To address this problem, a real-time deblur network (RT-Deblur) based on adversarial generative networks has been proposed. Specifically, a fast Fourier transform residual (FFTRes) block was designed to allow the model to achieve better performance while using fewer residual blocks, and to reduce the number of parameters to meet real-time requirements. In order to improve the deblurring performance and the accuracy of object detection after deblurring, a weighted loss function was developed, which includes the discriminator loss, MSE loss, multi-scale frequency reconstruction loss and perceptual YOLO loss. To validate the effectiveness of deblurring for object detection, we produced two blurred object detection datasets based on REDS and GoPro. A large number of comparative experiments on object detection before and after deblurring using YOLOv5s have been done, and the results show that our network achieves leading \(mAP\) scores. RT-Deblur improved the \(mAP\) scores from 0.486 to 0.566 and from 0.17 to 0.433 on the two blurred datasets, respectively.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
References
Nishiyama, M., Hadid, A., Takeshima, H., Shotton, J., Kozakaya, T., Yamaguchi, O.: Facial deblur inference using subspace analysis for recognition of blurred faces. IEEE Trans. Pattern Anal. Mach. Intell. 33, 838–845 (2011)
Ben-Ezra, M., Nayar, S.K.: Motion-based motion deblurring. IEEE Trans. Pattern Anal. Mach. Intell. 26, 689–698 (2004)
Fortunato, H.E., Oliveira, M.M.: Fast high-quality non-blind deconvolution using sparse adaptive priors. Vis. Comput. 30, 661–671 (2014)
Dong, J.X., Roth, S., Schiele, B.: DWDN: deep wiener deconvolution network for non-blind image deblurring. IEEE Trans. Pattern Anal. Mach. Intell. 44, 9960–9976 (2022)
Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. Acm Trans. Graph. 25, 787–794 (2006)
Kim, T.H., Lee, K.M.: IEEE: Generalized video deblurring for dynamic scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5426–5434. (2015)
Dong, J., Pan, J., Su, Z.: Blur kernel estimation via salient edges and low rank prior for blind image deblurring. Signal Process.-Image Commun. 58, 134–145 (2017)
Bai, H., Cheng, S., Tang, J., Pan, J., Soc, I.C.: Learning a cascaded non-local residual network for super-resolving blurry images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 223–232. (2021)
Chan, K.C.K., Wang, X., Yu, K., Dong, C., Loy, C.C.: IEEE Comp, S.O.C.: BasicVSR: the search for essential components in video super-resolution and beyond. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4945–4954. (2021)
Zou, W., Jiang, M., Zhang, Y., Chen, L., Lu, Z., Wu, Y., Soc, I.C.: SDWNet: A Straight Dilated Network with Wavelet Transformation for image Deblurring. In: 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1895–1904. (2021)
Zhang, Z., Chen, H., Yin, X., Deng, J., Li, W.: Dynamic selection of proper kernels for image deblurring: a multistrategy design. Vis. Comput. 39, 1375–1390 (2022)
Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: IEEE: DeblurGAN: blind motion deblurring using conditional adversarial networks. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8183–8192. (2018)
Kupyn, O., Martyniuk, T., Wu, J., Wang, Z.: IEEE: DeblurGAN-v2: Deblurring (Orders-of-Magnitude) faster and better. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8877–8886. (2019)
Liu, X., Du, H., Xu, J., Qiu, B.: DBGAN: a dual-branch generative adversarial network for undersampled MRI reconstruction. Magn. Reson. Imaging 89, 77–91 (2022)
Cho, S.-J., Ji, S.-W., Hong, J.-P., Jung, S.-W., Ko, S.-J.: IEEE: rethinking coarse-to-fine approach in single image deblurring. In: 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4621–4630. (2021)
Nah, S., Kim, T.H., Lee, K.M.: IEEE: deep multi-scale convolutional neural network for dynamic scene deblurring. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 257–265. (2017)
Nah, S., Baik, S., Hong, S., Moon, G., Son, S., Timofte, R., Lee, K.M.: IEEE: NTIRE 2019 challenge on video deblurring and super-resolution: dataset and study. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1996–2005. (2019)
Zhang, H., Dai, Y., Li, H., Koniusz, P., Soc, I.C.: Deep stacked hierarchical multi-patch network for image deblurring. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5971–5979. (2019)
Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: IEEE Comp, S.O.C.: multi-stage progressive image restoration. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14816–14826. (2021)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Commun. ACM 63, 139–144 (2020)
Pan, J.S., Dong, J.X., Liu, Y., Zhang, J.W., Ren, J.M., Tang, J.H., Tai, Y.W., Yang, M.H.: Physics-based generative adversarial models for image restoration and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 43, 2449–2462 (2021)
Zhang, K., Luo, W., Zhong, Y., Ma, L., Stenger, B., Liu, W., Li, H.: IEEE: deblurring by realistic blurring. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2734–2743. (2020)
He, K.M., Zhang, X.Y., Ren, S.Q., Sun, J.: IEEE: deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. (2016)
Suin, M., Purohit, K., Rajagopalan, A.N.: IEEE: spatially-attentive patch-hierarchical network for adaptive motion deblurring. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3603–3612. (2020)
Zhou, L.H., Min, W.D., Lin, D.Y., Han, Q., Liu, R.K.: Detecting motion blurred vehicle logo in IoV Using filter-DeblurGAN and VL-YOLO. IEEE Trans. Veh. Technol. 69, 3604–3614 (2020)
Batchuluun, G., Kang, J.K., Nguyen, D.T., Pham, T.D., Arsalan, M., Park, K.R.: Deep learning-based thermal image reconstruction and object detection. IEEE Access 9, 5951–5971 (2021)
Truong, N.Q., Lee, Y.W., Owais, M., Nguyen, D.T., Batchuluun, G., Pham, T.D., Park, K.R.: SlimDeblurGAN-based motion deblurring and marker detection for autonomous drone landing. Sensors 20, 3918 (2020)
Xie, G., Li, Z., Bhattacharyya, S., Mehmood, A.: IEEE: plug-and-play deblurring for robust object detection. In: IEEE International Conference on Visual Communications and Image Processing (VCIP) - Visual Communications in the Era of AI and Limited Resources. (2021)
Tao, X., Gao, H.Y., Shen, X.Y., Wang, J., Jia, J.Y.: IEEE: scale-recurrent network for deep image deblurring. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8174–8182. (2018)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: 18th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234–241. (2015)
Isola, P., Zhu, J.Y., Zhou, T.H., Efros, A.A.: IEEE: image-to-image translation with conditional adversarial networks. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976. (2017)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: 34th International Conference on Machine Learning. (2017)
Wu, Y.L., Shuai, H.H., Tam, Z.R., Chiu, H.Y.: IEEE: Gradient normalization for generative adversarial networks. In: 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6353–6362. (2021)
Johnson, J., Alahi, A., Li, F.F.: Perceptual losses for real-time style transfer and super-resolution. In: 14th European Conference on Computer Vision (ECCV), pp. 694–711. (2016)
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.M., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J.J., Chintala, S.: PyTorch: An imperative style, high-performance deep learning library. In: 33rd Conference on Neural Information Processing Systems (NeurIPS). (2019)
Nah, S., Son, S., Lee, S., Timofte, R., Lee, K.M., Chen, L.Y., Zhang, J., Lu, X., Chu, X.J., Chen, C.P., Xiong, Z.W., Xu, R.K., Xiao, Z.Y., Huang, J., Zhang, Y.Y., Xi, S., Wei, J., Bai, H.R., Cheng, S.S., Wei, H., Sun, L., Tang, J.H., Pan, J.S., Lee, D., Lee, C., Kim, T., Wang, X.B., Zhang, D.F., Pan, Z.H., Lin, T.W., Wu, W.H., He, D.L., Li, B.P., Li, B.Y., Xi, T., Zhang, G., Liu, J.T., Han, J.Y., Ding, E.R., Tao, G.P., Chu, W.Q., Cao, Y., Luo, D.H., Tai, Y., Lu, T., Wang, C.J., Li, J.L., Huang, F.Y., Chen, H.T., Chen, S.J., Guo, T.Y., Wang, Y.H., Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Shao, L., Zuo, Y.S., Ou, Y.M., Chai, Y.J., Shi, L., Liu, S., Lei, L., Feng, C.Y., Zeng, K., Yao, Y.Y., Liu, X.R., Zhang, Z.Z., Huang, H.C., Zhang, Y.C., Jiang, M.C., Zou, W.B., Miao, S., Kim, Y., Sun, Y.J., Deng, S.Y., Ren, W.Q., Cao, X.C., Wang, T., Suin, M., Rajagopalan, A.N., Duong, V.V., Nguyen, T.H., Yim, J., Jeon, B., Li, R., Xie, J.W., Han, J.W., Choi, J.H., Kim, J.H., Lee, J.S., Zhang, J.X., Peng, F., Svitov, D., Pakulich, D., Kim, J., Jeong, J., Soc, I.C.: NTIRE 2021 challenge on image deblurring. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 149–165. (2021)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004)
Xu, R.K., Xiao, Z.Y., Huang, J., Zhang, Y.Y., Xiong, Z.W., Soc, I.C.: EDPN: Enhanced deep pyramid network for blurry image restoration. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 414–423. (2021)
Purohit, K., Rajagopalan, A.N.: Assoc Advancement Artificial, I.: Region-adaptive dense network for efficient motion deblurring. In: 34th AAAI Conference on Artificial Intelligence / 32nd Innovative Applications of Artificial Intelligence Conference / 10th AAAI Symposium on Educational Advances in Artificial Intelligence, pp. 11882–11889. (2020)
Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pp. 17–33. (2022)
Blalock, D., Gonzalez Ortiz, J.J., Frankle, J., Guttag, J.: What is the state of neural network pruning? Proc. Mach. Learn. Syst. 2, 129–146 (2020)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Heikkila, J., Silvén, O.: A four-step camera calibration procedure with implicit image correction. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, pp. 1106–1112. (1997)
Acknowledgements
This work was supported by National Natural Science Foundation of China (62072246).
Funding
National Natural Science Foundation of China, 62072246.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, H., Hu, C., Qian, W. et al. RT-Deblur: real-time image deblurring for object detection. Vis Comput 40, 2873–2887 (2024). https://doi.org/10.1007/s00371-023-02991-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-023-02991-y