Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

TSID-Net: a two-stage single image dehazing framework with style transfer and contrastive knowledge transfer: TSID-Net: a two-stage single image dehazing framework...

Published: 07 June 2024 Publication History

Abstract

Haze-free images have become a prerequisite for many computer vision tasks; therefore, single image dehazing is particularly important in the field of computer vision. However, existing deep learning dehazing methods face two main problems. First, existing dehazing methods are mostly trained based on paired images, but obtaining paired data of the same scene in the real world is challenging, which limits their dehazing performance. Second, most existing dehazing methods are primarily result-driven, which disregards the intermediate process of dehazing, and the rich prior knowledge present in clear and hazy images is not fully utilized, resulting in significant deviations between the dehazed results and the ground truth. Therefore, we propose a novel two-stage single image dehazing network, TSID-Net, to address the above two issues. In the first stage, we consider hazy images as a form of hazy artistic style, while clear images serve as the content information of the artwork. By combining style transfer, we generate high-quality and diverse paired images. This approach significantly mitigates the challenge of acquiring paired data and provides an ample training sample for the second stage. In the second stage, we utilize abundant clear and hazy images to train positive and negative teacher networks with strong robust prior learning capabilities. By combining knowledge transfer, contrastive learning and process-oriented mechanism, we achieve effective knowledge transfer and contrastive knowledge transfer of the intermediate features in the student network. Additionally, we propose a style version bank and incorporate curricular contrastive regularization to achieve dual contrastive learning of both the process and results for student network. Extensive experimental results demonstrate that TSID-Net effectively removes haze and produces visually pleasing results. Code is available at: https://github.com/wsl666/TSID-Net.git.

References

[1]
Chen Z, He Z, and Lu Z-M Dea-net: single image dehazing based on detail-enhanced convolution and content-guided attention IEEE Trans. Image Process. 2024 33 1002-1015
[2]
Qiu, Y., Zhang, K., Wang, C., Luo, W., Li, H., Jin, Z.: Mb-taylorformer: multi-branch efficient transformer expanded by Taylor formula for image dehazing. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12756–12767 (2023).
[3]
Tran L-A and Park D-C Encoder–decoder networks with guided transmission map for effective image dehazing Vis. Comput. 2024
[4]
Qin, X., Wang, Z., Bai, Y., Xie, X., Jia, H.: Ffa-net: feature fusion attention network for single image dehazing. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11908–11915 (2020).
[5]
Liu P and Liu J Knowledge-guided multi-perception attention network for image dehazing Vis. Comput. 2023
[6]
Zhou Y, Chen Z, Li P, Song H, Chen CLP, and Sheng B Fsad-net: feedback spatial attention dehazing network IEEE Trans. Neural Netw. Learn. Syst. 2023 34 10 7719-7733
[7]
Cantor A Optics of the atmosphere-scattering by molecules and particles IEEE J. Quantum Electron. 1978 14 9 698-699
[8]
Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M.-H.: Single image dehazing via multi-scale convolutional neural networks. In: Computer Vision-ECCV 2016: 14th European conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part II 14, pp. 154–169. Springer (2016)
[9]
Shao, Y., Li, L., Ren, W., Gao, C., Sang, N.: Domain adaptation for image dehazing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2805–2814 (2020).
[10]
Mallick T, Das PP, and Majumdar AK Characterizations of noise in kinect depth images: a review IEEE Sens. J. 2014 14 6 1731-1740
[11]
Sweeney, C., Izatt, G., Tedrake, R.: A supervised approach to predicting noise in depth images. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 796–802 (2019).
[12]
Torbunov, D., Huang, Y., Yu, H., Huang, J., Yoo, S., Lin, M., Viren, B., Ren, Y.: Uvcgan: Unet vision transformer cycle-consistent gan for unpaired image-to-image translation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 702–712 (2023)
[13]
Pernuš M, Štruc V, and Dobrišek S Maskfacegan: high-resolution face editing with masked gan latent code optimization IEEE Trans. Image Process. 2023 32 5893-5908
[14]
Jiang, Y., Jiang, L., Yang, S., Loy, C.C.: Scenimefy: learning to craft anime scene via semi-supervised image-to-image translation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7357–7367 (2023)
[15]
Cheema MN, Nazir A, Yang P, Sheng B, Li P, Li H, Wei X, Qin J, Kim J, and Feng DD Modified gan-caed to minimize risk of unintentional liver major vessels cutting by controlled segmentation using cta/spet-ct IEEE Trans. Ind. Inf. 2021 17 12 7991-8002
[16]
Engin, D., Genç, A., Kemal Ekenel, H.: Cycle-dehaze: enhanced cyclegan for single image dehazing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 825–833 (2018)
[17]
Li J, Li Y, Zhuo L, Kuang L, and Yu T Usid-net: unsupervised single image dehazing network via disentangled representations IEEE Trans. Multimed. 2023 25 3587-3601
[18]
Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017).
[19]
Park, T., Efros, A.A., Zhang, R., Zhu, J.-Y.: Contrastive learning for unpaired image-to-image translation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pp. 319–345. Springer (2020)
[20]
Zhang C, Lin Z, Xu L, Li Z, Tang W, Liu Y, Meng G, Wang L, and Li L Density-aware haze image synthesis by self-supervised content-style disentanglement IEEE Trans. Circuits Syst. Video Technol. 2022 32 7 4552-4572
[21]
Lin, X., Ren, C., Liu, X., Huang, J., Lei, Y.: Unsupervised image denoising in real-world scenarios via self-collaboration parallel generative adversarial branches. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12608–12618 (2023).
[22]
Chen H, Wang Z, Zhang H, Zuo Z, Li A, Xing W, Lu D, et al. Artistic style transfer with internal-external learning and contrastive learning Adv. Neural. Inf. Process. Syst. 2021 34 26561-26573
[23]
Li S, Zhou Y, Ren W, and Xiang W Pfonet: a progressive feedback optimization network for lightweight single image dehazing IEEE Trans. Image Process. 2023 32 6558-6569
[24]
Bai H, Pan J, Xiang X, and Tang J Self-guided image dehazing using progressive feature fusion IEEE Trans. Image Process. 2022 31 1217-1229
[25]
Song Y, He Z, Qian H, and Du X Vision transformers for single image dehazing IEEE Trans. Image Process. 2023 32 1927-1941
[26]
Song X, Zhou D, Li W, Dai Y, Shen Z, Zhang L, and Li H Tusr-net: triple unfolding single image dehazing with self-regularization and dual feature to pixel attention IEEE Trans. Image Process. 2023 32 1231-1244
[27]
Chen, W.-T., Huang, Z.-K., Tsai, C.-C., Yang, H.-H., Ding, J.-J., Kuo, S.-Y.: Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: toward a unified model. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17632–17641 (2022).
[28]
Liu X, Shi Z, Wu Z, Chen J, and Zhai G Griddehazenet+: an enhanced multi-scale network with intra-task knowledge transfer for single image dehazing IEEE Trans. Intell. Transp. Syst. 2023 24 1 870-884
[29]
Wu, H., Liu, J., Xie, Y., Qu, Y., Ma, L.: Knowledge transfer dehazing network for nonhomogeneous dehazing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1975–1983 (2020).
[30]
Zheng, Y., Zhan, J., He, S., Dong, J., Du, Y.: Curricular contrastive regularization for physics-aware single image dehazing. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5785–5794 (2023).
[31]
He K, Sun J, and Tang X Single image haze removal using dark channel prior IEEE Trans. Pattern Anal. Mach. Intell. 2011 33 12 2341-2353
[32]
Liu J, Liu RW, Sun J, and Zeng T Rank-one prior: real-time scene recovery IEEE Trans. Pattern Anal. Mach. Intell. 2023 45 7 8845-8860
[33]
Ling P, Chen H, Tan X, Jin Y, and Chen E Single image dehazing using saturation line prior IEEE Trans. Image Process. 2023 32 3238-3253
[34]
Cai B, Xu X, Jia K, Qing C, and Tao D Dehazenet: an end-to-end system for single image haze removal IEEE Trans. Image Process. 2016 25 11 5187-5198
[35]
Zhang, H., Patel, V.M.: Densely connected pyramid dehazing network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3194–3203 (2018).
[36]
Su YZ, Cui ZG, He C, Li AH, Wang T, and Cheng K Prior guided conditional generative adversarial network for single image dehazing Neurocomputing 2021 423 620-638
[37]
Wang N, Cui Z, Su Y, He C, Lan Y, and Li A SMGAN: a self-modulated generative adversarial network for single image dehazing AIP Adv. 2021 11 8 085227
[38]
Su YZ, He C, Cui ZG, Li AH, and Wang N Physical model and image translation fused network for single-image dehazing Pattern Recogn. 2023 142 109700
[39]
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423 (2016).
[40]
Li, C., Wand, M.: Combining Markov random fields and convolutional neural networks for image synthesis. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2479–2486 (2016).
[41]
Lu, M., Zhao, H., Yao, A., Chen, Y., Xu, F., Zhang, L.: A closed-form solution to universal style transfer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5952–5961 (2019)
[42]
Wu, Z., Song, C., Zhou, Y., Gong, M., Huang, H.: Efanet: exchangeable feature alignment network for arbitrary style transfer. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12305–12312 (2020)
[43]
Zhang, Y., Fang, C., Wang, Y., Wang, Z., Lin, Z., Fu, Y., Yang, J.: Multimodal style transfer via graph cuts. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5942–5950 (2019).
[44]
Sanakoyeu, A., Kotovenko, D., Lang, S., Ommer, B.: A style-aware content loss for real-time hd style transfer. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 698–714 (2018)
[45]
Kotovenko, D., Sanakoyeu, A., Ma, P., Lang, S., Ommer, B.: A content transformation block for image style transfer. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10024–10033 (2019).
[46]
Chen, H., Zhao, L., Wang, Z., Zhang, H., Zuo, Z., Li, A., Xing, W., Lu, D.: Dualast: dual style-learning networks for artistic style transfer. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 872–881 (2021).
[47]
Chen H, Wang Z, Zhang H, Zuo Z, Li A, Xing W, Lu D, et al. Artistic style transfer with internal-external learning and contrastive learning Adv. Neural. Inf. Process. Syst. 2021 34 26561-26573
[48]
Zhou J, Zeng S, and Zhang B Two-stage knowledge transfer framework for image classification Pattern Recogn. 2020 107 107529
[49]
He, S., Guo, T., Dai, T., Qiao, R., Shu, X., Ren, B., Xia, S.-T.: Open-vocabulary multi-label classification via multi-modal knowledge transfer. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 808–816 (2023)
[50]
Karambakhsh A, Sheng B, Li P, Li H, Kim J, Jung Y, and Chen CLP Sparsevoxnet: 3-d object recognition with sparsely aggregation of 3-d dense blocks IEEE Trans. Neural Netw. Learn. Syst. 2024 35 1 532-546
[51]
Li Y, Chen Y, Qi X, Li Z, Sun J, and Jia J Unifying voxel-based representation with transformer for 3d object detection Adv. Neural. Inf. Process. Syst. 2022 35 18442-18455
[52]
Li Z, Xu P, Chang X, Yang L, Zhang Y, Yao L, and Chen X When object detection meets knowledge distillation: a survey IEEE Trans. Pattern Anal. Mach. Intell. 2023 45 8 10555-10579
[53]
Lan Y, Cui Z, Su Y, Wang N, Li A, Zhang W, Li Q, and Zhong X Online knowledge distillation network for single image dehazing Sci. Rep. 2022 12 1 14927
[54]
Wang, N., Cui, Z., Li, A., Su, Y., Lan, Y.: Multi-priors guided dehazing network based on knowledge distillation. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 15–26 (2022). Springer
[55]
Hong, M., Xie, Y., Li, C., Qu, Y.: Distilling image dehazing with heterogeneous task imitation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3459–3468 (2020).
[56]
Lan Y, Cui Z, Su Y, Wang N, Li A, and Han D Physical-model guided self-distillation network for single image dehazing Front. Neurorobot. 2022 16 1036465
[57]
Lan Y, Cui Z, Su Y, Wang N, Li A, Li Q, Zhong X, and Zhang C Sskdn: a semisupervised knowledge distillation network for single image dehazing J. Electron. Imaging 2023 32 1 013002-013002
[58]
Henaff, O.: Data-efficient image recognition with contrastive predictive coding. In: International Conference on Machine Learning, pp. 4182–4192. PMLR (2020)
[59]
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726–9735 (2020).
[60]
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
[61]
Grill J-B, Strub F, Altché F, Tallec C, Richemond P, Buchatskaya E, Doersch C, Avila Pires B, Guo Z, Gheshlaghi Azar M, et al. Bootstrap your own latent-a new approach to self-supervised learning Adv. Neural. Inf. Process. Syst. 2020 33 21271-21284
[62]
Yi W, Dong L, Liu M, Hui M, Kong L, and Zhao Y Towards compact single image dehazing via task-related contrastive network Expert Syst. Appl. 2024 235 121130
[63]
Wang Y, Yan X, Wang FL, Xie H, Yang W, Zhang X-P, Qin J, and Wei M Ucl-dehaze: toward real-world image dehazing via unsupervised contrastive learning IEEE Trans. Image Process. 2024 33 1361-1374
[64]
Yi W, Dong L, Liu M, Hui M, Kong L, and Zhao Y Sid-net: single image dehazing network using adversarial and contrastive learning Multimed. Tools Appl. 2024
[65]
Cheng D, Li Y, Zhang D, Wang N, Sun J, and Gao X Progressive negative enhancing contrastive learning for image dehazing and beyond IEEE Trans. Multimed. 2024
[66]
Wu, H., Qu, Y., Lin, S., Zhou, J., Qiao, R., Zhang, Z., Xie, Y., Ma, L.: Contrastive learning for compact single image dehazing. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10546–10555 (2021).
[67]
Li, B., Peng, X., Wang, Z., Xu, J., Feng, D.: An all-in-one network for dehazing and beyond. arXiv preprint arXiv:1707.06543 (2017)
[68]
Park, D.Y., Lee, K.H.: Arbitrary style transfer with style-attentional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5880–5888 (2019)
[69]
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27 (2014)
[70]
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
[71]
Li B, Ren W, Fu D, Tao D, Feng D, Zeng W, and Wang Z Benchmarking single-image dehazing and beyond IEEE Trans. Image Process. 2019 28 1 492-505
[72]
Ancuti, C., Ancuti, C.O., Timofte, R., De Vleeschouwer, C.: I-haze: A dehazing benchmark with real hazy and haze-free indoor images. In: Advanced Concepts for Intelligent Vision Systems: 19th International Conference, ACIVS 2018, Poitiers, France, September 24–27, 2018, Proceedings 19, pp. 620–631 (2018). Springer
[73]
Ancuti, C.O., Ancuti, C., Timofte, R., De Vleeschouwer, C.: O-haze: a dehazing benchmark with real hazy and haze-free outdoor images. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 867–8678 (2018).
[74]
Zhao S, Zhang L, Huang S, Shen Y, and Zhao S Dehazing evaluation: real-world benchmark datasets, criteria, and baselines IEEE Trans. Image Process. 2020 29 6947-6962
[75]
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
[76]
Zhu Q, Mai J, and Shao L A fast single image haze removal algorithm using color attenuation prior IEEE Trans. Image Process. 2015 24 11 3522-3533
[77]
Berman, D., Treibitz, T., Avidan, S.: Non-local image dehazing. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1674–1682 (2016).
[78]
Galdran A Image dehazing by artificial multiple-exposure image fusion Signal Process. 2018 149 135-147
[79]
Chen, Z., Wang, Y., Yang, Y., Liu, D.: Psd: principled synthetic-to-real dehazing guided by physical priors. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7176–7185 (2021).
[80]
Zhao S, Zhang L, Shen Y, and Zhou Y Refinednet: a weakly supervised refinement framework for single image dehazing IEEE Trans. Image Process. 2021 30 3391-3404
[81]
Li, B., Liu, X., Hu, P., Wu, Z., Lv, J., Peng, X.: All-in-one image restoration for unknown corruption. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17431–17441 (2022).

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image The Visual Computer: International Journal of Computer Graphics
The Visual Computer: International Journal of Computer Graphics  Volume 41, Issue 3
Feb 2025
629 pages

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 07 June 2024
Accepted: 24 May 2024

Author Tags

  1. Single image dehazing
  2. Style transfer
  3. Comparative knowledge transfer
  4. Unsupervised learning

Author Tag

  1. Information and Computing Sciences
  2. Artificial Intelligence and Image Processing

Qualifiers

  • Research-article

Funding Sources

  • National Natural Science Foundation of China
  • Engineering Project for Improving the Innovation Capability of Technology-oriented Small and Medium-sized Enterprises

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 23 Feb 2025

Other Metrics

Citations

View Options

View options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media