Nothing Special   »   [go: up one dir, main page]
Skip to main content

Advertisement

SpringerLink
Log in

Ghost-free multi-exposure high dynamic range imaging based on feedback network

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In multi-exposure HDR imaging of dynamic scenes, two challenging problems need to be addressed: one is the removal of ghost artifacts, and the other is the reconstruction of lost information. Researchers usually adopt feed-forward networks to deal with the two problems in the existing deep learning-based multi-exposure HDR imaging methods. However, feedback mechanism plays an important role in human visual system, and feedback networks achieves efficient performance on other imaging tasks. Therefore, we presented a novel deep learning algorithm based on feedback mechanism to achieve ghost-free multi-exposure HDR imaging. To our best knowledge, it is the first time that feedback mechanism is applied to multi-exposure HDR imaging. We first use an attention-guided encoder to adjust the shallow features of the input images to avoid redundant features that interfere with feature fusion. Then, the adjusted features are fused by a feature merging network. It consists of two blocks: a feedback block and a global feature extracting block. In the feedback block, a hidden state is used in a recurrent structure to achieve the feedback mechanism. It enables the feedback block to use high-level features to enhance its feature fusion capability, and the reconfiguration capability of the network gradually improves during iterations. In the global feature extracting block, the extracted overall information increases the receptive field of the network and helps in the recovery of lost details. Extensive experiments show the superior performance of the proposed method in high-quality HDR images reconstruction, and outperforms state-of-the-art methods considered in comparison.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data will be provided if requested.

References

  1. Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH 2008 Classes, pp. 1–10 (2008)

  2. Mitsunaga, T., Nayar, S.K.: Radiometric self calibration. In: Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), vol. 1, pp. 374–380. IEEE (1999)

  3. Robertson, M.A., Borman, S., Stevenson, R.L.: Dynamic range improvement through multiple exposures. In: Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348), vol. 3, pp. 159–163. IEEE (1999)

  4. Gallo, O., Gelfandz, N., Chen, W.-C., Tico, M., Pulli, K.: Artifact-free high dynamic range imaging. In: 2009 IEEE International Conference on Computational Photography (ICCP), pp. 1–7 (2009). https://doi.org/10.1109/ICCPHOT.2009.5559003

  5. Jacobs, K., Loscos, C., Ward, G.: Automatic high-dynamic range image generation for dynamic scenes. IEEE Comput. Graphi. Appl. 28(2), 84–93 (2008). https://doi.org/10.1109/MCG.2008.23

    Article  Google Scholar 

  6. Zhang, W., Cham, W.-K.: Gradient-directed composition of multi-exposure images. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 530–536 (2010). https://doi.org/10.1109/CVPR.2010.5540168

  7. Khan, E.A., Akyuz, A.O., Reinhard, E.: Ghost removal in high dynamic range images. In: 2006 International Conference on Image Processing, pp. 2005–2008 (2006). https://doi.org/10.1109/ICIP.2006.312892

  8. Heo, Y.S., Lee, K.M., Lee, S.U., Moon, Y., Cha, J.: Ghost-free high dynamic range imaging. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) Computer Vision – ACCV 2010 (2011)

  9. Zimmer, H., Bruhn, A., Weickert, J.: Freehand HDR imaging of moving scenes with simultaneous resolution enhancement. Comput. Graph. Forum (Proceedings of Eurographics) 30(2), 405–414 (2011)

    Article  Google Scholar 

  10. Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graph. (TOG) (Proceedings of SIGGRAPH Asia 2012) 31(6), 203–120311 (2012)

    Google Scholar 

  11. Hu, J., Gallo, O., Pulli, K., Sun, X.: Hdr deghosting: How to deal with saturation? In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1163–1170 (2013). https://doi.org/10.1109/CVPR.2013.154

  12. Xia, S., Guo, S., Qu, Z., Yang, Y.: A coarse-to-fine ghost removal scheme for hdr imaging. Vis. Comput. 1–14 (2022)

  13. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks (2018)

  14. Huang, J., Zhu, P., Geng, M., Ran, J., Zhou, X., Xing, C., Wan, P., Ji, X.: Range scaling global u-net for perceptual image enhancement on mobile devices. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018)

  15. Kalantari, N.K., Ramamoorthi, R., et al.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. 36(4), 144–1 (2017)

    Article  Google Scholar 

  16. Liu, C., et al.: Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. PhD thesis, Massachusetts Institute of Technology (2009)

  17. Wu, S., Xu, J., Tai, Y.-W., Tang, C.-K.: Deep high dynamic range imaging with large foreground motions. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 117–132 (2018)

  18. Yan, Q., Gong, D., Shi, Q., Hengel, A.v.d., Shen, C., Reid, I., Zhang, Y.: Attention-guided network for ghost-free high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1751–1760 (2019)

  19. Yan, Q., Zhang, L., Liu, Y., Zhu, Y., Sun, J., Shi, Q., Zhang, Y.: Deep hdr imaging via a non-local network. IEEE Trans. Image Process. 29, 4308–4322 (2020)

    Article  Google Scholar 

  20. Niu, Y., Wu, J., Liu, W., Guo, W., Lau, R.W.: Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions. IEEE Trans. Image Process. 30, 3885–3896 (2021)

    Article  Google Scholar 

  21. Yan, Q., Gong, D., Shi, Q., Hengel, A., Shen, C., Reid, I., Zhang, Y.: Attention-guided network for ghost-free high dynamic range imaging (2019)

  22. Yan, Q., Gong, D., Shi, J.Q., van den Hengel, A., Shen, C., Reid, I., Zhang, Y.: Dual-attention-guided network for ghost-free high dynamic range imaging. Int. J. Comput. Vis. 130(1), 76–94 (2022)

  23. Pérez-Pellitero, E., Catley-Chandar, S., Shaw, R., Leonardis, A., Timofte, R., Zhang, Z., Liu, C., Peng, Y., Lin, Y., Yu, G., et al.: Ntire 2022 challenge on high dynamic range imaging: Methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1009–1023 (2022)

  24. Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)

    Article  Google Scholar 

  25. Sutskever, I., Martens, J., Hinton, G.E.: Generating text with recurrent neural networks. In: ICML (2011)

  26. Zamir, A.R., Wu, T.-L., Sun, L., Shen, W.B., Shi, B.E., Malik, J., Savarese, S.: Feedback networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1308–1317 (2017)

  27. Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., Wu, W.: Feedback network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3867–3876 (2019)

  28. Banterle, F., Artusi, A., Debattista, K., Chalmers, A.: Advanced High Dynamic Range Imaging: Theory and Practice, 2nd edn. AK Peters (CRC Press), Natick, MA, USA (2017)

    Book  Google Scholar 

  29. Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-based Lighting. Morgan Kaufmann, Burlington (2010)

    Google Scholar 

  30. Pece, F., Kautz, J.: Bitmap movement detection: Hdr for dynamic scenes. In: 2010 Conference on Visual Media Production, pp. 1–8 (2010). https://doi.org/10.1109/CVMP.2010.8

  31. Grosch, T.: Fast and robust high dynamic range image generation with camera and object movement. Vis. Model. Vis. RWTH Aachen 277284(3), 2 (2006)

    Google Scholar 

  32. Lee, C., Li, Y., Monga, V.: Ghost-free high dynamic range imaging via rank minimization. IEEE Signal Process. Lett. 21(9), 1045–1049 (2014). https://doi.org/10.1109/LSP.2014.2323404

    Article  Google Scholar 

  33. Oh, T.H., Lee, J.Y., Tai, Y.W., Kweon, I.S.: Robust high dynamic range imaging by rank minimization. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1219–1232 (2015)

    Article  Google Scholar 

  34. Grossberg, M.D., Nayar, S.K.: Modeling the space of camera response functions. IEEE Trans. Pattern Anal. Mach. Intell. 26(10), 1272–1282 (2004)

    Article  Google Scholar 

  35. Granados, M., Kim, K.I., Tompkin, J., Theobalt, C.: Automatic noise modeling for ghost-free HDR reconstruction. ACM Trans. Graph. 32(6), 201–120110 (2013). https://doi.org/10.1145/2508363.2508410

    Article  Google Scholar 

  36. Liu, Z., Wang, Y., Zeng, B., Liu, S.: Ghost-free high dynamic range imaging with context-aware transformer. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIX, pp. 344–360. Springer (2022)

  37. Chung, H., Cho, N.I.: High dynamic range imaging of dynamic scenes with saturation compensation but without explicit motion compensation. In: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 61–71 (2022). https://doi.org/10.1109/WACV51458.2022.00014

  38. Lee, K., Jang, Y.I., Cho, N.I.: Disentangled feature-guided multi-exposure high dynamic range imaging. In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2465–2469 (2022). https://doi.org/10.1109/ICASSP43922.2022.9747329

  39. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1664–1673 (2018)

  40. Khan, Z., Khanna, M., Raman, S.: FHDR: HDR image reconstruction from a single LDR image using feedback network. CoRR arxiv:1912.11463 (2019)

  41. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

  42. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR (2015) arxiv:1512.03385

  43. Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2016)

  44. Prabhakar, K.R., Arora, R., Swaminathan, A., Singh, K.P., Babu, R.V.: A fast, scalable, and reliable deghosting method for extreme exposure fusion. In: 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1–8. IEEE (2019)

  45. Tursun, O.T., Akyüz, A., Erdem, A., Erdem, E.: An objective deghosting quality metric for hdr images. Comput. Graph. Forum 35(2), 139–152 (2016)

  46. Aydın, T.O., Mantiuk, R., Seidel, H.-P.: Extending quality metrics to full luminance range images. In: Human Vision and Electronic Imaging Xiii, vol. 6806, p. 68060. International Society for Optics and Photonics (2008)

  47. Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. (TOG) 30(4), 1–14 (2011)

    Article  Google Scholar 

  48. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2017)

  49. Marnerides, D., Bashford-Rogers, T., Hatchett, J., Debattista, K.: Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content. CoRR arxiv:1803.02266 (2018)

  50. Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R., Unger, J.: HDR image reconstruction from a single exposure using deep CNNs. ACM Trans. Graph. (TOG) 36(6), 1–15 (2017)

    Article  Google Scholar 

Download references

Funding

This study was funded by the Sichuan Province Science and Technology Achievement Transfer Demonstration Project (Grant Number 2020ZHCG0056).

Author information

Authors and Affiliations

  1. School of Information and Communication Engineering, University of Electronic Science and Technology of China, No. 2006, Xiyuan Avenue, Chengdu, 611731, China

    Jing Gan & Yongqing Huo

Authors
  1. Jing Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongqing Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yongqing Huo.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gan, J., Huo, Y. Ghost-free multi-exposure high dynamic range imaging based on feedback network. Vis Comput 40, 4115–4132 (2024). https://doi.org/10.1007/s00371-023-03072-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03072-w

Keywords

Search

Navigation