Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Geometry-Aware Single-Image Full-Body Human Relighting

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13676))

Included in the following conference series:

Abstract

Single-image human relighting aims to relight a target human under new lighting conditions by decomposing the input image into albedo, shape and lighting. Although plausible relighting results can be achieved, previous methods suffer from both the entanglement between albedo and lighting and the lack of hard shadows, which significantly decrease the realism. To tackle these two problems, we propose a geometry-aware single-image human relighting framework that leverages single-image geometry reconstruction for joint deployment of traditional graphics rendering and neural rendering techniques. For the de-lighting, we explore the shortcomings of UNet architecture and propose a modified HRNet, achieving better disentanglement between albedo and lighting. For the relighting, we introduce a ray tracing-based per-pixel lighting representation that explicitly models high-frequency shadows and propose a learning-based shading refinement module to restore realistic shadows (including hard cast shadows) from the ray-traced shading maps. Our framework is able to generate photo-realistic high-frequency shadows such as cast shadows under challenging lighting conditions. Extensive experiments demonstrate that our proposed method outperforms previous methods on both synthetic and real images.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. https://www.blender.org/

  2. https://web.twindom.com/

  3. https://polyhaven.com/hdris

  4. Barron, J.T., Malik, J.: Color constancy, intrinsic images, and shape estimation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 57–70. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33765-9_5

    Chapter  Google Scholar 

  5. Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. 37(8), 1670–1687 (2014)

    Article  Google Scholar 

  6. Baslamisli, A.S., Le, H.A., Gevers, T.: CNN based learning using reflection and retinex models for intrinsic image decomposition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6674–6683 (2018)

    Google Scholar 

  7. Bonneel, N., Kovacs, B., Paris, S., Bala, K.: Intrinsic decompositions for image editing. In: Computer Graphics Forum, vol. 36, pp. 593–609. Wiley Online Library (2017)

    Google Scholar 

  8. Chabert, C.F., et al.: Relighting human locomotion with flowed reflectance fields. In: ACM SIGGRAPH 2006 Sketches, p. 76 (2006)

    Google Scholar 

  9. Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4641–4650 (2021)

    Google Scholar 

  10. Christou, C.G., Koenderink, J.J.: Light source dependence in shape from shading. Vision. Res. 37(11), 1441–1449 (1997)

    Article  Google Scholar 

  11. Tajima, D., Kanamori, Y., Endo, Y.: Relighting humans in the wild: monocular full-body human relighting with domain adaptation. Comput. Graph. Forum 40(7), 205–216 (2021)

    Google Scholar 

  12. Debevec, P.: The light stages and their applications to photoreal digital actors. SIGGRAPH Asia 2(4), 1–6 (2012)

    Google Scholar 

  13. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 145–156 (2000)

    Google Scholar 

  14. Debevec, P., Wenger, A., Tchou, C., Gardner, A., Waese, J., Hawkins, T.: A lighting reproduction approach to live-action compositing. ACM Trans. Graphics (TOG) 21(3), 547–556 (2002)

    Article  Google Scholar 

  15. Ding, S., Sheng, B., Hou, X., Xie, Z., Ma, L.: Intrinsic image decomposition using multi-scale measurements and sparsity. In: Computer Graphics Forum, vol. 36, pp. 251–261. Wiley Online Library (2017)

    Google Scholar 

  16. Egger, B., et al.: Occlusion-aware 3d morphable models and an illumination prior for face image analysis. Int. J. Comput. Vision 126(12), 1269–1287 (2018)

    Article  Google Scholar 

  17. Gardner, M.A., et al.: Learning to predict indoor illumination from a single image. ACM Trans. Graph. (TOG) 36(6), 1–14 (2017)

    Article  Google Scholar 

  18. Genova, K., Cole, F., Maschinot, A., Sarna, A., Vlasic, D., Freeman, W.T.: Unsupervised training for 3D morphable model regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8377–8386 (2018)

    Google Scholar 

  19. Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., Li, S.Z.: Towards fast, accurate and stable 3D dense face alignment. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12364, pp. 152–168. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58529-7_10

    Chapter  Google Scholar 

  20. Guo, K., et al.: The relightables: volumetric performance capture of humans with realistic relighting. ACM Trans. Graph. (TOG) 38(6), 1–19 (2019)

    Google Scholar 

  21. Hawkins, T., Cohen, J., Debevec, P.: A photometric approach to digitizing cultural artifacts. In: Proceedings of the 2001 Conference on Virtual Reality, Archeology, and Cultural Heritage, pp. 333–342 (2001)

    Google Scholar 

  22. Hou, A., Zhang, Z., Sarkis, M., Bi, N., Tong, Y., Liu, X.: Towards high fidelity face relighting with realistic shadows. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14719–14728 (2021)

    Google Scholar 

  23. Imber, J., Guillemaut, J.-Y., Hilton, A.: Intrinsic textures for relightable free-viewpoint video. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 392–407. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10605-2_26

    Chapter  Google Scholar 

  24. Jafarian, Y., Park, H.S.: Learning high fidelity depths of dressed humans by watching social media dance videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12753–12762 (2021)

    Google Scholar 

  25. Kanamori, Y., Endo, Y.: Relighting humans: occlusion-aware inverse rendering for full-body human images. ACM Trans. Graph. (TOG) 37(6), 1–11 (2018)

    Article  Google Scholar 

  26. Ke, Z., et al.: Is a green screen really necessary for real-time portrait matting? (2020)

    Google Scholar 

  27. Laffont, P.Y., Bazin, J.C.: Intrinsic decomposition of image sequences from local temporal variations. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 433–441 (2015)

    Google Scholar 

  28. Lagunas, M., et al.: Single-image full-body human relighting. arXiv preprint arXiv:2107.07259 (2021)

  29. Land, E.H., McCann, J.J.: Lightness and retinex theory. Josa 61(1), 1–11 (1971)

    Google Scholar 

  30. Li, C., Zhou, K., Wu, H.T., Lin, S.: Physically-based simulation of cosmetics via intrinsic image decomposition with facial priors. IEEE Trans. Pattern Anal. Mach. Intell. 41(6), 1455–1469 (2018)

    Article  Google Scholar 

  31. Li, G., et al.: Capturing relightable human performances under general uncontrolled illumination. In: Comput. Graph. Forum, vol. 32, pp. 275–284. Wiley Online Library (2013)

    Google Scholar 

  32. Li, Y., Liu, M.-Y., Li, X., Yang, M.-H., Kautz, J.: A closed-form solution to photorealistic image stylization. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 468–483. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_28

    Chapter  Google Scholar 

  33. Lin, J., Yuan, Y., Shao, T., Zhou, K.: Towards high-fidelity 3D face reconstruction from in-the-wild images using graph convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5891–5900 (2020)

    Google Scholar 

  34. Lopez-Moreno, J., Hadap, S., Reinhard, E., Gutierrez, D.: Light source detection in photographs. In: CEIG, pp. 161–167 (2009)

    Google Scholar 

  35. Meka, A., et al.: Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)

    Article  Google Scholar 

  36. Meka, A., et al.: Deep relightable textures: volumetric performance capture with neural rendering. ACM Trans. Graph. (TOG) 39(6), 1–21 (2020)

    Article  Google Scholar 

  37. Nagano, K., et al.: Deep face normalization. ACM Trans. Graph. (TOG) 38(6), 1–16 (2019)

    Article  Google Scholar 

  38. Nestmeyer, T., Lalonde, J.F., Matthews, I., Lehrmann, A.: Learning physics-guided face relighting under directional light. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5124–5133 (2020)

    Google Scholar 

  39. Okatani, T., Deguchi, K.: Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection center. Comput. Vis. Image Underst. 66(2), 119–131 (1997)

    Article  Google Scholar 

  40. Pandey, R., et al.: Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph. (TOG) 40(4), 1–21 (2021)

    Article  Google Scholar 

  41. Ramachandran, V.S.: Perception of shape from shading. Nature 331(6152), 163–166 (1988)

    Article  Google Scholar 

  42. Saito, S., Simon, T., Saragih, J., Joo, H.: PIFuHD: multi-level pixel-aligned implicit function for high-resolution 3D human digitization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 84–93 (2020)

    Google Scholar 

  43. Sengupta, S., Kanazawa, A., Castillo, C.D., Jacobs, D.W.: SfSNet: learning shape, reflectance and illuminance of faces ‘in the wild’. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6296–6305 (2018)

    Google Scholar 

  44. Shahlaei, D., Blanz, V.: Realistic inverse lighting from a single 2D image of a face, taken under unknown and complex lighting. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–8. IEEE (2015)

    Google Scholar 

  45. Sheng, B., Li, P., Jin, Y., Tan, P., Lee, T.Y.: Intrinsic image decomposition with step and drift shading separation. IEEE Trans. Visual Comput. Graphics 26(2), 1332–1346 (2018)

    Article  Google Scholar 

  46. Shu, Z., Hadap, S., Shechtman, E., Sunkavalli, K., Paris, S., Samaras, D.: Portrait lighting transfer using a mass transport approach. ACM Trans. Graph. (TOG) 36(4), 1 (2017)

    Article  Google Scholar 

  47. Shu, Z., Yumer, E., Hadap, S., Sunkavalli, K., Shechtman, E., Samaras, D.: Neural face editing with intrinsic image disentangling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5541–5550 (2017)

    Google Scholar 

  48. Sorkine, O.: Laplacian mesh processing. In: Eurographics (State of the Art Reports), pp. 53–70. Citeseer (2005)

    Google Scholar 

  49. Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)

    Google Scholar 

  50. Sun, T., et al.: Single image portrait relighting. ACM Trans. Graph. 38(4), 1–79 (2019)

    Article  Google Scholar 

  51. Tewari, A., et al.: MoFA: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1274–1283 (2017)

    Google Scholar 

  52. Wang, Z., Yu, X., Lu, M., Wang, Q., Qian, C., Xu, F.: Single image portrait relighting via explicit multiple reflectance channel modeling. ACM Trans. Graph. (TOG) 39(6), 1–13 (2020)

    Google Scholar 

  53. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., Debevec, P.: Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Trans. Graph. (TOG) 24(3), 756–764 (2005)

    Article  Google Scholar 

  54. Weyrich, T., et al.: Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. (ToG) 25(3), 1013–1024 (2006)

    Article  Google Scholar 

  55. Whitted, T.: An improved illumination model for shaded display. In: Proceedings of the 6th annual conference on Computer graphics and interactive techniques, p. 14 (1979)

    Google Scholar 

  56. Ye, G., Garces, E., Liu, Y., Dai, Q., Gutierrez, D.: Intrinsic video and applications. ACM Trans. Graph. (ToG) 33(4), 1–11 (2014)

    Article  Google Scholar 

  57. Zhang, L., Zhang, Q., Wu, M., Yu, J., Xu, L.: Neural video portrait relighting in real-time via consistency modeling. arXiv preprint arXiv:2104.00484 (2021)

  58. Zhang, X., et al.: Portrait shadow manipulation. ACM Trans. Graph. (TOG) 39(4), 1–78 (2020)

    Article  Google Scholar 

  59. Zhou, H., Hadap, S., Sunkavalli, K., Jacobs, D.W.: Deep single-image portrait relighting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7194–7202 (2019)

    Google Scholar 

Download references

Acknowledgement

This paper is supported by National Key R &D Program of China (2021ZD0113501) and the NSFC project No.62125107, No.62171255 and No.61827805.

Author information

Authors and Affiliations

  1. Department of Automation, Tsinghua University, Beijing, China

    Chaonan Ji, Tao Yu & Yebin Liu

  2. Meta Reality Labs, Redmond, USA

    Kaiwen Guo

  3. Guangdong OPPO Mobile Telecommunications Corporation Limited, Dongguan, China

    Jingxin Liu

Authors
  1. Chaonan Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kaiwen Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jingxin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yebin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yebin Liu .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4431 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ji, C., Yu, T., Guo, K., Liu, J., Liu, Y. (2022). Geometry-Aware Single-Image Full-Body Human Relighting. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13676. Springer, Cham. https://doi.org/10.1007/978-3-031-19787-1_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19787-1_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19786-4

  • Online ISBN: 978-3-031-19787-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation