Polarimetric BSSRDF Acquisition of Dynamic Faces
Article No.: 275, Pages 1 - 11
Abstract
Acquisition and modeling of polarized light reflection and scattering help reveal the shape, structure, and physical characteristics of an object, which is increasingly important in computer graphics. However, current polarimetric acquisition systems are limited to static and opaque objects. Human faces, on the other hand, present a particularly difficult challenge, given their complex structure and reflectance properties, the strong presence of spatially-varying subsurface scattering, and their dynamic nature. We present a new polarimetric acquisition method for dynamic human faces, which focuses on capturing spatially varying appearance and precise geometry, across a wide spectrum of skin tones and facial expressions. It includes both single and heterogeneous subsurface scattering, index of refraction, and specular roughness and intensity, among other parameters, while revealing biophysically-based components such as inner- and outer-layer hemoglobin, eumelanin and pheomelanin. Our method leverages such components' unique multispectral absorption profiles to quantify their concentrations, which in turn inform our model about the complex interactions occurring within the skin layers. To our knowledge, our work is the first to simultaneously acquire polarimetric and spectral reflectance information alongside biophysically-based skin parameters and geometry of dynamic human faces. Moreover, our polarimetric skin model integrates seamlessly into various rendering pipelines.
References
[1]
Carlos Aliaga, Christophe Hery, and Mengqi Xia. 2022. Estimation of spectral biophysical skin properties from captured RGB albedo. arXiv preprint arXiv:2201.10695 (2022).
[2]
Carlos Aliaga, Menqi Xia, Xao Xie, Adrian Jarabo, Gustav Braun, and Christophe Hery. 2023. A Hyperspectral Space of Skin Tones for Inverse Rendering of Biophysical Skin Properties. In Computer Graphics Forum, Vol. 42. Wiley Online Library.
[3]
Sarah Alotaibi and William AP Smith. 2017. A Biophysical 3D Morphable Model of Face Appearance. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 824--832.
[4]
Gary A. Atkinson and Edwin R. Hancock. 2006. Recovery of Surface Orientation from Diffuse Polarization. IEEE transactions on image processing 15, 6 (2006), 1653--1664. Publisher: IEEE.
[5]
Dejan Azinović, Olivier Maury, Christophe Hery, Matthias Nießner, and Justus Thies. 2023. High-Res Facial Appearance Capture from Polarized Smartphone Images. In Proc. the IEEE/CVF CVPR.
[6]
Yunhao Ba, Alex Gilbert, Franklin Wang, Jinfa Yang, Rui Chen, Yiqin Wang, Lei Yan, Boxin Shi, and Achuta Kadambi. 2020. Deep Shape from Polarization. In Proc. ECCV 2020. Springer, 554--571.
[7]
Seung-Hwan Baek and Felix Heide. 2021. Polarimetric Spatio-Temporal Light Transport Probing. ACM Transactions on Graphics 40, 6 (2021), 18 pages.
[8]
Seung-Hwan Baek and Felix Heide. 2022. All-photon Polarimetric Time-of-Flight Imaging. In Proc. IEEE/CVF CVPR. 17876--17885.
[9]
Seung-Hwan Baek, Daniel S. Jeon, Xin Tong, and Min H. Kim. 2018. Simultaneous acquisition of polarimetric SVBRDF and normals. ACM Trans. Graph. 37, 6 (2018), 268--1.
[10]
Seung-Hwan Baek, Tizian Zeltner, Hyunjin Ku, Inseung Hwang, Xin Tong, Wenzel Jakob, and Min H. Kim. 2020. Image-Based Acquisition and Modeling of Polarimetric Reflectance. ACM Transactions on Graphics 39, 4 (2020), 139.
[11]
Alexey N. Bashkatov, E. A. Genina, V. I. Kochubey, and V. V. Tuchin. 2005. Optical properties of human skin, subcutaneous and mucous tissues in the wavelength range from 400 to 2000 nm. Journal of Physics D: Applied Physics 38, 15 (2005), 2543.
[12]
Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-Quality Single-Shot Capture of Facial Geometry. ACM Transactions on Graphics 29, 4 (2010), 9 pages.
[13]
Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W. Sumner, and Markus Gross. 2011. High-Quality Passive Facial Performance Capture Using Anchor Frames. ACM Transactions on Graphics 30, 4 (2011), 10 pages.
[14]
Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei, Kevyn Mcphail, Ravi Ramamoorthi, Yaser Sheikh, and Jason Saragih. 2021. Deep Relightable Appearance Models for Animatable Faces. ACM Transactions on Graphics 40, 4 (2021).
[15]
Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010. High Resolution Passive Facial Performance Capture. ACM Transactions on Graphics 29, 4 (2010).
[16]
Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler. 2015. Real-Time High-Fidelity Facial Performance Capture. ACM Transactions on Graphics 34, 4 (2015), 9 pages.
[17]
Xu Cao, Hiroaki Santo, Fumio Okura, and Yasuyuki Matsushita. 2023. Multi-View Azimuth Stereo via Tangent Space Consistency. In Proc. IEEE/CVF CVPR. 825--834.
[18]
Tenn F Chen, Gladimir VG Baranoski, Bradley W Kimmel, and Erik Miranda. 2015. Hyperspectral modeling of skin appearance. ACM Transactions on Graphics (TOG) 34, 3 (2015), 1--14.
[19]
Zhaopeng Cui, Viktor Larsson, and Marc Pollefeys. 2019. Polarimetric Relative Pose Estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2671--2680.
[20]
Akshat Dave, Yongyi Zhao, and Ashok Veeraraghavan. 2022. Pandora: Polarization-aided neural decomposition of radiance. In European Conference on Computer Vision. Springer, 538--556.
[21]
Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the Reflectance Field of a Human Face. In Annual Conference Series (Proceedings of SIGGRAPH). 145--156.
[22]
Eugene d'Eon, David Luebke, and Eric Enderton. 2007. Efficient Rendering of Human Skin. In Proceedings of the 18th Eurographics conference on Rendering Techniques. Citeseer, 147--157.
[23]
Valentin Deschaintre, Yiming Lin, and Abhijeet Ghosh. 2021. Deep Polarization Imaging for 3D Shape and SVBRDF Acquisition. In Proc. IEEE/CVF CVPR. 15567--15576.
[24]
Craig Donner and Henrik Wann Jensen. 2005. Light Diffusion in Multi-Layered Translucent Materials. ACM Transactions on Graphics 24, 3 (2005), 1032--1039.
[25]
Craig Donner and Henrik Wann Jensen. 2006. A Spectral BSSRDF for Shading Human Skin. Rendering techniques 2006 (2006), 409--418. Publisher: Citeseer.
[26]
Craig Donner, Tim Weyrich, Eugene d'Eon, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2008. A Layered, Heterogeneous Reflectance Model for Acquiring and Rendering Human Skin. ACM Transactions on Graphics 27, 5 (2008), 12 pages.
[27]
Ashraf Fawzy, Tianshi David Wu, Kunbo Wang, Matthew L. Robinson, Jad Farha, Amanda Bradke, Sherita H. Golden, Yanxun Xu, and Brian T. Garibaldi. 2022. Racial and ethnic discrepancy in pulse oximetry and delayed identification of treatment eligibility among patients with COVID-19. JAMA internal medicine 182, 7 (2022), 730--738. Publisher: American Medical Association.
[28]
Graham Fyffe. 2010. Single-shot photometric stereo by spectral multiplexing. In ACM SIGGRAPH ASIA 2010 Sketches. 1--2.
[29]
Graham Fyffe, Paul Graham, Borom Tunwattanapong, Abhijeet Ghosh, and Paul Debevec. 2016. Near-Instant Capture of High-Resolution Facial Geometry and Reflectance. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 353--363.
[30]
Graham Fyffe, Tim Hawkins, Chris Watts, Wan-Chun Ma, and Paul Debevec. 2011. Comprehensive Facial Performance Capture. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 425--434.
[31]
Pablo Garrido, Levi Valgaert, Chenglei Wu, and Christian Theobalt. 2013. Reconstructing Detailed Dynamic Face Geometry from Monocular Video. ACM Transactions on Graphics 32, 6 (2013), 10 pages.
[32]
Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. 2010. Circularly polarized spherical illumination reflectometry. In ACM SIGGRAPH Asia 2010 papers. 1--12.
[33]
Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen, and Paul Debevec. 2008. Practical Modeling and Acquisition of Layered Facial Reflectance. ACM Transactions on Graphics 27, 5 (2008), 10 pages.
[34]
Yuliya Gitlina, Giuseppe Claudio Guarnera, Daljit Singh Dhillon, Jan Hansen, Alexander Lattas, Dinesh Pai, and Abhijeet Ghosh. 2020. Practical measurement and reconstruction of spectral skin reflectance. In Computer graphics forum, Vol. 39. Wiley Online Library, 75--89. Issue: 4.
[35]
Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2018. Practical Dynamic Facial Appearance Modeling and Acquisition. ACM Transactions on Graphics 37, 6 (2018), 13 pages.
[36]
Cong Phuoc Huynh, Antonio Robles-Kelly, and Edwin R. Hancock. 2013. Shape and Refractive Index from Single-View Spectro-Polarimetric Images. International journal of computer vision 101, 1 (2013), 64--94. Publisher: Springer.
[37]
Inseung Hwang, Daniel S. Jeon, Adolfo Muñoz, Diego Gutierrez, Xin Tong, and Min H. Kim. 2022. Sparse Ellipsometry: Portable Acquisition of Polarimetric SVBRDF and Shape with Unstructured Flash Photography. ACM Transactions on Graphics 41, 4 (2022), 14 pages.
[38]
Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015. Dynamic 3D Avatar Creation from Hand-Held Video Input. ACM Transactions on Graphics 34, 4 (2015), 14 pages.
[39]
Jose A. Iglesias-Guitian, Carlos Aliaga, Adrian Jarabo, and Diego Gutierrez. 2015. A Biophysically-Based Model of the Optical Properties of Skin Aging. Computer Graphics Forum (EUROGRAPHICS 2015) 34, 2 (2015).
[40]
Francisco H. Imai. 1998. Multi-spectral image acquisition and spectral reconstruction using a trichromatic digital camera system associated with absorption filters. Munsell Color Science Laboratory, Rochester Institute of Technology 2 (1998).
[41]
Steven L. Jacques. 1998. Skin Optics Summary. Oregon Medical Laser Center News (1998).
[42]
Jorge Jimenez, Timothy Scully, Nuno Barbosa, Craig Donner, Xenxo Alvarez, Teresa Vieira, Paul Matts, Verónica Orvalho, Diego Gutierrez, and Tim Weyrich. 2010. A Practical Appearance Model for Dynamic Facial Color. ACM Transactions on Graphics 29, 6 (2010), 10 pages.
[43]
Achuta Kadambi, Vage Taamazyan, Boxin Shi, and Ramesh Raskar. 2015. High-Quality Depth Sensing with Polarization Cues. In Proc. IEEE/CVF CVPR. 3370--3378.
[44]
Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. In Proc. Eurographics symposium on Geometry processing, Vol. 7. 0.
[45]
Aravind Krishnaswamy and Gladimir VG Baranoski. 2004. A biophysically-based spectral model of light interaction with human skin. In Computer graphics forum, Vol. 23. Wiley Online Library, 331--340.
[46]
Chloe LeGendre, Kalle Bladin, Bipin Kishore, Xinglei Ren, Xueming Yu, and Paul Debevec. 2018. Efficient Multispectral Facial Capture with Monochrome Cameras. In ACM SIGGRAPH 2018 Posters. Association for Computing Machinery, 2 pages.
[47]
Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, and Qifeng Chen. 2022. Shape from Polarization for Complex Scenes in the Wild. In Proc. IEEE/CVF CVPR. 12632--12641.
[48]
Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, and Jun Xing. 2020. Learning Formation of Physically-Based Face Attributes. In Proc. IEEE/CVF CVPR. 3410--3419.
[49]
Xiaohui Li, Giuseppe Claudio Guarnera, Arvin Lin, and Abhijeet Ghosh. 2024. Practical Measurement and Neural Encoding of Hyperspectral Skin Reflectance. In Proc. Int. Conf. on 3D Vision (3DV). 1301--1309.
[50]
Lahav Lipson, Zachary Teed, and Jia Deng. 2021. Raft-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching. In Proc. Int. Conf. on 3D Vision (3DV). 218--227.
[51]
Shichen Liu, Yunxuan Cai, Haiwei Chen, Yichao Zhou, and Yajie Zhao. 2022. Rapid Face Asset Acquisition with Recurrent Feature Alignment. ACM Transactions on Graphics 41, 6 (2022), 1--17.
[52]
Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul E. Debevec. 2007. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. Rendering Techniques 2007, 9 (2007).
[53]
Daisuke Miyazaki, Robby T. Tan, Kenji Hara, and Katsushi Ikeuchi. 2003. Polarization-based Inverse Rendering from a Single View. In Proc. IEEE ICCV, Vol. 3. 982--982.
[54]
Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbič, Hao Li, Abhijeet Ghosh, and Paul Debevec. 2015. Skin Microstructure Deformation with Displacement Map Convolution. ACM Transactions on Graphics 34, 4, Article 109 (2015), 10 pages.
[55]
Scott Prahl. 1999. Optical Absorption of Hemoglobin. http://omlc.ogi.edu/spectra/hemoglobin. Publisher: Oregon Medical Laser Center, Accessed: Sept. 08, 2024.
[56]
Stephen J. Preece and Ela Claridge. 2004. Spectral filter optimization for the recovery of parameters which describe human skin. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 7 (2004), 913--922.
[57]
Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2020. Single-Shot High-Quality Facial Geometry and Skin Appearance Capture. ACM Transactions on Graphics 39, 4 (2020), 12 pages.
[58]
Jérémy Riviere, Ilya Reshetouski, Luka Filipi, and Abhijeet Ghosh. 2017. Polarization imaging reflectometry in the wild. ACM Transactions on Graphics 36, 6 (2017), 1--14.
[59]
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018. SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild. In Proc. IEEE/CVF CVPR. 6296--6305.
[60]
Chunhu Shi, Mark Goodall, Jo Dumville, James Hill, Gill Norman, Oliver Hamer, Andrew Clegg, Caroline Leigh Watkins, George Georgiou, Alexander Hodkinson, Catherine Elizabeth Lightbody, Paul Dark, and Nicky Cullum. 2022. The accuracy of pulse oximetry in measuring oxygen saturation by levels of skin pigmentation: a systematic review and meta-analysis. BMC Medicine 20, 1 (Aug. 2022), 267.
[61]
Fuhao Shi, Hsiang-Tao Wu, Xin Tong, and Jinxiang Chai. 2014. Automatic Acquisition of High-Fidelity Facial Performances Using Monocular Videos. ACM Transactions on Graphics 33, 6 (2014), 13 pages.
[62]
Raju Shrestha and Jon Yngve Hardeberg. 2010. Multispectral image capture using two RGB cameras. In 2010 18th European Signal Processing Conference. IEEE, 1801--1805.
[63]
Silvia Tozza, William AP Smith, Dizhong Zhu, Ravi Ramamoorthi, and Edwin R. Hancock. 2017. Linear Differential Constraints for Photo-Polarimetric Height Estimation. In Proc. IEEE/CVF CVPR. 2279--2287.
[64]
Anh Tuan Tran, Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, and Gerard Medioni. 2018. Extreme 3D Face Reconstruction: Seeing Through Occlusions. In Proc. IEEE/CVF CVPR. 3935--3944.
[65]
Luan Tran, Feng Liu, and Xiaoming Liu. 2019. Towards High-Fidelity Nonlinear 3D Face Morphable Model. In Proc. IEEE/CVF CVPR. 1126--1135.
[66]
Luan Tran and Xiaoming Liu. 2019. On Learning 3D Face Morphable Model from In-the-wild Images. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 1 (2019), 157--171.
[67]
Norimichi Tsumura, Hideaki Haneishi, and Yoichi Miyake. 1999. Independent-component analysis of skin color image. JOSA A 16, 9 (1999), 2169--2176.
[68]
Norimichi Tsumura, Nobutoshi Ojima, Kayoko Sato, Mitsuhiro Shiraishi, Hideto Shimizu, Hirohide Nabeshima, Syuuichi Akazaki, Kimihiko Hori, and Yoichi Miyake. 2003. Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin. In ACM SIGGRAPH 2003 Papers. 770--779.
[69]
Levi Valgaerts, Chenglei Wu, Andrés Bruhn, Hans-Peter Seidel, and Christian Theobalt. 2012. Lightweight Binocular Facial Performance Capture under Uncontrolled Lighting. ACM Transactions on Graphics 31, 6 (2012), 11 pages.
[70]
Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. 2006. Analysis of Human Faces Using a Measurement-Based Skin Reflectance Model. ACM Transactions on Graphics 25, 3 (2006), 1013--1024.
[71]
Alexander Wilkie and Andrea Weidlich. 2012. Polarised Light in Computer Graphics. In SIGGRAPH Asia 2012 Courses. 87 pages.
[72]
Stephen J. Wright. 2015. Coordinate descent algorithms. Mathematical programming 151, 1 (2015), 3--34. Publisher: Springer.
[73]
Longwen Zhang, Chuxiao Zeng, Qixuan Zhang, Hongyang Lin, Ruixiang Cao, Wei Yang, Lan Xu, and Jingyi Yu. 2022. Video-driven Neural Physically-based Facial Asset for Production. ACM Transactions on Graphics 41, 6 (2022), 1--16.
[74]
Jinyu Zhao, Yusuke Monno, and Masatoshi Okutomi. 2022. Polarimetric Multi-View Inverse Rendering. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022). Publisher: IEEE.
[75]
Dizhong Zhu and William AP Smith. 2019. Depth From a Polarisation + RGB Stereo Pair. In Proc. IEEE/CVF CVPR. 7586--7595.
Index Terms
- Polarimetric BSSRDF Acquisition of Dynamic Faces
Recommendations
Simultaneous acquisition of polarimetric SVBRDF and normals
Capturing appearance often requires dense sampling in light-view space, which is often achieved in specialized, expensive hardware setups. With the aim of realizing a compact acquisition setup without multiple angular samples of light and view, we ...
A BSSRDF model for efficient rendering of fur with global illumination
Physically-based hair and fur rendering is crucial for visual realism. One of the key effects is global illumination, involving light bouncing between different fibers. This is very time-consuming to simulate with methods like path tracing. Efficient ...
Practical dynamic facial appearance modeling and acquisition
We present a method to acquire dynamic properties of facial skin appearance, including dynamic diffuse albedo encoding blood flow, dynamic specular intensity, and per-frame high resolution normal maps for a facial performance sequence. The method ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 November 2024
Published in TOG Volume 43, Issue 6
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Korea NRF
- MSIT/IITP of Korea
- Microsoft Research Asia
- Lig
- Samsung Electronics
- Gobierno de Aragon predoctoral grant
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 7Total Downloads
- Downloads (Last 12 months)7
- Downloads (Last 6 weeks)7
Reflects downloads up to 19 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in