research-article

Full 3D reconstruction of transparent objects

Authors:

Hui HuangAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 4

Article No.: 103, Pages 1 - 11

https://doi.org/10.1145/3197517.3201286

Published: 30 July 2018 Publication History

Abstract

Numerous techniques have been proposed for reconstructing 3D models for opaque objects in past decades. However, none of them can be directly applied to transparent objects. This paper presents a fully automatic approach for reconstructing complete 3D shapes of transparent objects. Through positioning an object on a turntable, its silhouettes and light refraction paths under different viewing directions are captured. Then, starting from an initial rough model generated from space carving, our algorithm progressively optimizes the model under three constraints: surface and refraction normal consistency, surface projection and silhouette consistency, and surface smoothness. Experimental results on both synthetic and real objects demonstrate that our method can successfully recover the complex shapes of transparent objects and faithfully reproduce their light refraction properties.

Supplementary Material

MP4 File (a103-wu.mp4)

Download
261.28 MB

References

[1]

Kfir Aberman, Oren Katzir, Qiang Zhou, Zegang Luo, Andrei Sharf, Chen Greif, Baoquan Chen, and Daniel Cohen-Or. 2017. Dip Transform for 3D Shape Reconstruction. ACM Trans. on Graphics (Proc. of SIGGRAPH) 36, 4 (2017), 79:1--79:11.

Digital Library

[2]

Bradley Atcheson, Ivo Ihrke, Wolfgang Heidrich, Art Tevs, Derek Bradley, Marcus Magnor, and Hans-Peter Seidel. 2008. Time-resolved 3D Capture of Non-stationary Gas Flows. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 27, 5 (2008), 132:1--132:9.

Digital Library

[3]

Moshe Ben-Ezra and Shree K. Nayar. 2003. What Does Motion Reveal About Transparency? Proc. Int. Conf. on Computer Vision (2003), 1025--1032.

Digital Library

[4]

Matthew Berger, Joshua A. Levine, Luis Gustavo Nonato, Gabriel Taubin, and Claudio T. Silva. 2013. A Benchmark for Surface Reconstruction. ACM Trans. on Graphics 32, 2 (2013), 20:1--20:17.

Digital Library

[5]

Matthew Berger, Andrea Tagliasacchi, Lee M. Seversky, Pierre Alliez, Joshua A. Levine, Andrei Sharf, and Claudio Silva. 2014. State of the Art in Surface Reconstruction from Point Clouds. Eurographics STAR (2014), 165--185.

[6]

Max Born and Emil Wolf. 2013. Principles of optics: electromagnetic theory of propagation, interference and diffraction of light. Elsevier.

[7]

Tongbo Chen, Hendrik PA Lensch, Christian Fuchs, and Hans-Peter Seidel. 2007. Polarization and Phase-Shifting for 3D Scanning of Translucent Objects. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2007), 1--8.

[8]

Yung-Yu Chuang, Douglas E. Zongker, Joel Hindorff, Brian Curless, David H. Salesin, and Richard Szeliski. 2000. Environment Matting Extensions: Towards Higher Accuracy and Real-time Capture. ACM Trans. on Graphics (Proc. of SIGGRAPH) (2000), 121--130.

Digital Library

[9]

Massimiliano Corsini, Paolo Cignoni, and Roberto Scopigno. 2012. Efficient and Flexible Sampling with Blue Noise Properties of Triangular Meshes. IEEE Trans. Visualization & Computer Graphics 18, 6 (2012), 914--924.

Digital Library

[10]

Zhaopeng Cui, Jinwei Gu, Boxin Shi, Ping Tan, and Jan Kautz. 2017. Polarimetric Multi-View Stereo. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2017), 1558--1567.

[11]

Qi Duan, Jianfei Cai, and Jianmin Zheng. 2015. Compressive Environment Matting. The Visual Computer 31, 12 (2015), 1587--1600.

Digital Library

[12]

Silvano Galliani, Katrin Lasinger, and Konrad Schindler. 2015. Massively Parallel Multiview Stereopsis by Surface Normal Diffusion. Proc. Int. Conf. on Computer Vision (2015), 873--881.

Digital Library

[13]

James Gregson, Michael Krimerman, Matthias B. Hullin, and Wolfgang Heidrich. 2012. Stochastic Tomography and Its Applications in 3D Imaging of Mixing Fluids. ACM Trans. on Graphics (Proc. of SIGGRAPH) 31, 4 (2012), 52:1--52:10.

Digital Library

[14]

Hui Huang, Dan Li, Hao Zhang, Uri Ascher, and Daniel Cohen-Or. 2009. Consolidation of Unorganized Point Clouds for Surface Reconstruction. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 28, 5 (2009), 176:1--176:7.

Digital Library

[15]

Hui Huang, Shihao Wu, Minglun Gong, Daniel Cohen-Or, Uri Ascher, and Hao Zhang. 2013. Edge-aware Point Set Resampling. ACM Trans. on Graphics 32, 1 (2013), 9:1--9:12.

Digital Library

[16]

Matthias B. Hullin, Martin Fuchs, Ivo Ihrke, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2008. Fluorescent Immersion Range Scanning. ACM Trans. on Graphics (Proc. of SIGGRAPH) 27, 3 (2008), 87:1--87:10.

Digital Library

[17]

Cong Phuoc Huynh, Antonio Robles-Kelly, and Edwin Hancock. 2010. Shape and refractive index recovery from single-view polarisation images. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2010), 1229--1236.

[18]

Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik Lensch, Marcus Magnor, and Wolfgang Heidrich. 2010. Transparent and specular object reconstruction. Computer Graphics Forum 29, 8 (2010), 2400--2426.

[19]

Ivo Ihrke and Marcus Magnor. 2004. Image-based Tomographic Reconstruction of Flames. Proc. Eurographics Symp. on Computer Animation (2004), 365--373.

Digital Library

[20]

Julian Iseringhausen, Bastian Goldlücke, Nina Pesheva, Stanimir Iliev, Alexander Wender, Martin Fuchs, and Matthias B. Hullin. 2017. 4D Imaging Through Spray-on Optics. ACM Trans. on Graphics (Proc. of SIGGRAPH) 36, 4 (2017), 35:1--35:11.

Digital Library

[21]

Yu Ji, Jinwei Ye, and Jingyi Yu. 2013. Reconstructing gas flows using light-path approximation. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2013), 2507--2514.

Digital Library

[22]

Michael Kazhdan and Hugues Hoppe. 2013. Screened Poisson Surface Reconstruction. ACM Trans. on Graphics 32, 3 (2013), 29:1--29:13.

Digital Library

[23]

Jaewon Kim, Ilya Reshetouski, and Abhijeet Ghosh. 2017. Acquiring Axially-Symmetric Transparent Objects Using Single-View Transmission Imaging. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2017), 1484--1492.

[24]

Kiriakos N. Kutulakos and Steven M. Seitz. 2000. A Theory of Shape by Space Carving. Int. J. Computer Vision 38, 3 (2000), 199--218.

Digital Library

[25]

Kiriakos N. Kutulakos and Eron Steger. 2008. A theory of refractive and specular 3D shape by light-path triangulation. Int. J. Computer Vision 76, 1 (2008), 13--29.

Digital Library

[26]

Aviad Levis, Yoav Y Schechner, Amit Aides, and Anthony B Davis. 2015. Airborne three-dimensional cloud tomography. Proc. Int. Conf. on Computer Vision (2015), 3379--3387.

Digital Library

[27]

Aviad Levis, Yoav Y Schechner, and Anthony B Davis. 2017. Multiple-scattering microphysics tomography. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2017), 5797--5806.

[28]

Yaron Lipman, Daniel Cohen-Or, David Levin, and Hillel Tal-Ezer. 2007. Parameterization-free Projection for Geometry Reconstruction. ACM Trans. on Graphics (Proc. of SIGGRAPH) 26, 3 (2007), 22:1--22:6.

Digital Library

[29]

Daisuke Miyazaki and Katsushi Ikeuchi. 2005. Inverse polarization raytracing: estimating surface shapes of transparent objects. Proc. IEEE Conf. on Computer Vision & Pattern Recognition 2 (2005), 910--917.

Digital Library

[30]

Nigel JW Morris and Kiriakos N. Kutulakos. 2007. Reconstructing the surface of inhomogeneous transparent scenes by scatter-trace photography. Proc. Int. Conf. on Computer Vision (2007), 1--8.

[31]

Nigel JW Morris and Kiriakos N. Kutulakos. 2011. Dynamic refraction stereo. IEEE Trans. Pattern Analysis & Machine Intelligence 33, 8 (2011), 1518--1531.

Digital Library

[32]

Pieter Peers and Philip Dutré. 2003. Wavelet environment matting. Proc. Eurographics Workshop on Rendering (2003), 157--166.

Digital Library

[33]

Yiming Qian, Minglun Gong, and Yee Hong Yang. 2016. 3D Reconstruction of Transparent Objects with Position-Normal Consistency. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2016), 4369--4377.

[34]

Yiming Qian, Minglun Gong, and Yee-Hong Yang. 2015. Frequency-based environment matting by compressive sensing. Proc. Int. Conf. on Computer Vision (2015), 3532--3540.

Digital Library

[35]

Yiming Qian, Minglun Gong, and Yee-Hong Yang. 2017. Stereo-Based 3D Reconstruction of Dynamic Fluid Surfaces by Global Optimization. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2017), 6650--6659.

[36]

Yuliy Schwartzburg, Romain Testuz, Andrea Tagliasacchi, and Mark Pauly. 2014. High-contrast Computational Caustic Design. ACM Trans. on Graphics (Proc. of SIGGRAPH) 33, 4 (2014), 74:1--74:11.

Digital Library

[37]

Qi Shan, Sameer Agarwal, and Brian Curless. 2012. Refractive height fields from single and multiple images. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2012), 286--293.

Digital Library

[38]

Kenichiro Tanaka, Yasuhiro Mukaigawa, Hiroyuki Kubo, Yasuyuki Matsushita, and Yasushi Yagi. 2016. Recovering Transparent Shape from Time-of-Flight Distortion. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2016), 4387--4395.

[39]

Borislav Trifonov, Derek Bradley, and Wolfgang Heidrich. 2006. Tomographic reconstruction of transparent objects. Proc. Eurographics Conf. on Rendering Techniques (2006), 51--60.

Digital Library

[40]

Chia-Yin Tsai, Ashok Veeraraghavan, and Aswin C Sankaranarayanan. 2015. What does a single light-ray reveal about a transparent object? Proc. IEEE Int. Conf. on Image Processing (2015), 606--610.

Digital Library

[41]

Gordon Wetzstein, David Roodnick, Wolfgang Heidrich, and Ramesh Raskar. 2011. Refractive shape from light field distortion. Proc. Int. Conf. on Computer Vision (2011), 1180--1186.

Digital Library

[42]

Yonatan Wexler, Andrew. W. Fitzgibbon, and Andrew. Zisserman. 2002. Image-based Environment Matting. Proc. Eurographics Workshop on Rendering (2002), 279--290.

Digital Library

[43]

Shihao Wu, Hui Huang, Minglun Gong, Matthias Zwicker, and Daniel Cohen-Or. 2015a. Deep Points Consolidation. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 34, 6 (2015), 176:1--176:13.

Digital Library

[44]

Shihao Wu, Wei Sun, Pinxin Long, Hui Huang, Daniel Cohen-Or, Minglun Gong, Oliver Deussen, and Baoquan Chen. 2014. Quality-driven Poisson-guided Autoscanning. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 33, 6 (2014), 203:1--203:12.

Digital Library

[45]

Zhaohui Wu, Zhong Zhou, Delei Tian, and Wei Wu. 2015b. Reconstruction of Three-dimensional Flame with Color Temperature. The Visual Computer 31, 5 (2015), 613--625.

Digital Library

[46]

Sai-Kit Yeung, Tai-Pang Wu, Chi-Keung Tang, Tony F Chan, and Stanley Osher. 2011. Adequate reconstruction of transparent objects on a shoestring budget. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2011), 2513--2520.

Digital Library

[47]

Yonghao Yue, Kei Iwasaki, Bing-Yu Chen, Yoshinori Dobashi, and Tomoyuki Nishita. 2014. Poisson-Based Continuous Surface Generation for Goal-Based Caustics. ACM Trans. on Graphics 33, 3 (2014), 31:1--31:7.

Digital Library

[48]

Mingjie Zhang, Xing Lin, Mohit Gupta, Jinli Suo, and Qionghai Dai. 2014. Recovering Scene Geometry under Wavy Fluid via Distortion and Defocus Analysis. Proc. Euro. Conf. on Computer Vision (2014), 234--250.

[49]

Zhengyou Zhang. 2000. A Flexible New Technique for Camera Calibration. IEEE Trans. Pattern Analysis & Machine Intelligence 22, 11 (2000), 1330--1334.

Digital Library

[50]

Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. 1997. Algorithm 778: L-BFGS-B: Fortran Subroutines for Large-scale Bound-constrained Optimization. ACM Trans. Mathematical Software 23, 4 (1997), 550--560.

Digital Library

[51]

Douglas E. Zongker, Dawn M. Werner, Brian Curless, and David H. Salesin. 1999. Environment Matting and Compositing. ACM Trans. on Graphics (Proc. of SIGGRAPH) (1999), 205--214.

Digital Library

[52]

Xinxin Zuo, Chao Du, Sen Wang, Jiangbin Zheng, and Ruigang Yang. 2015. Interactive visual hull refinement for specular and transparent object surface reconstruction. Proc. Int. Conf. on Computer Vision (2015), 2237--2245.

Digital Library

Cited By

Iwaguchi TKubo HKawasaki H(2024)Specular Object Reconstruction Behind Frosted Glass by Differentiable Rendering2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00400(4035-4044)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00400
Deng WCampbell DSun CKanitkar SShaffer MGould S(2024)Ray Deformation Networks for Novel View Synthesis of Refractive Objects2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00309(3106-3116)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00309
Shao MXia CDuan DWang X(2024)Polarimetric Inverse Rendering for Transparent Shapes ReconstructionIEEE Transactions on Multimedia10.1109/TMM.2024.337179226(7801-7811)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3371792
Show More Cited By

Index Terms

Full 3D reconstruction of transparent objects
1. Computing methodologies
  1. Computer graphics
    1. Shape modeling
      1. Point-based models

Recommendations

Differentiable refraction-tracing for mesh reconstruction of transparent objects

Capturing the 3D geometry of transparent objects is a challenging task, ill-suited for general-purpose scanning and reconstruction techniques, since these cannot handle specular light transport phenomena. Existing state-of-the-art methods, designed ...
Matting and compositing of transparent and refractive objects

This article introduces a new approach for matting and compositing transparent and refractive objects in photographs. The key to our work is an image-based matting model, termed the Attenuation-Refraction Matte (ARM), that encodes plausible refractive ...
3D Reconstruction by Shadow Carving: Theory and Practical Evaluation

Cast shadows are an informative cue to the shape of objects. They are particularly valuable for discovering object's concavities which are not available from other cues such as occluding boundaries. We propose a new method for recovering shape from ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 4

August 2018

1670 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3197517

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2018

Published in TOG Volume 37, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
789
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)6

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Iwaguchi TKubo HKawasaki H(2024)Specular Object Reconstruction Behind Frosted Glass by Differentiable Rendering2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00400(4035-4044)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00400
Deng WCampbell DSun CKanitkar SShaffer MGould S(2024)Ray Deformation Networks for Novel View Synthesis of Refractive Objects2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00309(3106-3116)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00309
Shao MXia CDuan DWang X(2024)Polarimetric Inverse Rendering for Transparent Shapes ReconstructionIEEE Transactions on Multimedia10.1109/TMM.2024.337179226(7801-7811)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3371792
Yang CFang CLi ZWang QZhang X(2024)An Improved SFS Method for Achieving Fast, High-Precision, and Widely Adaptable 3-D ReconstructionIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.337075573(1-9)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3370755
Jiang JCao GDeng JDo TLuo S(2024)Robotic Perception of Transparent Objects: A ReviewIEEE Transactions on Artificial Intelligence10.1109/TAI.2023.33261205:6(2547-2567)Online publication date: Jun-2024
https://doi.org/10.1109/TAI.2023.3326120
Tang TLiu JZhang JFu HXu WLu C(2024)RFTrans: Leveraging Refractive Flow of Transparent Objects for Surface Normal Estimation and ManipulationIEEE Robotics and Automation Letters10.1109/LRA.2024.33648379:4(3735-3742)Online publication date: Apr-2024
https://doi.org/10.1109/LRA.2024.3364837
Hodan TSundermeyer MLabbé YNguyen VWang GBrachmann EDrost BLepetit VRother CMatas J(2024)BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00570(5610-5619)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00570
Deng WCampbell DSun CKanitkar SShaffer MGould S(2024)Differentiable Neural Surface Refinement for Modeling Transparent Objects2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01916(20268-20277)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01916
Song YHuang GYin JWang D(2024)Three-dimensional reconstruction of bubble geometry from single-perspective images based on ray tracing algorithmMeasurement Science and Technology10.1088/1361-6501/ad7e4336:1(016010)Online publication date: 24-Oct-2024
https://doi.org/10.1088/1361-6501/ad7e43
Remondino FKarami AYan ZMazzacca GRigon SQin R(2023)A Critical Analysis of NeRF-Based 3D ReconstructionRemote Sensing10.3390/rs1514358515:14(3585)Online publication date: 18-Jul-2023
https://doi.org/10.3390/rs15143585
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents