Using 2.5D Sketches for 3D Point Cloud Reconstruction from A Single Image
Authors: 
Dongyi Yao, Fengqi Li, Yi Wang, Hong Yang, Xiuyun LiAuthors Info & Claims
Pages 92 - 98
Abstract
3D reconstruction from a single image is a highly uncertain problem. Unknown information such as and depth of the self-occluded part of an object requires strong prior knowledge of the object. In order to make better use of prior information and generate high-quality 3D model, we first propose a 3D point cloud generator to obtain prior knowledge of the 3D point cloud in datasets. We then design a model to recover 3D shape of the object from a 2D image, which first estimates a 2.5D sketch from the input 2D image and then transfer the knowledge obtained in the 3D point cloud domain to the 2D image domain through the 2.5D sketch. Because 2.5D sketches are easier to obtain from 2D images and the information in a 2.5D sketch is more abundant than that in a 2D image Experiments demonstrate that our method is highly competitive to state-of-art works in 3D point cloud reconstruction on both synthetic datasets and real datasets.
References
[1]
Harry Barrow, J Tenenbaum, A Hanson, and E Riseman. 1978. Recovering intrinsic scene characteristics. Comput. Vis. Syst 2, 3-26 (1978), 2.
[2]
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, 2015. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012(2015).
[3]
Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 2016. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision. Springer, 628–644.
[4]
Haoqiang Fan, Hao Su, and Leonidas J Guibas. 2017. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition. 605–613.
[5]
Daniele Ferdani, Bruno Fanini, Maria Claudia Piccioli, Fabiana Carboni, and Paolo Vigliarolo. 2020. 3D reconstruction and validation of historical background for immersive VR applications and games: The case study of the Forum of Augustus in Rome. Journal of Cultural Heritage(2020).
[6]
JunYoung Gwak, Christopher B Choy, Manmohan Chandraker, Animesh Garg, and Silvio Savarese. 2017. Weakly supervised 3d reconstruction with adversarial constraint. In 2017 International Conference on 3D Vision (3DV). IEEE, 263–272.
[7]
Antonios Kargas, Georgios Loumos, and Dimitrios Varoutas. 2019. Using different ways of 3D reconstruction of historical cities for gaming purposes: The case study of Nafplio. Heritage 2, 3 (2019), 1799–1811.
[8]
Aldo Laurentini. 2002. The visual hull concept for silhouette-based image understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence 16, 2(2002), 150–162.
[9]
Bin Li, Yonghan Zhang, Bo Zhao, and Hongyao Shao. 2020. 3D-ReConstnet: A Single-View 3D-Object Point Cloud Reconstruction Network. IEEE Access 8(2020), 83782–83790.
[10]
Shubao Liu and David B. Cooper. 2010. Ray Markov Random Fields for image-based 3D modeling: Model and efficient inference. In Computer Vision and Pattern Recognition.
[11]
Priyanka Mandikal and R. Venkatesh Babu. 2019. Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).
[12]
Priyanka Mandikal, KL Navaneet, Mayank Agarwal, and R Venkatesh Babu. 2018. 3D-LMNet: Latent embedding matching for accurate and diverse 3D point cloud reconstruction from a single image. arXiv preprint arXiv:1807.07796(2018).
[13]
Priyanka Mandikal, K. L. Navaneet, and R. Venkatesh Babu. 2019. 3D-PSRNet: Part Segmented 3D Point Cloud Reconstruction from a Single Image: Munich, Germany, September 8-14, 2018, Proceedings, Part III.
[14]
Wojciech Matusik, Chris Buehler, Ramesh Raskar, Steven J Gortler, and Leonard McMillan. 2000. Image-based visual hulls. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 369–374.
[15]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[16]
Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. 2018. Pix3d: Dataset and methods for single-image 3d shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2974–2983.
[17]
Yun-Chao Tang, Chenglin Wang, Lufeng Luo, Xiangjun Zou, 2020. Recognition and localization methods for vision-based fruit picking robots: a review. Frontiers in Plant Science 11 (2020), 510.
[18]
Marshall F Tappen, William T Freeman, and Edward H Adelson. 2005. Recovering intrinsic images from a single image. IEEE transactions on pattern analysis and machine intelligence 27, 9(2005), 1459–1472.
[19]
Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. 2017. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2626–2634.
[20]
MD Vision. 1982. A Computational Investigation into the Human Representation. Processings of visual Information, Freeman WH and Company. San Francisco (1982).
[21]
Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum. 2017. Marrnet: 3d shape reconstruction via 2.5 d sketches. In Advances in neural information processing systems. 540–550.
[22]
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information processing systems 29 (2016), 82–90.
[23]
Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum. 2018. Learning shape priors for single-view 3d completion and reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV). 646–662.
[24]
Bo Yang, Hongkai Wen, Sen Wang, Ronald Clark, Andrew Markham, and Niki Trigoni. 2017. 3d object reconstruction from a single depth view with adversarial learning. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 679–688.
[25]
Bo Yang, Hongkai Wen, Sen Wang, Ronald Clark, Andrew Markham, and Niki Trigoni. 2017. 3D Object Reconstruction from a Single Depth View with Adversarial Learning. (2017).
[26]
Wei Zeng, Sezer Karaoglu, and Theo Gevers. 2018. Inferring Point Clouds from Single Monocular Images by Depth Intermediation. (2018).
[27]
Jing Zhang, Ling Rui Gong, Keping Yu, Xin Qi, and San Hlaing Myint. 2020. 3D Reconstruction for Super-Resolution CT Images in the Internet of Health Things Using Deep Learning. IEEE Access PP, 99 (2020), 1–1.
Index Terms
- Using 2.5D Sketches for 3D Point Cloud Reconstruction from A Single Image
Index terms have been assigned to the content through auto-classification.
Recommendations
3D-ARNet: An accurate 3D point cloud reconstruction network from a single-image
AbstractGenerating a more realistic 3D reconstruction point cloud is an ill-posed problem. It is a challenging task to infer 3D shape from a single image. In this paper, a two-stage training network that can reconstruct point cloud from a single image is ...
3D-PSRNet: Part Segmented 3D Point Cloud Reconstruction from a Single Image
Computer Vision – ECCV 2018 WorkshopsAbstractWe propose a mechanism to reconstruct part annotated 3D point clouds of objects given just a single input image. We demonstrate that jointly training for both reconstruction and segmentation leads to improved performance in both the tasks, when ...
Tomographic surface reconstruction from point cloud
Inspired by computed tomography (CT), this paper presents a novel surface reconstruction algorithm, tomographic surface reconstruction, to reconstruct a surface mesh from a point cloud equipped with oriented normals. In the process of scanning a real ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2021
246 pages
ISBN:9781450388634
DOI:10.1145/3461353
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 September 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- National Key RD Program of China
- S&T Program of Hebei
Conference
ICIAI 2021
ICIAI 2021: 2021 the 5th International Conference on Innovation in Artificial Intelligence
March 5 - 8, 2021
Xia men, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 142Total Downloads
- Downloads (Last 12 months)21
- Downloads (Last 6 weeks)1
Reflects downloads up to 22 Sep 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format