research-article

Deep Surface Light Fields

Authors:

Yingliang Zhang,

Jingyi YuAuthors Info & Claims

Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 1, Issue 1

Article No.: 14, Pages 1 - 17

https://doi.org/10.1145/3203192

Published: 25 July 2018 Publication History

Abstract

A surface light field represents the radiance of rays originating from any points on the surface in any directions. Traditional approaches require ultra-dense sampling to ensure the rendering quality. In this paper, we present a novel neural network based technique called deep surface light field or DSLF to use only moderate sampling for high fidelity rendering. DSLF automatically fills in the missing data by leveraging different sampling patterns across the vertices and at the same time eliminates redundancies due to the network's prediction capability. For real data, we address the image registration problem as well as conduct texture-aware remeshing for aligning texture edges with vertices to avoid blurring. Comprehensive experiments show that DSLF can further achieve high data compression ratio while facilitating real-time rendering on the GPU.

References

[1]

Chris Buehler, Michael Bosse, Leonard Mcmillan, Steven Gortler, and Michael Cohen. 2001. Unstructured lumigraph rendering. In Conference on Computer Graphics and Interactive Techniques. 425--432.

Digital Library

[2]

Wei-Chao Chen, Jean-Yves Bouguet, Michael H Chu, and Radek Grzeszczuk. 2002. Light field mapping: efficient representation and hardware rendering of surface light fields. ACM Transactions on Graphics (TOG) 21, 3 (2002), 447--456.

Digital Library

[3]

Paul Debevec, Christoph Bregler, Michael Cohen, and Leonard Mcmillan. 1998. Image-based modeling and rendering. 299.

[4]

Martin A. Fischler and Robert C. Bolles. 1987. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Readings in Computer Vision (1987), 726--740.

Digital Library

[5]

Xiao Shan Gao, Xiao Rong Hou, Jianliang Tang, and Hang Fei Cheng. 2003. Complete solution classification for the perspective-three-point problem. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 8 (2003), 930--943.

Digital Library

[6]

M.W Gardner and S.R Dorling. 1998. Artificial neural networks (the multilayer perceptron)-a review of applications in the atmospheric sciences. Atmospheric Environment 32, 14-15 (1998), 2627--2636.

[7]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. (2015), 770--778.

[8]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[9]

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators. Neural networks 2, 5 (1989), 359--366.

Digital Library

[10]

Liang Hu, Pedro V. Sander, and Hugues Hoppe. 2010. Parallel View-Dependent Level-of-Detail Control. IEEE Transactions on Visualization and Computer Graphics 16, 5 (2010), 718--728.

Digital Library

[11]

Tomasz Jannson. 1980. Radiance transfer function. Journal of the Optical Society of America 70, 12 (1980), 1544--1549.

[12]

Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[13]

Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics 22, 1 (1951), 79--86.

[14]

Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. 2009. EPnP: An Accurate O(n) Solution to the PnP Problem. International Journal of Computer Vision 81, 2 (2009), 155--166.

Digital Library

[15]

Marc Levoy. 1996. Light field rendering. In Conference on Computer Graphics and Interactive Techniques. 31--42.

Digital Library

[16]

M. Lounsbery, T. Derose, and J. Warren. 1997. Multersolution surfaces of arbitrary topological type. (1997).

[17]

Yi Ma, Stefano Soatto, Jana KoEck, and S. Shankar Sastry. 2004. An invitation to 3-D vision:. Springer,. 526 pages.

[18]

Marcus Magnor and Bernd Girod. 1999. Adaptive Block-based Light Field Coding. In PROC. 3RD INTERNATIONAL WORKSHOP ON SYNTHETIC AND NATURAL HYBRID CODING AND THREE-DIMENSIONAL IMAGING IWSNHC3DI;99, SANTORINI, GREECE. 140--143.

[19]

Leonard Mcmillan and Gary Bishop. 1995. Plenoptic modeling: an image-based rendering system. In Conference on Computer Graphics and Interactive Techniques. 39--46.

Digital Library

[20]

Ehsan Miandji, Joel Kronander, and Jonas Unger. 2013. Learning based compression of surface light fields for real-time rendering of global illumination scenes. In SIGGRAPH Asia 2013 Technical Briefs. ACM, 24.

Digital Library

[21]

Gavin Miller, Steven Rubin, and Dulce Ponceleon. 1998. Lazy decompression of surface light fields for precomputed global illumination. In Rendering Techniques '98, Proceedings of the Eurographics Workshop in Vienna, Austria, June 29-July. 281--292.

[22]

Jorge J More. 1977. The Levenberg-Marquardt algorithm: Implementation and theory. Lecture Notes in Mathematics 630 (1977), 105--116.

[23]

Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In International Conference on International Conference on Machine Learning. 807--814.

Digital Library

[24]

Damien Porquet, Jean-Michel Dischler, and Djamchid Ghazanfarpour. 2005. Real-time high-quality view-dependent texture mapping using per-pixel visibility. In Proceedings of the 3rd international conference on Computer graphics and interactive techniques in Australasia and South East Asia. ACM, 213--220.

Digital Library

[25]

Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).

[26]

Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. 2015. Image based relighting using neural networks. ACM Transactions on Graphics (TOG) 34, 4 (2015), 111.

Digital Library

[27]

Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global illumination with radiance regression functions. ACM Transactions on Graphics (TOG) 32, 4 (2013), 130.

Digital Library

[28]

Johannes L. Schonberger, Enliang Zheng, Jan Michael Frahm, and Marc Pollefeys. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In European Conference on Computer Vision. 501--518.

[29]

Michael Van den Bergh, Xavier Boix, Gemma Roig, Benjamin de Capitani, and Luc Van Gool. 2012. SEEDS: Superpixels extracted via energy-driven sampling. In European conference on computer vision. Springer, 13--26.

Digital Library

[30]

Daniel N Wood, Daniel I Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle. 2000. Surface light fields for 3D photography. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 287--296.

Digital Library

Cited By

Li ZMa YZhou JZhou P(2024)Spiking-NeRF: Spiking Neural Network for Energy-Efficient Neural RenderingACM Journal on Emerging Technologies in Computing Systems10.1145/367580820:3(1-23)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3675808
Reiser CGarbin SSrinivasan PVerbin DSzeliski RMildenhall BBarron JHedman PGeiger A(2024)Binary Opacity Grids: Capturing Fine Geometric Detail for Mesh-Based View SynthesisACM Transactions on Graphics10.1145/365813043:4(1-14)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658130
Kim MGu GChoo J(2024)LensNeRF: Rethinking Volume Rendering based on Thin-Lens Camera Model2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00315(3170-3179)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00315
Show More Cited By

Index Terms

Deep Surface Light Fields
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Opacity light fields: interactive rendering of surface light fields with view-dependent opacity
I3D '03: Proceedings of the 2003 symposium on Interactive 3D graphics

We present new hardware-accelerated techniques for rendering surface light fields with opacity hulls that allow for interactive visualization of objects that have complex reflectance properties and elaborate geometrical details. The opacity hull is a ...
Rendering for an interactive 360° light field display
SIGGRAPH '07: ACM SIGGRAPH 2007 papers

We describe a set of rendering techniques for an autostereoscopic light field display able to present interactive 3D graphics to multiple simultaneous viewers 360 degrees around the display. The display consists of a high-speed video projector, a ...
Surface light fields for 3D photography
SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques

A surface light field is a function that assigns a color to each ray originating on a surface. Surface light fields are well suited to constructing virtual images of shiny objects under complex lighting conditions. This paper presents a framework for ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques

Proceedings of the ACM on Computer Graphics and Interactive Techniques Volume 1, Issue 1

July 2018

378 pages

EISSN:2577-6193

DOI:10.1145/3242771

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2018

Accepted: 01 June 2018

Revised: 01 March 2018

Received: 01 November 2017

Published in PACMCGIT Volume 1, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Science and Technology Commission of Shanghai Municipality

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
539
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)7

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li ZMa YZhou JZhou P(2024)Spiking-NeRF: Spiking Neural Network for Energy-Efficient Neural RenderingACM Journal on Emerging Technologies in Computing Systems10.1145/367580820:3(1-23)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3675808
Reiser CGarbin SSrinivasan PVerbin DSzeliski RMildenhall BBarron JHedman PGeiger A(2024)Binary Opacity Grids: Capturing Fine Geometric Detail for Mesh-Based View SynthesisACM Transactions on Graphics10.1145/365813043:4(1-14)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658130
Kim MGu GChoo J(2024)LensNeRF: Rethinking Volume Rendering based on Thin-Lens Camera Model2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00315(3170-3179)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00315
Chen HZhu PLiu YChu XLiu S(2024)Multi-View 3D Reconstruction of Vessel Hull with Spatial Density Weighted NeRFOCEANS 2024 - Singapore10.1109/OCEANS51537.2024.10682398(1-7)Online publication date: 15-Apr-2024
https://doi.org/10.1109/OCEANS51537.2024.10682398
You ZGeiger AChen A(2024)NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01875(19833-19843)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01875
Wang LYao KGuo CZhang ZHu QYu JXu LWu M(2024)VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00052(470-481)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00052
Xu KLi JLi ZCao Y(2024)SG-NeRF: Sparse-Input Generalized Neural Radiance Fields for Novel View SynthesisJournal of Computer Science and Technology10.1007/s11390-024-4157-639:4(785-797)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1007/s11390-024-4157-6
Bako SSen PKaplanyan A(2023)Deep Appearance PrefilteringACM Transactions on Graphics10.1145/357032742:2(1-23)Online publication date: 16-Jan-2023
https://dl.acm.org/doi/10.1145/3570327
Ju YShi BChen YZhou HDong JLam K(2023)GR-PSN: Learning to Estimate Surface Normal and Reconstruct Photometric Stereo ImagesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332981730:9(6192-6207)Online publication date: 3-Nov-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3329817
Wang GZhang JZhang KHuang RFang L(2023)GiganticNVS: Gigapixel Large-Scale Neural Rendering With Implicit Meta-Deformed ManifoldIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.332306946:1(338-353)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3323069
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents