PVLI: potentially visible layered image for real-time ray tracing
Abstract
Novel view synthesis is frequently employed in video streaming, temporal upsampling, or virtual reality. We propose a new representation, potentially visible layered image (PVLI), that uses a combination of a potentially visible set of the scene geometry and layered color images. PVLI encodes the depth implicitly and enables cheap run-time reconstruction. Furthermore, PVLI can also be used to reconstruct pixel and layer connectivities, which is crucial for filtering and post-processing of the rendered images. We use PVLIs to achieve local and server-based real-time ray tracing. In the first case, PVLIs are used as a basis for temporal and spatial upsampling of ray-traced illumination. In the second case, PVLIs are compressed, streamed over the network, and then used by a thin client to perform temporal and spatial upsampling and to hide latency. To shade the view, we use path tracing, accounting for effects such as soft shadows, global illumination, and physically based refraction. Our method supports dynamic lighting, and up to a limited extent, it also handles view-dependent surface interactions.
References
[1]
Andersson P, Nilsson J, Akenine-Möller T, Oskarsson M, Åström K, and Fairchild MD Flip: a difference evaluator for alternating images Proc. ACM Comput. Graph. Interact. Technol. 2020 3 2 1-15
[2]
Chang, C.F., Bishop, G., Lastra, A.: Ldi tree: a hierarchical representation for image-based rendering. In: Proceedings of SIGGRAPH’99, pp. 291–298 (1999)
[3]
Chen, S.E., Williams, L.: View interpolation for image synthesis. In: Proceedings of SIGGRAPH’93, pp. 279–288 (1993)
[4]
Cohen-Or D, Chrysanthou Y, Silva C, and Durand F A survey of visibility for walkthrough applications IEEE Trans. Vis. Comput. Graph. 2003 9 3 412-431
[5]
Didyk P, Eisemann E, Ritschel T, Myszkowski K, and Seidel HP Perceptually-motivated real-time temporal upsampling of 3D content for high-refresh-rate displays Comput. Graph. Forum 2010 29 2 713-722
[6]
Didyk, P., Ritschel, T., Eisemann, E., Myszkowski, K., Seidel, H.P.: Adaptive image-space stereo view synthesis. In: Vision, Modeling and Visualization Workshop, pp. 299–306. Siegen, Germany (2010)
[7]
Gribble, C.: Multi-hit ray tracing in DXR. In: Ray Tracing Gems, chap. 9. Apress (2019)
[8]
Hladky J, Seidel HP, and Steinberger M The camera offset space: real-time potentially visible set computations for streaming rendering ACM Trans. Graph. 2019 38 6 66
[9]
Hladky J, Seidel HP, and Steinberger M Tessellated shading streaming Comput. Graph. Forum 2019 38 4 171-182
[10]
Hladky, J., Seidel, H.P., Steinberger, M.: Snakebinning: efficient temporally coherent triangle packing for shading streaming. In: Computer Graphics Forum, vol. 40, pp. 475–488. Wiley Online Library (2021)
[11]
Huang JB, Kang SB, Ahuja N, and Kopf J Temporally coherent completion of dynamic video ACM Trans. Graph. 2016 35 6 1-11
[12]
Işık M, Mullia K, Fisher M, Eisenmann J, and Gharbi M Interactive Monte Carlo denoising using affinity of neural features ACM Trans. Graph. 2021 40 4 1-13
[13]
Koch T and Wimmer M Guided visibility sampling++ Proc. ACM Comput. Graph. Interact. Technol. 2021 4 1 66
[14]
Kondapaneni I, Vévoda P, Grittmann P, Skřivan T, Slusallek P, and Křivánek J Optimal multiple importance sampling ACM Trans. Graph. 2019 38 4 1-14
[15]
Koskela M, Immonen K, Mäkitalo M, Foi A, Viitanen T, Jääskeläinen P, Kultala H, and Takala J Blockwise multi-order feature regression for real-time path-tracing reconstruction ACM Trans. Graph. 2019 38 5 1-14
[16]
Luebke, D., Reddy, M., Cohen, J.D., Varshney, A., Watson, B., Huebner, R.: Level of Detail for 3D Graphics. Morgan Kaufmann (2003)
[17]
Mara, M., McGuire, M., Luebke, D.: Lighting deep g-buffers: single-pass, layered depth images with minimum separation applied to indirect illumination. Tech. rep, NVIDIA (2013)
[18]
Mark, W.R., McMillan, L., Bishop, G.: Post-rendering 3d warping. In: Proceedings of the 1997 Symposium on Interactive 3D Graphics, p. 7-ff (1997)
[19]
Muddala SM, Sjöström M, and Olsson R Virtual view synthesis using layered depth image generation and depth-based inpainting for filling disocclusions and translucent disocclusions J. Vis. Commun. Image Represent. 2016 38 351-366
[20]
Mueller JH, Neff T, Voglreiter P, Steinberger M, and Schmalstieg D Temporally adaptive shading reuse for real-time rendering and virtual reality ACM Trans. Graph. 2021 40 2 66
[21]
Mueller JH, Voglreiter P, Dokter M, Neff T, Makar M, Steinberger M, and Schmalstieg D Shading atlas streaming ACM Trans. Graph. 2018 37 6 66
[22]
Müller, T., Gross, M., Novák, J.: Practical path guiding for efficient light-transport simulation. In: Computer Graphics Forum, vol. 36, pp. 91–100 (2017)
[23]
Noimark Y and Cohen-Or D Streaming scenes to mpeg-4 video-enabled devices IEEE Comput. Graph. Appl. 2003 23 1 58-64
[24]
OculusVR: Rendering to the oculus rift. https://developer.oculus.com/documentation/pcsdk/latest/concepts/dg-render/ (2020). Accessed 15 Jan 2020
[25]
Ouyang, Y., Liu, S., Kettunen, M., Pharr, M., Pantaleoni, J.: Restir gi: path resampling for real-time path tracing. In: Computer Graphics Forum, vol. 40, pp. 17–29 (2021)
[26]
Pajak, D., Herzog, R., Eisemann, E., Myszkowski, K., Seidel, H.P.: Scalable remote rendering with depth and motion-flow augmented streaming. In: Computer Graphics Forum, vol. 30, pp. 415–424 (2011)
[27]
Park SY and Kim SD Efficient depth compression based on partial surface for 3-d object represented by layered depth image IEEE Signal Process. Lett. 2010 17 10 839-842
[28]
Reinert B, Kopf J, Ritschel T, Cuervo E, Chu D, and Seidel HP Proxy-guided image-based rendering for mobile devices Comput. Graph. Forum 2016 6 66
[29]
Samet H The quadtree and related hierarchical data structures ACM Comput. Surv. 1984 66 2
[30]
Schied, C., Kaplanyan, A., Wyman, C., Patney, A., Chaitanya, C.R.A., Burgess, J., Liu, S., Dachsbacher, C., Lefohn, A., Salvi, M.: Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination. In: Proceedings of HPG (2017)
[31]
Schied C, Peters C, and Dachsbacher C Gradient estimation for real-time adaptive temporal filtering Proc. ACM Comput. Graph. Interact. Tech. 2018 1 2 1-16
[32]
Shade, J., Gortler, S., He, L.w., Szeliski, R.: Layered depth images. In: Proceedings of SIGGRAPH’98, pp. 231–242 (1998)
[33]
Shi S and Hsu CH A survey of interactive remote rendering systems ACM Comput. Surv. 2015 47 4 66
[34]
Shi S, Nahrstedt K, and Campbell R A real-time remote rendering system for interactive mobile graphics ACM Trans. Multimed. Comput. Commun. Appl. 2012 8 3s 66
[35]
Strengert, M., Kraus, M., Ertl, T.: Pyramid methods in gpu-based image processing. In: Proceedings Vision, Modeling, and Visualization, vol. 2006, pp. 169–176 (2006)
[36]
Tauber Z, Li ZN, and Drew MS Review and preview: disocclusion by inpainting for image-based rendering IEEE Trans. Syst. Man Cybernet. Part C Appl. Rev. 2007 37 4 527-540
[37]
Walter, B., Drettakis, G., Parker, S.: Interactive rendering using the render cache. In: Eurographics Workshop on Rendering Techniques, pp. 19–30. Springer (1999)
[38]
Wang Z, Bovik A, Sheikh H, and Simoncelli E Image quality assessment: from error visibility to structural similarity IEEE Trans. Image Process. 2004 13 4 600-612
[39]
Wonka P, Wimmer M, Zhou K, Maierhofer S, Hesina G, and Reshetov A Guided visibility sampling ACM Trans. Graph. 2006 25 3 494-502
[40]
Yang, L., Tse, Y.C., Sander, P.V., Lawrence, J., Nehab, D., Hoppe, H., Wilkins, C.L.: Image-based bidirectional scene reprojection. In: Proceedings of the 2011 SIGGRAPH Asia Conference, pp. 1–10 (2011)
[41]
Zheng S, Zheng F, Xu K, and Yan LQ Ensemble denoising for Monte Carlo renderings ACM Trans. Graph. 2021 40 6 1-17
[42]
Zitnick CL, Kang SB, Uyttendaele M, Winder S, and Szeliski R High-quality video view interpolation using a layered representation ACM Trans. Graph. 2004 23 3 600-608
Recommendations
Real-Time Volume Caustics with Image-Based Photon Tracing
CADGRAPHICS '11: Proceedings of the 2011 12th International Conference on Computer-Aided Design and Computer GraphicsRendering of volume caustics in participating media is often expensive, even with different acceleration approaches. Basic volume photon tracing is used to render such effects, but rather slow due to its massive quantity of photons to be traced. In this ...
Ray tracing-based interactive diffuse indirect illumination
Despite great efforts in recent years to accelerate global illumination computation, the real-time ray tracing of fully dynamic scenes to support photorealistic indirect illumination effects has yet to be achieved in computer graphics. In this paper, we ...
Distributed ray tracing
Ray tracing is one of the most elegant techniques in computer graphics. Many phenomena that are difficult or impossible with other techniques are simple with ray tracing, including shadows, reflections, and refracted light. Ray directions, however, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 28 July 2023
Accepted: 12 June 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
- Grantová Agentura České Republiky
- Research Center for Informatics, Czech Technical University in Prague
- Grant Agency of the Czech Technical University in Prague
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in