research-article

Shading atlas streaming

Authors:

Joerg H. Mueller,

Philip Voglreiter,

Markus Steinberger,

Dieter SchmalstiegAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 6

Article No.: 199, Pages 1 - 16

https://doi.org/10.1145/3272127.3275087

Published: 04 December 2018 Publication History

Abstract

Streaming high quality rendering for virtual reality applications requires minimizing perceived latency. We introduce Shading Atlas Streaming (SAS), a novel object-space rendering framework suitable for streaming virtual reality content. SAS decouples server-side shading from client-side rendering, allowing the client to perform framerate upsampling and latency compensation autonomously for short periods of time. The shading information created by the server in object space is temporally coherent and can be efficiently compressed using standard MPEG encoding. Our results show that SAS compares favorably to previous methods for remote image-based rendering in terms of image quality and network bandwidth efficiency. SAS allows highly efficient parallel allocation in a virtualized-texture-like memory hierarchy, solving a common efficiency problem of object-space shading. With SAS, untethered virtual reality headsets can benefit from high quality rendering without paying in increased latency.

Supplementary Material

ZIP File (a199-mueller.zip)

Supplemental files.

Download
603.02 KB

MP4 File (a199-mueller.mp4)

Download
397.91 MB

References

[1]

John M. Airey, John H. Rohlf, and Frederick P. Brooks, Jr. 1990. Towards Image Realism with Interactive Update Rates in Complex Virtual Building Environments. In Proceedings ACM I3D. 41--50.

Digital Library

[2]

Tomas Akenine-Möller and Timo Aila. 2005. Conservative and Tiled Rasterization Using a Modified Triangle Set-Up. Journal of Graphics Tools 10, 3 (Jan 2005), 1--8.

[3]

Magnus Andersson, Jon Hasselgren, Robert Toth, and Tomas Akenine-Möiler. 2014. Adaptive texture space shading for stochastic rendering. Computer Graphics Forum 33, 2 (may 2014), 341--350.

Digital Library

[4]

Dan Baker. 2016. Object Space Lighting. Talk at Game Developers Conference. (2016). http://www.cogsci.rpi.edu/~destem/gamearch/gdc16/Object-Space-Lighting-Rev-21.pptx

[5]

Paul Bao and Douglas Gourlay. 2004. Remote walkthrough over mobile networks using 3-D image warping and streaming. IEE Proceedings - Vision, Image and Signal Processing 151, 4 (Aug 2004), 329--336.

[6]

Louis Bavoil and Miguel Sainz. 2009. Multi-layer dual-resolution screen-space ambient occlusion. In SIGGRAPH 2009: Talks. ACM, 45.

Digital Library

[7]

Kevin Boos, David Chu, and Eduardo Cuervo. 2016. FlashBack: Immersive Virtual Reality on Mobile Devices via Rendering Memoization. In Proc. MobiSys. 291--304.

Digital Library

[8]

Mario Botsch, Andreas Wiratanaya, and Leif Kobbelt. 2002. Efficient High Quality Rendering of Point Sampled Geometry. In Proceedings of the 13th Eurographics Workshop on Rendering. 53--64.

Digital Library

[9]

Huw Bowles, Kenny Mitchell, Robert W. Sumner, Jeremy Moore, and Markus Gross. 2012. Iterative Image Warping. Computer Graphics Forum 31, 2pt1 (2012), 237--246.

[10]

Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured Lumigraph Rendering. In Proceedings SIGGRAPH. 425--432.

Digital Library

[11]

Christopher A. Burns, Kayvon Fatahalian, and William R. Mark. 2010. A Lazy Object-space Shading Architecture with Decoupled Sampling. In Proceedings HPG. 19--28.

Digital Library

[12]

Nathan A. Carr and John C. Hart. 2002. Meshed atlases for real-time procedural solid texturing. ACM Transactions on Graphics 21, 2 (apr 2002), 106--131.

Digital Library

[13]

Cem Cebenoyan. 2014. Real Virtual Texturing Taking Advantage of DirectX11.2 Tiled Resources. Game Developer Conference. (2014).

[14]

Matthäus Chadjas, Christian Eisenacher, Marc Stamminger, and Sylvain Lefebvre. 2010. Virtual Texture Mapping 101. (2010).

[15]

Chun-Fa Chang and Shyh-Haur Ger. 2002. Enhancing 3D Graphics on Mobile Devices by Image-Based Rendering. Springer Berlin Heidelberg, Berlin, Heidelberg, 1105--1111.

Digital Library

[16]

Ka Chen. 2015. Adaptive Virtual Texture Rendering in Far Cry 4. Talk at Game Developers Conference. (March 2015). http://twvideo01.ubm-us.net/o1/vault/gdc2015/presentations/Chen_Ka_AdaptiveVirtualTexture.pdf

[17]

Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei. 2011. Measuring the Latency of Cloud Gaming Systems. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). 1269--1272.

Digital Library

[18]

Shenchang Eric Chen and Lance Williams. 1993. View interpolation for image synthesis. In Proceedings SIGGRAPH. 279--288.

Digital Library

[19]

Sharon. Choy, Bernard. Wong, Gwendal. Simon, and Catherine. Rosenberg. 2012. The brewing storm in cloud gaming: A measurement study on cloud to end-user latency. In Workshop on Network and Systems Support for Games (NetGames). 1--6.

Digital Library

[20]

Petrik Clarberg, Robert Toth, Jon Hasselgren, Jim Nilsson, and Tomas Akenine-Möller. 2014. AMFS: Adaptive Multi-Frequency Shading for Future Graphics Processors. ACM Transactions on Graphics 33, 4 (jul 2014), 1--12.

Digital Library

[21]

Petrik Clarberg, Robert Toth, and Jacob Munkberg. 2013. A sort-based deferred shading architecture for decoupled sampling. ACM Transactions on Graphics 32, 4 (jul 2013).

Digital Library

[22]

Daniel Cohen-Or, Yair Mann, and Shachar Fleishman. 1999. Deep Compression for Streaming Texture Intensive Animations. In Proceedings SIGGRAPH. 261--267.

Digital Library

[23]

Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-quality Streamable Free-viewpoint Video. ACM Transactions on Graphics 34, 4 (July 2015).

Digital Library

[24]

Robert L. Cook, Loren Carpenter, and Edwin Catmull. 1987. The Reyes Image Rendering Architecture. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '87). ACM, New York, NY, USA, 95--102.

Digital Library

[25]

Cyril Crassin, David Luebke, Michael Mara, Morgan McGuire, Brent Oster, Peter Shirley, Peter-Pike Sloan, and Chris Wyman. 2015. CloudLight: A System for Amortizing Indirect Lighting in Real-Time Rendering. Journal of Computer Graphics Techniques (JCGT) 4, 4 (15 October 2015), 1--27. http://jcgt.org/published/0004/04/01/

[26]

Eduardo Cuervo, Alec Wolmany, Landon P. Coxz, Kiron Lebeck, Ali Razeenz, Stefan Saroiuy, and Madanlal Musuvathi. 2015. Kahawai: High-Quality Mobile Gaming Using GPU Offload. In Proceedings MobiSys. 121--135.

Digital Library

[27]

Piotr Didyk, Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2010. Adaptive Image-space Stereo View Synthesis. In 15th International Workshop on Vision, Modeling and Visualization Workshop. Siegen, Germany, 299--306.

[28]

Karl. E. Hillesland and J. C. Yang. 2016. Texel Shading. In Proceedings of the 37th Annual Conference of the European Association for Computer Graphics: Short Papers. 73--76.

Digital Library

[29]

Martin Kraus and Thomas Ertl. 2002. Adaptive Texture Maps. In Proceedings ACM SIGGRAPH/EUROGRAPHICS Conference on Graphics Hardware. 7--15.

Digital Library

[30]

Kyungmin Lee, David Chu, Eduardo Cuervo, Johannes Kopf, Yury Degtyarev, Sergey Grizan, Alec Wolman, and Jason Flinn. 2015. Outatime - Using speculation to enable low-latency continuous interaction for mobile cloud gaming. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services.

Digital Library

[31]

Sylvain Lefebvre, Jerome Darbon, and Fabrice Neyret. 2004. Unified Texture Management for Arbitrary Meshes. INRIA Research report 5210. (2004).

[32]

Marc Levoy. 1995. Polygon-assisted JPEG and MPEG Compression of Synthetic Images. In Proceedings SIGGRAPH. 21--28.

Digital Library

[33]

Gábor Liktor and Carsten Dachsbacher. 2012. Decoupled deferred shading for hardware rasterization. In Proceedings ACM I3D. 143--150.

Digital Library

[34]

Yair Mann and Daniel Cohen-Or. 1997. Selective Pixel Transmission for Navigating in Remote Virtual Environments. Computer Graphics Forum 16 (1997), C201--C206.

[35]

Marc Manzano, José A. Hernandez, Manuel Uruena, and Eusebi Calle. 2012. An empirical study of Cloud Gaming. In 2012 11th Annual Workshop on Network and Systems Support for Games (NetGames). 1--2.

Digital Library

[36]

William R. Mark, Leonard McMillan, and Gary Bishop. 1997. Post-rendering 3D warping. In Proceedings of the 1997 Symposium on Interactive 3D Graphics.

Digital Library

[37]

Colt McAnlis. 2009. Halo Wars: The Terrain of Next-Gen. Talk at Game Developers Conference. (2009).

[38]

Martin Mittring. 2007. Finding next gen: Cryengine 2. In ACM SIGGRAPH 2007 courses. ACM, 97--121.

Digital Library

[39]

Martin Mittring. 2008. Advanced Virtual Texture Topics. In Advances in Real-Time Rendering in 3D Graphics and Games Course SIGGRAPH 2008. 23--51. http://portal.acm.org/citation.cfm?id=1404435.1404438

Digital Library

[40]

Yuval Noimark and Daniel Cohen-Or. 2003. Streaming Scenes to MPEG-4 Video-Enabled Devices. IEEE Computer Graphics and Applications 23 (01 2003), 58--64.

Digital Library

[41]

OculusVR. 2018. Rendering to the Oculus Rift. (2018). https://developer.oculus.com/documentation/pcsdk/latest/concepts/dg-render/ Visited on March 30, 2018.

[42]

Dawid Pajak, Robert Herzog, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2011. Scalable Remote Rendering with Depth and Motion-flow Augmented Streaming. Computer Graphics Forum 30, 2 (2011), 415--424.

[43]

Jonathan Ragan-Kelley, Jaakko Lehtinen, Jiawen Chen, Michael Doggett, and Frédo Durand. 2011. Decoupled sampling for graphics pipelines. ACM Transactions on Graphics 30, 3 (may 2011), 1--17.

Digital Library

[44]

Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, and Hans-Peter Seidel. 2016. Proxy-guided Image-based Rendering for Mobile Devices. Computer Graphics Forum 35, 7 (oct 2016), 353--362.

Digital Library

[45]

Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered Depth Images. In Proceedings SIGGRAPH. 231--242.

Digital Library

[46]

Bin Sheng, Wei-Liang Meng, Han-Qiu Sun, and En-Hua Wu. 2011. MCGIM-Based Model Streaming for Realtime Progressive Rendering. Journal of Computer Science and Technology 26, 1 (jan 2011), 166--175.

Digital Library

[47]

Shu Shi and Cheng-Hsin Hsu. 2015. A Survey of Interactive Remote Rendering Systems. ACM Comput. Surv. 47, 4, Article 57 (May 2015).

Digital Library

[48]

Shu Shi, Klara Nahrstedt, and Roy Campbell. 2012. A Real-time Remote Rendering System for Interactive Mobile Graphics. ACM Trans. Multimedia Comput. Commun. Appl. 8, 3s, Article 46 (Oct. 2012), 20 pages.

Digital Library

[49]

Markus Steinberger, Michael Kenzel, Bernhard Kainz, and Dieter Schmalstieg. 2012. ScatterAlloc: Massively parallel dynamic memory allocation for the GPU. In 2012 Innovative Parallel Computing (InPar). 1--10.

[50]

Christopher C. Tanner, Christopher J. Migdal, and Michael T. Jones. 1998. The Clipmap: A Virtual Mipmap. In Proceedings SIGGRAPH. 151--158.

Digital Library

[51]

Eyal Teler and Dani Lischinski. 2001. Streaming of Complex 3D Scenes for Remote Walkthroughs. Computer Graphics Forum 20, 3 (2001), 17--25.

[52]

J. M. P. van Waveren. 2009. id Tech 5 Challenges - From Texture Virtualization to Massive Parallelization. In SIGGRAPH 2009 Course: Beyond Programmable Shading.

[53]

Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (apr 2004), 600--612. Issue 4.

Digital Library

[54]

Lei Yang, Yu-Chiu Tse, Pedro V. Sander, Jason Lawrence, Diego Nehab, Hugues Hoppe, and Clara L. Wilkins. 2011. Image-based bidirectional scene reprojection. In Proceedings of the 2011 SIGGRAPH Asia Conference. Article 150.

Digital Library

[55]

Ilmi Yoon and Ulrich Neumann. 2000. Web-Based Remote Rendering with IBRAC (Image-Based Rendering Acceleration and Compression). Computer Graphics Forum 19, 3 (2000), 321--330.

[56]

Cem Yuksel. 2017. Mesh Color Textures. In Proceedings of High Performance Graphics (HPG '17). Article 17, 11 pages.

Digital Library

Cited By

Feng YLin WLiu ZLeng JGuo MZhao HHou XZhao JZhu Y(2024)Potamoi: Accelerating Neural Rendering via a Unified Streaming ArchitectureACM Transactions on Architecture and Code Optimization10.1145/368934021:4(1-25)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3689340
Coomans ADominci EDöring CMueller JHladky JSteinberger M(2024)Real‐time Neural Rendering of Dynamic Light FieldsComputer Graphics Forum10.1111/cgf.1501443:2Online publication date: 23-Apr-2024
https://doi.org/10.1111/cgf.15014
Jin TDasa MSmith CApicharttrisorn KSeshan SRowe A(2024)MeshReduce: Scalable and Bandwidth Efficient 3D Scene Capture2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00026(20-30)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00026
Show More Cited By

Index Terms

Shading atlas streaming
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
    2. Rendering

Recommendations

Temporally Adaptive Shading Reuse for Real-Time Rendering and Virtual Reality
Temporal coherence has the potential to enable a huge reduction of shading costs in rendering. Existing techniques focus either only on spatial shading reuse or cannot adaptively choose temporal shading frequencies. We find that temporal shading reuse is ...
Shading atlas streaming demonstration
SIGGRAPH '19: ACM SIGGRAPH 2019 Emerging Technologies

Streaming high quality rendering for virtual reality applications requires minimizing perceived latency. Shading Atlas Streaming (SAS) [Mueller et al. 2018] is a novel object-space rendering framework suitable for streaming virtual reality content. SAS ...
Lighting sensitive display

Although display devices have been used for decades, they have functioned without taking into account the illumination of their environment. We present the concept of a lighting sensitive display (LSD)---a display that measures the incident illumination ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 6

December 2018

1401 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3272127

Editor:
Takeo Igarashi
The University of Tokyo, Japan

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2018

Published in TOG Volume 37, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
740
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)8

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Feng YLin WLiu ZLeng JGuo MZhao HHou XZhao JZhu Y(2024)Potamoi: Accelerating Neural Rendering via a Unified Streaming ArchitectureACM Transactions on Architecture and Code Optimization10.1145/368934021:4(1-25)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3689340
Coomans ADominci EDöring CMueller JHladky JSteinberger M(2024)Real‐time Neural Rendering of Dynamic Light FieldsComputer Graphics Forum10.1111/cgf.1501443:2Online publication date: 23-Apr-2024
https://doi.org/10.1111/cgf.15014
Jin TDasa MSmith CApicharttrisorn KSeshan SRowe A(2024)MeshReduce: Scalable and Bandwidth Efficient 3D Scene Capture2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00026(20-30)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00026
Piochowiak MDachsbacher C(2024)Fast Compressed Segmentation Volumes for Scientific VisualizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332657330:1(12-22)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3326573
Feng YLiu ZLeng JGuo MZhu Y(2024)Cicero: Addressing Algorithmic and Architectural Bottlenecks in Neural Rendering by Radiance Warping and Memory Optimizations2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00096(1293-1308)Online publication date: 29-Jun-2024
https://doi.org/10.1109/ISCA59077.2024.00096
Lim JTreder MChadha AAndreopoulos Y(2024)Adaptive Render-Video Streaming for Virtual Environments2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00583(5742-5751)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00583
Geringer SGeiselhart FBäuerle ADec DOdenthal OReina GRopinski TWeiskopf D(2024)mint: Integrating scientific visualizations into virtual realityJournal of Visualization10.1007/s12650-024-01011-y27:6(1143-1169)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s12650-024-01011-y
Voglreiter PKerbl BWeinrauch AMueller JNeff TSteinberger MSchmalstieg D(2023)Trim Regions for Online Computation of From-Region Potentially Visible SetsACM Transactions on Graphics10.1145/359243442:4(1-15)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592434
Weinrauch ATatzgern WStadlbauer PCrickx AHladky JCoomans AWinter MMueller JSteinberger M(2023)Effect-based Multi-viewer Caching for Cloud-native RenderingACM Transactions on Graphics10.1145/359243142:4(1-16)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592431
Cantory VRingo N(2023)An Image-Space Split-Rendering Approach to Accelerate Low-Powered Virtual Reality2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00289(893-894)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00289
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents