research-article

Virtual Reality Telepresence: 360-Degree Video Streaming with Edge-Compute Assisted Static Foveated Compression

Authors:

Xincheng Huang,

Robert XiaoAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 29, Issue 11

Pages 4525 - 4534

https://doi.org/10.1109/TVCG.2023.3320255

Published: 03 October 2023 Publication History

Abstract

Real-time communication with immersive 360° video can enable users to be telepresent within a remotely streamed environment. Increasingly, users are shifting to mobile devices and connecting to the Internet via mobile-cellular networks. As the ideal media for 360° videos, some VR headsets now also come with cellular capacity, giving them potential for mobile applications. However, streaming high-quality 360° live video poses challenges for network bandwidth, particularly on cellular connections. To reduce bandwidth requirements, videos can be compressed using viewport-adaptive streaming or foveated rendering techniques. Such approaches require very low latency in order to be effective, which has previously limited their applications on traditional cellular networks. In this work, we demonstrate an end-to-end virtual reality telepresence system that streams ∼6K 360° video over 5G millimeter-wave (mmW) radio. Our use of 5G technologies, in conjunction with mobile edge compute nodes, substantially reduces latency when compared with existing 4G networks, enabling high-efficiency foveated compression over modern cellular networks on par with WiFi. We performed a technical evaluation of our system's visual quality post-compression with peak signal-to-noise ratio (PSNR) and FOVVideoVDP. We also conducted a user study to evaluate users' sensitivity to compressed video. Our findings demonstrate that our system achieves visually indistinguishable video streams while using up to 80% less data when compared with un-foveated video. We demonstrate our video compression system in the context of an immersive, telepresent video calling application.

References

[1]

Essential concepts. [Online]. Available: https://developer.tobii.com/xr/learn/foveation/rendering/essential-concepts/. Accessed: 2023-6-14. 1, 2.

[2]

Fisheye projection. [Online]. Available: https://wiki.panotools.org/Fisheye_Projection. Accessed: 2023-6-14. 4.

[3]

Qualcomm snapdragon XR2 5G platform. [Online]. Available: https://www.qualcomm.com/products/mobile/snapdragon/xr-vr-ar/snapdragon-xr2-5g-platform. Accessed: 2023-3-24. 1.

[4]

R. Albert, A. Patney, D. Luebke, and J. Kim. Latency requirements for foveated rendering in virtual reality. ACM Trans. Appl. Percept., 14 (4): pp. 1–13, Sept. 2017. 1, 2, 9.

Digital Library

[5]

D. V. G. Alcabaza, Polytechnic University of the Philippines - Santa Rosa Campus, M. E. Legaspi, T. L. Muyot, K. L. C. Ofren, J. A. D. Panganiban, and R. E. Tolentino. Real-time realistic telepresence using a 360 degree camera and a virtual reality box. Int. j. inf. technol. comput. sci., 11 (3): pp. 46–52, Mar. 2019. 1, 2.

[6]

M. Antonov. Asynchronous timewarp examined. [Online]. Available: https://developer.oculus.com/blog/asynchronous-timewarp-examined/. Accessed: 2023-6-17. 9.

[7]

J. Chakareski, M. Khan, T. Ropitault, and S. Blandino. 6DOF virtual reality dataset and performance evaluation of millimeter wave vs. Free-Space-Optical indoor communications systems for lifelike mobile VR streaming. In 2020 54th Asilomar Conference on Signals, Systems, and Computers, pp. 1051–1058. ieeexplore.ieee.org, Nov. 2020. 2.

[8]

K. Doppler, E. Torkildson, and J. Bouwen. On wireless networks for the era of mixed reality. In 2017 European Conference on Networks and Communications (EuCNC), pp. 1–5, June 2017. 2.

[9]

M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler. Edge computing meets millimeter-wave enabled VR: Paving the way to cutting the cord, 2018. 1, 2.

[10]

M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler. Toward Low-Latency and Ultra-Reliable virtual reality. IEEE Netw., 32 (2): pp. 78–84, Mar. 2018. 2.

[11]

X. Feng, V. Swaminathan, and S. Wei. Viewport prediction for live 360-degree mobile video streaming using user-content hybrid motion tracking. in Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 3 (2), jun 2019. 2.

Digital Library

[12]

W. S. Geisler and J. S. Perry. Real-time foveated multiresolution system for low-bandwidth video communication. In B. E. Rogowitz and T. N. Pappas, eds., Human Vision and Electronic Imaging III. SPIE, July 1998. 1, 2.

[13]

B. Guenter, M. Finch, S. Drucker, D. Tan, and J. Snyder. Foveated 3D graphics. ACM Trans. Graph., 31 (6): pp. 1–10, Nov. 2012. 1, 2.

Digital Library

[14]

A. Gupta and R. K. Jha. A survey of 5g network: Architecture and emerging technologies. IEEE Access, 3: pp. 1206–1232, 2015. 2.

[15]

S. Gupta, J. Chakareski, and P. Popovski. Millimeter wave meets edge computing for mobile vr with high-fidelity 8k scalable 360 video. In 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP), pp. 1–6, 2019. 2.

[16]

M. Hosseini. View-aware tile-based adaptations in 360 virtual reality video streaming. In 2017 IEEE Virtual Reality (VR), pp. 423–424, Mar. 2017. 2.

[17]

M. Hosseini and V. Swaminathan. Adaptive 360 VR video streaming: Divide and conquer. In 2016 IEEE International Symposium on Multimedia (ISM), pp. 107–110, Dec. 2016. 2.

[18]

G. Illahi, T. Van Gemert, M. Siekkinen, E. Masala, A. Oulasvirta, and A. Ylä-Jääski. Cloud gaming with foveated graphics, 2018. 1, 2, 9.

[19]

A. S. Kaplanyan, A. Sochenov, T. Leimkühler, M. Okunev, T. Goodall, and G. Rufo. DeepFovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Trans. Graph., 38 (6): pp. 1–13, Nov. 2019. 1, 2, 9.

Digital Library

[20]

S. Kasahara and J. Rekimoto. JackIn head: immersive visual telepresence system with omnidirectional wearable camera for remote collaboration. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology, VRST '15, pp. 217–225. Association for Computing Machinery, New York, NY, USA, Nov. 2015. 2.

[21]

M. A. Khan, E. Baccour, Z. Chkirbene, A. Erbad, R. Hamila, M. Hamdi, and M. Gabbouj. A survey on mobile edge computing for video streaming: Opportunities and challenges. IEEE Access, 10: pp. 120514–120550, 2022. 2.

[22]

H. Kim, J. Yang, J. Lee, S. Yoon, Y. Kim, M. Choi, J. Yang, E.-S. Ryu, and W. Park. Eye Tracking-Based 360 vr Foveated/Tiled video rendering. In 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1–1, July 2018. 2.

[23]

H.-W. Kim, T. T. Le, and E.-S. Ryu. 360-degree video off loading using millimeter-wave communication for cyberphysical system. Trans. emerg. telecommun. technol., 30 (4): p. e3506, Apr. 2019. 1, 2.

Digital Library

[24]

J. Kim, Y. Jeong, M. Stengel, K. Akşit, R. Albert, B. Boudaoud, T. Greer, J. Kim, W. Lopes, Z. Majercik, P. Shirley, J. Spjut, M. McGuire, and D. Luebke. Foveated ar: Dynamically-foveated augmented reality display. ACM Trans. Graph., 38 (4), jul 2019. 1, 2.

Digital Library

[25]

T.-T. Le, D. N. Van, and E.-S. Ryu. Real-time 360-degree video streaming over millimeter wave communication. In 2018 International Conference on Information Networking (ICOIN), pp. 857–862, Jan. 2018. 1, 2.

[26]

T. T. Le, D. Van Nguyen, and E.-S. Ryu. Computing offloading over mmwave for mobile VR: Make 360 video streaming alive. IEEE Access, 6: pp. 66576–66589, 2018. 2.

[27]

D. Li, R. Du, A. Babu, C. D. Brumar, and A. Varshney. A Log-Rectilinear transformation for foveated 360-degree video streaming. IEEE Trans. Vis. Comput. Graph., 27 (5): pp. 2638–2647, May 2021. 1, 2, 5, 9.

[28]

Z. Li, T. Teo, L. Chan, G. Lee, M. Adcock, M. Billinghurst, and H. Koike. OmniGlobe Vr: A collaborative 360-degree communication system for V R. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, DIS '20, pp. 615–625. Association for Computing Machinery, New York, NY, USA, July 2020. 1, 2.

[29]

X. Liu, Q. Xiao, V. Gopalakrishnan, B. Han, F. Qian, and M. Varvello. 360° innovations for panoramic video streaming. In Proceedings of the 16th ACM Workshop on Hot Topics in Networks, HotNets-XVI, pp. 50–56. Association for Computing Machinery, New York, NY, USA, Nov. 2017. 1,2.

[30]

Y. Liu, J. Liu, A. Argyriou, and S. Ci. MEC-Assisted panoramic VR video streaming over millimeter wave mobile networks, 2019. 2.

[31]

P. Lyu and H. Hua. Design of a statically foveated display based on a perceptual-driven approach. Opt. Express, 31 (2): pp. 2088–2101, Jan 2023. 1, 2.

[32]

R. K. Mantiuk, G. Denes, A. Chapiro, A. Kaplanyan, G. Rufo, R. Bachy, T. Lian, and A. Patney. Fovvideovdp: A visible difference predictor for wide field-of-view video. ACM Trans. Graph., 40 (4), jul 2021. 5, 6.

Digital Library

[33]

X. Meng, R. Du, M. Zwicker, and A. Varshney. Kernel foveated rendering. in Proc. ACM Comput. Graph. Interact. Tech., 1 (1): pp. 1–20, July 2018. 2.

[34]

A. T. Nasrabadi, A. Mahzari, J. D. Beshay, and R. Prakash. Adaptive 360-degree video streaming using layered video coding. In 2017 IEEE Virtual Reality (VR), pp. 347–348, Mar. 2017. 2.

[35]

D. V. Nguyen, H. T. T. Tran, and T. C. Thang. An evaluation of tile selection methods for Viewport-Adaptive streaming of 360-degree video. ACM Trans. Multimedia Comput. Commun. Appl., 16 (1): pp. 1–24, Mar. 2020. 9.

Digital Library

[36]

J. Orlosky, K. Kiyokawa, and H. Takemura. Virtual and augmented reality on the 5G highway. Journal of Information Processing, 25: pp. 133–141, 2017. 2.

[37]

J. Park and K. Nahrstedt. Navigation graph for tiled media streaming. In Proceedings of the 27th ACM International Conference on Multimedia, MM '19, pp. 447–455. Association for Computing Machinery, New York, NY, USA, Oct. 2019. 2.

[38]

T. Piumsomboon, G. A. Lee, A. Irlitti, B. Ens, B. H. Thomas, and M. Billinghurst. On the shoulder of the giant: A Multi-Scale mixed reality collaboration with 360 video sharing and tangible interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–17. Association for Computing Machinery, New York, NY, USA, May 2019. 1, 2.

[39]

F. Qian, B. Han, Q. Xiao, and V. Gopalakrishnan. Flare: Practical Viewport-Adaptive 360-degree video streaming for mobile devices. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, MobiCom '18, pp. 99–114. Association for Computing Machinery, New York, NY, USA, Oct. 2018. 1, 2, 9.

[40]

F. Qian, L. Ji, B. Han, and V. Gopalakrishnan. Optimizing 360 video delivery over cellular networks. In Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges, ATC '16, pp. 1–6. Association for Computing Machinery, New York, NY, USA, Oct. 2016. 2, 9.

[41]

J. Ryoo, K. Yun, D. Samaras, S. R. Das, and G. Zelinsky. Design and evaluation of a foveated video streaming service for commodity client devices. In Proceedings of the 7th International Conference on Multimedia Systems, number Article 6 in MMSys '16, pp. 1–11. Association for Computing Machinery, New York, NY, USA, May 2016. 1, 2.

[42]

M. Y. Saraiji, K. Minamizawa, and S. Tachi. Foveated streaming: Optimizing video streaming for telexistence systems using eye-gaze based foveation. Virtual Reality Society of Japan, Sept. 2017. 1, 2.

[43]

N. Stein, D. C. Niehorster, T. Watson, F. Steinicke, K. Rifai, S. Wahl, and M. Lappe. A comparison of eye tracking latencies among several commercial head-mounted displays. i-Perception, 12 (1): p. 2041669520983338, 2021. 33628410. 1, 2.

[44]

M. K. Stern and J. H. Johnson. Just Noticeable Difference, pp. 1–2. John Wiley & Sons, Ltd, 2010. 6.

[45]

L. Sun, F. Duanmu, Y. Liu, Y. Wang, Y. Ye, H. Shi, and D. Dai. Multi-path multi-tier 360-degree video streaming in 5G networks. In Proceedings of the 9th ACM Multimedia Systems Conference, MMSys '18, pp. 162–173. Association for Computing Machinery, New York, NY, USA, June 2018. 2.

[46]

A. Tang, O. Fakourfar, C. Neustaedter, and S. Bateman. Collaboration with 360 videochat: Challenges and opportunities. In Proceedings of the 2017 Conference on Designing Interactive Systems, DIS'17, pp. 1327–1339. Association for Computing Machinery, New York, NY, USA, June 2017. 1, 2.

[47]

T. Teo, L. Lawrence, G. A. Lee, M. Billinghurst, and M. Adcock. Mixed reality remote collaboration combining 360 video and 3D reconstruction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–14. Association for Computing Machinery, New York, NY, USA, May 2019. 1, 2.

[48]

E. Turner, H. Jiang, D. Saint-Macary, and B. Bastani. Phase-Aligned foveated rendering for virtual reality headsets. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 1–2, Mar. 2018. 2.

[49]

O. Wiedemann, V. Hosu, H. Lin, and D. Saupe. Foveated video coding for Real-Time streaming applications. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6, May 2020. 1, 2.

[50]

S.-C. Yen, C.-L. Fan, and C.-H. Hsu. Streaming 360° videos to head-mounted virtual reality using DASH over QUIC transport protocol. In Proceedings of the 24th ACM Workshop on Packet Video, PV '19, pp. 7–12. Association for Computing Machinery, New York, NY, USA, June 2019. 1, 2.

[51]

X. Yu, F. Xu, J. Cai, X.-Y. Dang, and K. Wang. Computation efficiency optimization for Millimeter-Wave mobile edge computing networks with NOMA, 2022. 2.

Cited By

Huang XYin MXia ZXiao R(2024)VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual ReplicasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676377(1-12)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676377
Huang XXiao R(2024)SurfShareProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314187:4(1-24)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631418

Recommendations

Towards Viewport-dependent 6DoF 360 Video Tiled Streaming for Virtual Reality Systems
MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Previous studies of 360-degree video streaming with regard to virtual reality allowed users to move their head freely, while their position is fixed according to the camera's location in virtual reality. One of the approaches to overcome the problem is ...
Implementing 360 video tiled streaming system
MMSys '18: Proceedings of the 9th ACM Multimedia Systems Conference

The computing power and bandwidth of the current VR are limited when compared to the high-quality VR. To overcome these limits, this study proposes a new viewport dependent streaming method that transmits 360-degree videos using the high efficiency ...
Foveated streaming of virtual reality videos
MMSys '18: Proceedings of the 9th ACM Multimedia Systems Conference

While Virtual Reality (VR) represents a revolution in the user experience, current VR systems are flawed on different aspects. The difficulty to focus naturally in current headsets incurs visual discomfort and cognitive overload, while high-end headsets ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 29, Issue 11

Nov. 2023

465 pages

ISSN:1077-2626

Issue’s Table of Contents

1077-2626 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 03 October 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang XYin MXia ZXiao R(2024)VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual ReplicasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676377(1-12)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676377
Huang XXiao R(2024)SurfShareProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314187:4(1-24)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631418

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents