Scene Graph Driven Hybrid Interactive VR Teleconferencing
Authors: 
Mingyuan Wu, Ruifan Ji, Haozhen Zheng, Jiaxi Li, Beitong Tian, Bo Chen, Ruixiao Zhang, Jacob Chakareski, Michael Zink, 
Ramesh Sitaraman, 
Klara NahrstedtAuthors Info & Claims
Pages 11276 - 11278
Abstract
We propose an interactive and intelligent hybrid teleconferencing system compatible with Virtual Reality devices. Our system understands meeting contexts and leverages user interactions to enhance better system configuration. Employing interactive scene graphs [11], the system extracts and transmits essential meeting context to users while relaying user interactions back to the streaming systems for user-involved adaptive streaming and foveated rendering. We demonstrate the system's real-time performance and compatibility with commercial VR devices such as the Meta Quest 3.
Supplemental Material
MKV File - Scene Graph Driven Hybrid Interactive VR Teleconferencing
An introduction and welcome video from our demo paper, Scene Graph Driven Hybrid Interactive VR Teleconferencing
- Download
- 9.06 MB
References
[1]
Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard Newcombe, and Vasileios Balntas. 2024. SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model. arxiv: 2403.13064 [cs.CV]
[2]
Bo Chen, Zhisheng Yan, Haiming Jin, and Klara Nahrstedt. 2019. Event-Driven Stitching for Tile-Based Live 360 Video Streaming. In Proceedings of the 10th ACM Multimedia Systems Conference. Association for Computing Machinery, 1--12.
[3]
Matthias De Fré, Jeroen van der Hooft, Tim Wauters, and Filip De Turck. 2024. Scalable MDC-Based Volumetric Video Delivery for Real-Time One-to-Many WebRTC Conferencing. In Proceedings of the 15th ACM Multimedia Systems Conference (, Bari, Italy,) (MMSys '24). Association for Computing Machinery, New York, NY, USA, 121--131. https://doi.org/10.1145/3625468.3647617
[4]
Simon N.B. Gunkel, Rick Hindriks, Yonatan Shiferaw, Sylvie Dijkstra-Soudarissanane, and Omar Niamut. 2024. VP9 bitstream-based Tiled Multipoint Control Unit: Scaling simultaneous RGBD user streams in an immersive 3D communication system. In Proceedings of the 15th ACM Multimedia Systems Conference (, Bari, Italy,) (MMSys '24). Association for Computing Machinery, New York, NY, USA, 23--33. https://doi.org/10.1145/3625468.3647608
[5]
Simon N. B. Gunkel, Rick Hindriks, Karim M. El Assal, Hans M. Stokking, Sylvie Dijkstra-Soudarissanane, Frank ter Haar, and Omar Niamut. 2021. VRComm: an end-to-end web system for real-time photorealistic social VR communication. In Proceedings of the 12th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys '21). Association for Computing Machinery, New York, NY, USA, 65--79. https://doi.org/10.1145/3458305.3459595
[6]
Anh Nguyen and Zhisheng Yan. 2023. Enhancing 360 Video Streaming through Salient Content in Head-Mounted Displays. Sensors, Vol. 23, 8 (2023).
[7]
Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST '16). Association for Computing Machinery, New York, NY, USA, 741--754. https://doi.org/10.1145/2984511.2984517
[8]
Jounsup Park, Mingyuan Wu, Kuan-Ying Lee, Bo Chen, Klara Nahrstedt, Michael Zink, and Ramesh Sitaraman. 2020. SEAWARE: Semantic Aware View Prediction System for 360-degree Video Streaming. In 2020 IEEE International Symposium on Multimedia (ISM). 57--64. https://doi.org/10.1109/ISM.2020.00016
[9]
Jangwoo Son and Eun-Seok Ryu. 2018. Tile-based 360-degree video streaming for mobile virtual reality in cyber physical system. Computers & Electrical Engineering, Vol. 72 (2018), 361--368.
[10]
Atsushi Tagami, Kazuaki Ueda, Rikisenia Lukita, Jacopo De Benedetto, Mayutan Arumaithurai, Giulio Rossi, Andrea Detti, and Toru Hasegawa. 2019. Tile-Based Panoramic Live Video Streaming on ICN. In 2019 IEEE International Conference on Communications Workshops (ICC Workshops). 1--6.
[11]
Mingyuan Wu, Yuhan Lu, Shiv Trivedi, Bo Chen, Qian Zhou, Lingdong Wang, Simran Singh, Michael Zink, Ramesh Sitaraman, Jacob Chakareski, and Klara Nahrstedt. 2023. Interactive Scene Graph Analysis for Future Intelligent Teleconferencing Systems. In 2023 IEEE International Symposium on Multimedia (ISM). 251--255. https://doi.org/10.1109/ISM59092.2023.00048
[12]
Lan Xie, Zhimin Xu, Yixuan Ban, Xinggong Zhang, and Zongming Guo. 2017. 360ProbDASH: Improving QoE of 360 Video Streaming Using Tile-Based HTTP Adaptive Streaming. In Proceedings of the 25th ACM International Conference on Multimedia. 315--323.
Index Terms
- Scene Graph Driven Hybrid Interactive VR Teleconferencing
Recommendations
DreamStream: Immersive and Interactive Spectating in VR
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing SystemsToday spectating and streaming virtual reality (VR) activities typically involves spectators viewing a 2D stream of the VR user’s view. Streaming 2D videos of the game play is popular and well-supported by platforms such as Twitch. However, the generic ...
VR Grabbers: Ungrounded Haptic Retargeting for Precision Grabbing Tools
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyHaptic feedback in VR is important for realistic simulation in virtual reality. However, recreating the haptic experience for hand tools in VR traditionally requires hardware with precise actuators, adding complexity to the system. We propose Ungrounded ...
Measurement of exceptional motion in VR video contents for VR sickness assessment using deep convolutional autoencoder
VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and TechnologyThis paper proposes a new objective metric of exceptional motion in VR video contents for VR sickness assessment. In VR environment, VR sickness can be caused by several factors which are mismatched motion, field of view, motion parallax, viewing angle, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Abstract
Funding Sources
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 46Total Downloads
- Downloads (Last 12 months)46
- Downloads (Last 6 weeks)46
Reflects downloads up to 14 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in