Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3458305.3459595acmconferencesArticle/Chapter ViewAbstractPublication PagesmmsysConference Proceedingsconference-collections
research-article

VRComm: an end-to-end web system for real-time photorealistic social VR communication

Published: 15 July 2021 Publication History

Abstract

Tools and platforms that enable remote communication and collaboration provide a strong contribution to societal challenges. Virtual meetings and conferencing, in particular, can help to reduce commutes and lower our ecological footprint, and can alleviate physical distancing measures in case of global pandemics. In this paper, we outline how to bridge the gap between common video conferencing systems and emerging social VR platforms to allow immersive communication in Virtual Reality (VR). We present a novel VR communication framework that enables remote communication in virtual environments with real-time photorealistic user representation based on colour-and-depth (RGBD) cameras and web browser clients, deployed on common off-the-shelf hardware devices. The paper's main contribution is threefold: (a) a new VR communication framework, (b) a novel approach for real-time depth data transmitting as a 2D grayscale for 3D user representation, including a central MCU-based approach for this new format and (c) a technical evaluation of the system with respect to processing delay, CPU and GPU usage.

References

[1]
D. S. Alexiadis, A. Chatzitofis, N. Zioulis, O. Zoidi, G. Louizis, D. Zarpalas, and P. Daras. 2017. An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing. IEEE Transactions on Circuits and Systems for Video Technology 27, 4 (2017), 798--813.
[2]
Alessandro Amirante, Tobia Castaldi, Lorenzo Miniero, and Simon Pietro Romano. 2015. Performance Analysis of the Janus WebRTC Gateway. In Proceedings of the 1st Workshop on All-Web Real-Time Systems (Bordeaux, France) (AWeS '15). Association for Computing Machinery, New York, NY, USA, Article 4, 7 pages.
[3]
Jeremy N Bailenson and Nick Yee. 2005. Digital chameleons: Automatic assimilation of nonverbal gestures in immersive virtual environments. Psychological science 16, 10 (2005), 814--819.
[4]
Steve Benford, Chris Greenhalgh, Tom Rodden, and James Pycock. 2001. Collaborative virtual environments. Commun. ACM 44, 7 (2001), 79--85.
[5]
Jim Blascovich and Jeremy Bailenson. 2011. Infinite reality: Avatars, eternal life, new worlds, and the dawn of the virtual revolution. William Morrow & Co.
[6]
Xavier P Burgos-Artizzu, Julien Fleureau, Olivier Dumas, Thierry Tapie, François LeClerc, and Nicolas Mollet. 2015. Real-time expression-sensitive hmd face reconstruction. In SIGGRAPH Asia 2015 Technical Briefs. 1--4.
[7]
Gianluca Cernigliaro, Marc Martos, Mario Montagud, Amir Ansari, and Sergi Fernandez. 2020. PC-MCU: Point Cloud Multipoint Control Unit for Multi-User Holoconferencing Systems. In Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (Istanbul, Turkey) (NOSSDAV '20). Association for Computing Machinery, New York, NY, USA, 47--53.
[8]
S. Cho, S. Kim, J. Lee, J. Ahn, and J. Han. 2020. Effects of volumetric capture avatars on social presence in immersive virtual environments. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 26--34.
[9]
Carlos Coelho, JG Tichon, Trevor J Hine, GM Wallis, and Giuseppe Riva. 2006. Media presence and inner presence: the sense of presence in virtual reality technologies. In From communication to presence: Cognition, emotions and culture towards the ultimate communicative experience. IOS Press, Amsterdam, 25--45.
[10]
Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-Quality Streamable Free-Viewpoint Video. ACM Trans. Graph. 34, 4, Article 69 (July 2015), 13 pages.
[11]
Francesca De Simone, Jie Li, Henrique Galvan Debarba, Abdallah El Ali, Simon NB Gunkel, and Pablo Cesar. 2019. Watching videos together in social virtual reality: An experimental study on user's QoE. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 890--891.
[12]
Sylvie Dijkstra-Soudarissanane, Karim El Assal, Simon Gunkel, Frank ter Haar, Rick Hindriks, Jan Willem Kleinrouweler, and Omar Niamut. 2019. Multi-sensor capture and network processing for virtual reality conferencing. In Proceedings of the 10th ACM Multimedia Systems Conference. 316--319.
[13]
Sam Ekong, Christoph W. Borst, Jason Woodworth, and Terrence L. Chambers. 2016. Teacher-Student VR Telepresence with Networked Depth Camera Mesh and Heterogeneous Displays. In Advances in Visual Computing, George Bebis, Richard Boyle, Bahram Parvin, Darko Koracin, Fatih Porikli, Sandra Skaff, Alireza Entezari, Jianyuan Min, Daisuke Iwai, Amela Sadagic, Carlos Scheidegger, and Tobias Isenberg (Eds.). Springer International Publishing, Cham, 246--258.
[14]
Carmine Elvezio, Mengu Sukan, Ohan Oda, Steven Feiner, and Barbara Tversky. 2017. Remote Collaboration in AR and VR Using Virtual Replicas. In ACM SIGGRAPH 2017 VR Village (Los Angeles, California) (SIGGRAPH '17). Association for Computing Machinery, New York, NY, USA, Article 13, 2 pages.
[15]
Christian Feldmann, Christopher Bulla, and Bastian Cellarius. 2013. Efficient stream-reassembling for video conferencing applications using tiles in HEVC. In Proc. of International Conferences on Advances in Multimedia (MMEDIA). 130--135.
[16]
Leonor Fermoselle, Simon Gunkel, Frank ter ter Haar, Sylvie Dijkstra-Soudarissanane, Alexander Toet, Omar Niamut, and Nanda van van der Stap. 2020. Let's Get in Touch! Adding Haptics to Social VR. In ACM International Conference on Interactive Media Experiences (Cornella, Barcelona, Spain) (MX '20). Association for Computing Machinery, New York, NY, USA, 174--179.
[17]
Christian Frueh, Avneesh Sud, and Vivek Kwatra. 2017. Headset removal for virtual and mixed reality. In ACM SIGGRAPH 2017 Talks. 1--2.
[18]
Maia Garau, Mel Slater, Vinoba Vinayagamoorthy, Andrea Brogni, Anthony Steed, and M Angela Sasse. 2003. The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment. In Proceedings of the SIGCHI conference on Human factors in computing systems. 529--536.
[19]
Juan C Granda, Pelayo Nuño, Francisco J Suárez, and Daniel F García. 2015. Overlay network based on WebRTC for interactive multimedia communications. In 2015 International Conference on computer, information and telecommunication systems (CITS). IEEE, 1--5.
[20]
RA Grier, H Thiruvengada, SR Ellis, P Havig, KS Hale, and JG Hollands. 2012. Augmented Reality-implications toward virtual reality, human perception and performance. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 56. SAGE Publications Sage CA: Los Angeles, CA, 1351--1355.
[21]
S.N.B Gunkel, Stokking H., T. De Koninck, and OA Niamut. 2019. Everyday Photo-Realistic Social VR: Communicate and Collaborate with an Enhanced Co-Presence and Immersion. In Technical Papers International Broadcasting Convention (IBC).
[22]
Simon NB Gunkel, Hans M Stokking, Martin J Prins, Nanda van der Stap, Frank B ter Haar, and Omar A Niamut. 2018. Virtual Reality Conferencing: Multi-user immersive VR experiences on the web. In Proceedings of the 9th ACM Multimedia Systems Conference. 498--501.
[23]
Michitaka Hirose, Tetsuro Ogi, and Toshio Yamada. 1999. Integrating live video for immersive environments. IEEE MultiMedia 6, 3 (1999), 14--22.
[24]
Jack Jansen. 2014. VideoLat. Proceedings of the ACM International Conference on Multimedia - MM 14 (2014).
[25]
Jack Jansen, Shishir Subramanyam, Romain Bouqueau, Gianluca Cernigliaro, Marc Martos Cabré, Fernando Pérez, and Pablo Cesar. 2020. A pipeline for multiparty volumetric video conferencing: transmission of point clouds over low latency DASH. In Proceedings of the 11th ACM Multimedia Systems Conference. 341--344.
[26]
Peter Kauff and Oliver Schreer. 2002. An immersive 3D video-conferencing system using shared virtual team user environments. In Proceedings of the 4th international conference on Collaborative virtual environments. 105--112.
[27]
Gregorij Kurillo and Ruzena Bajcsy. 2013. 3D teleimmersion for collaboration and interaction of geographically distributed users. Virtual Reality 17, 1 (2013), 29--43.
[28]
Jason Leigh, Thomas A DeFanti, A Johnson, Maxine Brown, and D Sandin. 1997. Global tele-immersion: Better than being there. In Proceedings of ICAT, Vol. 97. 3--5.
[29]
Jason Leigh, Andrew E Johnson, Thomas A DeFanti, Maxine Brown, M Dastagir Ali, Stuart Bailey, Andy Banerjee, P Benerjee, Jim Chen, Kevin Curry, et al. 1999. A review of tele-immersive applications in the CAVE research network. In Proceedings IEEE Virtual Reality (Cat. No. 99CB36316). IEEE, 180--187.
[30]
Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei, Tristan Trutna, Pei-Lun Hsieh, Aaron Nicholls, and Chongyang Ma. 2015. Facial performance sensing head-mounted display. ACM Transactions on Graphics (ToG) 34, 4 (2015), 1--9.
[31]
Jyh-Ming Lien, Gregorij Kurillo, and Ruzena Bajcsy. 2010. Multi-camera tele-immersion system with real-time model driven data compression. The Visual Computer 26, 1 (2010), 3.
[32]
Yunpeng Liu, Stephan Beck, Renfang Wang, Jin Li, Huixia Xu, Shijie Yao, Xiaopeng Tong, and Bernd Froehlich. 2015. Hybrid Lossless-Lossy Compression for Real-Time Depth-Sensor Streams in 3D Telepresence Applications. 442--452.
[33]
R. Mekuria, K. Blom, and P. Cesar. 2017. Design, Implementation, and Evaluation of a Point Cloud Codec for Tele-Immersive Video. IEEE Transactions on Circuits and Systems for Video Technology 27, 4 (2017), 828--842.
[34]
Mario Montagud, Juan Antonio De Rus, Rafael Fayos-Jordan, Miguel Garcia-Pineda, and Jaume Segura-Garcia. 2020. Open-Source Software Tools for Measuring Resources Consumption and DASH Metrics. In Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys '20). Association for Computing Machinery, New York, NY, USA, 261--266.
[35]
Nels Numan, Frank Haar, and Pablo Cesar. 2021. Generative RGB-D Face Completion for Head-Mounted Display Removal. In 2021 IEEE Virtual Humans and Crowds for Immersive Environments (VHCIE). IEEE, IEEE.
[36]
Pelayo Nuño, Francisco G Bulnes, Juan C Granda, Francisco J Suárez, and Daniel F García. 2018. A Scalable WebRTC Platform based on Open Technologies. In 2018 International Conference on Computer, Information and Telecommunication Systems (CITS). IEEE, 1--5.
[37]
Tetsuro Ogi, Toshio Yamada, Ken Tamagawa, Makoto Kano, and Michitaka Hirose. 2001. Immersive telecommunication using stereo video avatar. In Proceedings IEEE Virtual Reality 2001. IEEE, 45--51.
[38]
Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al. 2016. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 741--754.
[39]
J. Outlaw and B. Duckles. 2017. Why Woman Don't Like Social Virtual Reality. https://extendedmind.io/social-vr
[40]
J. Park, P. A. Chou, and J. Hwang. 2019. Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 1 (2019), 149--162.
[41]
Fabrizio Pece, Jan Kautz, and Tim Weyrich. 2011. Adapting standard video codecs for depth streaming. In EGVE/EuroVR. 59--66.
[42]
Tomislav Pejsa, Julian Kantor, Hrvoje Benko, Eyal Ofek, and Andrew Wilson. 2016. Room2room: Enabling life-size telepresence in a projected augmented reality environment. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing. 1716--1725.
[43]
Stefano Petrangeli, Dries Pauwels, Jeroen van der Hooft, Tim Wauters, Filip De Turck, and Jürgen Slowack. 2018. Improving quality and scalability of WebRTC video collaboration applications. In Proceedings of the 9th ACM Multimedia Systems Conference. 533--536.
[44]
O. Schreer, I. Feldmann, S. Renault, M. Zepp, M. Worchel, P. Eisert, and P. Kauff. 2019. Capture and 3D Video Processing of Volumetric Video. In 2019 IEEE International Conference on Image Processing (ICIP). 4310--4314.
[45]
Ralph Schroeder. 2012. The social life of avatars: Presence and interaction in shared virtual environments. Springer Science & Business Media.
[46]
S. Schwarz, M. Preda, V. Baroncini, M. Budagavi, P. Cesar, P. A. Chou, R. A. Cohen, M. Krivokuća, S. Lasserre, Z. Li, J. Llach, K. Mammou, R. Mekuria, O. Nakagami, E. Siahaan, A. Tabatabai, A. M. Tourapis, and V. Zakharchenko. 2019. Emerging MPEG Standards for Point Cloud Compression. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 1 (2019), 133--148.
[47]
Renata M. Sheppard, Mahsa Kamali, Raoul Rivas, Morihiko Tamai, Zhenyu Yang, Wanmin Wu, and Klara Nahrstedt. 2008. Advancing Interactive Collaborative Mediums through Tele-Immersive Dance (TED): A Symbiotic Creativity and Design Environment for Art and Computer Science. In Proceedings of the 16th ACM International Conference on Multimedia (Vancouver, British Columbia, Canada) (MM '08). Association for Computing Machinery, New York, NY, USA, 579--588.
[48]
Mel Slater. 2018. Immersion and the illusion of presence in virtual reality. British Journal of Psychology 109, 3 (2018), 431--433.
[49]
S. Subramanyam, J. Li, I. Viola, and P. Cesar. 2020. Comparing the Quality of Highly Realistic Digital Humans in 3DoF and 6DoF: A Volumetric Video Case Study. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 127--136.
[50]
Masayuki Takemura and Yuichi Ohta. 2005. Generating High-Definition Facial Video for Shared Mixed Reality. In MVA. 422--425.
[51]
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2017. Demo of FaceVR: real-time facial reenactment and eye gaze control in virtual reality. In ACM SIGGRAPH 2017 Emerging Technologies. 1--2.
[52]
J. v. d. Hooft, M. T. Vega, T. Wauters, C. Timmerer, A. C. Begen, F. D. Turck, and R. Schatz. 2020. From Capturing to Rendering: Volumetric Media Delivery with Six Degrees of Freedom. IEEE Communications Magazine 58, 10 (2020), 49--55.
[53]
M. Westerlund and S. Wenger. 2015. RTP Topologies. RFC 7667. RFC Editor. https://tools.ietf.org/html/rfc7667
[54]
Andrew D. Wilson and Hrvoje Benko. 2016. Projected Augmented Reality with the RoomAlive Toolkit. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (Niagara Falls, Ontario, Canada) (ISS '16). Association for Computing Machinery, New York, NY, USA, 517--520.
[55]
Zhenyu Yang, Bin Yu, Klara Nahrstedt, and Ruzena Bajscy. 2006. A multi-stream adaptation framework for bandwidth management in 3D tele-immersion. In Proceedings of the 2006 international workshop on Network and operating systems support for digital audio and video. 1--6.
[56]
Zhenyu Yang, K. Nahrstedt, Yi Cui, Bin Yu, Jin Liang, Sang-hack Jung, and R. Bajscy. 2005. TEEVE: the next generation architecture for tele-immersive environments. In Seventh IEEE International Symposium on Multimedia (ISM'05). 8 pp.-.
[57]
N. Zioulis, D. Alexiadis, A. Doumanoglou, G. Louizis, K. Apostolakis, D. Zarpalas, and P. Daras. 2016. 3D tele-immersion platform for interactive immersive experiences between remote users. In 2016 IEEE International Conference on Image Processing (ICIP). 365--369.

Cited By

View all
  • (2024)Volumetric Video on the Web: a platform prototype and empirical studyProceedings of the 29th International ACM Conference on 3D Web Technology10.1145/3665318.3677170(1-10)Online publication date: 25-Sep-2024
  • (2024)Open-Sourcing VR2Gather: A Collaborative Social VR System for Adaptive Multi-Party Real Time CommunicationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3685515(11210-11213)Online publication date: 28-Oct-2024
  • (2024)Scene Graph Driven Hybrid Interactive VR TeleconferencingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3684996(11276-11278)Online publication date: 28-Oct-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
MMSys '21: Proceedings of the 12th ACM Multimedia Systems Conference
June 2021
254 pages
ISBN:9781450384346
DOI:10.1145/3458305
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 July 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. VR
  2. WebRTC
  3. WebVR
  4. WebXR
  5. communication
  6. conferencing
  7. immersive media
  8. social VR
  9. virtual reality

Qualifiers

  • Research-article

Funding Sources

  • European Commission

Conference

MMSys '21
Sponsor:
MMSys '21: 12th ACM Multimedia Systems Conference
September 28 - October 1, 2021
Istanbul, Turkey

Acceptance Rates

MMSys '21 Paper Acceptance Rate 18 of 55 submissions, 33%;
Overall Acceptance Rate 176 of 530 submissions, 33%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)168
  • Downloads (Last 6 weeks)20
Reflects downloads up to 14 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Volumetric Video on the Web: a platform prototype and empirical studyProceedings of the 29th International ACM Conference on 3D Web Technology10.1145/3665318.3677170(1-10)Online publication date: 25-Sep-2024
  • (2024)Open-Sourcing VR2Gather: A Collaborative Social VR System for Adaptive Multi-Party Real Time CommunicationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3685515(11210-11213)Online publication date: 28-Oct-2024
  • (2024)Scene Graph Driven Hybrid Interactive VR TeleconferencingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3684996(11276-11278)Online publication date: 28-Oct-2024
  • (2024)MeshReduce: Scalable and Bandwidth Efficient 3D Scene Capture2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00026(20-30)Online publication date: 16-Mar-2024
  • (2024)On Cross-Layer Optimization for Real-Time Remote Communication in 5G and Beyond2024 27th Conference on Innovation in Clouds, Internet and Networks (ICIN)10.1109/ICIN60470.2024.10494420(37-41)Online publication date: 11-Mar-2024
  • (2024)Metaverse Technologies in Painting Education and Their Influence on Student Creativity and Self-ExpressionInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2381926(1-9)Online publication date: 29-Jul-2024
  • (2023)Neural Network Assisted Depth Map Packing for Compression Using Standard Hardware Video CodecsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358844019:5s(1-20)Online publication date: 7-Jun-2023
  • (2023)Addressing Scalability for Real-time Multiuser Holo-portation: Introducing and Assessing a Multipoint Control Unit (MCU) for Volumetric VideoProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613777(9243-9251)Online publication date: 26-Oct-2023
  • (2023)VQBA: Visual-Quality-Driven Bit Allocation for Low-Latency Point Cloud StreamingProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612486(9143-9151)Online publication date: 26-Oct-2023
  • (2023)VR2Gather: A Collaborative, Social Virtual Reality System for Adaptive, Multiparty Real-Time CommunicationIEEE MultiMedia10.1109/MMUL.2023.326394330:2(48-59)Online publication date: 1-Apr-2023
  • Show More Cited By

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media