research-article

VR2Gather: A Collaborative, Social Virtual Reality System for Adaptive, Multiparty Real-Time Communication

Authors:

Shishir Subramanyam,

Ignacio Reimat,

Pablo CesarAuthors Info & Claims

IEEE MultiMedia, Volume 30, Issue 2

Pages 48 - 59

https://doi.org/10.1109/MMUL.2023.3263943

Published: 01 April 2023 Publication History

Abstract

Virtual reality telecommunication systems promise to overcome the limitations of current real-time teleconferencing solutions by enabling a better sense of immersion and fostering more natural interpersonal interactions. Many solutions that currently enable immersive teleconferencing employ synthetic avatars to represent their users. However, photorealistic reconstructions have been shown to increase the sense of presence with respect to synthetic avatars in teleimmersive scenarios. In this article, we present VR2Gather, a costumizable, end-to-end system to transmit volumetric contents in multiparty, real-time communication. We present the architecture and evaluate the costs and benefits of using different modules and transport mechanisms in terms of CPU usage, latency, and bandwidth. Moreover, we report the user experience based on applications the system has been used for and how it was customized to meet the requirements using different acquisition and rendering modules.

References

[1]

S. N. B. Gunkel et al., “VRComm: An end-to-end web system for real-time photorealistic social VR communication,” in Proc. 12th ACM Multimedia Syst. Conf., Istanbul, Turkey: Association for Computing Machinery, Jul. 2021, pp. 65–79.

Digital Library

[2]

J. van der Hooft et al., “From capturing to rendering: Volumetric media delivery with six degrees of freedom,” IEEE Commun. Mag., vol. 58, no. 10, pp. 49–55, Oct. 2020.

[3]

E. d’Eon, B. Harrison, T. Myers, and P. A. Chou, “8i voxelized full bodies - A voxelized point cloud dataset,” presented at the ISO/IEC JTC1/SC29 Joint WG11/WG1 (MPEG/JPEG) Input Document WG11M40059/WG1M74006, Geneva, Switzerland, Jan. 2017.

[4]

N. Zioulis et al., “3D tele-immersion platform for interactive immersive experiences between remote users,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Phoenix, AZ, USA: IEEE Press, Sep. 2016, pp. 365–369.

[5]

S. Orts-Escolano et al., “Holoportation: Virtual 3D teleportation in real-time,” in Proc. 29th Annu. Symp. User Interface Softw. Technol., New York, NY, USA: Association for Computing Machinery, Oct. 2016, pp. 741–754.

Digital Library

[6]

J. Son et al., “Split rendering for mixed reality: Interactive volumetric video in action,” in Proc. SIGGRAPH Asia 2020 XR, Virtual Event, Republic of Korea: Association for Computing Machinery, Dec. 2020, pp. 1–3.

Digital Library

[7]

P. Carballeira et al., “FVV live: A real-time free-viewpoint video system with consumer electronics hardware,” IEEE Trans. Multimedia, vol. 24, pp. 2378–2391, May 2021.

Digital Library

[8]

J. Jansen et al., “A pipeline for multiparty volumetric video conferencing: Transmission of point clouds over low latency DASH,” in Proc. 11th ACM Multimedia Syst. Conf., Istanbul, Turkey: Association for Computing Machinery, May 2020, pp. 341–344.

Digital Library

[9]

I. Viola, and P. Cesar, “Volumetric video streaming: Current approaches and implementations,” in Immersive Media Technologies, G. Valenzise, M. Alain, E. Zerman, and C. Ozcinar, Eds. New York, NY, USA: Elsevier, 2023, pp. 425–443.

[10]

S. Schwarz et al., “Emerging MPEG standards for point cloud compression,” IEEE J. Emerg. Sel. Topics Circuits Syst., vol. 9, no. 1, pp. 133–148, 2019.

[11]

G. Cernigliaro, M. Martos, M. Montagud, A. Ansari, and S. Fernandez, “PC-MCU: Point cloud multipoint control unit for multi-user holoconferencing systems,” in Proc. 30th ACM Workshop Netw. Oper. Syst. Support Digit. Audio Video, New York, NY, USA: Association for Computing Machinery, Jun. 2020, pp. 47–53.

Digital Library

[12]

D. Zhang, B. Han, P. Pathak, and H. Wang, “Innovating multi-user volumetric video streaming through cross-layer design,” in Proc. 20th ACM Workshop Hot Topics Netw., Virtual Event, U.K.: Association for Computing Machinery, Nov. 2021, pp. 16–22.

Digital Library

[13]

R. Mekuria, K. Blom, and P. Cesar, “Design, implementation, and evaluation of a point cloud codec for tele-immersive video,” IEEE Trans. Circuits Syst. Video Technol., vol. 27, no. 4, pp. 828–842, Apr. 2017.

Digital Library

[14]

S. Subramanyam, I. Viola, J. Jansen, E. Alexiou, A. Hanjalic, and P. Cesar, “Evaluating the impact of tiled user-adaptive real-time point cloud streaming on VR remote communication,” in Proc. 30th ACM Int. Conf. Multimedia, New York, NY, USA: Association for Computing Machinery, 2022, pp. 3094–3103.

Digital Library

[15]

I. Reimat, E. Alexiou, J. Jansen, I. Viola, S. Subramanyam, and P. Cesar, “CWIPC-SXR: Point cloud dynamic human dataset for social XR,” in Proc. 12th ACM Multimedia Syst. Conf., New York, NY, USA: Association for Computing Machinery, 2021, pp. 300–306.

Digital Library

[16]

S. Subramanyam, I. Viola, A. Hanjalic, and P. Cesar, “User centered adaptive streaming of dynamic point clouds with low complexity tiling,” in Proc. 28th ACM Int. Conf. Multimedia, Seattle, WA, USA: Association for Computing Machinery, Oct. 2020, pp. 3669–3677.

Digital Library

[17]

J. Tam, E. Carter, S. Kiesler, and J. Hodgins, “Video increases the perception of naturalness during remote interactions with latency,” in Proc. CHI '12 Extended Abstr. Hum. Factors Comput. Syst., Jan. 2012, pp. 2045–2050.

Digital Library

[18]

N. I. Fisher and R. E. Kordupleski, “Good and bad market research: A critical review of net promoter score,” Appl. Stochastic Models Bus. Ind., vol. 35, no. 1, pp. 138–151, 2019.

[19]

A. Bangor, P. Kortum, and J. Miller, “Determining what individual sus scores mean: Adding an adjective rating scale,” J. Usability Stud., vol. 4, no. 3, pp. 114–123, May 2009.

Digital Library

[20]

J. Sauro, “The challenges and opportunities of measuring the user experience,” J. Usability Stud., vol. 12, no. 1, pp. 1–7, Nov. 2016.

Digital Library

Cited By

Zhou XViola IYin RCesar PLi JGao XLe Callet PJanowski LLu WYang JWang JLi JZhang J(2024)Visual-Saliency Guided Multi-modal Learning for No Reference Point Cloud Quality AssessmentProceedings of the 3rd Workshop on Quality of Experience in Visual Multimedia Applications10.1145/3689093.3689183(39-47)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3689093.3689183
Cabrero Barros SElosegi ATamayo IDominguez AZorrilla M(2024)Volumetric Video on the Web: a platform prototype and empirical studyProceedings of the 29th International ACM Conference on 3D Web Technology10.1145/3665318.3677170(1-10)Online publication date: 25-Sep-2024
https://dl.acm.org/doi/10.1145/3665318.3677170
Li NZhu MWang SLiu Y(2024)A Comparative Study of K-Planes vs. V-PCC for 6-DoF Volumetric Video RepresentationProceedings of the 16th International Workshop on Immersive Mixed and Virtual Environment Systems10.1145/3652212.3652227(92-98)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3652212.3652227
Show More Cited By

Recommendations

Collaborative Augmented Reality: Multi-user Interaction in Urban Simulation
IVIC '09: Proceedings of the 1st International Visual Informatics Conference on Visual Informatics: Bridging Research and Practice

Augmented reality (AR) environment allows user or multi-user to interact with 2D and 3D data. AR simply can provide a collaborative interactive AR environment for urban simulation, where users can interact naturally and intuitively. AR collaboration ...
Real-time facial tracking in virtual reality
SA '16: SIGGRAPH ASIA 2016 VR Showcase

Virtual reality (VR) emerges as the next social computing platform. For realizing immersive social interactions, projecting facial expressions onto the virtual avatar a crucial component. This is a challenge in VR as it requires capturing the facial ...
Communication Behavior in Embodied Virtual Reality
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Embodied virtual reality faithfully renders users' movements onto an avatar in a virtual 3D environment, supporting nuanced nonverbal behavior alongside verbal communication. To investigate communication behavior within this medium, we had 30 dyads ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE MultiMedia

IEEE MultiMedia Volume 30, Issue 2

April-June 2023

86 pages

ISSN:1070-986X

Issue’s Table of Contents

1070-986X © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Computer Society Press

Washington, DC, United States

Publication History

Published: 01 April 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhou XViola IYin RCesar PLi JGao XLe Callet PJanowski LLu WYang JWang JLi JZhang J(2024)Visual-Saliency Guided Multi-modal Learning for No Reference Point Cloud Quality AssessmentProceedings of the 3rd Workshop on Quality of Experience in Visual Multimedia Applications10.1145/3689093.3689183(39-47)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3689093.3689183
Cabrero Barros SElosegi ATamayo IDominguez AZorrilla M(2024)Volumetric Video on the Web: a platform prototype and empirical studyProceedings of the 29th International ACM Conference on 3D Web Technology10.1145/3665318.3677170(1-10)Online publication date: 25-Sep-2024
https://dl.acm.org/doi/10.1145/3665318.3677170
Li NZhu MWang SLiu Y(2024)A Comparative Study of K-Planes vs. V-PCC for 6-DoF Volumetric Video RepresentationProceedings of the 16th International Workshop on Immersive Mixed and Virtual Environment Systems10.1145/3652212.3652227(92-98)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3652212.3652227
Usón JCortés CMuñoz VHernando TBerjón DMorán FCabrera JGarcía N(2024)Untethered Real-Time Immersive Free Viewpoint VideoProceedings of the 16th International Workshop on Immersive Mixed and Virtual Environment Systems10.1145/3652212.3652214(45-49)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3652212.3652214
Usón JCabrera J(2024)Analysis and Development of Deep Learning Depth Estimation Techniques for Volumetric Capture and Free Viewpoint VideoProceedings of the 15th ACM Multimedia Systems Conference10.1145/3625468.3652913(520-523)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3625468.3652913
Lee SViola IRossi SGuo ZReimat IŁawicka KStriner ACesar P(2024)Designing and Evaluating a VR Lobby for a Socially Enriching Remote Opera Watching ExperienceIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337208130:5(2055-2065)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372081
Cortés CViola IGutiérrez JJansen JSubramanyam SAlexiou EPérez PGarcía NCésar P(undefined)Delay threshold for social interaction in volumetric eXtended Reality communicationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3651164
https://dl.acm.org/doi/10.1145/3651164

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents