Abstract
Despite remarkable advances, current augmented and virtual reality (AR/VR) applications are a largely individual and local experience. Interconnected AR/VR, where participants can virtually interact across vast distances, remains a distant dream. The great barrier that stands between current technology and such applications is the stringent end-to-end latency requirement, which should not exceed 20 ms in order to avoid motion sickness and other discomforts. Bringing AR/VR to the next level to enable immersive interconnected AR/VR will require significant advances towards 5G ultra-reliable low-latency communication (URLLC) and a Tactile Internet of Things (IoT). In this article, we articulate the technical challenges to enable a future AR/VR end-to-end architecture, that combines 5G URLLC and Tactile IoT technology to support this next generation of interconnected AR/VR applications. Through the use of IoT sensors and actuators, AR/VR applications will be aware of the environmental and user context, supporting human-centric adaptations of the application logic, and lifelike interactions with the virtual environment. We present potential use cases and the required technological building blocks. For each of them, we delve into the current state of the art and challenges that need to be addressed before the dream of remote AR/VR interaction can become reality.
Similar content being viewed by others
References
Bastug, E., Bennis, M., Medard, M., Debbah, M.: Toward interconnected virtual reality: Opportunities, challenges, and enablers. IEEE Commun. Magaz. 55(6), 110–117 (2017). https://doi.org/10.1109/MCOM.2017.1601089
Aijaz, A., Dohler, M., Aghvami, A.H., Friderikos, V., Frodigh, M.: Realizing the tactile internet: Haptic communications over next generation 5G cellular networks. IEEE Wireless Commun. 24(2), 82–89 (2017)
Mangiante, S., Klas, G., Navon, A., GuanHua, Z., Ran, J., Dias Silva, M.: VR is on the edge: How to deliver 360-videos in mobile networks. In: Workshop on Virtual Reality and Augmented Reality Network (VR/AR Network), pp. 30–35 (2017). https://doi.org/10.1145/3097895.3097901
Elbamby, M.S., Perfecto, C., Bennis, M., Doppler, K.: Toward low-latency and ultra-reliable virtual reality. IEEE Netw. 32(2), 78–84 (2018). https://doi.org/10.1109/MNET.2018.1700268
Finn, N.: Introduction to time-sensitive networking. IEEE Commun. Stand. Magaz. 2(2), 22–28 (2018)
Zhang, H., Elmokashfi, A., Yang, Z., Mohapatra, P.: Wireless access to ultimate virtual reality 360-degree video at home. In: International Conference on Internet of Things Design and Implementation, pp. 271–272 (2019)
Baños-Gonzalez, V., Afaqui, M., Lopez-Aguilera, E., Garcia-Villegas, E.: IEEE 802.11ah: A technology to face the IoT challenge. Sensors 16(11) (2016). https://doi.org/10.3390/s16111960
Khorov, E., Kiryanov, A., Lyakhov, A., Bianchi, G.: A tutorial on IEEE 802.11ax high efficiency WLANs. IEEE Commun. Surv. Tutor. 21(1), 197–216 (2019). https://doi.org/10.1109/COMST.2018.2871099
Zhou, P., Cheng, K., Han, X., Fang, X., Fang, Y., He, R., Long, Y., Liu, Y.: IEEE 802.11ay-based mmWave WLANs: design challenges and solutions. IEEE Commun. Surv. Tutor. 20(3), 1654–1681 (2018). https://doi.org/10.1109/COMST.2018.2816920
Beyene, Y.D., Jantti, R., Tirkkonen, O., Ruttik, K., Iraji, S., Larmo, A., Tirronen, T., Torsner, J.: NB-IoT technology overview and experience from cloud-RAN implementation. IEEE Wireless Commun. 24(3), 26–32 (2017). https://doi.org/10.1109/MWC.2017.1600418
Lien, S.Y., Shieh, S.L., Huang, Y., Su, B., Hsu, Y.L., Wei, H.Y.: 5G new radio: waveform, frame structure, multiple access, and initial access. IEEE Commun. Magaz. 55(6), 64–71 (2017). https://doi.org/10.1109/MCOM.2017.1601107
Lopez, A.V., Chervyakov, A., Chance, G., Verma, S., Tang, Y.: Opportunities and challenges of mmWave NR. IEEE Wireless Commun. 26(2), 4–6 (2019). https://doi.org/10.1109/MWC.2019.8700132
Parvez, I., Rahmati, A., Guvenc, I., Sarwat, A.I., Dai, H.: A survey on low latency towards 5G: RAN, core network and caching solutions. IEEE Commun. Surv. Tutor. 20(4), 3098–3130 (2018). https://doi.org/10.1109/COMST.2018.2841349
Barbarossa, S., Ceci, E., Merluzzi, M.: Overbooking radio and computation resources in mmW-mobile edge computing to reduce vulnerability to channel intermittency. In: European Conference on Networks and Communications (EuCNC) (2017). https://doi.org/10.1109/EuCNC.2017.7980746
di Pietro, N., Merluzzi, M., Calvanese Strinati, E., Barbarossa, S.: Resilient design of 5G mobile-edge computing over intermittent mmWave links (2019)
Nielsen, J.J., Liu, R., Popovski, P.: Ultra-reliable low latency communication using interface diversity. IEEE Trans. Commun. 66(3), 1322–1334 (2018)
Drago, M., Azzino, T., Polese, M., Stefanović, C., Zorzi, M.: Reliable video streaming over mmWave with multi connectivity and network coding. In: International Conference on Computing, Networking and Communications (ICNC), pp. 508–512 (2018)
De Schepper, T., Bosch, P., Zeljkovic, E., Mahfoudhi, F., Haxhibeqiri, J., Hoebeke, J., Famaey, J., Latre, S.: ORCHESTRA: Enabling inter-technology network management in heterogeneous wireless networks. IEEE Trans. Netw. Serv. Manag. 15(4), 1733–1746 (2018)
Sur, S., Venkateswaran, V., Zhang, X., Ramanathan, P.: 60 GHz indoor networking through flexible beams: A link-level profiling. SIGMETRICS Perform. Eval. Rev. 43(1), 71–84 (2015)
Palacios, J., Casari, P., Assasa, H., Widmer, J.: LEAP: Location estimation and predictive handover with consumer-grade mmWave devices. In: IEEE Conference on Computer Communications (INFOCOM), pp. 2377–2385 (2019)
Braden, B., Zhang, L., Berson, S., Herzog, S., Jamin, S.: Resource ReSerVation Protocol (RSVP) - Version 1 Functional Specification. RFC 2205, (1997)
Nasrallah, A., Balasubramanian, V., Thyagaturu, A., Reisslein, M., ElBakoury, H.: TSN Algorithms for Large Scale Networks: A Survey and Conceptual Comparison (2019)
Messenger, J.L.: Time-sensitive networking: an introduction. IEEE Commun. Stand. Magaz. 2(2), 29–33 (2018)
Kua, J., Armitage, G., Branch, P.: A survey of rate adaptation techniques for dynamic adaptive streaming over http. IEEE Commun. Surv. Tutor. 19, 1842–1866 (2017)
He, D., Westphal, C., Garcia-Luna-Aceves, J.: Network support for ar/vr and immersive video application: A survey. pp. 359–369 (2018). https://doi.org/10.5220/0006941703590369
Abdallah, M., Griwodz, C., Chen, K.T., Simon, G., Wang, P.C., Hsu, C.H.: Delay-sensitive video computing in the cloud: a survey. ACM Trans. Multimedia Comput. Commun. Appl. 14, 3 (2018). https://doi.org/10.1145/3212804
Lakiotakis, E., Liaskos, C., Dimitropoulos, X.: Improving networked music performance systems using application-network collaboration. Concurrency and Computation: Practice and Experience (2018). https://doi.org/10.1002/cpe.4730
Wang, M., Cui, Y., Wang, G., Xiao, S., Jiang, J.: Machine learning for networking: Workflow, advances and opportunities. IEEE Network PP (2017). https://doi.org/10.1109/MNET.2017.1700200
Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V.F., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gülçehre, Ç., Song, H.F., Ballard, A.J., Gilmer, J., Dahl, G.E., Vaswani, A., Allen, K.R., Nash, C., Langston, V., Dyer, C., Heess, N., Wierstra, D., Kohli, P., Botvinick, M., Vinyals, O., Li, Y., Pascanu, R.: Relational inductive biases, deep learning, and graph networks. CoRR abs/1806.01261 (2018) URL arXiv:1806.01261
Davie, B., Koponen, T., Pettit, J., Pfaff, B., Casado, M., Gude, N., Padmanabhan, A., Petty, T., Duda, K., Chanda, A.: A database approach to sdn control plane design. SIGCOMM Comput. Commun. Rev. 47(1), 15–26 (2017). https://doi.org/10.1145/3041027.3041030
Mestres, A., Rodriguez-Natal, A., Carner, J., Barlet-Ros, P., Alarcón, E., Solé, M., Muntés-Mulero, V., Meyer, D., Barkai, S., Hibbett, M.J., et al.: Knowledge-defined networking. SIGCOMM Comput. Commun. Rev. 47(3), 2–10 (2017). https://doi.org/10.1145/3138808.3138810
Lakiotakis, E., Liaskos, C., Dimitropoulos, X.A.: Improving networked music performance systems using application-network collaboration. CoRR abs/1808.09405 (2018) URL arXiv:1808.09405
Jiang, C., Zhang, H., Ren, Y., Han, Z., Chen, K., Hanzo, L.: Machine learning paradigms for next-generation wireless networks. IEEE Wireless Commun. 24(2), 98–105 (2017). https://doi.org/10.1109/MWC.2016.1500356WC
Suárez-Varela, J., Mestres, A., Yu, J., Kuang, L., Feng, H., Cabellos-Aparicio, A., Barlet-Ros, P.: Routing in optical transport networks with deep reinforcement learning. J. Opt. Commun. Netw. 11(11), 547–558 (2019). https://doi.org/10.1364/JOCN.11.000547. http://jocn.osa.org/abstract.cfm?URI=jocn-11-11-547
Afolabi, I., Taleb, T., Samdanis, K., Ksentini, A., Flinck, H.: Network slicing and softwarization: a survey on principles, enabling technologies, and solutions. IEEE Commun. Surv. Tutor. 20(3), 2429–2453 (2018). https://doi.org/10.1109/COMST.2018.2815638
Zhang, H., Liu, N., Chu, X., Long, K., Aghvami, A., Leung, V.C.M.: Network slicing based 5g and future mobile networks: mobility, resource management, and challenges. IEEE Commun. Magaz. 55(8), 138–145 (2017). https://doi.org/10.1109/MCOM.2017.1600940
Li, R., Zhao, Z., Sun, Q., C, I., Yang, C., Chen, X., Zhao, M., Zhang, H.: Deep reinforcement learning for resource management in network slicing. IEEE Access. 6, 74429–74441 (2018). https://doi.org/10.1109/ACCESS.2018.2881964
Li, R., Zhao, Z., Zhou, X., Ding, G., Chen, Y., Wang, Z., Zhang, H.: Intelligent 5g: when cellular networks meet artificial intelligence. IEEE Wireless Commun. 24(5), 175–183 (2017). https://doi.org/10.1109/MWC.2017.1600304WC
Luong, N.C., Hoang, D.T., Gong, S., Niyato, D., Wang, P., Liang, Y.C., Kim, D.I.: Applications of deep reinforcement learning in communications and networking: A survey. IEEE Commun. Surv. Tutor. 21(4), 3133–3174 (2019)
Bellemare, M.G., Dabney, W., Munos, R.: A distributional perspective on reinforcement learning. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 449–458. JMLR. org (2017)
Hua, Y., Li, R., Zhao, Z., Chen, X., Zhang, H.: Gan-powered deep distributional reinforcement learning for resource management in network slicing. IEEE Journal on Selected Areas in Communications (2019)
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Müller, K.R.: Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recogn. 65, 211–222 (2017)
Zhang, C., Patras, P., Haddadi, H.: Deep learning in mobile and wireless networking: a survey. IEEE Commun. Surv. Tutor. 21(3), 2224–2287 (2019)
Liaskos, C., Tsioliaridou, A., Ioannidis, S.: The socket store: An app model for the application-network interaction. In: IEEE ISCC 2017 (2017)
Cloud, J., Leith, D., Médard, M.: A coded generalization of selective repeat arq. In: 2015 IEEE Conference on Computer Communications (INFOCOM), pp. 2155–2163 (2015). https://doi.org/10.1109/INFOCOM.2015.7218601
Papadopoulos, I., Papanikos, N., Papapetrou, E., Kondi, L.: Network-wide md and network coding for heterogeneous video multicast. In: 2013 IEEE 24th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC), pp. 3578–3582 (2013). https://doi.org/10.1109/PIMRC.2013.6666770
Zhu, Q., Wang, R., Chen, Q., Liu, Y., Qin, W.: IOT Gateway: Bridging Wireless Sensor Networks into Internet of Things. In: IEEE/IFIP International Conference on Embedded and Ubiquitous Computing, pp. 347–352 (2010)
Chen, H., Jia, X., Li, H.: A brief introduction to IoT gateway. In: Communication Technology and Application (ICCTA 2011), IET International Conference on, pp. 610 – 613 (2011)
Chellough, S.A., El-Zawawy, M.A.: Middleware for internet of things: survey and challenges. Intell. Autom. Soft Comput. 24(2), 309–318 (2018)
Dickerson, K., Heinz, C., García-Castro, R., et al.: Analysis of Standardisation Context and Recommendations for Standards Involvement (2016). https://vicinity2020.eu/vicinity/sites/default/files/documents/vicinity_d2.1_analysis_of_standardisation_context_and_recommendations_for_standards_involvement.pdf
Nakhuva, B., Champaneria, T.: Study of various internet of things platforms. Int. J. Comput. Sci. Eng. Surv. 6(6), 61–74 (2015)
Gomes, P., Cavalcante, E., Batista, T., Taconet, C., Conan, D., Chabridon, S., Delicato, F.C., Pires, P.F.: A semantic-based discovery service for the internet of things. J. Internet Serv. Appl. (2019). https://doi.org/10.1186/s13174-019-0109-8
Song, Z., Cardenas, A., Masuoka, R.: Semantic middleware for the internet of things. In: 2010 Internet of Things (IOT), pp. 1–8 (2011). https://doi.org/10.1109/IOT.2010.5678448
Wang, W., Lee, K., Guo, J., Murray, D.: Discovering objects and services in context-aware iot environments. Int. J. Serv. Technol. Manag. 25(3/4), 326–347 (2019). https://doi.org/10.1504/IJSTM.2019.10021608
Kostelnik, P., Sarnovsky, M., Furdík, K.: The semantic middleware for networked embedded systems applied in the internet of things and services domain. Scalable Comput. 12(3), 307–315 (2011)
Guan, Y., Vasquez, J.C., Guerrero, J.M., Samovich, N., Vanya, S., Oravec, V., Garcí-Castro, R., Serena, F., Poveda-Villalón, M., Radojicic, C., Heinz, C., Grimm, C., Tryferidis, A., Tzovaras, D., Dickerson, K., Paralic, M., Skokan, M., Sabol, T.: An open virtual neighbourhood network to connect iot infrastructures and smart objects — vicinity: Iot enables interoperability as a service. In: 2017 Global Internet of Things Summit (GIoTS), pp. 1–6 (2017). https://doi.org/10.1109/GIOTS.2017.8016233
Alam, M.F., Katsikas, S., Beltramello, O., Hadjiefthymiades, S.: Augmented and virtual reality based monitoring and safety system: A prototype iot platform. Journal of Network and Computer Applications 89, 109–119 (2017). https://doi.org/10.1016/j.jnca.2017.03.022. http://www.sciencedirect.com/science/article/pii/S1084804517301315. Emerging Services for Internet of Things (IoT)
Antonakoglou, K., Xu, X., Steinbach, E., Mahmoodi, T., Dohler, M.: Toward haptic communications over the 5g tactile internet. IEEE Commun. Surv. Tutor. 20(4), 3034–3059 (2018)
Hooft, J.V., Vega, M., Petrangeli, S., Wauters, T., Turck, F.D.: Tile-based adaptive streaming for virtual reality video. ACM Trans. Multimedia Comput. Commun. Appl. 15, 4 (2019). https://doi.org/10.1145/3362101
Clemm, A., Torres Vega, M., Ravuri, H.K., Wauters, T., Turck, F.D.: Toward truly immersive holographic-type communication: challenges and solutions. IEEE Commun. Magaz. 58(1), 93–99 (2020)
Palmer, M., Krüger, T., Chandrasekaran, B., Feldmann, A.: The quic fix for optimal video streaming. In: Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC, EPIQ’18, p. 43–49. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3284850.3284857
King, H.H., Hannaford, B., Kammerly, J., Steinbachy, E.: Establishing multimodal telepresence sessions using the session initiation protocol (sip) and advanced haptic codecs. In: 2010 IEEE Haptics Symposium, pp. 321–325 (2010)
Nasir, Q., Khalil, E.: Perception based adaptive haptic communication protocol (pahcp). In: 2012 International Conference on Computer Systems and Industrial Informatics, pp. 1–6 (2012)
Venkatraman, K., Vellingiri, S., Prabhakaran, B., Nguyen, N.: Mpeg media transport (mmt) for 3d tele-immersion systems. In: 2014 IEEE International Symposium on Multimedia, pp. 279–282 (2014)
Rossol, N., Cheng, I., Bischof, W.F., Basu, A.: A framework for adaptive training and games in virtual reality rehabilitation environments. In: Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, VRCAI ’11, p. 343–346. Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/2087756.2087810
Power, D.J., Sharda, R.: Model-driven decision support systems: Concepts and research directions. Decision Support Systems 43(3), 1044–1061 (2007). https://doi.org/10.1016/j.dss.2005.05.030. http://www.sciencedirect.com/science/article/pii/S0167923605000953. Integrated Decision Support
Vaughan, N., Gabrys, B., Dubey, V.N.: An overview of self-adaptive technologies within virtual reality training. Computer Science Review 22, 65–87 (2016). https://doi.org/10.1016/j.cosrev.2016.09.001. http://www.sciencedirect.com/science/article/pii/S1574013716300259
Luzanin, O., Plancak, M.: Hand gesture recognition using low-budget data glove and cluster-trained probabilistic neural network. Assembly Autom. 34(1), 94–105 (2014). https://doi.org/10.1108/AA-03-2013-020
Yamashita, M.: Assistive driving simulator with haptic manipulator using model predictive control and admittance control. Procedia Computer Science 39, 107–114 (2014). https://doi.org/10.1016/j.procs.2014.11.016. http://www.sciencedirect.com/science/article/pii/S1877050914014343. The 6th international conference on Intelligent Human Computer Interaction, IHCI 2014
Van Damme, S., Torres Vega, M., De Turck, F.: Human-centric Quality Management of Immersive Multimedia Applications. In: in proceedings of the fourth Quality of Experience Management Workshop, collocated with NetSoft 2020. Ghent, Belgium (2020)
Simsek, M., Aijaz, A., Dohler, M., Sachs, J., Fettweis, G.: 5g-enabled tactile internet. IEEE J. Select. Areas Commun. 34(3), 460–473 (2016)
Skorin-Kapov, L., Varela, M., Hoßfeld, T., Chen, K.T.: A survey of emerging concepts and challenges for qoe management of multimedia services. ACM Trans. Multimedia Comput. Commun. Appl. 14, 2 (2018). https://doi.org/10.1145/3176648
ITU-T: Recommendation P.910 (09/99) ITU-T RECOMMENDATION P.910: Subjective video quality assessment methods for multimedia applications (1999)
Alexiou, E., Ebrahimi, T.: On subjective and objective quality evaluation of point cloud geometry. In: 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–3 (2017)
Tran, H.T.T., Ngoc, N.P., Bui, C.M., Pham, M.H., Thang, T.C.: An evaluation of quality metrics for 360 videos. In: 2017 Ninth International Conference on Ubiquitous and Future Networks (ICUFN), pp. 7–11 (2017)
van der Hooft, J., Torres Vega, M., Petrangeli, S., Wauters, T., De Turck, F.: Quality Assessment for Adaptive Virtual Reality Video Streaming: A Probabilistic Approach on the User’s Gaze. In: 2019 22nd Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN), pp. 19–24 (2019)
Alexiou, E., Viola, I., Borges, T.M., Fonseca, T.A., de Queiroz, R.L., Ebrahimi, T.: A comprehensive study of the rate-distortion performance in MPEG point cloud compression. APSIPA Trans. Sign. Inform. Process 8, e27 (2019). https://doi.org/10.1017/ATSIP.2019.20
Narbutt, M., Allen, A., Skoglund, J., Chinen, M., Hines, A.: AMBIQUAL - a full reference objective quality metric for ambisonic spatial audio. In: 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6 (2018)
Sakr, N., Georganas, N.D., Zhao, J.: A Perceptual Quality Metric for Haptic Signals. In: 2007 IEEE International Workshop on Haptic, Audio and Visual Environments and Games, pp. 27–32 (2007)
Hassen, R., Steinbach, E.: HSSIM: An Objective Haptic Quality Assessment Measure for Force-Feedback Signals. In: 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6 (2018)
Wang, Zhou, Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
De Coninck, Q., Bonaventure, O.: Multipath quic: Design and evaluation. In: Proceedings of the 13th International Conference on Emerging Networking EXperiments and Technologies, CoNEXT ’17, p. 160–166. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3143361.3143370
Acknowledgements
Maria Torres Vega is funded by the Research Foundation Flanders (FWO), grant number 12W4819N. This work has been partially supported by the Spanish Ministry of Economy and Competitiveness under Contract TEC2017-90034-C2-1-R (ALLIANCE project) that receives funding from FEDER.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Torres Vega, M., Liaskos, C., Abadal, S. et al. Immersive Interconnected Virtual and Augmented Reality: A 5G and IoT Perspective. J Netw Syst Manage 28, 796–826 (2020). https://doi.org/10.1007/s10922-020-09545-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10922-020-09545-w