Subjective Evaluation of User Quality of Experience for Omnidirectional Video Streaming
Abstract
This paper reports on the results of subjective testing of user Quality of Experience (QoE) for omnidirectional video (ODV) streaming quality. The test was conducted among 20 test subjects who watched three ODVs using a Head Mounted Display (HMD) system. The length of the videos was between two and three minutes. The first video was used for training purposes and contained no quality degradations. The quality of the other two ODVs was degraded by manipulating the resolution or by introducing different frame drop patterns. While watching the pre-prepared videos the subjects indicated if they noticed the changes in the quality and then rated it. After watching each video, the subjects completed a separate questionnaire, which evaluated their level of enjoyment and discomfort with the video. The results showed that the degradation of both objective parameters (video resolution and frame rate) impacted the subjects’ perception of quality; however, the impact was somewhat alleviated in ODV which contained dynamic scenes and fast camera movements.
References
Ericsson Consumerlab. Merged Reality: Understanding How Virtual and Augmented Realities Could Transform Everyday Reality. Ericsson; 2017. Available from: https://www.ericsson.com/en/trends-and-nsights/consumerlab/
consumer-insights/reports/merged-reality [Accessed June 2019].
Mrvelj Š, Matulin M. Impact of Packet Loss on the Perceived Quality of UDP-based Multimedia Streaming: A Study of User Quality of Experience in Real-life Environments. Multimedia Systems. 2018;24(1): 33-53. Available from: doi:10.1007/s00530-016-0531-8
Matulin M, Mrvelj Š. Modelling User Quality of Experience from Objective and Subjective Data Sets using Fuzzy Logic. Multimedia Systems. 2018;24(6): 645-667 Available from: doi:10.1007/s00530-018-0590-0
https://www.fpz.unizg.hr/qoe4vr/ [Accessed September 2019].
Cisco. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2017-2022. White Paper. Cisco; 2019. Available from: https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-738429.html [Accessed June 2019].
Yu M, Lakshman H, Girod B. A Framework to Evaluate Omnidirectional Video Coding Schemes, Mixed and Augmented Reality. Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 29 September – 3 October 2015, Fukuoka, Japan; 2015. p. 31-36. Available from: doi:10.1109/ISMAR.2015.12
Sun Y, Lu A, Yu L. WS-PSNR for 360 Video Objective Quality Evaluation. MPEG Joint Video Exploration Team. Vol. 116. Chengdu; 2016.
Sun Y, Lu A, Yu L. AHG8: WS-PSNR for 360 Video Objective Quality Evaluation. Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JVET-D0040, 4th Meeting. Chengdu; 2016.
He Y, Vishwanath B, Xiu X, Ye Y. AHG8: InterDigital’s Projection Format Conversion Tool. Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JVET-D0021, 4th Meeting. Chengdu; 2016.
Zakharchenko V, Alshina E, Singh A, Dsouza A. AHG8: Suggested Testing Procedure for 360-degree Video. Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JVET-D0027, 4th Meeting. Chengdu; 2016.
Wu S, Chen X, Fu J, Chen Z. Efficient VR Video Representation and Quality Assessment. Journal of Visual Communication and Image Representation. 2018;57: 107-117. Available from: doi:10.1016/j.jvcir.2018.10.018
Chen S, Zhang Y, Li Y, Chen Z, Wang Z. Spherical Structural Similarity Index for Objective Omnidirectional Video Quality Assessment. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), 23-27 July 2018, San Diego, United States; 2018. p. 1-6. Available from: doi:10.1109/ICME.2018.8486584
Wang Z, Simoncelli EP, Bovik AC. Multiscale Structural Similarity for Image Quality Assessment. Proceedings of the 37th Asilomar Conference on Signals, Systems & Computers, 9-12 November 2003, Pacific Grove, United States; 2003. p. 1398-1402. Available from: doi:10.1109/ACSSC.2003.1292216
Ozcinar C, Cabrera J, Smolic A. Omnidirectional Video Streaming Using Visual Attention-Driven Dynamic Tiling for VR. Proceedings of the IEEE International Conference on Visual Communications and Image Processing (VCIP) 2018, 9-12 December 2018, Taichung, Taiwan; 2018. p. 1-4. Available from: doi:10.1109/VCIP.2018.8698638
Ozcinar C, Cabrera J, Smolic A. Visual Attention-Aware Omnidirectional Video Streaming Using Optimal Tiles for Virtual Reality. IEEE Journal on Emerging and Selected Topics in Circuits and Systems. 2019;9(1): 217-230. Available from: doi:10.1109/JETCAS.2019.2895096
Xu M, Li C, Chen Z, Wang Z, Guan Z. Assessing Visual Quality of Omnidirectional Videos. IEEE Transactions on Circuits and Systems for Video Technology. 2019. Available from: doi:10.1109/TCSVT.2018.2886277
Birkbeck N, Brown C, Suderman R. Quantitative Evaluation of Omnidirectional Video Quality. Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX), 31 May - 2 June 2017, Erfurt, Germany; 2017. Available from: doi:10.1109/QoMEX.2017.7965684
Zakharchenko V, Pyo Choi K, Alshina E, Hoon Park J. Omnidirectional Video Quality Metrics and Evaluation Process. Proceedings of the Data Compression Conference (DCC), 4-7 April 2017, Snowbird, United States; 2017. p. 472-472. Available from: doi:10.1109/DCC.2017.90
Chen Z, Li Y, Zhang Y. Recent Advances in Omnidirectional Video Coding for virtual Reality: Projection and Evaluation. Signal Processing. 2018;146: 66-78. Available from: doi:10.1016/j.sigpro.2018.01.004
Upenik E, Řeřábek M, Ebrahimi T. Testbed for Subjective Evaluation of Omnidirectional Visual Content. Picture Coding Symposium (PCS), 4-7 December 2016, Nuremberg, Germany; 2016. p. 1-5. Available from: doi:10.1109/PCS.2016.7906378
Pérez P, Escobar J. MIRO360: A Tool for Subjective Assessment of 360 Degree Video for ITU-T P.360-VR. Proceedings of the 11th International Conference on Quality of Multimedia Experience (QoMEX), 5-7 June 2019, Berlin, Germany; 2019. Available from: doi:10.1109/QoMEX.2019.8743216
Sassatelli L, Winckler M, Fisichella T, Dezarnaud A, Lemaire J, Aparicio-Pardo R, Trevisan D. New Interactive Strategies for Virtual Reality Streaming in Degraded Context of Use. Computers & Graphics. 2019. Available from: doi:10.1016/j.cag.2019.10.005
Schatz R, Sackl A, Timmerer C, Gardlo B. Towards Subjective Quality of Experience Assessment for Omnidirectional Video Streaming. Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX), 31 May - 2 June 2017, Erfurt, Germany; 2017. Available from: doi:10.1109/QoMEX.2017.7965657
Yao S-H, Fan C-L, Hsu C-H. Towards Quality-of-Experience Models for Watching 360° Videos in Head-Mounted Virtual Reality. Proceedings of the 11th International Conference on Quality of Multimedia Experience (QoMEX), 5-7 June 2019, Berlin, Germany; 2019. Available from: doi:10.1109/QoMEX.2019.8743198
Liotou E, Tsolkas D, Passas N. A roadmap on QoE metrics and models. Proceedings of the 23rd International Conference on Telecommunications (ICT), 16-18 May 2016, Thessaloniki, Greece; 2016. Available from: doi:10.1109/ICT.2016.7500363
Datta P, Izdebski L, Kumar N, Suh K. “It came to me in a stream…” The upward arc of online video, driven by consumers. Cisco white paper. Available from: https://www.cisco.com/c/dam/en_us/about/ac79/docs/sp/Online-
Video-Consumption_Consumers.pdf [Accessed 22nd Feb 2020].
Farrokhi F, Mahmoudi-Hamidabad A. Rethinking convenience sampling: defining quality criteria. Theory and Practice in Language Studies. 2012;2(4): 784-792. Available from: doi:10.4304/tpls.2.4.784-792
Fiedler M, Hossfeld T, Tran-Gia P. A Generic Quantitative Relationship between Quality of Experience and Quality of Service. IEEE Network. 2010;24(2): 36-41. Available from: doi:10.1109/MNET.2010.5430142
Copyright (c) 2020 Štefica Mrvelj, Marko Matulin, Sergo Martirosov
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).