research-article

Deep dive: deep-neural-network-based video extension for immersive head-mounted display experiences

Authors:

Michinari Kono,

Jun RekimotoAuthors Info & Claims

PerDis '19: Proceedings of the 8th ACM International Symposium on Pervasive Displays

Article No.: 22, Pages 1 - 7

https://doi.org/10.1145/3321335.3324932

Published: 12 June 2019 Publication History

Abstract

Immersion is an important factor in video experiences. Therefore, various methods and video viewing systems have been proposed. Head-mounted displays (HMDs) are home-friendly pervasive devices, which can provide an immersive video experience owing to their wide field-of-view (FoV) and separation of users from the outside environment. They are often used for viewing panoramic and stereoscopic recorded videos or virtually generated environments, but the demand for viewing standard plane videos with HMDs has increased. However, the theater mode, which restricts the FoV, is basically used for viewing plane videos. Thus, the advantages of HMDs are not fully utilized. Therefore, we explored a method for viewing plane videos by an HMD, in combination with view augmentation by LED implants to the HMD. We have constructed a system for viewing plane videos using an HMD with a deep neural network (DNN) model optimized for generating and extending images for peripheral vision and wide FoV customization. We found that enlarging the original video and extending the video with our DNN model can improve the user experience. However, our method provided more comfortable viewing by preventing motion sickness in a first-person-view video.

References

[1]

A. Aides, T. Avraham, and Y. Y. Schechner. 2011. Multiscale ultrawide foveated video extrapolation. In 2011 IEEE International Conference on Computational Photography (ICCP). 1--8.

[2]

T. Avraham and Y. Y. Schechner. 2011. Ultrawide Foveated Video Extrapolation. IEEE Journal of Selected Topics in Signal Processing 5, 2 (April 2011), 321--334.

[3]

Hong-Yu Chang, Wen-Jie Tseng, Chia-En Tsai, Hsin-Yu Chen, Roshan Lalintha Peiris, and Liwei Chan. 2018. FacePush: Introducing Normal Force on Face with Head-Mounted Displays. In The 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 927--935.

Digital Library

[4]

Carolina Cruz-Neira, Daniel J. Sandin, and Thomas A. DeFanti. 1993. Surround-screen Projection-based Virtual Reality: The Design and Implementation of the CAVE. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '93). ACM, New York, NY, USA, 135--142.

Digital Library

[5]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 2672--2680. http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf

Digital Library

[6]

Uwe Gruenefeld, Tim Claudius Stratmann, Wilko Heuten, and Susanne Boll. 2017. PeriMR: A Prototyping Tool for Head-mounted Peripheral Light Displays in Mixed Reality. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 51, 6 pages.

Digital Library

[7]

Jan Gugenheimer, Dennis Wolf, Eythor R. Eiriksson, Pattie Maes, and Enrico Rukzio. 2016. GyroVR: Simulating Inertia in Virtual Reality Using Head Worn Flywheels. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 227--232.

Digital Library

[8]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and Locally Consistent Image Completion. ACM Trans. Graph. 36, 4, Article 107 (July 2017), 14 pages.

Digital Library

[9]

P. Isola, J. Zhu, T. Zhou, and A. A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5967--5976.

[10]

ITU-R. 2007. Methodology for the subjective assessment of video quality in multimedia applications. Recommendation BT.1788-0 (Jan. 2007), 1--13.

[11]

Brett R. Jones, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2013. IllumiRoom: Peripheral Projected Illusions for Interactive Experiences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 869--878.

Digital Library

[12]

Naoki Kimura, Michinari Kono, and Jun Rekimoto. 2018. Using Deep-neural-network to Extend Videos for Head-mounted Display Experiences. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST '18). ACM, New York, NY, USA, Article 128, 2 pages.

Digital Library

[13]

Naoki Kimura and Jun Rekimoto. 2018. ExtVision: Augmentation of Visual Experiences with Generation of Context Images for a Peripheral Vision Using Deep Neural Network. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 427, 10 pages.

Digital Library

[14]

Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (201 4). arXiv:1412.6980 http://arxiv.org/abs/1412.6980

[15]

Koninklijke Philips N.V. 2015. Philips Ambilight TV. https://www.philips.co.uk/c-m-so/tv/p/ambilight

[16]

Koninklijke Philips N.V. 2016. Philips Ambilux. https://www.philips.com/c-cs/tv/ambilux.html

[17]

Michinari Kono, Takashi Miyaki, and Jun Rekimoto. 2018. In-pulse: Inducing Fear and Pain in Virtual Experiences. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST '18). ACM, New York, NY, USA, Article 40, 5 pages.

Digital Library

[18]

Bernhard Kratzwald, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool. 2017. Towards an Understanding of Our World by GANing Videos in the Wild. CoRR abs/1711.11453 (2017). arXiv:1711.11453 http://arxiv.org/abs/1711.11453

[19]

Yi-Chia Nina Lee, Li-Ting Shan, and Chien-Hsu Chen. 2013. System Development of Immersive Technology Theatre in Museum. In Virtual, Augmented and Mixed Reality. Systems and Applications, Randall Shumaker (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 400--408.

[20]

Yung-Ta Lin, Yi-Chi Liao, Shan-Yuan Teng, Yi-Ju Chung, Liwei Chan, and Bing-Yu Chen. 2017. Outside-In: Visualizing Out-of-Sight Regions-of-Interest in a 360° Video Using Spatial Picture-in-Picture Previews. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 255--265.

Digital Library

[21]

Paul Lubos, Gerd Bruder, Oscar Ariza, and Frank Steinicke. 2016. Ambiculus: LED-based Low-resolution Peripheral Display Extension for Immersive Head-mounted Displays. In Proceedings of the 2016 Virtual Reality International Conference (VRIC '16). ACM, New York, NY, USA, Article 13, 4 pages.

Digital Library

[22]

Mindprobe. 2014. CINEVEO. http://www.mindprobelabs.com/

[23]

Netflix, Inc. 2017. Netflix. www.netflix.com

[24]

Daniel E. Novy. 2013. Computational immersive displays. Master's thesis. Massachusetts Institute of Technology. Department of Architecture. Program in Media Arts and Sciences. http://hdl.handle.net/1721.1/82430

[25]

Oculus VR, Inc. 2016. Oculus Rift. https://www.oculus.com/rift/

[26]

Roshan Lalintha Peiris, Wei Peng, Zikun Chen, Liwei Chan, and Kouta Minamizawa. 2017. ThermoVR: Exploring Integrated Thermal Haptic Feedback with Head Mounted Displays. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 5452--5456.

Digital Library

[27]

Ismo Rakkolainen, Roope Raisamo, Matthew Turk, and Tobias Höllerer. 2017. Field-of-view Extension for VR Viewers. In Proceedings of the 21st International Academic Mindtrek Conference (AcademicMindtrek '17). ACM, New York, NY, USA, 227--230.

Digital Library

[28]

Ismo Rakkolainen, Matthew Turk, and Tobias Hollerer. 2016. A Compact, wide-FOV Optical Design for Head-mounted Displays. In Proceedings of the 22Nd ACM Conference on Virtual Reality Software and Technology (VRST '16). ACM, New York, NY, USA, 293--294.

Digital Library

[29]

Naomi B. Robbins and Richard M. Heiberger. 2011. Plotting Likert and Other Rating Scales, In Proceedings of the 2011 Joint Statistical Meeting. Section on Survey Research Methods, 1058--1066.

[30]

Ruth Rosenholtz. 2016. Capabilities and Limitations of Peripheral Vision. Annual Review of Vision Science 2, 1 (2016), 437--457. arXiv: 28532349.

[31]

Raunak Sinha, Mrinalini Hoon, Jacob Baudin, Haruhisa Okawa, Rachel O.L. Wong, and Fred Rieke. 2017. Cellular and Circuit Mechanisms Shaping the Perceptual Properties of the Primate Fovea. Cell 168, 3 (2017), 413 -- 426.e12.

[32]

L. Turban, F. Urban, and P. Guillotel. 2017. Extrafoveal Video Extension for an Immersive Viewing Experience. IEEE Transactions on Visualization and Computer Graphics 23, 5 (May 2017), 1520--1533.

Digital Library

[33]

Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. 2016. Generating Videos with Scene Dynamics. CoRR abs/1609.02612 (2016). arXiv:1609.02612 http://arxiv.org/abs/1609.02612

Digital Library

[34]

Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. 2019. Video Inpainting by Jointly Learning Temporal Structure and Spatial Details. In Proceedings of the 33th AAAI Conference on Artificial Intelligence.

[35]

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. Video-to-Video Synthesis. In Advances in Neural Information ProcessingSystems (NIPS).

Digital Library

[36]

Robert Xiao and Hrvoje Benko. 2016. Augmenting the Field-of-View of Head-Mounted Displays with Sparse Peripheral Displays. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1221--1232.

Digital Library

[37]

Wataru Yamada and Hiroyuki Manabe. 2016. Expanding the Field-of-View of Head-Mounted Displays with Peripheral Blurred Images. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16 Adjunct). ACM, New York, NY, USA, 141--142.

Digital Library

[38]

YouTube, LLC. 2005. Youtube. https://www.youtube.com/

Cited By

Šilić MSužnjević MSkorin-Kapov LSkorin-Kapov NLorenzana M(2024)The impact of video encoding parameters on QoE of simulated FPV drone controlMultimedia Tools and Applications10.1007/s11042-024-18442-283:28(71525-71557)Online publication date: 8-Feb-2024
https://doi.org/10.1007/s11042-024-18442-2
Traparic DLarabi MBellatreche L(2023)Towards Automatic Content Generation for Immersive Cinema Theater Based on Artificial Intelligence2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP59012.2023.10337685(1-6)Online publication date: 27-Sep-2023
https://doi.org/10.1109/MMSP59012.2023.10337685
Shirazi MUddin RKim M(2021)Supervised Learning Based Peripheral Vision System for Immersive Visual Experiences for Extended DisplayApplied Sciences10.3390/app1111472611:11(4726)Online publication date: 21-May-2021
https://doi.org/10.3390/app11114726
Show More Cited By

Index Terms

Deep dive: deep-neural-network-based video extension for immersive head-mounted display experiences
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Recommendations

Using deep-neural-network to extend videos for head-mounted display experiences
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

Immersion is an important factor in video experiences. Therefore, various methods and video viewing systems have been proposed so far. Although head-mounted displays (HMDs) are home-friendly and more available among these devices, they can provide an ...
Do virtual reality head-mounted displays make a difference? A comparison of presence and self-efficacy between head-mounted displays and desktop computer-facilitated virtual environments
Abstract
Virtual reality (VR) has made it possible for users to access novel digital experiences. An interesting question that arises in the context of VR is whether it appears or feels different to users when different virtual environments are used. This ...
Towards integration of user-centered designed tutorials for better virtual reality immersion
ICIGP '19: Proceedings of the 2nd International Conference on Image and Graphics Processing

Virtual reality (VR) has been contributing to education, health sciences and entertainment of late. The technology has made itself flexible enough to cater to users of different ages. Additionally head-mounted displays (HMDs) has led to the increase of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

PerDis '19: Proceedings of the 8th ACM International Symposium on Pervasive Displays

June 2019

223 pages

ISBN:9781450367516

DOI:10.1145/3321335

General Chairs:
Mohamed Khamis
University of Glasgow, United Kingdom
,
Salvatore Sorce
Università degli Studi di Palermo, Italy
,
Program Chairs:
Jessica R. Cauchard
Ben Gurion University of the Negev, Israel
,
Vito Gentile
Università degli Studi di Palermo, Italy

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

PerDis '19

PerDis '19: The 8th ACM International Symposium on Pervasive Displays

June 12 - 14, 2019

Palermo, Italy

Acceptance Rates

PerDis '19 Paper Acceptance Rate 26 of 67 submissions, 39%;

Overall Acceptance Rate 213 of 384 submissions, 55%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
244
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Šilić MSužnjević MSkorin-Kapov LSkorin-Kapov NLorenzana M(2024)The impact of video encoding parameters on QoE of simulated FPV drone controlMultimedia Tools and Applications10.1007/s11042-024-18442-283:28(71525-71557)Online publication date: 8-Feb-2024
https://doi.org/10.1007/s11042-024-18442-2
Traparic DLarabi MBellatreche L(2023)Towards Automatic Content Generation for Immersive Cinema Theater Based on Artificial Intelligence2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP59012.2023.10337685(1-6)Online publication date: 27-Sep-2023
https://doi.org/10.1109/MMSP59012.2023.10337685
Shirazi MUddin RKim M(2021)Supervised Learning Based Peripheral Vision System for Immersive Visual Experiences for Extended DisplayApplied Sciences10.3390/app1111472611:11(4726)Online publication date: 21-May-2021
https://doi.org/10.3390/app11114726
Endo ITakashima KInoue MFujita KKiyokawa KKitamura Y(2021)ModularHMD: A Reconfigurable Mobile Head-Mounted Display Enabling Ad-hoc Peripheral Interactions with the Real WorldThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474738(100-117)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474738
Endo ITakashima KInoue MFujita KKiyokawa KKitamura Y(2021)A Reconfigurable Mobile Head-Mounted Display Supporting Real World InteractionsExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411763.3451765(1-7)Online publication date: 8-May-2021
https://dl.acm.org/doi/10.1145/3411763.3451765

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents