research-article

ExtVision: Augmentation of Visual Experiences with Generation of Context Images for a Peripheral Vision Using Deep Neural Network

Authors:

Jun RekimotoAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 427, Pages 1 - 10

https://doi.org/10.1145/3173574.3174001

Published: 21 April 2018 Publication History

Abstract

We propose a system, called ExtVision, to augment visual experiences by generating and projecting context-images onto the periphery of the television or computer screen. A peripheral projection of the context-image is one of the most effective techniques to enhance visual experiences. However, the projection is not commonly used at present, because of the difficulty in preparing the context-image. In this paper, we propose a deep neural network-based method to generate context-images for peripheral projection. A user study was performed to investigate the manner in which the proposed system augments traditional visual experiences. In addition, we present applications and future prospects of the developed system.

Supplementary Material

suppl.mov (pn3578-file3.mp4)

Supplemental video

Download
20.77 MB

suppl.mov (pn3578-file5.mp4)

Supplemental video

Download
15.82 MB

References

[1]

Amit Aides, Tamar Avraham, and Yoav Y. Schechner. 2011. Multiscale ultrawide foveated video extrapolation. In 2011 IEEE International Conference on Computational Photography (ICCP), 1--8.

[2]

Tamar Avraham and Yoav Y. Schechner. 2011. Ultrawide Foveated Video Extrapolation. IEEE Journal of Selected Topics in Signal Processing 5, 2: 321--334.

[3]

Patrick Baudisch, Nathaniel Good, and Paul Stewart. 2001. Focus plus context screens. In Proceedings of the 14th annual ACM symposium on User interface software and technology - UIST '01, 31.

Digital Library

[4]

Carolina Cruz-Neira, Daniel J. Sandin, and Thomas A. DeFanti. 1993. Surround-screen projection-based virtual reality. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques - SIGGRAPH '93, 135--142.

Digital Library

[5]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Figure 11. Percentage of each answer to some questions of "Drama". Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. Advances in Neural Information Processing Systems 27: 2672--2680.

Digital Library

[6]

Yixiang Huang, Xuan F. Zha, Jay Lee, and Chengliang Liu. 2013. Discriminant diffusion maps analysis: A robust manifold learner for dimensionality reduction and its applications in machine condition monitoring and fault diagnosis. Mechanical Systems and Signal Processing 34, 1--2: 277--297.

[7]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and Locally Consistent Image Completion. ACM Transactions on Graphics ACM Trans. Graph. Article 36, 13.

Digital Library

[8]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics (Proc. of SIGGRAPH 2016) 35, 4.

Digital Library

[9]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5967--5976.

[10]

Brett R. Jones, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2013. IllumiRoom: Peripheral Projected Illusions for Interactive Experiences. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13: 869.

Digital Library

[11]

Mehdi Mirza and Simon Osindero. 2014. Conditional Generative Adversarial Nets. Retrieved January 9, 2018 from http://arxiv.org/abs/1411.1784

[12]

Daniel E Novy. 2013. COMPUTATIONAL IMMERSIVE DISPLAYS. Retrieved January 9, 2018 from http://excedrin.media.mit.edu/wpcontent/uploads/sites/10/2013/07/novyms.pdf

[13]

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context Encoders: Feature Learning by Inpainting.

[14]

Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.

[15]

Recommendation Itu-. Methodology for the subjective assessment of video quality in multimedia applications. Retrieved January 9, 2018 from https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-RECBT.1788-0--200701-I!!PDF-E.pdf

[16]

Laura Turban, Fabrice Urban, and Philippe Guillotel. 2017. Extrafoveal Video Extension for an Immersive Viewing Experience. IEEE Transactions on Visualization and Computer Graphics 23, 5: 1520-- 1533.

Digital Library

[17]

Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum.

Digital Library

[18]

Kentaro Yoshida, Seki Inoue, Yasutoshi Makino, and Hiroyuki Shinoda. 2017. VibVid: VIBration Estimation from VIDeo by using Neural Network. 37--44.

[19]

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence: 1--1.

[20]

SCREENX. Retrieved January 9, 2018 from https://screenx.co.kr/

[21]

Ready 2 Escape -- The ultimate immersive cinema experience. Retrieved January 9, 2018 from https://ready2escape.com/

[22]

Philips TV. Experience Ambilight | Philips. Retrieved January 9, 2018 from https://www.philips.co.uk/c-mso/televisions/p/ambilight

[23]

Phillipi. 2017. phillipi/pix2pix. (December 2017). Retrieved January 8, 2018 from https://github.com/phillipi/pix2pix

Cited By

Takami TSaito TKameoka TKajimoto H(2023)ExtEdge: Haptic Augmentation of Visual Experiences of a Smartphone by Electro-Tactile Sensation Through the EdgesProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614543(1-8)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1145/3607822.3614543
Hirzle TMüller FDraxler FSchmitz MKnierim PHornbæk K(2023)When XR and AI Meet - A Scoping Review on Extended Reality and Artificial IntelligenceProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581072(1-45)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581072
Traparic DLarabi MBellatreche L(2023)Towards Automatic Content Generation for Immersive Cinema Theater Based on Artificial Intelligence2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP59012.2023.10337685(1-6)Online publication date: 27-Sep-2023
https://doi.org/10.1109/MMSP59012.2023.10337685
Show More Cited By

Index Terms

ExtVision: Augmentation of Visual Experiences with Generation of Context Images for a Peripheral Vision Using Deep Neural Network
1. Information systems
  1. Information systems applications
    1. Multimedia information systems

Recommendations

Using deep-neural-network to extend videos for head-mounted display experiences
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

Immersion is an important factor in video experiences. Therefore, various methods and video viewing systems have been proposed so far. Although head-mounted displays (HMDs) are home-friendly and more available among these devices, they can provide an ...
Deep dive: deep-neural-network-based video extension for immersive head-mounted display experiences
PerDis '19: Proceedings of the 8th ACM International Symposium on Pervasive Displays

Immersion is an important factor in video experiences. Therefore, various methods and video viewing systems have been proposed. Head-mounted displays (HMDs) are home-friendly pervasive devices, which can provide an immersive video experience owing to ...
Comparative Study of HMD-based Virtual and Augmented Realities for Immersive Museums: User Acceptance, Medium, and Learning
Technologies like Head Mounted Display (HMD)-based Virtual Reality (VR) and Augmented Reality (AR) have made HMD-based immersive museums possible. To investigate the user acceptance, medium, and interaction experience for HMD-based immersive museums, an ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
1,467
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)9

Reflects downloads up to 26 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Takami TSaito TKameoka TKajimoto H(2023)ExtEdge: Haptic Augmentation of Visual Experiences of a Smartphone by Electro-Tactile Sensation Through the EdgesProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614543(1-8)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1145/3607822.3614543
Hirzle TMüller FDraxler FSchmitz MKnierim PHornbæk K(2023)When XR and AI Meet - A Scoping Review on Extended Reality and Artificial IntelligenceProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581072(1-45)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581072
Traparic DLarabi MBellatreche L(2023)Towards Automatic Content Generation for Immersive Cinema Theater Based on Artificial Intelligence2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP59012.2023.10337685(1-6)Online publication date: 27-Sep-2023
https://doi.org/10.1109/MMSP59012.2023.10337685
Chatterjee JVega M(2023)360-GAN: Cycle-Consistent GAN for Extrapolating 360-Degree Field-of-View2023 11th European Workshop on Visual Information Processing (EUVIP)10.1109/EUVIP58404.2023.10323054(1-6)Online publication date: 11-Sep-2023
https://doi.org/10.1109/EUVIP58404.2023.10323054
Inami MUriu DKashino ZYoshida SSaito HMaekawa AKitazaki M(2022)Cyborgs, Human Augmentation, Cybernetics, and JIZAI BodyProceedings of the Augmented Humans International Conference 202210.1145/3519391.3519401(230-242)Online publication date: 13-Mar-2022
https://dl.acm.org/doi/10.1145/3519391.3519401
Saeghe PWeir BMcGill MClinch SStevens R(2022)Augmenting a Nature Documentary with a Lifelike Hologram in Virtual RealityProceedings of the 2022 ACM International Conference on Interactive Media Experiences10.1145/3505284.3532974(275-280)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3505284.3532974
Pamparău CVatavu R(2022)The User Experience of Journeys in the Realm of Augmented Reality TelevisionProceedings of the 2022 ACM International Conference on Interactive Media Experiences10.1145/3505284.3529969(161-174)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3505284.3529969
Saeghe PMcGill MWeir BClinch SStevens R(2022)Evaluating and Updating a Design Space for Augmented Reality TelevisionProceedings of the 2022 ACM International Conference on Interactive Media Experiences10.1145/3505284.3529965(79-94)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3505284.3529965
Hara TMukuta YHarada T(2022)Spherical Image Generation From a Few Normal-Field-of-View Images by Considering Scene SymmetryIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.3215933(1-15)Online publication date: 2022
https://doi.org/10.1109/TPAMI.2022.3215933
Dastjerdi MHold-Geoffroy YEisenmann JKhodadadeh SLalonde J(2022)Guided Co-Modulated GAN for 360° Field of View Extrapolation2022 International Conference on 3D Vision (3DV)10.1109/3DV57658.2022.00059(475-485)Online publication date: Sep-2022
https://doi.org/10.1109/3DV57658.2022.00059
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents