research-article

One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities

Authors:

Martin HachetAuthors Info & Claims

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

Pages 787 - 795

https://doi.org/10.1145/3126594.3126638

Published: 20 October 2017 Publication History

Abstract

Most of our daily activities take place in the physical world, which inherently imposes physical constraints. In contrast, the digital world is very flexible, but usually isolated from its physical counterpart. To combine these two realms, many Mixed Reality (MR) techniques have been explored, at different levels in the continuum. In this work we present an integrated Mixed Reality ecosystem that allows users to incrementally transition from pure physical to pure virtual experiences in a unique reality. This system stands on a conceptual framework composed of 6 levels. This paper presents these levels as well as the related interaction techniques.

Supplementary Material

suppl.mov (uistf3584-file3.mp4)

Supplemental video

Download
68.83 MB

References

[1]

Dragomir Anguelov, Carole Dulong, Daniel filip, Christian Frueh, Stéphane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, and Josh Weaver. 2010. Google street view: Capturing the world at street level. Computer 43, 6 (2010), 32--38.

Digital Library

[2]

Albert Barille. 1987. Once Upon a Time... Life. (1987).

[3]

Hrvoje Benko, Edward W Ishak, and Steven Feiner. 2004. Collaborative mixed reality visualization of an archaeological excavation. In Proceedings of the 3rd IEEE/ACM international Symposium on Mixed and Augmented Reality. IEEE Computer Society, 132--140.

Digital Library

[4]

Hrvoje Benko, Eyal Ofek, Feng Zheng, and Andrew D Wilson. 2015. Fovear: Combining an optically see-through near-eye display with projector-based spatial augmented reality. In UIST '15. ACM, 129--135.

Digital Library

[5]

Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. 2001. The magicbook-moving seamlessly between reality and virtuality. IEEE CGandA 21, 3 (2001), 6--8.

Digital Library

[6]

Oliver Bimber and Ramesh Raskar. 2006. Modern approaches to augmented reality. In ACM SIGGRAPH 2006 Courses. ACM, 1.

Digital Library

[7]

Mark Blythe. 2014. Research through design fiction: narrative in real and imaginary abstracts. In CHI '14. ACM, 703--712.

Digital Library

[8]

Sophia Sophia Agnes Brueckner. 2014. Out of network: technologies to connect with strangers. Ph.D. Dissertation. MA Institute of Technology.

[9]

D Alex Butler, Shahram Izadi, Otmar Hilliges, David Molyneaux, Steve Hodges, and David Kim. 2012. Shake 'n' sense: reducing interference for overlapping structured light depth cameras. In CHI '12. ACM, 1933--1936.

Digital Library

[10]

Andreas Butz, Tobias Hollerer, Steven Feiner, Blair MacIntyre, and Clifford Beshers. 1999. Enveloping users and computers in a collaborative 3D augmented reality. In Augmented Reality, 1999.(IWAR'99) Proceedings. 2nd IEEE and ACM International Workshop on. IEEE, 35--44.

Digital Library

[11]

Joanna Cole and Bruce Degen. 1994. The Magic School Bus. (1994).

[12]

Carolina Cruz-Neira, Daniel J Sandin, and Thomas A DeFanti. 1993. Surround-screen projection-based virtual reality: the design and implementation of the CAVE. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques. ACM, 135--142.

Digital Library

[13]

Nicolas J Dedual, Ohan Oda, and Steven K Feiner. 2011. Creating hybrid user interfaces with a 2D multi-touch tabletop and a 3D see-through head-worn display. In ISMAR 2011. IEEE, 231--232.

Digital Library

[14]

Mike Eissele, Oliver Siemoneit, and Thomas Ertl. 2006. Transition of mixed, virtual, and augmented reality in smart production environments-an interdisciplinary view. In Robotics, Automation and Mechatronics, 2006 IEEE Conference on. IEEE, 1--6.

[15]

Renaud Gervais, Joan Sol Roo, and Martin Hachet. 2016. Tangible Viewports: Getting Out of Flatland in Desktop Environments. In TEI'16. ACM, 176--184.

Digital Library

[16]

Valentin Heun, Shunichi Kasahara, and Pattie Maes. 2013. Smarter objects: using AR technology to program physical objects and their interactions. In CHI'13 Extended Abstracts. ACM, 961--966.

Digital Library

[17]

David Holman and Roel Vertegaal. 2008. Organic User Interfaces: Designing Computers in Any Way, Shape, or Form. Commun. ACM 51, 6 (June 2008), 48--55.

Digital Library

[18]

David Holman, Roel Vertegaal, Mark Altosaar, Nikolaus Troje, and Derek Johns. 2005. Paper windows: interaction techniques for digital paper. In CHI '05. ACM, 591--599.

Digital Library

[19]

Hikaru Ibayashi, Yuta Sugiura, Daisuke Sakamoto, Natsuki Miyata, Mitsunori Tada, Takashi Okuma, Takeshi Kurata, Masaaki Mochimaru, and Takeo Igarashi. 2015. Dollhouse VR: a multi-view, multi-user collaborative design workspace with VR technology. In SIGGRAPH Asia 2015 Emerging Technologies. ACM, 8.

Digital Library

[20]

Adrian Ilie, Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs, and Bruce Cairns. 2004. Combining head-mounted and projector-based displays for surgical training. Presence: Teleoperators and Virtual Environments 13, 2 (2004), 128--145.

Digital Library

[21]

Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune. 2012. Radical atoms: beyond tangible bits, toward transformable materials. interactions 19, 1 (2012), 38--51.

Digital Library

[22]

Robert JK Jacob, Audrey Girouard, Leanne M Hirshfield, Michael S Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-based interaction: a framework for post-WIMP interfaces. In CHI '08. ACM, 201--210.

Digital Library

[23]

Brett Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, and Lior Shapira. 2014. RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units. In UIST '14. ACM, 637--644.

Digital Library

[24]

Shunichi Kasahara, Ryuma Niiyama, Valentin Heun, and Hiroshi Ishii. 2013. exTouch: Spatially-aware Embodied Manipulation of Actuated Objects Mediated by Augmented Reality. In TEI '13 (TEI '13). ACM, New York, NY, USA, 223--228.

Digital Library

[25]

Kiyoshi Kiyokawa, Haruo Takemura, and Naokazu Yokoya. 2000. Seamless Design for 3D object creation. IEEE multimedia 7, 1 (2000), 22--33.

Digital Library

[26]

Ryohei Komiyama, Takashi Miyaki, and Jun Rekimoto. 2017. JackIn Space: Designing a Seamless Transition Between first and Third Person View for Effective Telepresence Collaborations. In Proceedings of the 8th Augmented Human International Conference (AH '17). ACM, New York, NY, USA, Article 14, 9 pages.

Digital Library

[27]

Jérémy Laviole and Martin Hachet. 2012. PapARt: interactive 3D graphics and multi-touch augmented paper for artistic creation. In 3D User Interfaces (3DUI), 2012 IEEE Symposium on. IEEE, 3--6.

[28]

Johnny C Lee, Scott E Hudson, Jay W Summet, and Paul H Dietz. 2005. Moveable interactive projected displays using projector based tracking. In UIST '05. ACM, 63--72.

Digital Library

[29]

Michael R Marner, Bruce H Thomas, and Christian Sandor. 2009. Physical-virtual tools for spatial augmented reality user interfaces. In ISMAR, Vol. 9.

Digital Library

[30]

Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. 1995. Augmented reality: A class of displays on the reality-virtuality continuum. In Photonics for industrial applications. International Society for Optics and Photonics, 282--292.

[31]

Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition.

[32]

Ben Piper, Carlo Ratti, and Hiroshi Ishii. 2002. Illuminating clay: a 3-D tangible interface for landscape analysis. In CHI '02. ACM, 355--362.

Digital Library

[33]

Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. 1998b. The office of the future: A unified approach to image-based modeling and spatially immersive displays. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 179--188.

Digital Library

[34]

Ramesh Raskar, Greg Welch, and Henry Fuchs. 1998a. Spatially augmented reality. In first IEEE Workshop on Augmented Reality (IWAR '98). Citeseer, 11--20.

Digital Library

[35]

Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay. 2001. Shader lamps: Animating real objects with image-based illumination. In Rendering Techniques 2001. Springer, 89--102.

Digital Library

[36]

Jun Rekimoto and Katashi Nagao. 1995. The world through the computer: Computer augmented interaction with real world environments. In UIST '95. ACM, 29--36.

Digital Library

[37]

Jun Rekimoto and Masanori Saitoh. 1999. Augmented surfaces: a spatially continuous work space for hybrid computing environments. In CHI'99. ACM, 378--385.

Digital Library

[38]

Joan Sol Roo, Renaud Gervais, Jérémy Frey, and Martin Hachet. 2017. Inner Garden: Connecting Inner States to a Mixed Reality Sandbox for Mindfulness. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1459--1470.

Digital Library

[39]

Joan Sol Roo and Martin Hachet. 2017. Towards a hybrid space combining Spatial Augmented Reality and virtual reality. In 3DUI'17. IEEE, 195--198.

[40]

Carl Sagan, Steven Soter, and Ann Druyan. 1989. Cosmos: A personal voyage. (1989).

[41]

Ari Shapiro, Andrew Feng, Ruizhe Wang, Hao Li, Mark Bolas, Gerard Medioni, and Evan Suma. 2014. Rapid avatar capture and simulation using commodity depth sensors. Computer Animation and Virtual Worlds 25 (2014).

Digital Library

[42]

Ross T Smith, Guy Webber, Maki Sugimoto, Michael Marner, and Bruce H Thomas. 2013. {Invited Paper} Automatic Sub-pixel Projector Calibration. ITE Transactions on Media Technology and Applications 1, 3 (2013), 204--213.

[43]

Ivan E Sutherland. 1965. The ultimate display. Multimedia: From Wagner to virtual reality (1965).

[44]

Brygg Ullmer and Hiroshi Ishii. 1997. The metaDESK: models and prototypes for tangible user interfaces. In UIST '97. ACM, 223--232.

Digital Library

[45]

Brygg Ullmer and Hiroshi Ishii. 2000. Emerging frameworks for tangible user interfaces. IBM systems journal 39, 3.4 (2000), 915--931.

Digital Library

[46]

Bret Victor. 2014. Humane Representation of Thought: A Trail Map for the 21st Century. In UIST '14. ACM, New York, NY, USA, 1.

Digital Library

[47]

James A Walsh, Stewart von Itzstein, and Bruce H Thomas. 2014. Ephemeral interaction using everyday objects. In Proceedings of the fifteenth Australasian User Interface Conference-Volume 150. Australian Computer Society, Inc., 29--37.

Digital Library

[48]

Mark Weiser. 1993. Some computer science issues in ubiquitous computing. Commun. ACM 36, 7 (1993).

Digital Library

[49]

Mark Weiser and John Seely Brown. 1997. The coming age of calm technology. In Beyond calculation. Springer.

[50]

Pierre Wellner. 1993. Interacting with Paper on the DigitalDesk. Commun. ACM 36, 7 (July 1993), 87--96.

Digital Library

[51]

Andrew Wilson, Hrvoje Benko, Shahram Izadi, and Otmar Hilliges. 2012. Steerable augmented reality with the beamatron. In UIST '12. ACM, 413--422.

Digital Library

[52]

Robert Xiao, Chris Harrison, and Scott E. Hudson. 2013. WorldKit: Rapid and Easy Creation of Ad-hoc Interactive Applications on Everyday Surfaces. In CHI '13 (CHI '13). ACM, New York, NY, USA, 879--888.

Digital Library

[53]

Robert Xiao, Chris Harrison, and Scott E Hudson. 2013. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In CHI '13. ACM, 879--888.

Digital Library

[54]

Jianlong Zhou, Ivan Lee, Bruce Thomas, Roland Menassa, Anthony Farrant, and Andrew Sansome. 2011. Applying Spatial Augmented Reality to Facilitate In-situ Support for Automotive Spot Welding Inspection. In Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI '11). ACM, New York, NY, USA, 195--200.

Digital Library

Cited By

Tavernaro ASant’Ana MMarti HAlmeida JSoares L(2024)Shared Reconstructed Environments in Extended RealityProceedings of the 26th Symposium on Virtual and Augmented Reality10.1145/3691573.3691590(161-167)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3691573.3691590
Chiossi FGruenefeld UHou BNewn JOu CLiao RWelsch RMayer S(2024)Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Using Fixation-Related Potentials and Eye Tracking FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36765288:MHCI(1-33)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676528
Mayer EChiossi FMayer S(2024)Crossing Mixed Realities: A Review for Transitional Interfaces DesignProceedings of Mensch und Computer 202410.1145/3670653.3677481(629-634)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1145/3670653.3677481
Show More Cited By

Index Terms

One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Information systems
  1. Information systems applications
    1. Multimedia information systems

Recommendations

Understanding Users' Capability to Transfer Information between Mixed and Virtual Reality: Position Estimation across Modalities and Perspectives
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Mixed Reality systems combine physical and digital worlds, with great potential for the future of HCI. It is possible to design systems that support flexible degrees of virtuality by combining complementary technologies. In order for such systems to ...
Garden: A Mixed Reality Experience Combining Virtual Reality and 3D Reconstruction
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

Garden is a Mixed Reality (MR) experience that combines both Virtual Reality (VR) and Augmented Reality (AR), and lets players transform their environment into a virtual garden they can play in. This is done by doing both stereoscopic rendering and 3D ...
Merging environments for shared spaces in mixed reality
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

In virtual reality a real walking interface limits the extent of a virtual environment to our local walkable space. As local spaces are specific to each user, sharing a virtual environment with others for collaborative work or games becomes complicated. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

October 2017

870 pages

ISBN:9781450349819

DOI:10.1145/3126594

General Chair:
Krzysztof Gajos
Harvard University
,
Program Chairs:
Jennifer Mankoff
Carnegie Mellon University
,
Chris Harrison
Carnegie Mellon University

Copyright © 2017 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

ANR

Conference

UIST '17

Sponsor:

UIST '17: The 30th Annual ACM Symposium on User Interface Software and Technology

October 22 - 25, 2017

QC, Québec City, Canada

Acceptance Rates

UIST '17 Paper Acceptance Rate 73 of 324 submissions, 23%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

79
Total Citations
View Citations
2,215
Total Downloads

Downloads (Last 12 months)199
Downloads (Last 6 weeks)24

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tavernaro ASant’Ana MMarti HAlmeida JSoares L(2024)Shared Reconstructed Environments in Extended RealityProceedings of the 26th Symposium on Virtual and Augmented Reality10.1145/3691573.3691590(161-167)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3691573.3691590
Chiossi FGruenefeld UHou BNewn JOu CLiao RWelsch RMayer S(2024)Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Using Fixation-Related Potentials and Eye Tracking FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36765288:MHCI(1-33)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676528
Mayer EChiossi FMayer S(2024)Crossing Mixed Realities: A Review for Transitional Interfaces DesignProceedings of Mensch und Computer 202410.1145/3670653.3677481(629-634)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1145/3670653.3677481
Cools RBalcı ONguyen BVande Moere ASimeone A(2024)How a Vertical Surface Supports Cross-Reality Transitional Interface Tasks at Different Virtuality LevelsProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656762(1-5)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656762
Wang NZielasko DMaurer F(2024)User Preferences for Interactive 3D Object Transitions in Cross Reality - An Elicitation StudyProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656698(1-9)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656698
Muff FFill HCombemale BWimmer MChechik MEgyed A(2024)M2AR: A Web-based Modeling Environment for the Augmented Reality Workflow Modeling LanguageProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3687779(1-5)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3652620.3687779
Chang RHung CChen BJain DGuo A(2024)SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality AwarenessProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661556(116-132)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661556
Zhou QSyiem BLi BGoncalves JVelloso E(2024)Reflected RealityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314317:4(1-28)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631431
Zhang ZZhang ZJiao ZSu YLiu HWang WZhu S(2024)On the Emergence of Symmetrical Reality2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00084(639-649)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00084
Lenz LFender AChatain JHolz C(2024)Comparing Synchronous and Asynchronous Task Delivery in Mixed Reality EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337203430:5(2776-2784)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372034
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents