research-article

ARchitect: Building Interactive Virtual Experiences from Physical Affordances by Bringing Human-in-the-Loop

Authors:

Xiaojuan MaAuthors Info & Claims

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Pages 1 - 13

https://doi.org/10.1145/3313831.3376614

Published: 23 April 2020 Publication History

Abstract

Automatic generation of Virtual Reality (VR) worlds which adapt to physical environments have been proposed to enable safe walking in VR. However, such techniques mainly focus on the avoidance of physical objects as obstacles and overlook their interaction affordances as passive haptics. Current VR experiences involving interaction with physical objects in surroundings still require verbal instruction from an assisting partner. We present ARchitect, a proof-of-concept prototype that allows flexible customization of a VR experience with human-in-the-loop. ARchitect brings in an assistant to map physical objects to virtual proxies of matching affordances using Augmented Reality (AR). In a within-subjects study (9 user pairs) comparing ARchitect to a baseline condition, assistants and players experienced decreased workload and players showed increased VR presence and trust in the assistant. Finally, we defined design guidelines of ARchitect for future designers and implemented three demonstrative experiences.

Supplementary Material

MP4 File (paper487pv.mp4)

Preview video

Download
11.58 MB

MP4 File (pn6922vf.mp4)

Supplemental video

Download
112.91 MB

References

[1]

2019. Keep Talking and Nobody Explodes. (2019). Retrieved June 15, 2019 from http://www.keeptalkinggame.com/presskit/sheet.php?p=keep_talking_and_nobody_explodes

[2]

2019. Touch Gesture Reference Guide. (2019). Retrieved July 6, 2019 from https://www.lukew.com/ff/entry.asp?1071

[3]

2019. The VOID | A Virtual Reality Experience. (2019). Retrieved June 15, 2019 from https://www.thevoid.com/

[4]

Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, and Matthias Niessner. 2019. Scan2CAD: Learning CAD Model Alignment in RGB-D Scans. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]

Sylvain Chagué and Caecilia Charbonnier. 2016. Real Virtuality: A Multi-user Immersive Platform Connecting Real and Virtual Worlds. In Proceedings of the 2016 Virtual Reality International Conference (VRIC '16). ACM, NY, NY, USA, Article 4, 3 pages.

Digital Library

[6]

Lung-Pan Cheng, Li Chang, Sebastian Marwecki, and Patrick Baudisch. 2018. iTurk: Turning Passive Haptics into Active Haptics by Making Users Reconfigure Props in Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 89, 10 pages.

Digital Library

[7]

Lung-Pan Cheng, Sebastian Marwecki, and Patrick Baudisch. 2017. Mutual Human Actuation. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, NY, NY, USA, 797--805.

Digital Library

[8]

Lung-Pan Cheng, E. Ofek, C. Holz, and A. D. Wilson. 2019. VRoamer: Generating On-The-Fly VR Experiences While Walking inside Large, Unknown Real-World Building Environments. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 359--366.

[9]

Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch, Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes Jasper, Jonas Kemper, and Patrick Baudisch. 2015. TurkDeck: Physical Virtual Reality Based on People. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, NY, NY, USA, 417--426.

Digital Library

[10]

Inrak Choi, Eyal Ofek, Hrvoje Benko, Mike Sinclair, and Christian Holz. 2018. CLAW: A Multifunctional Handheld Haptic Controller for Grasping, Touching, and Triggering in Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 654, 13 pages.

Digital Library

[11]

Angela Dai and Matthias Nießner. 2018. 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[12]

J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. (June 2009).

[13]

Deven Fosberry and Pratik Lakhotia. 2014. Playing ?The Floor is Lava' in Real Life. Journal of Interdisciplinary Science 3 (03 2014).

[14]

James Jerry Gibson. 2013. James J. Gibson From: the Ecological Approach to Visual Perception Chapter 8 the Theory of Affordances.

[15]

Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality Between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 4021--4033.

Digital Library

[16]

Jan Gugenheimer, Evgeny Stemasov, Harpreet Sareen, and Enrico Rukzio. 2018. FaceDisplay: Towards Asymmetric Multi-User Interaction for Nomadic Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 54, 13 pages.

Digital Library

[17]

Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139 -- 183.

[18]

Jeremy Hartmann, Christian Holz, Eyal Ofek, and Andrew D. Wilson. 2019. RealityCheck: Blending Virtual Environments with Situated Physical Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 347, 12 pages.

Digital Library

[19]

Zhenyi He, Fengyuan Zhu, Ken Perlin, and Xiaojuan Ma. 2018. Manifest the Invisible: Design for Situational Awareness of Physical Environments in Virtual Reality. arXiv e-prints (Sep 2018), arXiv:1809.05837.

[20]

Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects As Tangible Proxies in Augmented Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 1957--1967.

Digital Library

[21]

Christian Hirt, Markus Zank, and Andreas Kunz. 2017. Real-time Wall Outline Extraction for Redirected Walking. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (VRST '17). ACM, NY, NY, USA, Article 72, 2 pages.

Digital Library

[22]

H. G. Hoffman. 1998. Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments. In Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180). 59--63.

[23]

John Holmes and John Rempel. 1985. Trust in Close Relationships. Journal of Personality and Social Psychology 49 (07 1985).

[24]

Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv e-prints (Apr 2017), arXiv:1704.04861.

[25]

Hikaru Ibayashi, Yuta Sugiura, Daisuke Sakamoto, Natsuki Miyata, Mitsunori Tada, Takashi Okuma, Takeshi Kurata, Masaaki Mochimaru, and Takeo Igarashi. 2015. Dollhouse VR: A Multi-view, Multi-user Collaborative Design Workspace with VR Technology. In SIGGRAPH Asia 2015 Emerging Technologies (SA '15). ACM, NY, NY, USA, Article 8, 2 pages.

Digital Library

[26]

Brent Edward Insko. 2001. Passive Haptics Significantly Enhances Virtual Environments. Ph.D. Dissertation. Advisor(s) Brooks,Jr., Frederick P. AAI3007820.

[27]

Eun Kwon, Gerard J. Kim, and Sangyoon Lee. 2009. Effects of Sizes and Shapes of Props in Tangible Augmented Reality. In Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR '09). IEEE Computer Society, Washington, DC, USA, 201--202.

Digital Library

[28]

Jaeyeon Lee, Mike Sinclair, Mar Gonzalez-Franco, Eyal Ofek, and Christian Holz. 2019. TORC: A Virtual Reality Controller for In-Hand High-Dexterity Finger Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 71, 13 pages.

Digital Library

[29]

Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2014. Microsoft COCO: Common Objects in Context. arXiv e-prints (May 2014), arXiv:1405.0312.

[30]

R. W. Lindeman, J. L. Sibert, and J. K. Hahn. 1999. Hand-held windows: towards effective 2D interaction in immersive virtual environments. In Proceedings IEEE Virtual Reality (Cat. No. 99CB36316). 205--212.

[31]

Kok-Lim Low, Greg Welch, Anselmo Lastra, and Henry Fuchs. 2001. Life-sized Projector-based Dioramas. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '01). ACM, NY, NY, USA, 93--101.

Digital Library

[32]

Sebastian Marwecki and Patrick Baudisch. 2018. Scenograph: Fitting Real-Walking VR Experiences into Various Tracking Volumes. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, NY, NY, USA, 511--520.

Digital Library

[33]

Benjamin Nuernberger, Eyal Ofek, Hrvoje Benko, and Andrew D. Wilson. 2016. SnapToReality: Aligning Augmented Reality to the Real World. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 1233--1244.

Digital Library

[34]

Ohan Oda, Carmine Elvezio, Mengu Sukan, Steven Feiner, and Barbara Tversky. 2015. Virtual Replicas for Remote Assistance in Virtual and Augmented Reality. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, NY, NY, USA, 405--415.

Digital Library

[35]

J. Pair, U. Neumann, D. Piepol, and B. Swartout. 2003. FlatWorld: combining Hollywood set-design techniques with VR. IEEE Computer Graphics and Applications 23, 1 (Jan 2003), 12--15.

Digital Library

[36]

L. Shapira and D. Freedman. 2016. Reality Skins: Creating Immersive and Tactile Virtual Environments. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 115--124.

[37]

Adalberto L. Simeone, Eduardo Velloso, and Hans Gellersen. 2015. Substitutional Reality: Using the Physical Environment to Design Virtual Reality Experiences. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 3307--3316.

Digital Library

[38]

Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, and Gordon Wetzstein. 2018. Saliency in VR: How Do People Explore Virtual Environments? IEEE Transactions on Visualization and Computer Graphics 24, 4 (April 2018), 1633--1642.

Digital Library

[39]

Mel Slater, Martin Usoh, and Anthony Steed. 1995. Taking Steps: The Influence of a Walking Technique on Presence in Virtual Reality. ACM Trans. Comput.-Hum. Interact. 2, 3 (Sept. 1995), 201--219.

Digital Library

[40]

Shuran Song and Jianxiong Xiao. 2014. Sliding Shapes for 3D Object Detection in Depth Images. In Proceedings of the European Conference on Computer Vision (ECCV).

[41]

Misha Sra, Sergio Garrido-Jurado, and Pattie Maes. 2017. Oasis: Procedurally Generated Social Virtual Spaces from 3D Scanned Real Spaces. IEEE Transactions on Visualization and Computer Graphics PP (10 2017), 1--1.

[42]

Aaron Stafford, Wayne Piekarski, and Bruce Thomas. 2006. Implementation of God-like Interaction Techniques for Supporting Collaboration Between Outdoor AR and Indoor Tabletop Users. In Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR '06). IEEE Computer Society, Washington, DC, USA, 165--172.

Digital Library

[43]

Qi Sun, Anjul Patney, Li-Yi Wei, Omer Shapira, Jingwan Lu, Paul Asente, Suwen Zhu, Morgan Mcguire, David Luebke, and Arie Kaufman. 2018. Towards Virtual Reality Infinite Walking: Dynamic Saccadic Redirection. ACM Trans. Graph. 37, 4, Article 67 (July 2018), 13 pages.

Digital Library

[44]

C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1--9.

[45]

Martin Usoh, Kevin Arthur, Mary C. Whitton, Rui Bastos, Anthony Steed, Mel Slater, and Frederick P. Brooks, Jr. 1999. Walking --> Walking-in-place --> Flying, in Virtual Environments. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 359--364.

Digital Library

[46]

Martin Usoh, Ernest Catena, Sima Arman, and Mel Slater. 2000. Using Presence Questionnaires in Reality. Presence: Teleoper. Virtual Environ. 9, 5 (Oct. 2000), 497--503.

Digital Library

[47]

David H. D. Warren and Wallace T. Cleaves. 1971. Visual-proprioceptive interaction under large amounts of conflict. Journal of experimental psychology 90 2 (1971), 206--14.

Cited By

Çetin Er CÖzcan O(2024)Learning from Users: Everyday Playful Interactions to Support Architectural Spatial ChangesProceedings of the ACM on Human-Computer Interaction10.1145/36770858:CHI PLAY(1-25)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677085
Lee LLin Z(2024)Danger, Nuisance, Disregard: Analyzing User-Generated Videos for Augmented Reality Gameplay on Hand-held DevicesProceedings of the ACM on Human-Computer Interaction10.1145/36770638:CHI PLAY(1-33)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677063
Auda JGruenefeld UFaltaous SMayer SSchneegass S(2023)A Scoping Survey on Cross-reality SystemsACM Computing Surveys10.1145/361653656:4(1-38)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3616536
Show More Cited By

Index Terms

ARchitect: Building Interactive Virtual Experiences from Physical Affordances by Bringing Human-in-the-Loop
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces
      2. Virtual reality

Recommendations

Substitutional Reality: Using the Physical Environment to Design Virtual Reality Experiences
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Experiencing Virtual Reality in domestic and other uncontrolled settings is challenging due to the presence of physical objects and furniture that are not usually defined in the Virtual Environment. To address this challenge, we explore the concept of ...
Physical hand interaction for controlling multiple virtual objects in virtual reality
IWISC '18: Proceedings of the 3rd International Workshop on Interactive and Spatial Computing

Immersive virtual reality systems often strive to support realistic physical interaction through direct hand control, but it is not often possible to configure the physical environment to exactly match the virtual environment. We study methods that ...
VR Grabbers: Ungrounded Haptic Retargeting for Precision Grabbing Tools
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Haptic feedback in VR is important for realistic simulation in virtual reality. However, recreating the haptic experience for hand tools in VR traditionally requires hardware with precise actuators, adding complexity to the system. We propose Ungrounded ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

April 2020

10688 pages

ISBN:9781450367080

DOI:10.1145/3313831

General Chairs:
Regina Bernhaupt
Eindhoven University of Technology, Netherlands
,
Florian 'Floyd' Mueller
Monash University, Australia
,
David Verweij
Newcastle University, UK
,
Josh Andres
RMIT, Australia
,
Program Chairs:
Joanna McGrenere
University of British Columbia, Canada
,
Andy Cockburn
University of Canterbury, New Zealand
,
Ignacio Avellino
University of Maryland Baltimore County, USA
,
Alix Goguey
Grenoble Alpes University, France
,
Pernille Bjørn
University of Copenhagen, Denmark
,
Shengdong (Shen) Zhao
National University of Singapore, Singapore
,
Briane Paul Samson
Future University Hakodate, Japan & De La Salle University, Philippines
,
Rafal Kocielnik
University of Washington, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Research Grants Council of the Hong Kong SAR China

Conference

CHI '20

Sponsor:

SIGCHI

CHI '20: CHI Conference on Human Factors in Computing Systems

April 25 - 30, 2020

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
1,305
Total Downloads

Downloads (Last 12 months)151
Downloads (Last 6 weeks)13

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Çetin Er CÖzcan O(2024)Learning from Users: Everyday Playful Interactions to Support Architectural Spatial ChangesProceedings of the ACM on Human-Computer Interaction10.1145/36770858:CHI PLAY(1-25)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677085
Lee LLin Z(2024)Danger, Nuisance, Disregard: Analyzing User-Generated Videos for Augmented Reality Gameplay on Hand-held DevicesProceedings of the ACM on Human-Computer Interaction10.1145/36770638:CHI PLAY(1-33)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677063
Auda JGruenefeld UFaltaous SMayer SSchneegass S(2023)A Scoping Survey on Cross-reality SystemsACM Computing Surveys10.1145/361653656:4(1-38)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3616536
Cheng YGebhardt CHolz C(2023)InteractionAdapt: Interaction-driven Workspace Adaptation for Situated Virtual Reality EnvironmentsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606717(1-14)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606717
Rajaram SChen CRoesner FNebeling M(2023)Eliciting Security & Privacy-Informed Sharing Techniques for Multi-User Augmented RealityProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581089(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581089
Wu XHung HBabu SChuang J(2023)Novel Design and Evaluation of Redirection Controllers Using Optimized Alignment and Artificial Potential FieldIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332020829:11(4556-4566)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3320208
Matsumuro MMori SKataoka YIgarashi FShibata FKimura A(2023)Modified Egocentric Viewpoint for Softer Seated Experience in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324705629:5(2230-2238)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247056
Wang CChen BChan L(2022)RealityLens: A User Interface for Blending Customized Physical World View into Virtual RealityProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545686(1-11)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545686
Beever LJohn N(2022)LevelEd SR: A Substitutional Reality Level Design Workflow2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)10.1109/VR51125.2022.00031(130-138)Online publication date: Mar-2022
https://doi.org/10.1109/VR51125.2022.00031
Kwon SJeon SHwang JCho YPark JLee I(2022)Infinite Virtual Space Exploration Using Space Tiling and Perceivable Reset at Fixed Positions2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR55827.2022.00094(758-767)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR55827.2022.00094
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents