poster

Public Access

VRFromX: From Scanned Reality to Interactive Virtual Experience with Human-in-the-Loop

Authors:

Subramanian Chidambaram,

Karthik RamaniAuthors Info & Claims

CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 289, Pages 1 - 7

https://doi.org/10.1145/3411763.3451747

Published: 08 May 2021 Publication History

All formats PDF

Abstract

There is an increasing trend of Virtual-Reality (VR) applications found in education, entertainment, and industry. Many of them utilize real world tools, environments, and interactions as bases for creation. However, creating such applications is tedious, fragmented, and involves expertise in authoring VR using programming and 3D-modelling softwares. This hinders VR adoption by decoupling subject matter experts from the actual process of authoring while increasing cost and time. We present VRFromX, an in-situ Do-It-Yourself (DIY) platform for content creation in VR that allows users to create interactive virtual experiences. Using our system, users can select region(s) of interest (ROI) in scanned point cloud or sketch in mid-air using a brush tool to retrieve virtual models and then attach behavioral properties to them. We ran an exploratory study to evaluate usability of VRFromX and the results demonstrate feasibility of the framework as an authoring tool. Finally, we implemented three possible use-cases to showcase potential applications.

Supplemental Material

MP4 File

Supplemental video

Transcript for: Supplemental video

ZIP File

Supplemental material

Download
3.63 MB

References

[1]

Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey. 2019. PointNetLK: Robust & efficient point cloud registration using PointNet. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Long Beach, CA). IEEE, New York, NY, USA, 7163–7172.

[2]

Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, and Matthias Nießner. 2019. Scan2CAD: Learning CAD Model Alignment in RGB-D Scans. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE (Long Beach, CA). IEEE, New York, NY, USA, 2609–2618.

[3]

Esther Z Barsom, Maurits Graafland, and Marlies P Schijven. 2016. Systematic review on the effectiveness of augmented reality applications in medical training. Surgical endoscopy 30, 10 (2016), 4174–4183.

[4]

John Brooke 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.

[5]

Kang Chen, Yu-Kun Lai, and Shi-Min Hu. 2015. 3D indoor scene modeling from RGB-D data: a survey. Computational Visual Media 1, 4 (2015), 267–278.

[6]

Lung-Pan Cheng, Eyal Ofek, Christian Holz, and Andrew D Wilson. 2019. VRoamer: Generating On-The-Fly VR Experiences While Walking inside Large, Unknown Real-World Building Environments. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (Osaka, Japan). IEEE, New York, NY, USA, 359–366.

[7]

Brent Cowan and Bill Kapralos. 2014. A survey of frameworks and game engines for serious game development. In 2014 IEEE 14th International Conference on Advanced Learning Technologies (Athens, Greece). IEEE, New York, NY, USA, 662–664.

Digital Library

[8]

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, 2020. DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology(Minneapolis, Minnesota, USA). ACM, New York, NY, USA, 829–843.

Digital Library

[9]

Barrett Ens, Fraser Anderson, Tovi Grossman, Michelle Annett, Pourang Irani, and George Fitzmaurice. 2017. Ivy: Exploring spatially situated visual programming for authoring and understanding intelligent environments. In Proceedings of the 43rd Graphics Interface Conference. CHCCS/SCDHM, Edmonton, Alberta, 156–162.

[10]

James J Gibson. 1977. The theory of affordances. Hilldale, USA 1, 2 (1977), 67–82.

[11]

Daniele Giunchi, Stuart James, and Anthony Steed. 2018. 3D sketching for interactive model retrieval in virtual reality. In Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering. ACM, New York, NY, United States, 1–12.

Digital Library

[12]

Hirokazu Kato and Mark Billinghurst. 1999. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99) (San Francisco, CA, USA). IEEE, New York, NY, USA, 85–94.

[13]

Ashish Khurana, Bhanu Prakash Lohani, and Vimal Bibhu. 2019. AI Frame-Worked Virtual World Application-The Ramification of Virtual World on Real World. In 2019 International Conference on Automation, Computational and Technology Management (ICACTM) (London, UK). IEEE, New York, NY, USA, 582–585.

[14]

Konstantina Kilteni, Raphaela Groten, and Mel Slater. 2012. The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments 21, 4(2012), 373–387.

Digital Library

[15]

Laan Labs. 2020. 3D Scanner App. https://www.3dscannerapp.com/.

[16]

Valerie D Lehner and Thomas A DeFanti. 1997. Distributed virtual reality: Supporting remote collaboration in vehicle design. IEEE Computer Graphics and Applications 17, 2 (1997), 13–17.

Digital Library

[17]

Lingxiao Li, Minhyuk Sung, Anastasia Dubrovina, Li Yi, and Leonidas J Guibas. 2019. Supervised fitting of geometric primitives to 3D point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Long Beach, CA). IEEE, New York, NY, USA, 2652–2660.

[18]

Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018. PointCNN: Convolution on X-transformed points. In Advances in neural information processing systems. NIPS 2018, Montreal, Canada, 820–830.

[19]

Chuan-en Lin, Ta Ying Cheng, and Xiaojuan Ma. 2020. ARchitect: Building Interactive Virtual Experiences from Physical Affordances by Bringing Human-in-the-Loop. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI). ACM, New York, NY, USA, 1–13.

Digital Library

[20]

David Lindlbauer and Andy D Wilson. 2018. Remixed reality: manipulating space and time in augmented reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal, Canada). ACM, New York, NY, USA, 1–13.

Digital Library

[21]

Huan Luo, Cheng Wang, Chenglu Wen, Ziyi Chen, Dawei Zai, Yongtao Yu, and Jonathan Li. 2018. Semantic labeling of mobile LiDAR point clouds via active learning and higher order MRF. IEEE Transactions on Geoscience and Remote Sensing 56, 7 (2018), 3631–3644.

[22]

Sina Masnadi, Andrés N Vargas González, Brian Williamson, and Joseph J LaViola. 2020. AffordIt!: A Tool for Authoring Object Component Behavior in VR. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (Atlanta, GA, USA). IEEE, New York, NY, USA, 741–742.

[23]

Muzammal Naseer, Salman Khan, and Fatih Porikli. 2018. Indoor scene understanding in 2.5/3D for autonomous agents: A survey. IEEE Access 7(2018), 1859–1887.

[24]

Gary Ng, Joon Gi Shin, Alexander Plopski, Christian Sandor, and Daniel Saakes. 2018. Situated game level editing in Augmented Reality. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction (Stockholm, Sweden). ACM, New York, NY, USA, 409–418.

Digital Library

[25]

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (Honolulu, HI, USA). IEEE, New York, NY, USA, 652–660.

[26]

Udo Schultheis, Jason Jerald, Fernando Toledo, Arun Yoganandan, and Paul Mlyniec. 2012. Comparison of a two-handed interface to a wand interface and a mouse interface for fundamental 3D tasks. In 2012 IEEE Symposium on 3D User Interfaces (3DUI)(Costa Mesa, CA, USA). IEEE, New York, NY, USA, 117–124.

[27]

Will Seidelman, Michael Lee, C Melody Carswell, Travis Kent, Bo Fu, and Ruigang Yang. 2014. User centered design of a hybrid-reality display for weld monitoring. In CHI’14 Extended Abstracts on Human Factors in Computing Systems. ACM, New York, NY, USA, 2059–2064.

[28]

Tianjia Shao, Weiwei Xu, Kun Zhou, Jingdong Wang, Dongping Li, and Baining Guo. 2012. An interactive approach to semantic modeling of indoor scenes with an RGBD camera. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1–11.

Digital Library

[29]

Lior Shapira and Daniel Freedman. 2016. Reality Skins: Creating Immersive and Tactile Virtual Environments. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (Merida, Mexico). IEEE, New York, NY, USA, 115–124.

[30]

Misha Sra, Sergio Garrido-Jurado, and Pattie Maes. 2017. Oasis: Procedurally Generated Social Virtual Spaces from 3D Scanned Real Spaces. IEEE transactions on visualization and computer graphics 24, 12(2017), 3174–3187.

[31]

Harald Steinlechner, Bernhard Rainer, Michael Schwärzler, Georg Haaser, Attila Szabo, Stefan Maierhofer, and Michael Wimmer. 2019. Adaptive point cloud segmentation for assisted interactions. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (Montreal, Quebec, Canada). ACM, New York, NY, USA, 1–9.

Digital Library

[32]

Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, and Philip Torr. 2015. Semanticpaint: Interactive 3D labeling and learning at your fingertips. ACM Transactions on Graphics (TOG) 34, 5 (2015), 1–17.

Digital Library

[33]

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition (Boston, Massachusetts). ACM, New York, NY, USA, 1912–1920.

[34]

Jackie Yang, Christian Holz, Eyal Ofek, and Andrew D Wilson. 2019. DreamWalker: Substituting real-world walking experiences with a virtual reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology(New Orleans, LA, USA). ACM, New York, NY, USA, 1093–1107.

Digital Library

[35]

Faizal Amin Nur Yunus, Jamil Abd Baser, Saiful Hadi Masran, Nizamuddin Razali, and Bekri Rahim. 2011. Virtual reality simulator developed welding technology skills. Journal of Modern Education Review 1, 1 (2011), 57–62.

[36]

Xiaoqiang Zhu, Lei Song, Lihua You, Mengyao Zhu, Xiangyang Wang, and Xiaogang Jin. 2017. Brush2Model: Convolution surface-based brushes for 3D modelling in head-mounted display-based virtual environments. Computer Animation and Virtual Worlds 28, 3-4 (2017), e1764.

Cited By

Chamusca ICai YSilva PFerreira CMurari TApolinario AWinkler I(2024)Evaluating Design Guidelines for Intuitive, Therefore Sustainable, Virtual Reality Authoring ToolsSustainability10.3390/su1605174416:5(1744)Online publication date: 20-Feb-2024
https://doi.org/10.3390/su16051744
Mora PGarcia CIvorra EOrtega MAlcañiz M(2024)Virtual Experience Toolkit: An End-to-End Automated 3D Scene Virtualization Framework Implementing Computer Vision TechniquesSensors10.3390/s2412383724:12(3837)Online publication date: 13-Jun-2024
https://doi.org/10.3390/s24123837
Liebers CMegarajan PAuda JStratmann TPfingsthorn MGruenefeld USchneegass S(2024)Keep the Human in the Loop: Arguments for Human Assistance in the Synthesis of Simulation Data for Robot TrainingMultimodal Technologies and Interaction10.3390/mti80300188:3(18)Online publication date: 1-Mar-2024
https://doi.org/10.3390/mti8030018
Show More Cited By

Recommendations

Interactive Augmented Live Virtual Reality Streaming: A Health Care Application
ICMHI '18: Proceedings of the 2nd International Conference on Medical and Health Informatics

Virtual Reality (VR) technology has been around for decades; however, we have probably only begun to realize the practical applications until recent years. With increase of computing power, decrease of cost and physical dimension, a new class of ...
Multi-sensorial virtual reality and augmented human food interaction
MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction

In the field of virtual reality (VR) research, media technologies to create a realistic feeling of being present in a real/virtual world by duplicating multi-sensory information have been studied over a long period. Recently, technologies for multi-...
Visualizing the keyboard in virtual reality for enhancing immersive experience
SIGGRAPH '17: ACM SIGGRAPH 2017 Posters

Recently, virtual reality (VR) becomes more and more popular and provides users an immersive experience with a head-mounted display (HMD). However, in some applications, users have to interact with physical objects while immersed in VR. With a non-see-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

2965 pages

ISBN:9781450380959

DOI:10.1145/3411763

Editors:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2021

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
1,218
Total Downloads

Downloads (Last 12 months)293
Downloads (Last 6 weeks)47

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chamusca ICai YSilva PFerreira CMurari TApolinario AWinkler I(2024)Evaluating Design Guidelines for Intuitive, Therefore Sustainable, Virtual Reality Authoring ToolsSustainability10.3390/su1605174416:5(1744)Online publication date: 20-Feb-2024
https://doi.org/10.3390/su16051744
Mora PGarcia CIvorra EOrtega MAlcañiz M(2024)Virtual Experience Toolkit: An End-to-End Automated 3D Scene Virtualization Framework Implementing Computer Vision TechniquesSensors10.3390/s2412383724:12(3837)Online publication date: 13-Jun-2024
https://doi.org/10.3390/s24123837
Liebers CMegarajan PAuda JStratmann TPfingsthorn MGruenefeld USchneegass S(2024)Keep the Human in the Loop: Arguments for Human Assistance in the Synthesis of Simulation Data for Robot TrainingMultimodal Technologies and Interaction10.3390/mti80300188:3(18)Online publication date: 1-Mar-2024
https://doi.org/10.3390/mti8030018
Chidambaram SWilliams ABai MVirk SHaffner PLease MLi E(2024)Annorama: Enabling Immersive At-Desk Annotation Experiences in Virtual Reality with 3D Point Cloud DioramasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682081(1-13)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682081
Jiang HSong LWeng DSun ZLi HDongye XZhang ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)In Situ 3D Scene Synthesis for Ubiquitous Embodied InterfacesProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681616(3666-3675)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681616
Kim MAlghofaili RLi CYu L(2024)Dragon's Path: Synthesizing User-Centered Flying Creature Animation Paths for Outdoor Augmented Reality ExperiencesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657397(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657397
Van Brummelen JUrwin LJohnston OSayed MBrostow G(2024)Don’t Look Now: Audio/Haptic Guidance for 3D Scanning of LandmarksProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642271(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642271
Stemasov EDemharter SRädler MGugenheimer JRukzio E(2024)pARam: Leveraging Parametric Design in Extended Reality to Support the Personalization of Artifacts for Personal FabricationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642083(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642083
Chidambaram SJain RReddy SUnmesh ARamani K(2024)AnnotateXR: An Extended Reality Workflow for Automating Data Annotation to Support Computer Vision ApplicationsJournal of Computing and Information Science in Engineering10.1115/1.4066180(1-13)Online publication date: 7-Aug-2024
https://doi.org/10.1115/1.4066180
Kumar HKhargonkar NPrabhakaran B(2024)CIS2VR: CNN-based Indoor Scan to VR Environment Authoring Framework2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00025(128-137)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00025
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents