abstract

Resolving occlusion for 3D object manipulation with hands in mixed reality

Authors:

Qi Feng,

Hubert P. H. Shum,

Shigeo MorishimaAuthors Info & Claims

VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

Article No.: 119, Pages 1 - 2

https://doi.org/10.1145/3281505.3283390

Published: 28 November 2018 Publication History

Get Access

Abstract

Due to the need to interact with virtual objects, the hand-object interaction has become an important element in mixed reality (MR) applications. In this paper, we propose a novel approach to handle the occlusion of augmented 3D object manipulation with hands by exploiting the nature of hand poses combined with tracking-based and model-based methods, to achieve a complete mixed reality experience without necessities of heavy computations, complex manual segmentation processes or wearing special gloves. The experimental results show a frame rate faster than real-time and a great accuracy of rendered virtual appearances, and a user study verifies a more immersive experience compared to past approaches. We believe that the proposed method can improve a wide range of mixed reality applications that involve hand-object interactions.

Supplementary Material

MP4 File (a119-feng.mp4)

Download
33.63 MB

References

[1]

Ronald T. Azuma. 2017. Making Augmented Reality a Reality, In Imaging and Applied Optics 2017. Imaging and Applied Optics 2017, JTu1F.1. http://www.osapublishing.org/abstract.cfm?URI=AIO-2017-JTu1F.1

Google Scholar

[2]

Benjamin Bach, Ronell Sicat, Johanna Beyer, Maxime Cordeil, and Hanspeter Pfister. 2018. The Hologram in My Hand: How Effective is Interactive Exploration of 3D Visualizations in Immersive Tangible Augmented Reality? IEEE Transactions on Visualization & Computer Graphics 1 (2018), 1--1.

Google Scholar

[3]

Max Krichenbauer, Goshiro Yamamoto, Takafumi Taketom, Christian Sandor, and Hirokazu Kato. 2018. Augmented Reality versus Virtual Reality for 3D Object Manipulation. IEEE transactions on visualization and computer graphics 24, 2 (2018), 1038--1048.

Google Scholar

[4]

Hui Liang, Junsong Yuan, Daniel Thalmann, and Nadia Magnenat Thalmann. 2015. Ar in hand: Egocentric palm pose tracking and gesture recognition for augmented reality applications. In Proceedings of the 23rd ACM international conference on Multimedia. ACM, 743--744.

Digital Library

Google Scholar

[5]

Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on. IEEE, 127--136.

Digital Library

Google Scholar

[6]

Yuan Tian, Tao Guan, and Cheng Wang. 2010. Real-time occlusion handling in augmented reality based on an object tracking approach. Sensors 10, 4 (2010), 2885--2900.

Crossref

Google Scholar

[7]

Yuan Tian, Yan Long, Dan Xia, Huang Yao, and Jincheng Zhang. 2015. Handling occlusions in augmented reality based on 3D reconstruction method. Neurocomputing 156 (2015), 96--104.

Digital Library

Google Scholar

[8]

Frank Weichert, Daniel Bachmann, Bartholomäus Rudak, and Denis Fisseler. 2013. Analysis of the accuracy and robustness of the leap motion controller. Sensors 13, 5 (2013), 6380--6393.

Crossref

Google Scholar

Cited By

View all

Feld NZielasko DWeyers B(2024)Effects of Hand Occlusion in Radial Mid-Air Menu Interaction in Augmented Reality2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00106(551-558)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00106
Ishihara AAga HIshihara YIchikawa HKaji HKawasaki KKobayashi DKobayashi TNishida KHamasaki TMori HMorikubo Y(2023)Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted DisplayIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324746029:5(2826-2836)Online publication date: 27-Feb-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247460
Macedo MApolinario A(2023)Occlusion Handling in Augmented Reality: Past, Present and FutureIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311786629:2(1590-1609)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TVCG.2021.3117866
Show More Cited By

Index Terms

Resolving occlusion for 3D object manipulation with hands in mixed reality
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality

Recommendations

Resolving occlusion in augmented reality
I3D '95: Proceedings of the 1995 symposium on Interactive 3D graphics

Current state-of-the-art augmented reality systems simply overlay computer-generated visuals on the real-world imagery, for example via video or optical see-through displays. However, overlays are not effective when displaying data in three dimensions, ...
Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences

The Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
Real-time 3D video avatar in mixed reality: an implementation for immersive telecommunication
Symposium: virtual reality simulation

This article presents an implementation of a real-time dynamic 3D avatar from multiview cameras for immersive telecommunication. Immersive telecommunication is a new challenging field that enables a user to share a virtual space with remote participants,...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

November 2018

570 pages

ISBN:9781450360869

DOI:10.1145/3281505

Editor:
Stephen N. Spencer
University of Washington
,
General Chair:
Shigeo Morishima
Waseda University
,
Program Chairs:
Yuichi Itoh
Osaka University
,
Takaaki Shiratori
Facebook Reality Labs
,
Yonghao Yue
The University of Tokyo
,
Rob Lindeman
University of Canterbury

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2018

Check for updates

Author Tags

Qualifiers

Abstract

Conference

VRST '18

Sponsor:

VRST '18: 24th ACM Symposium on Virtual Reality Software and Technology

November 28 - December 1, 2018

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
401
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)3

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Feld NZielasko DWeyers B(2024)Effects of Hand Occlusion in Radial Mid-Air Menu Interaction in Augmented Reality2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00106(551-558)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00106
Ishihara AAga HIshihara YIchikawa HKaji HKawasaki KKobayashi DKobayashi TNishida KHamasaki TMori HMorikubo Y(2023)Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted DisplayIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324746029:5(2826-2836)Online publication date: 27-Feb-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247460
Macedo MApolinario A(2023)Occlusion Handling in Augmented Reality: Past, Present and FutureIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311786629:2(1590-1609)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TVCG.2021.3117866
Mohanty RRaina AChaudhuri SQuek FSueda SKrishnamurthy V(2022)Spatial Manipulation in Virtual Peripersonal Space: A Study of Motor StrategiesJournal of Computing and Information Science in Engineering10.1115/1.405427723:2Online publication date: 20-May-2022
https://doi.org/10.1115/1.4054277
Tian YZhou XWang XWang ZYao H(2021)Registration and occlusion handling based on the FAST ICP-ORB method for augmented reality systemsMultimedia Tools and Applications10.1007/s11042-020-10342-5Online publication date: 12-Mar-2021
https://doi.org/10.1007/s11042-020-10342-5
Fanini BPagano APietroni EFerdani DDemetrescu EPalombini A(2021)Augmented Reality for Cultural HeritageSpringer Handbook of Augmented Reality10.1007/978-3-030-67822-7_16(391-411)Online publication date: 16-Dec-2021
https://doi.org/10.1007/978-3-030-67822-7_16
Tang XHu XFu CCohen-Or DIqbal SMacLean KChevalier FMueller S(2020)GrabAR: Occlusion-aware Grabbing Virtual Objects in ARProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379337.3415835(697-708)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3379337.3415835
Jiang F(2020)Study on the Artificial Intelligent based Layout of Interior Decorations in Chu Culture Homestay using Mixed Reality Platform2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA)10.1109/ICIRCA48905.2020.9182832(287-290)Online publication date: Jul-2020
https://doi.org/10.1109/ICIRCA48905.2020.9182832
Pike ASemwal S(2020)Implementing Material Changes in Augmented EnvironmentsAdvances in Information and Communication10.1007/978-3-030-39442-4_27(352-361)Online publication date: 13-Feb-2020
https://doi.org/10.1007/978-3-030-39442-4_27

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Resolving occlusion in augmented reality

Mixed Reality MIDI Keyboard Demonstration

Real-time 3D video avatar in mixed reality: an implementation for immersive telecommunication