research-article

Dynamic Occlusion Handling for Real-Time AR Applications

Authors:

Rafael Kuffner Dos Anjos,

Ricardo SilvaAuthors Info & Claims

VRCAI '19: Proceedings of the 17th International Conference on Virtual-Reality Continuum and its Applications in Industry

Article No.: 15, Pages 1 - 9

https://doi.org/10.1145/3359997.3365700

Published: 14 November 2019 Publication History

Abstract

Augmented reality (AR) allows computer generated graphics to be overlaid in images or video captured by a camera in real time. This technology is often used to enhance perception by providing extra information or simply by enriching the experience of the user. AR offers a significant potential in many applications such as industrial, medical, education and entertainment. However, for AR to achieve the maximum potential and become fully accepted, the real and virtual objects within the user’s environment must become seamlessly integrated. Three main types of problems arise when we try to achieve this effect: illumination issues, tracking difficulties and occlusion troubles. In this work we present an algorithm to handle AR occlusions in real time. Our approach uses raw depth information of the scene to realize a rough foreground / background segmentation. We use this information, as well as details from color data to estimate a blending coefficient and combine the virtual objects with the real objects into a single image. After experimenting with different scenes we show that our approach is able to produce consistent and aesthetically pleasing occlusions between virtual and real objects, with a low computational cost. Furthermore, we explore different alternatives to improving the quality of the final results while overcoming limitations of previous methods.

References

[1]

P A. Fortin and P Hebert. 2006. Handling Occlusions in Real-time Augmented Reality : Dealing with Movable Real and Virtual Objects. In Third Canadian Conference on Computer and Robot Vision (CRV). 54.

Digital Library

[2]

Richard A. Newcombe, Steven J. Lovegrove, and Andrew J. Davison. 2011. DTAM: Dense tracking and mapping in real-time. In IEEE International Conference on Computer Vision. 2320–2327.

[3]

A Berman, A Dadourian, and P Vlahos. 2000a. Method for removing from an image the background surrounding a selected object. (January 2000).

[4]

A Berman, P Vlahos, and A Dadourian. 2000b. Comprehensive method for removing from an image the background surrounding a selected object. (October 2000).

[5]

David Breen, Eric Rose, and Ross T. Whitaker. 1995. Interactive Occlusion and Collision of Real and Virtual Objects in Augmented Reality. Technical Report. European Computer-Industry Research Centre.

[6]

L. Chen, H. Lin, and S. Li. 2012. Depth image enhancement for Kinect using region growing and bilateral filter. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). 3070–3073.

[7]

J. Cho, T. Yamasaki, K. Aizawa, and K. H. Lee. 2011. Depth video camera based temporal alpha matting for natural 3D scene generation. In 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video. 1–4.

[8]

J. Davis, S. R. Marschner, M. Garr, and M. Levoy. 2002. Filling holes in complex surfaces using volumetric diffusion. In Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission. 428–441. https://doi.org/10.1109/TDPVT.2002.1024098

[9]

C. Du, Y. Chen, M. Ye, and L. Ren. 2016. Edge Snapping-Based Depth Enhancement for Dynamic Occlusion Handling in Augmented Reality. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 54–62.

[10]

Jan Fischer, Holger Regenbrecht, and Gregory Baratoff. 2003. Detecting dynamic occlusion in front of static backgrounds for AR scenes. In EGVE ’03 Workshop on Virtual environments. 153–161.

[11]

Kenichi Hayashi, Hirokazu Kato, and Shogo Nishida. 2005. Occlusion detection of real objects using contour based stereo matching. 180–186.

[12]

K. He, J. Sun, and X. Tang. 2013. Guided Image Filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 6 (June 2013), 1397–1409.

Digital Library

[13]

A. K. Hebborn, N. Höhner, and S. Müller. 2017. Occlusion Matting: Realistic Occlusion Handling for Augmented Reality Applications. In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 62–71. https://doi.org/10.1109/ISMAR.2017.23

[14]

Bradley Herbert, Barrett Ens, Amali Weerasinghe, Mark Billinghurst, and Grant Wigley. 2018. Design considerations for combining augmented reality with intelligent tutors. Computers & Graphics 77(2018), 166 – 182. https://doi.org/10.1016/j.cag.2018.09.017

[15]

Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard A. Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew J. Davison, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST). 559–568.

Digital Library

[16]

Peter J Burt. 1981. Fast Filter Transforms for Image Processing. Computer Graphics and Image Processing 16 (May 1981), 20–51.

[17]

Jubin Johnson, Ehsan Shahrian, Hisham Cholakkal, and Deepu Rajan. 2016. Sparse Coding for Alpha Matting. In IEEE Transactions on Image Processing, Vol. 25. 1.

[18]

Johannes Kopf, Michael Cohen, Dani Lischinski, and Matt Uyttendaele. 2007. Joint bilateral upsampling. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2007) 26 (July 2007), 96.

Digital Library

[19]

Adrian Leal, Leopoldo Altamirano Robles, and Jesus Gonzalez. 2013. Occlusion Handling in Video-Based Augmented Reality Using the Kinect Sensor for Indoor Registration. In Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications. 447–454.

[20]

Vincent Lepetit and Berger Marie-Odile. 2000. A Semi-Automatic Method for Resolving Occlusion in Augmented Reality. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2. 2225–2230.

[21]

Vincent Lepetit and Berger marie odile. 2000. A Semi-Automatic Method for Resolving Occlusion in Augmented Reality. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). 2225–2230.

[22]

Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A Closed-Form Solution to Natural Image Matting. In IEEE transactions on pattern analysis and machine intelligence, Vol. 30. 228–42.

[23]

Daniel Simões Lopes and Joaquim A. Jorge. 2019. Extending medical interfaces towards virtual reality and augmented reality. Annals of Medicine 51, sup1 (2019), 29–29. https://doi.org/10.1080/07853890.2018.1560068 arXiv:https://doi.org/10.1080/07853890.2018.1560068

[24]

Ting Lu and Shutao Li. 2012. Image matting with color and depth information. In International Conference on Pattern Recognition. 3787–3790.

[25]

Matthias M. Wloka and Brian G. Anderson. 1995. Resolving Occlusion in Augmented Reality. In Symposium on Interactive 3D graphics (I3D). 5–12.

[26]

Daniel Mendes, Daniel Medeiros, Maurício Sousa, Eduardo Cordeiro, Alfredo Ferreira, and Joaquim A. Jorge. 2017. Design and evaluation of a novel out-of-reach selection technique for VR using iterative refinement. Computers & Graphics 67(2017), 95 – 102. https://doi.org/10.1016/j.cag.2017.06.003

Digital Library

[27]

Joan Ogden, Edward Adelson, James Bergen, and Peter J. Burt. 1985. Pyramid-based Computer Graphics. RCA engineer 30 (September 1985), 4–15.

[28]

Bernhard Preim and Patrick Saalfeld. 2018. A survey of virtual human anatomy education systems. Computers & Graphics 71(2018), 132 – 153. https://doi.org/10.1016/j.cag.2018.01.005

[29]

M. A. Ruzon and C. Tomasi. 2000. Alpha estimation in natural images. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000, Vol. 1. 18–25.

[30]

Eduardo S. L. Gastal and Manuel Oliveira. 2010. Shared Sampling for Real-Time Alpha Matting. In Eurographics 2010, Vol. 29. 575–584.

[31]

Michael Schmeing and Xiaoyi Jiang. 2014. Edge-aware depth image filtering using color segmentation. Pattern Recognition Letters 50 (December 2014), 63–71.

[32]

J. Schmidt, H. Niemann, and S. Vogt. 2002. Dense disparity maps in real-time with an application to augmented reality. In Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings.225–230.

[33]

Susanne Schmidt, Gerd Bruder, and Frank Steinicke. 2019. Effects of virtual agent and object representation on experiencing exhibited artifacts. Computers & Graphics 83(2019), 1 – 10. https://doi.org/10.1016/j.cag.2019.06.002

Digital Library

[34]

Paul D. Schmirler, Thong T. Nguyen, Alex L. Nicoll, and David Vasko. 2018. Virtual reality and augmented reality for industrial automation. PATENT US20180131907A1. https://patents.google.com/patent/US20180131907A1/en

[35]

M. M. Shah, H. Arshad, and R. Sulaiman. 2012. Occlusion in augmented reality. In 2012 8th International Conference on Information Science and Digital Content Technology (ICIDT2012), Vol. 2. 372–378.

[36]

Ricardo Silva. 2018. Dynamic Occlusion Handling for Real-Time AR Applications. Master’s thesis. Instituto Superior Técnico da Universidade de Lisboa, Avenida Rovisco Pais s/n 1049-001, Lisboa, Portugal.

[37]

Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeung Shum. 2004. Poisson matting. In ACM SIGGRAPH, Vol. 23. 315–321.

[38]

C. Tomasi and R. Manduchi. 1998. Bilateral filtering for gray and color images. In Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271). 839–846.

Digital Library

[39]

Jue Wang and Michael Cohen. 2007a. Image and Video Matting: A Survey. Foundations and Trends in Computer Graphics and Vision 3 (January 2007), 97–175.

[40]

J. Wang and M. F. Cohen. 2007b. Optimized Color Sampling for Robust Matting. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. 1–8.

[41]

O. Wang, J. Finger, Q. Yang, J. Davis, and R. Yang. 2007. Automatic Natural Video Matting with Depth. In 15th Pacific Conference on Computer Graphics and Applications. 469–472.

[42]

Thomas Whelan, Stefan Leutenegger, Renato Salas Moreno, Ben Glocker, and Andrew Davison. 2015. ElasticFusion: Dense SLAM Without A Pose Graph. (July 2015).

[43]

Tian Yuan, Guan Tao, and Wang Cheng. 2010. Real-Time Occlusion Handling in Augmented Reality Based on an Object Tracking Approach. In Sensors, Vol. 10.

[44]

J. Zhu, Miao Liao, R. Yang, and Zhigeng Pan. 2009. Joint depth and alpha matte optimization via fusion of stereo and time-of-flight sensor. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 453–460.

[45]

Jiejie Zhu, Zhigeng Pan, Chao Sun, and Wenzhi Chen. 2010. Handling occlusions in video-based augmented reality using depth information. Journal of Visualization and Computer Animation 21 (September 2010), 509–521.

[46]

Ezequiel R. Zorzal, Maurício Sousa, Daniel Mendes, Rafael Kuffner dos Anjos, Daniel Medeiros, Soraia Figueiredo Paulo, Pedro Rodrigues, José João Mendes, Vincent Delmas, Jean-Francois Uhl, José Mogorrón, Joaquim Armando Jorge, and Daniel Simões Lopes. 2019. Anatomy studio: A tool for virtual dissection through augmented 3D reconstruction. Computers & Graphics(2019). https://doi.org/10.1016/j.cag.2019.09.006

Digital Library

Cited By

Duarte AFernandes FPereira JMoreira CNascimento JJorge J(2024)SelfredepthJournal of Real-Time Image Processing10.1007/s11554-024-01491-z21:4Online publication date: 4-Jul-2024
https://doi.org/10.1007/s11554-024-01491-z
Watson JSayed MQureshi ZBrostow GVicente SAodha OFirman M(2023)Virtual Occlusions Through Implicit Depth2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00874(9053-9064)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00874
Ji NZhang FZhang HZhao YYu D(2022)Mixed reality depth contour occlusion using binocular similarity matching and three-dimensional contour optimisationVirtual Reality10.1007/s10055-022-00695-727:2(887-901)Online publication date: 27-Sep-2022
https://doi.org/10.1007/s10055-022-00695-7
Show More Cited By

Recommendations

GrabAR: Occlusion-aware Grabbing Virtual Objects in AR
UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Existing augmented reality (AR) applications often ignore the occlusion between real hands and virtual objects when incorporating virtual objects in user's views. The challenges come from the lack of accurate depth and mismatch between real and virtual ...
Cooperatively Resolving Occlusion between Real and Virtual in Multiple Video Sequences
CHINAGRID '11: Proceedings of the 2011 Sixth Annual ChinaGrid Conference

The occlusion between real and virtual objects influences not only seamless merging of virtual and real environments but also users' visual perception of orientations & locations and spatial interactions in augmented reality. If there exist a large ...
Occlusion Handling in Augmented Reality System for Human-Assisted Assembly Task
Intelligent Robotics and Applications
Abstract
Augmented reality (AR) based human-machine interaction (HMI) provided a seamless interface between user and application environment, but occlusion handing remained as a tough problem in AR applications. Recently most AR occlusion handling ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VRCAI '19: Proceedings of the 17th International Conference on Virtual-Reality Continuum and its Applications in Industry

November 2019

354 pages

ISBN:9781450370028

DOI:10.1145/3359997

Conference Chairs:
Joaquim Jorge
INESC/ID & Univ Lisboa, Portugal
,
June Kim
University of New South Wales, Australia
,
Herman Van Eyken
Griffith University, Queensland, Australia
,
Editor:
Stephen N. Spencer
University of Washington
,
Program Chairs:
Mashhuda Glencross
University of Queensland, Australia
,
Kenny Mitchell
Disney Research, USA and Edinburgh Napier University, UK
,
Takuji Narumi
University of Tokyo, Japan

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

VRCAI '19

Sponsor:

SIGGRAPH

VRCAI '19: The 17th International Conference on Virtual-Reality Continuum and its Applications in Industry

November 14 - 16, 2019

QLD, Brisbane, Australia

Acceptance Rates

Overall Acceptance Rate 51 of 107 submissions, 48%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
480
Total Downloads

Downloads (Last 12 months)62
Downloads (Last 6 weeks)3

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Duarte AFernandes FPereira JMoreira CNascimento JJorge J(2024)SelfredepthJournal of Real-Time Image Processing10.1007/s11554-024-01491-z21:4Online publication date: 4-Jul-2024
https://doi.org/10.1007/s11554-024-01491-z
Watson JSayed MQureshi ZBrostow GVicente SAodha OFirman M(2023)Virtual Occlusions Through Implicit Depth2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00874(9053-9064)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00874
Ji NZhang FZhang HZhao YYu D(2022)Mixed reality depth contour occlusion using binocular similarity matching and three-dimensional contour optimisationVirtual Reality10.1007/s10055-022-00695-727:2(887-901)Online publication date: 27-Sep-2022
https://doi.org/10.1007/s10055-022-00695-7
Pfeuffer KAbdrabou YEsteves ARivu RAbdelrahman YMeitner SSaadi AAlt F(2021)ARtention: A design space for gaze-adaptive user interfaces in augmented realityComputers & Graphics10.1016/j.cag.2021.01.00195(1-12)Online publication date: Apr-2021
https://doi.org/10.1016/j.cag.2021.01.001
Luo TZhang MPan ZLi ZCai NMiao JChen YXu M(2020)Dream-Experiment: A MR User Interface with Natural Multi-channel Interaction for Virtual ExperimentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.302360226:12(3524-3534)Online publication date: Dec-2020
https://doi.org/10.1109/TVCG.2020.3023602

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents