research-article

Occlusion Leak Compensation for Optical See-Through Displays Using a Single-Layer Transmissive Spatial Light Modulator

Authors:

Takumi Hamasaki,

Maki SugimotoAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 23, Issue 11

Pages 2463 - 2473

https://doi.org/10.1109/TVCG.2017.2734427

Published: 01 November 2017 Publication History

Abstract

We propose an occlusion compensation method for optical see-through head-mounted displays (OST-HMDs) equipped with a singlelayer transmissive spatial light modulator (SLM), in particular, a liquid crystal display (LCD). Occlusion is an important depth cue for 3D perception, yet realizing it on OST-HMDs is particularly difficult due to the displays' semitransparent nature. A key component for the occlusion support is the SLM—a device that can selectively interfere with light rays passing through it. For example, an LCD is a transmissive SLM that can block or pass incoming light rays by turning pixels black or transparent. A straightforward solution places an LCD in front of an OST-HMD and drives the LCD to block light rays that could pass through rendered virtual objects at the viewpoint. This simple approach is, however, defective due to the depth mismatch between the LCD panel and the virtual objects, leading to blurred occlusion. This led existing OST-HMDs to employ dedicated hardware such as focus optics and multi-stacked SLMs. Contrary to these viable, yet complex and/or computationally expensive solutions, we return to the single-layer LCD approach for the hardware simplicity while maintaining fine occlusion—we compensate for a degraded occlusion area by overlaying a compensation image. We compute the image based on the HMD parameters and the background scene captured by a scene camera. The evaluation demonstrates that the proposed method reduced the occlusion leak error by 61.4% and the occlusion error by 85.7%.

References

[1]

K. Akeley, S.J. Watt, A.R. Girshick, and M.S. Banks. “A stereo display prototype with multiple focal distances”. In ACM TOG, volume 23, pages 804–813. ACM, 2004.

[2]

C. Benoît-Pasanau, F. Goudail, P. Chavel, J.-P. Cano, and J. Ballet. Minimization of diffraction peaks of spatial light modulators using voronoi diagrams. Optics Express, 18(14): pp. 15223–15235, 2010.

[3]

O. Bimber and B. Fröhlich. Occlusion shadows: Using projected light to generate realistic occlusion effects for view-dependent optical see-through displays. In 1st ISMAR, page 186. IEEE, 2002.

[4]

O. Bimber, A. Grundhöfer, G. Wetzstein, and S. Knödel. Consistent illumination within optical see-through augmented environments. In 2nd IEEE/ACM ISMAR, page 198. IEEE, 2003.

[5]

D.D. Bohn. Pixel opacity for augmented reality, Dec. 29 2015. US Patent 9,223,138.

[6]

O. Cakmakci, Y. Ha, and J.P. Rolland. A compact optical see-through head-worn display with occlusion support. In 3rd IEEE/ACM ISMAR, pages 16–25. IEEE, 2004.

[7]

H.-C. Chiang, T.-Y. Ho, and C.-R. Sheu. Structure for reducing the diffraction effect in periodic electrode arrangements and liquid crystal device including the same, Dec. 20 2005. US Patent 6,977,705.

[8]

D. Dunn, C. Tippets, K. Torell, P. Kellnhofer, K. Aksit, P. Didyk, K. Myszkowski, D. Luebke, and H. Fuchs. Wide field of view varifocal near-eye display using see-through deformable membrane mirrors. IEEE TVCG, 2017.

[9]

A. Fuhrmann, G. Hesina, F. Faure, and M. Gervautz. Occlusion in collaborative augmented environments. CG, 23(6): pp. 809–819, 1999.

[10]

C. Gao, Y. Lin, and H. Hua. Occlusion capable optical see-through head-mounted display using freeform optics. In 11th IEEE ISMAR, pages 281–282. IEEE, 2012.

[11]

C. Gao, Y. Lin, and H. Hua. Optical see-through head-mounted display with occlusion capability. In Proc. SPIE, volume 8735, page 87350F-1:9, 2013.

[12]

E.D. Guestrin and M. Eizenman. General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE Transactions on Biomedical Engineering, 53(6): pp. 1124–1133, 2006.

[13]

R.R. Hainich and O. Bimber. Displays: fundamentals & applications. CRC press, 2016.

[14]

Y. Hiroi, Y. Itoh, T. Hamasaki, and M. Sugimoto. AdaptiVisor: Assisting eye adaptation via occlusive optical see-through head-mounted displays. In 8th Augmented Human International Conference, AH '17, pages 9:1–9:9, New York, NY, USA, 2017. ACM.

[15]

H. Hua. Enabling focus cues in head-mounted displays. in Proceedings of the IEEE, 2017.

[16]

H. Hua, C. Gao, L.D. Brown, N. Ahuja, and J.P. Rolland. A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD). In IEEE VR, pages 81–89. IEEE, 2002.

[17]

F.-C. Huang, K. Chen, and G. Wetzstein. The light field stereoscope: immersive computer graphics via factored near-eye light field displays with focus cues. ACM TOG, 34(4): p. 60, 2015.

[18]

Y. Itoh, M. Dzitsiuk, T. Amano, and G. Klinker. Semi-parametric color re-production method for optical see-through head-mounted displays. IEEE TVCG, 21(11): pp. 1269–1278, 2015.

[19]

Y. Itoh and G. Klinker. Interaction-free calibration for optical see-through head-mounted displays based on 3D eye localization. In IEEE 3DUI, pages 75–82. IEEE, 2014.

[20]

H. Kakeya, S. Ishizuka, and Y. Sato. Realization of an aerial 3D image that occludes the background scenery. Optics Express, 22(20): pp. 24491–24496, 2014.

[21]

K. Kiyokawa. “Occlusion displays”. In Handbook of Visual Display Technology, pages 2251–2257. Springer, 2012.

[22]

K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods. An occlusion-capable optical see-through head mount display for supporting co-located collaboration. In 2nd IEEE/ACM ISMAR, page 133. IEEE, 2003.

[23]

K. Kiyokawa, Y. Kurata, and H. Ohno. An optical see-through display for mutual occlusion with a real-time stereovision system. Computers & Graphics, 25(5): pp. 765–779, 2001.

[24]

E. Kruijff, J.E. Swan, and S. Feiner. Perceptual issues in augmented reality revisited. In 9th IEEE ISMAR, pages 3–12. IEEE, 2010.

[25]

T. Langlotz, M. Cook, and H. Regenbrecht. Real-time radiometric compensation for optical see-through head-mounted displays. IEEE TVCG, 22(11): pp. 2385–2394, 2016.

[26]

D. Lanman and D. Luebke. Near-eye light field displays. ACM TOG, 32(6): p. 220, 2013.

Digital Library

[27]

B. Lee, J.-Y. Hong, C. Jang, J. Jeong, and C.-K. Lee. Holographic and light-field imaging for augmented reality. In Proc. SPIE, volume 10125, page 101251A-1, 2017.

[28]

P. Lincoln, A. Blate, M. Singh, T. Whitted, A. State, A. Lastra, and H. Fuchs. From motion to photons in 80 microseconds: Towards minimal latency for virtual and augmented reality. IEEE TVCG, 22(4): pp. 1367–1376, 2016.

[29]

S. Liu, D. Cheng, and H. Hua. An optical see-through head mounted display with addressable focal planes. In 7th IEEE/ACM ISMAR, pages 33–42. IEEE, 2008.

[30]

M.A. Livingston, J.E. Swan II, J.L. Gabbard, T.H. Höllerer, D. Hix, S.J. Julier, Y. Baillot, and D. Brown. Resolving multiple occluded layers in augmented reality. In 2nd IEEE/ACM ISMAR, page 56. IEEE, 2003.

[31]

A. Maimone and H. Fuchs. Computational augmented reality eyeglasses. In 12th IEEE ISMAR, pages 29–38. IEEE, 2013.

[32]

A. Maimone, A. Georgiou, and J. Kollin. Holographic near-eye displays for virtual and augmented reality. ACM TOG, 36(4): p. 85, 2017.

[33]

A. Maimone, D. Lanman, K. Rathinavel, K. Keller, D. Luebke, and H. Fuchs. “Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources”. In ACM SIGGRAPH Emerging Technologies, page 20. ACM, 2014.

[34]

B. Masia, G. Wetzstein, P. Didyk, and D. Gutierrez. A survey on computational displays: Pushing the boundaries of optics, computation, and perception. Computers & Graphics, 37(8): pp. 1012–1038, 2013.

Digital Library

[35]

E. Moon, M. Kim, J. Roh, H. Kim, and J. Hahn. Holographic head-mounted display with RGB light emitting diode light source. Optics Express, 22(6): pp. 6526–6534, 2014.

[36]

K.R. Moser, D.C. Rompapas, J.E. Swan, S. Ikeda, G. Yamamoto, T. Taketomi, C. Sandor, H. Kato, et al. SharpView: Improved clarity of defocused content on optical see-through head-mounted displays. In IEEE 3DUI, pages 173–181. IEEE, 2016.

[37]

J.D. Mulder. Realistic occlusion effects in mirror-based co-located augmented reality systems. In IEEE VR, pages 203–208. IEEE, 2005.

[38]

N. Padmanaban, R. Konrad, T. Stramer, E.A. Cooper, and G. Wetzstein. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. in Proc. of the NAS, pages 2183–2188, 2017.

[39]

W. Robinett and J.P. Rolland. A computational model for the stereoscopic optics of a head-mounted display. Presence, 1(1): pp. 45–62, 1992.

Digital Library

[40]

I.N. Robinson. Method and apparatus for augmented reality using a see-through head-mounted display, Mar. 14 2000. US Patent 6,037,914.

[41]

J.P. Rolland, R.L. Holloway, and H. Fuchs. “Comparison of optical and video see-through, head-mounted displays”. In Photonics for Industrial Applications, pages 293–307. SPIE, 1995.

[42]

D.C. Rompapas, A. Rovira, S. Ikeda, A. Plopski, T. Taketomi, C. Sandor, and H. Kato. EyeAR: Refocusable augmented reality content through eye measurements. In 15th IEEE ISMAR Adjunct paper, pages 334–335. IEEE, 2016.

[43]

P. Santos, T. Gierlinger, O. Machui, and A. Stork. The daylight blocking optical stereo see-through HMD. In Workshop on Immersive projection technologies/Emerging display technologies, page 4. ACM, 2008.

[44]

S. Suyama, M. Date, and H. Takada. Three-dimensional display system with dual-frequency liquid-crystal varifocal lens. Japanese Journal of Applied Physics, 39(2R): p. 480, 2000.

[45]

J.E. Swan, A. Jones, E. Kolstad, M.A. Livingston, and H.S. Smallman. Egocentric depth judgments in optical, see-through augmented reality. IEEE TVCG, 13(3): pp. 429–442, 2007.

[46]

A. Takagi, Y. Saito, N. Taniguchi, and T. Sudo. Image observing apparatus for observing outside information superposed with a display image, Aug. 7 2001. US Patent 6,271,895.

[47]

E.W. Tatham. Technical opinion: Getting the best of both real and virtual worlds. ACM Communucations, 42(9): pp. 96–98, Sept. 1999.

Digital Library

[48]

T. Uchida, K. Sato, and S. Inokuchi. An optical see-through MR display with digital micro-mirror device. Trans. of the Virtual Reality Society of Japan, 7(2), 2002.

[49]

M.M. Wloka and B.G. Anderson. Resolving occlusion in augmented reality. In Symp. on Interactive 3D Graphics, pages 5–12. ACM, 1995.

[50]

Y. Yamaguchi and Y. Takaki. See-through integral imaging display with background occlusion capability. Applied Optics, 55(3): pp. A144–A149, 2016.

Cited By

Hiroi YHiraki TItoh Y(2024)StainedSweeper: Compact, Variable-Intensity Light-Attenuation Display with Sweeping Tunable RetardersIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337205830:5(2682-2692)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372058
Uramune RSawamura KIkeda SIshizuka HOshiro OWard JMcGill MMarky K(2023)Gaze Depth Estimation for In-vehicle AR DisplaysProceedings of the Augmented Humans International Conference 202310.1145/3582700.3583707(323-325)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3583707
Zhang YHu XKiyokawa KYang X(2023)Add-on Occlusion: Turning Off-the-Shelf Optical See-through Head-mounted Displays Occlusion-capableIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324706429:5(2700-2709)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247064
Show More Cited By

Index Terms

Occlusion Leak Compensation for Optical See-Through Displays Using a Single-Layer Transmissive Spatial Light Modulator
1. Computing methodologies
  1. Computer graphics
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Occlusive optical see-through displays in a collaborative setup
SIGGRAPH '02: ACM SIGGRAPH 2002 conference abstracts and applications

There are several approaches to realize a 3D display that can be viewed by multiple co-located users, and each has its own pros and cons. A volumetric display utilizes afterimage effects, and it usually can only show transparent images within its ...
Near-eye light field displays
SIGGRAPH '13: ACM SIGGRAPH 2013 Talks

We propose a light-field-based approach to near-eye display that allows for thin, lightweight head-mounted displays capable of depicting accurate accommodation, convergence, and binocular disparity depth cues. Our near-eye light field displays depict ...
Add-on Occlusion: Turning Off-the-Shelf Optical See-through Head-mounted Displays Occlusion-capable
The occlusion-capable optical see-through head-mounted display (OC-OSTHMD) is actively developed in recent years since it allows mutual occlusion between virtual objects and the physical world to be correctly presented in augmented reality (AR). However, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 23, Issue 11

Nov. 2017

129 pages

ISSN:1077-2626

Issue’s Table of Contents

1077-2626 © 2017 IEEE.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 November 2017

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hiroi YHiraki TItoh Y(2024)StainedSweeper: Compact, Variable-Intensity Light-Attenuation Display with Sweeping Tunable RetardersIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337205830:5(2682-2692)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372058
Uramune RSawamura KIkeda SIshizuka HOshiro OWard JMcGill MMarky K(2023)Gaze Depth Estimation for In-vehicle AR DisplaysProceedings of the Augmented Humans International Conference 202310.1145/3582700.3583707(323-325)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3583707
Zhang YHu XKiyokawa KYang X(2023)Add-on Occlusion: Turning Off-the-Shelf Optical See-through Head-mounted Displays Occlusion-capableIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324706429:5(2700-2709)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247064
Sutton JLanglotz TPlopski AZollmann SItoh YRegenbrecht H(2022)Look over there! Investigating Saliency Modulation for Visual Guidance with Augmented Reality GlassesProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545633(1-15)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545633
Zhang YWang RPeng YHua WBao H(2022)Color Contrast Enhanced Rendering for Optical See-Through Head-Mounted DisplaysIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.309168628:12(4490-4502)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1109/TVCG.2021.3091686
Wilson AHua H(2022)Design of a Pupil-Matched Occlusion-Capable Optical See-Through Wearable DisplayIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.307606928:12(4113-4126)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1109/TVCG.2021.3076069
Itoh YLanglotz TSutton JPlopski A(2021)Towards Indistinguishable Augmented RealityACM Computing Surveys10.1145/345315754:6(1-36)Online publication date: 13-Jul-2021
https://dl.acm.org/doi/10.1145/3453157

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents