research-article

Open access

High resolution étendue expansion for holographic displays

Authors:

Andrew MaimoneAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 39, Issue 4

Article No.: 66, Pages 66:1 - 66:14

https://doi.org/10.1145/3386569.3392414

Published: 12 August 2020 Publication History

Abstract

Holographic displays can create high quality 3D images while maintaining a small form factor suitable for head-mounted virtual and augmented reality systems. However, holographic displays have limited étendue based on the number of pixels in their spatial light modulators, creating a tradeoff between the eyebox size and the field-of-view. Scattering-based étendue expansion, in which coherent light is focused into an image after being scattered by a static mask, is a promising avenue to break this tradeoff. However, to date, this approach has been limited to very sparse content consisting of, for example, only tens of spots.

In this work, we introduce new algorithms to scattering-based étendue expansion that support dense, photorealistic imagery at the native resolution of the spatial light modulator, offering up to a 20 dB improvement in peak signal to noise ratio over baseline methods. We propose spatial and frequency constraints to optimize performance for human perception, and performance is characterized both through simulation and a preliminary benchtop prototype. We further demonstrate the ability to generate content at multiple depths, and we provide a path for the miniaturization of our benchtop prototype into a sunglasses-like form factor.

Supplemental Material

MP4 File

Download
36.95 MB

References

[1]

Duygu Akbulut, Thomas J Huisman, Elbert G van Putten, Willem L Vos, and Allard P Mosk. 2011. Focusing light through random photonic media by binary amplitude modulation. Optics express 19, 5 (2011), 4017--4029.

[2]

Kiseung Bang, Changwon Jang, and Byoungho Lee. 2019. Compact noise-filtering volume gratings for holographic displays. Optics letters 44, 9 (2019), 2133--2136.

[3]

Amir Beck and Marc Teboulle. 2009. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences 2, 1 (2009), 183--202.

[4]

Alain Bergeron, Jonny Gauvin, François Gagnon, Denis Gingras, Henri H Arsenault. and Michel Doucet. 1995. Phase calibration and applications of a liquid-crystal spatial light modulator. Applied optics 34, 23 (1995), 5133--5139.

[5]

Edward Buckley, Adrian Cable, Nie Lawrence, and Tim Wilkinson. 2006. Viewing angle enhancement for two-and three-dimensional holographic displays with random superresolution phase masks. Applied optics 45, 28 (2006), 7334--7341.

[6]

Emmanuel J Candes, Xiaodong Li, and Mahdi Soltanolkotabi. 2015. Phase retrieval via Wirtinger flow: Theory and algorithms. IEEE Transactions on Information Theory 61, 4 (2015), 1985--2007.

Digital Library

[7]

Thomas Chaigne, Jérôme Gateau, Ori Katz, Emmanuel Bossy, and Sylvain Gigan. 2014. Light focusing and two-dimensional imaging through scattering media using the photoacoustic transmission matrix with an ultrasound array. Optics letters 39, 9 (2014), 2664--2667.

[8]

Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019. Wirtinger holography for near-eye displays. ACM Transactions on Graphics (TOG) 38, 6 (2019), 213.

Digital Library

[9]

Julio Chaves. 2017. Introduction to nonimaging optics. CRC press.

[10]

Myeong-Ho Choi, Yeon-Gyeong Ju, and Jae-Hyeung Park. 2020. Holographic near-eye display with continuously expanded eyebox using two-dimensional replication and angular spectrum wrapping. Opt. Express 28, 1 (Jan 2020), 533--547.

[11]

Donald B Conkey, Antonio M Caravaca-Aguirre, Jake D Dove, Hengyi Ju, Todd W Murray, and Rafael Piestun. 2015. Super-resolution photoacoustic imaging through a scattering wall. Nature communications 6 (2015), 7902.

[12]

Donald B Conkey, Antonio M Caravaca-Aguirre, and Rafael Piestun. 2012. High-speed scattering medium characterization with application to focusing light through turbid media. Optics express 20, 2 (2012), 1733--1740.

[13]

Yu Fang, Ryoichi Nakashima, Kazumichi Matsumiya, Ichiro Kuriki, and Satoshi Shioiri. 2015. Eye-head coordination for visual cognitive processing. PloS one 10, 3 (2015).

[14]

A Georgiou, J Christmas, N Collings, J Moore, and WA Crossland. 2008. Aspects of hologram calculation for video frames. Journal of Optics A: Pure and Applied Optics 10, 3 (2008), 035302.

[15]

Ralph W Gerchberg and W. . Saxton. 1972. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35 (1972), 237--246.

[16]

Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company Publishers.

[17]

Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics (TOG) 31, 6 (2012), 164.

Digital Library

[18]

Alain Hore and Djemel Ziou. 2010. Image quality metrics: PSNR vs. SSIM. In 2010 20th International Conference on Pattern Recognition. IEEE, 2366--2369.

Digital Library

[19]

Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2018. Holographic Near-Eye Display with Expanded Eye-Box. ACM Trans. Graph. 37, 6, Article Article 195 (Dec. 2018), 14 pages.

Digital Library

[20]

Anton S Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.

Digital Library

[21]

Mugeon Kim, Sungjin Lim, Geunseop Choi, Youngmin Kim, Hwi Kim, and Joonku Hahn. 2018. Expanded Exit-Pupil Holographic Head-Mounted Display With High-Speed Digital Micromirror Device. ETRI Journal 40, 3 (2018), 366--375. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.4218/etrij.2017-0166

[22]

Gang Li, Dukho Lee, Youngmo Jeong, Jaebum Cho, and Byoungho Lee. 2016. Holographic display for see-through augmented reality using mirror-lens holographic optical element. Optics letters 41, 11 (2016), 2486--2489.

[23]

Andrew Maimone, Andreas Georgiou, and Joel S Kollin. 2017. Holographic near-eye displays for virtual and augmented reality. ACM Transactions on Graphics (TOG) 36, 4 (2017), 85.

Digital Library

[24]

Kyoji Matsushima, Hagen Schimmel, and Frank Wyrowski. 2003. Fast calculation method for optical diffraction on tilted planes by use of the angular spectrum of plane waves. JOSA A 20, 9 (2003), 1755--1762.

[25]

Simon Moser, Monika Ritsch-Marte, and Gregor Thalhammer. 2019. Model-based compensation of pixel crosstalk in liquid crystal spatial light modulators. Optics express 27, 18 (2019), 25046--25063.

[26]

Jongchan Park, KyeoReh Lee, and YongKeun Park. 2019. Ultrathin wide-angle large-area digital 3D holographic display using a non-periodic photon sieve. Nature communications 10, 1 (2019), 1304.

[27]

Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG) 35, 6 (2016), 179.

Digital Library

[28]

SM Popoff, G Lerosey, R Carminati, M Fink, AC Boccara, and S Gigan. 2010. Measuring the transmission matrix in optics: an approach to the study and control of light propagation in disordered media. Physical review letters 104, 10 (2010), 100601.

[29]

Liang Shi, Fu-Chung Huang, Ward Lopes, Wojciech Matusik, and David Luebke. 2017. Near-eye light field holographic rendering with spherical waves for wide field of view interactive 3D computer graphics. ACM Transactions on Graphics (TOG) 36, 6 (2017), 236.

Digital Library

[30]

Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--13.

Digital Library

[31]

Rod Sterling. 2008. JVC D-ILA high resolution, high contrast projectors and applications. In Proceedings of the 2008 workshop on Immersive projection technologies/Emerging display technologiges. ACM, 10.

Digital Library

[32]

Xiaodong Tao, Dare Bodington, Marc Reinig, and Joel Kubby. 2015. High-speed scanning interferometric focusing by fast measurement of binary transmission matrix for channel demixing. Optics express 23, 11 (2015), 14168--14187.

[33]

Ivo M Vellekoop, Aart Lagendijk, and AP Mosk. 2010. Exploiting disorder for perfect focusing. Nature photonics 4, 5 (2010), 320.

[34]

Ivo M Vellekoop and AP Mosk. 2007. Focusing coherent light through opaque strongly scattering media. Optics letters 32, 16 (2007), 2309--2311.

[35]

Koki Wakunami, Po-Yuan Hsieh, Ryutaro Oi, Takanori Senoh, Hisayuki Sasaki, Yasuyuki Ichihashi, Makoto Okui, Yi-Pai Huang, and Kenji Yamamoto. 2016. Projection-type see-through holographic three-dimensional display. Nature communications 7, 1 (2016), 1--7.

[36]

Xiaodong Xun and Robert W Cohn. 2004. Phase calibration of spatially nonuniform spatial light modulators. Applied optics 43, 35 (2004), 6400--6406.

[37]

Fahri Yaraş, Hoonjong Kang, and Levent Onural. 2009. Real-time phase-only color holographic video display system using LED illumination. Applied optics 48, 34 (2009), H48--H53.

[38]

Li-Hao Yeh, Shwetadwip Chowdhury, Nicole A Repina, and Laura Waller. 2019. Speckle-structured illumination for 3D phase and fluorescence computational microscopy. Biomedical Optics Express 10, 7 (2019), 3635--3653.

[39]

Jonghee Yoon, KyeoReh Lee, Jongchan Park, and YongKeun Park. 2015. Measuring optical transmission matrices by wavefront shaping. Optics Express 23, 8 (2015), 10158--10167.

[40]

Hyeonseung Yu, KyeoReh Lee, Jongchan Park, and YongKeun Park. 2017b. Ultrahigh-definition dynamic 3D holographic display by active control of volume speckle fields. Nature Photonics 11, 3 (2017), 186.

[41]

HyeonSeung Yu, KyeoReh Lee, and YongKeun Park. 2017a. Ultrahigh enhancement of light focusing through disordered media controlled by mega-pixel modes. Optics express 25, 7 (2017), 8036--8047.

[42]

Jingzhao Zhang, Nicolas Pégard, Jingshan Zhong, Hillel Adesnik, and Laura Waller. 2017. 3D computer-generated holography by non-convex optimization. Optica 4, 10 (2017), 1306--1313.

Cited By

Cheng CWang JWu ZZhou JWang JChen C(2025)Pre-compensated annealing gradient descent for spherical holographyOptics Communications10.1016/j.optcom.2024.131049574(131049)Online publication date: Jan-2025
https://doi.org/10.1016/j.optcom.2024.131049
Gopakumar MLee GChoi SChao BPeng YKim JWetzstein G(2024)Full-colour 3D holographic augmented-reality displays with metasurface waveguidesNature10.1038/s41586-024-07386-0629:8013(791-797)Online publication date: 8-May-2024
https://doi.org/10.1038/s41586-024-07386-0
Tseng EKuo GBaek SMatsuda NMaimone ASchiffers FChakravarthula PFu QHeidrich WLanman DHeide F(2024)Neural étendue expander for ultra-wide-angle high-fidelity holographic displayNature Communications10.1038/s41467-024-46915-315:1Online publication date: 22-Apr-2024
https://doi.org/10.1038/s41467-024-46915-3
Show More Cited By

Index Terms

High resolution étendue expansion for holographic displays
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Displays and imagers

Recommendations

Wirtinger holography for near-eye displays

Near-eye displays using holographic projection are emerging as an exciting display approach for virtual and augmented reality at high-resolution without complex optical setups --- shifting optical complexity to computation. While precise phase ...
Holographic near-eye displays for virtual and augmented reality

We present novel designs for virtual and augmented reality near-eye displays based on phase-only holographic projection. Our approach is built on the principles of Fresnel holography and double phase amplitude encoding with additional hardware, phase ...
Neural Holographic Near-eye Displays for Virtual Reality
SIGGRAPH '23: ACM SIGGRAPH 2023 Emerging Technologies

By manipulating light as a wavefront, holographic displays have the potential to revolutionize virtual reality (VR) and augmented reality (AR) systems. These displays support 3D focus cues for visual comfort, vision correcting capabilities, and high ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 39, Issue 4

August 2020

1732 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3386569

Editor:
Szymon Rusinkiewicz
Princeton University

Issue’s Table of Contents

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2020

Published in TOG Volume 39, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
2,708
Total Downloads

Downloads (Last 12 months)709
Downloads (Last 6 weeks)81

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cheng CWang JWu ZZhou JWang JChen C(2025)Pre-compensated annealing gradient descent for spherical holographyOptics Communications10.1016/j.optcom.2024.131049574(131049)Online publication date: Jan-2025
https://doi.org/10.1016/j.optcom.2024.131049
Gopakumar MLee GChoi SChao BPeng YKim JWetzstein G(2024)Full-colour 3D holographic augmented-reality displays with metasurface waveguidesNature10.1038/s41586-024-07386-0629:8013(791-797)Online publication date: 8-May-2024
https://doi.org/10.1038/s41586-024-07386-0
Tseng EKuo GBaek SMatsuda NMaimone ASchiffers FChakravarthula PFu QHeidrich WLanman DHeide F(2024)Neural étendue expander for ultra-wide-angle high-fidelity holographic displayNature Communications10.1038/s41467-024-46915-315:1Online publication date: 22-Apr-2024
https://doi.org/10.1038/s41467-024-46915-3
Jang CBang KChae MLee BLanman D(2024)Waveguide holography for 3D augmented reality glassesNature Communications10.1038/s41467-023-44032-115:1Online publication date: 2-Jan-2024
https://doi.org/10.1038/s41467-023-44032-1
Wang ZPang YLiang LFeng QLv GWang AMing H(2024)Holographic near-eye display with improved image quality and depth cue based on pupil optimizationOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108098176(108098)Online publication date: May-2024
https://doi.org/10.1016/j.optlaseng.2024.108098
Shi LRyu DMatusik W(2024)Ergonomic‐Centric Holography: Optimizing Realism, Immersion, and Comfort for Holographic DisplayLaser & Photonics Reviews10.1002/lpor.20230065118:4Online publication date: 25-Feb-2024
https://doi.org/10.1002/lpor.202300651
Yu PLiu YWang ZLiang JLiu XLi YQiu CGong L(2023)Ultrahigh-density 3D holographic projection by scattering-assisted dynamic holographyOptica10.1364/OPTICA.48305710:4(481)Online publication date: 6-Apr-2023
https://doi.org/10.1364/OPTICA.483057
Wang ZLv GPang YFeng QWang AMing H(2023)Lens array-based holographic 3D display with an expanded field of view and eyeboxOptics Letters10.1364/OL.50518148:21(5559)Online publication date: 17-Oct-2023
https://doi.org/10.1364/OL.505181
Wang JWang JZhou JZhang YWu Y(2023)Crosstalk-free for multi-plane holographic display using double-constraint stochastic gradient descentOptics Express10.1364/OE.49959531:19(31142)Online publication date: 6-Sep-2023
https://doi.org/10.1364/OE.499595
Xia XWang WGuan FYang FShui XZheng HYu YPeng Y(2023)Exploring angular-steering illumination-based eyebox expansion for holographic displaysOptics Express10.1364/OE.49893831:19(31563)Online publication date: 8-Sep-2023
https://doi.org/10.1364/OE.498938
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents