article

Lighting sensitive display

Authors:

Shree K. Nayar,

Peter N. Belhumeur,

Terry E. BoultAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 23, Issue 4

Pages 963 - 979

https://doi.org/10.1145/1027411.1027414

Published: 01 October 2004 Publication History

Abstract

Although display devices have been used for decades, they have functioned without taking into account the illumination of their environment. We present the concept of a lighting sensitive display (LSD)---a display that measures the incident illumination and modifies its content accordingly. An ideal LSD would be able to measure the 4D illumination field incident upon it and generate a 4D light field in response to the illumination. However, current sensing and display technologies do not allow for such an ideal implementation. Our initial LSD prototype uses a 2D measurement of the illumination field and produces a 2D image in response to it. In particular, it renders a 3D scene such that it always appears to be lit by the real environment that the display resides in. The current system is designed to perform best when the light sources in the environment are distant from the display, and a single user in a known location views the display.

The displayed scene is represented by compressing a very large set of images (acquired or rendered) of the scene that correspond to different lighting conditions. The compression algorithm is a lossy one that exploits not only image correlations over the illumination dimensions but also coherences over the spatial dimensions of the image. This results in a highly compressed representation of the original image set. This representation enables us to achieve high quality relighting of the scene in real time. Our prototype LSD can render 640 × 480 images of scenes under complex and varying illuminations at 15 frames per second using a 2 GHz processor. We conclude with a discussion on the limitations of the current implementation and potential areas for future research.

References

[1]

Antwerp, J. C. V. 1985. Automatic Brightness Control Apparatus. United States Patent 4,514,727 (April).

[2]

Avidan, S. 2002. Eigensegments: A spatio-temporal decomposition of an ensemble of images. In European Conference on Computer Vision (ECCV). Copenhagen, Denmark, III: 747 ff.

[3]

Barbier, B., Ediar, S., and Brun, J. 1991. System for the Display of Luminous Data with Improved Readability. United States Patent 5,057,744 (October).

[4]

Basri, R. and Jacobs, D. 2001. Lambertian reflectance and linear subspaces. In International Conference on Computer Vision. 383--390.

[5]

Belhumeur, P. N. and Kriegman, D. J. 1996. What is the set of images of an object under all possible lighting conditions? In IEEE Conference on Computer Vision and Pattern Recognition. 270--277.

[6]

Biggs, A. J. 1965. Compensator for Color-Television Receivers for Chromacity Variations in Ambient Light. United States Patent 3,200,193 (August).

[7]

Constable, D. W. 1978. Ambient Light Contrast and Color Control Circuit. United States Patent 4,090,216 (May).

[8]

Debevec, P., Hawkins, T., Tchou, C., Duiker, H., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In SIGGRAPH 00. 145--156.

[9]

Dorsey, J., Sillion, F. X., and Greenberg, D. P. 1991. Design and simulation of opera lighting and projection effects. In SIGGRAPH 91. 41--50.

[10]

Duda, R. and Hart, P. 1973. Pattern Classification and Scene Analysis. Wiley, New York.

[11]

Durand, F. and Dorsey, J. 2000. Interactive tone mapping. In Rendering Techniques 2000: 11th Eurographics Workshop on Rendering. 219--230.

[12]

Epstein, R., Hallinan, P., and Yuille, A. 1995. 5 plus or minus 2 eigenimages suffice: An empirical investigation of low-dimensional lighting models. In IEEE Workshop on Physics-Based Modeling in Computer Vision. 108--116.

[13]

Fitzgibbon, J. J. 1982. Dimmer and Dimmer Override Control for a Display Device. United States Patent 4,355,334 (October).

[14]

Georghiades, A., Kriegman, D., and Belhumeur, P. 1998. Illumination cones for recognition under variable lighting: Faces. In IEEE Conference on Computer Vision and Pattern Recognition. 52--59.

[15]

Gibson, V. R. 1964. Automatic Brightness-Contrast Control Using Photoresistive Element to Control Brightness and AGC Voltages in Response to Ambient Light. United States Patent 3,147,341 (September).

[16]

Hallinan, P. 1994. A low-dimensional representation of human faces for arbitrary lighting conditions. In IEEE Conference on Computer Vision and Pattern Recognition. 995--999.

[17]

Heijligers, H. 1962. Circuit Arrangement for Automatically Adjusting the Brightness and the Contrast in a Television Receiver. United States Patent 3,027,421 (March).

[18]

Jensen, H. W. 2001. Realistic Image Synthesis Using Photon Mapping. A. K. Peters, Natick, Massachusetts.

[19]

Kochanski, G. P. 1998. Retro-Focus Type Lens and Projection-Type Display Apparatus. United States Patent 5,854,661 (December).

[20]

Korda, P. 1965. Automatic Contrast and Brilliance Control System for Television Receivers. United States Patent 3,165,582 (January).

[21]

Koudelka, M., Belhumeur, P., Magda, S., and Kriegman, D. 2001. Image-based modeling and rendering of surfaces with arbitrary BRDFs. In IEEE Conference on Computer Vision and Pattern Recognition. 568--575.

[22]

Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J., and Fulk, D. 2000. The Digital Michelangelo Project: 3D scanning of large statues. In SIGGRAPH 00. 131--144.

[23]

Lin, Z., Wong, T., and Shum, H. 2002. Relighting with the reflected irradiance field: Representation, sampling and reconstruction. Int. J. Comput. Vision 49, 2--3 (September), 229--246.

[24]

Malzbender, T., Gelb, D., and Wolters, H. 2001. Polynomial texture maps. In SIGGRAPH 01. 519--528.

[25]

Matusik, W., Pfister, H., Ngan, A., Beardsley, P., Ziegler, R., and McMillan, L. 2002. Image-based 3d photography using opacity hulls. In SIGGRAPH 02. 427--437.

[26]

Miller, G. 1995. Volumetric hyper-reality, a computer graphics holy grail for the 21st century? In Graphics Interface '95. Quebec, Canada, 56--64.

[27]

Nayar, S. and Murase, H. 1994. The dimensionality of illumination manifolds in eigenspace. Tech. Rep. CUCS021-94, Columbia University, New York. August.

[28]

Nayar, S. K., Belhumeur, P. N., and Boult, T. E. 2001. Lighting sensitive display. In Technical Sketch, SIGGRAPH 01. Los Angeles.

[29]

Nayar, S. K. and Branzoi, V. 2003. Adaptive dynamic range imaging: Optical control of pixel exposures over space and time. In International Conference on Computer Vision. 1168--1175.

[30]

Newman, A. J. 1972. Method and Means for Providing a Lighted Display Having a Constant Contrast Ratio. United States Patent 3,649,755 (March).

[31]

Nimeroff, J., Simoncelli, E., and Dorsey, J. 1994. Efficient re-rendering of naturally illuminated environments. In Fifth Eurographics Workshop on Rendering. 359--373.

[32]

Nishino, K., Sato, Y., and Ikeuchi, K. 1999. Eigen-texture method: Appearance compression based on 3D model. In IEEE Conference on Computer Vision and Pattern Recognition. 618--624.

[33]

Ottenstein, J. 1993. Arrangement for Automatically Controlling Brightness of Cockpit Displays. United States Patent 5,270,818 (December).

[34]

Pattanaik, S. N., Ferwerda, J. A., Fairchild, M. D., and Greenberg, D. P. 1998. A multiscale model of adaptation and spatial vision for realistic image display. In Proceedings of SIGGRAPH 98. Computer Graphics Proceedings, Annual Conference Series. 287--298.

[35]

Prince, J. S., Herz, H. L., and Coleman, E. P. 1986. Brightness Control System for CRT Video Display. United States Patent 4,589,022 (May).

[36]

Ramamoorthi, R. and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In SIGGRAPH 01. 117--128.

[37]

Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J. 2002. Photographic tone reproduction for digital images. ACM Trans. Graph. 21, 3 (July), 267--276.

Digital Library

[38]

Sato, I., Sato, Y., and Ikeuchi, K. 1999. Acquiring a radiance distribution to superimpose virtual objects onto a real scene. IEEE Trans. Visual. Comp. Graph. 5, 1, 1--12.

Digital Library

[39]

Shashua, A. 1997. On photometric issues in 3d visual recognition from a single 2d image. Int. J. Comput. Vision 21, 1--2, 99--122.

Digital Library

[40]

Sloan, P., Kautz, J., and Snyder, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In SIGGRAPH 02. 527--536.

[41]

Smith, B. and Rowe, L. 1996. Compressed domain processing of JPEG encoded images. Real-Time Imaging 2 2, 2, 3--17.

[42]

Sunnybrook Technologies. 2003. High dynamic range displays. http://www.sunnybrook.com.

[43]

Szermy, N. 1968. Automatic Contrast and Brightness Control for Television Receiver Utilizing a Light Dependent Resistor. United States Patent 3,404,226 (October).

[44]

Teo, P., Simoncelli, E., and Heeger, D. 1997. Efficient linear rerendering for interactive lighting design. Tech. Rep. STAN-CS-TN-97-60, Stanford.

[45]

Thomas, L. P. 1963. Television Receiver Circuits. United States Patent 3,096,399 (July).

[46]

Tumblin, J. and Rushmeier, H. E. 1993. Tone reproduction for realistic images. IEEE Computer Graphics & Applications 13, 6 (Nov.), 42--48.

Digital Library

[47]

Ward, G. 1994. A contrast-based scalefactor for luminance display. In Graphics Gems IV. 415--421.

[48]

Wood, D., Azuma, D., Aldinger, K., Curless, B., Duchamp, T., Salesin, D., and Stuetzle, W. 2000. Surface light fields for 3D photography. In SIGGRAPH 00. 287--296.

[49]

Yabuuchi, S. 1990. Display System with Luminance Calculation. United States Patent 4,952,917 (August).

Cited By

Wu XXu JZhang XBao HHuang QShen YTompkin JXu W(2023)ScaNeRF: Scalable Bundle-Adjusting Neural Radiance Fields for Large-Scale Scene RenderingACM Transactions on Graphics10.1145/361836942:6(1-18)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618369
Olwal ADementyev A(2022)Hidden Interfaces for Ambient Computing: Enabling Interaction in Everyday Materials through High-brightness Visuals on Low-cost Matrix DisplaysProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517674(1-20)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517674
Murmann LGharbi MAittala MDurand F(2019)A Dataset of Multi-Illumination Images in the Wild2019 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV.2019.00418(4079-4088)Online publication date: Oct-2019
https://doi.org/10.1109/ICCV.2019.00418
Show More Cited By

Index Terms

Lighting sensitive display
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
      2. Image and video acquisition
        3D imaging
  2. Computer graphics
    1. Animation
    2. Image compression

Recommendations

Spherical Piecewise Constant Basis Functions for All-Frequency Precomputed Radiance Transfer

This paper presents a novel basis function, called spherical piecewise constant basis function (SPCBF), for precomputed radiance transfer. SPCBFs have several desirable properties: rotatability, ability to represent all-frequency signals, and support ...
Rendering for an interactive 360° light field display
SIGGRAPH '07: ACM SIGGRAPH 2007 papers

We describe a set of rendering techniques for an autostereoscopic light field display able to present interactive 3D graphics to multiple simultaneous viewers 360 degrees around the display. The display consists of a high-speed video projector, a ...
Visibility Silhouettes for Semi-Analytic Spherical Integration

At each shade point, the spherical visibility function encodes occlusion from surrounding geometry, in all directions. Computing this function is difficult and point-sampling approaches, such as ray-tracing or hardware shadow mapping, are traditionally ...

Reviews

Reviewer: Christian Laforte

This paper introduces the concept of a lighting sensitive display (LSD), a display that can render a three-dimensional (3D) scene such that it appears to be lit by the real environment surrounding it. The authors built a prototype LSD, which accurately displays a relighted virtual or acquired scene at 15 frames per second (fps), using a regular liquid crystal display (LCD) panel, a camera with a 170-degree field of view, and an off-the-shelf personal computer (PC). For a given static scene with a fixed camera position, the relighting process first involves pre-acquiring or pre-rendering a collection of thousands of images in which the lighting direction is systematically varied. The collection of images is then compressed through several stages of singular value decomposition (SVD), to achieve local dimensionality reduction on constant-size image blocks, and to further exploit lighting coherence across a whole image. This novel compression scheme produces three sets of heavily compressed matrices. At runtime, the relighting algorithm is reduced to a series of matrix multiplications, with a downsampled estimate of the illumination field taken from the camera. A wide range of detailed, complex lighting effects, such as soft and hard shadows and subsurface scattering, can be simulated in real time using this approach. The relighting algorithm assumes that the viewpoint and the scene's material and object placement are constant; these limitations considerably restrict its potential real-world applications. Still, this paper covers a wide range of interesting ideas that could prove useful to computer graphics practitioners. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 23, Issue 4

October 2004

145 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1027411

Issue’s Table of Contents

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2004

Published in TOG Volume 23, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
1,194
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu XXu JZhang XBao HHuang QShen YTompkin JXu W(2023)ScaNeRF: Scalable Bundle-Adjusting Neural Radiance Fields for Large-Scale Scene RenderingACM Transactions on Graphics10.1145/361836942:6(1-18)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618369
Olwal ADementyev A(2022)Hidden Interfaces for Ambient Computing: Enabling Interaction in Everyday Materials through High-brightness Visuals on Low-cost Matrix DisplaysProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517674(1-20)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517674
Murmann LGharbi MAittala MDurand F(2019)A Dataset of Multi-Illumination Images in the Wild2019 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV.2019.00418(4079-4088)Online publication date: Oct-2019
https://doi.org/10.1109/ICCV.2019.00418
Ma WWang QShi DLiu SCheng CShi QLi QLi TXu YMorishima SItoh YShiratori TYue YLindeman R(2018)Walking into ancient paintings with virtual candlesProceedings of the 24th ACM Symposium on Virtual Reality Software and Technology10.1145/3281505.3283378(1-2)Online publication date: 28-Nov-2018
https://dl.acm.org/doi/10.1145/3281505.3283378
Xu ZSunkavalli KHadap SRamamoorthi R(2018)Deep image-based relighting from optimal sparse samplesACM Transactions on Graphics10.1145/3197517.320131337:4(1-13)Online publication date: 30-Jul-2018
https://dl.acm.org/doi/10.1145/3197517.3201313
Thanikachalam NBaboulaz LFirmenich DSusstrunk SVetterli M(2017)Handheld Reflectance Acquisition of PaintingsIEEE Transactions on Computational Imaging10.1109/TCI.2017.27491823:4(580-591)Online publication date: Dec-2017
https://doi.org/10.1109/TCI.2017.2749182
Lou YHu J(2017)Passive lighting responsive three-dimensional integral imagingOptics Communications10.1016/j.optcom.2017.06.057402(498-501)Online publication date: Nov-2017
https://doi.org/10.1016/j.optcom.2017.06.057
Miyashita LIshihara KWatanabe YIshikawa M(2016)ZoeMatropeACM Transactions on Graphics10.1145/2897824.292592535:4(1-11)Online publication date: 11-Jul-2016
https://dl.acm.org/doi/10.1145/2897824.2925925
Levin AMaron HYarom M(2016)Passive light and viewpoint sensitive display of 3D content2016 IEEE International Conference on Computational Photography (ICCP)10.1109/ICCPHOT.2016.7492881(1-15)Online publication date: May-2016
https://doi.org/10.1109/ICCPHOT.2016.7492881
Tompkin JMuff SMcCann JPfister HKautz JAlexa MMatusik WLatulipe CHartmann BGrossman T(2015)Joint 5D Pen Input for Light Field DisplaysProceedings of the 28th Annual ACM Symposium on User Interface Software & Technology10.1145/2807442.2807477(637-647)Online publication date: 5-Nov-2015
https://dl.acm.org/doi/10.1145/2807442.2807477
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents