research-article

Intrinsic images in the wild

Authors:

Noah SnavelyAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 4

Article No.: 159, Pages 1 - 12

https://doi.org/10.1145/2601097.2601206

Published: 27 July 2014 Publication History

Abstract

Intrinsic image decomposition separates an image into a reflectance layer and a shading layer. Automatic intrinsic image decomposition remains a significant challenge, particularly for real-world scenes. Advances on this longstanding problem have been spurred by public datasets of ground truth data, such as the MIT Intrinsic Images dataset. However, the difficulty of acquiring ground truth data has meant that such datasets cover a small range of materials and objects. In contrast, real-world scenes contain a rich range of shapes and materials, lit by complex illumination.

In this paper we introduce Intrinsic Images in the Wild, a large-scale, public dataset for evaluating intrinsic image decompositions of indoor scenes. We create this benchmark through millions of crowdsourced annotations of relative comparisons of material properties at pairs of points in each scene. Crowdsourcing enables a scalable approach to acquiring a large database, and uses the ability of humans to judge material comparisons, despite variations in illumination. Given our database, we develop a dense CRF-based intrinsic image algorithm for images in the wild that outperforms a range of state-of-the-art intrinsic image algorithms. Intrinsic image decomposition remains a challenging problem; we release our code and database publicly to support future research on this problem, available online at http://intrinsic.cs.cornell.edu/.

Supplementary Material

ZIP File (a159-bell.zip)

Supplemental material.

Download
180.96 MB

MP4 File (a159-sidebyside.mp4)

Download
25.16 MB

References

[1]

Adams, A., Baek, J., and Davis, A. 2010. Fast high-dimensional filtering using the permutohedral lattice. Computer Graphics Forum (Eurographics) 29, 2.

[2]

Barron, J. T., and Malik, J. 2012. Color constancy, intrinsic images, and shape estimation. In Proc. European Conference on Computer Vision.

Digital Library

[3]

Barron, J. T., and Malik, J. 2012. Shape, albedo, and illumination from a single image of an unknown object. In Proc. Computer Vision and Pattern Recognition.

Digital Library

[4]

Barron, J. T., and Malik, J. 2013. Intrinsic scene properties from a single RGB-D image. In Proc. Computer Vision and Pattern Recognition.

Digital Library

[5]

Barron, J. T., and Malik, J. 2013. Shape, illumination, and reflectance from shading. Tech. rep., UC Berkeley.

[6]

Bell, S., Upchurch, P., Snavely, N., and Bala, K. 2013. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Trans. on Graphics (SIGGRAPH) 32, 4.

Digital Library

[7]

Bostock, M., 2013. D3. http://d3js.org/. {Online; accessed 24-Mar-2014}.

[8]

Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. on Graphics (SIGGRAPH Asia) 28, 5.

Digital Library

[9]

Boyadzhiev, I., Paris, S., and Bala, K. 2013. User-assisted image compositing for photographic lighting. ACM Trans. on Graphics (SIGGRAPH) 32, 4.

Digital Library

[10]

Branson, S., Wah, C., Babenko, B., Schroff, F., Welinder, P., Perona, P., and Belongie, S. 2010. Visual recognition with humans in the loop. In Proc. European Conference on Computer Vision.

Digital Library

[11]

Carroll, R., Ramamoorthi, R., and Agrawala, M. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. on Graphics (SIGGRAPH).

Digital Library

[12]

Chen, Q., and Koltun, V. 2013. A simple model for intrinsic image decomposition with depth cues. In Proc. International Conference on Computer Vision.

Digital Library

[13]

Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., and Singh, M. 2009. How well do line drawings depict shape? ACM Trans. on Graphics (SIGGRAPH) 28, 3.

Digital Library

[14]

Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. In J. Roy. Statistical Society.

[15]

Felzenszwalb, P., McAllester, D., and Ramanan, D. 2008. A discriminatively trained, multiscale, deformable part model. In Proc. Computer Vision and Pattern Recognition.

[16]

Garces, E., Munoz, A., Lopez-Moreno, J., and Gutierrez, D. 2012. Intrinsic images by clustering. Computer Graphics Forum (Eurographics Symposium on Rendering) 31, 4.

Digital Library

[17]

Gehler, P., Rother, C., Kiefel, M., Zhang, L., and Scholkopf, B. 2011. Recovering intrinsic images with a global sparsity prior on reflectance. In Neural Information Processing Systems.

[18]

Gingold, Y., Shamir, A., and Cohen-Or, D. 2012. Micro perceptual human computation. ACM Trans. on Graphics 31, 5.

Digital Library

[19]

Grosse, R., Johnson, M. K., Adelson, E. H., and Freeman, W. T. 2009. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In Proc. International Conference on Computer Vision.

[20]

Haber, T., Fuchs, C., Bekaer, P., Seidel, H.-P., Goesele, M., and Lensch, H. P. 2009. Relighting objects from image collections. In Proc. Computer Vision and Pattern Recognition.

[21]

Hauagge, D., Wehrwein, S., Bala, K., and Snavely, N. 2013. Photometric ambient occlusion. Proc. Computer Vision and Pattern Recognition.

Digital Library

[22]

Krähenbühl, P., and Koltun, V. 2011. Efficient inference in fully connected CRFs with gaussian edge potentials. In Neural Information Processing Systems.

[23]

Krähenbühl, P., and Koltun, V. 2013. Parameter learning and convergent inference for dense random fields. In Proc. International Conference on Machine Learning.

[24]

Laffont, P.-Y., Bousseau, A., Paris, S., Durand, F., and Drettakis, G. 2012. Coherent intrinsic images from photo collections. ACM Trans. on Graphics (SIGGRAPH Asia) 31, 6.

Digital Library

[25]

Laffont, P., Bousseau, A., and Drettakis, G. 2013. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. on Visualization and Computer Graphics 19, 2.

Digital Library

[26]

Land, E. H., and McCann, J. J. 1971. Lightness and retinex theory. J. Opt. Soc. Am. 61, 1.

[27]

Liao, Z., Rock, J., Wang, Y., and Forsyth, D. 2013. Non-parametric filtering for geometric detail extraction and material representation. In Proc. Computer Vision and Pattern Recognition.

Digital Library

[28]

Liu, X., Jiang, L., Wong, T.-T., and Fu, C.-W. 2012. Statistical invariance for texture synthesis. IEEE Trans. on Visualization and Computer Graphics 18, 11.

Digital Library

[29]

Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proc. Conference on Comp. Graphics and Interactive Techniques (SIGGRAPH).

Digital Library

[30]

Omer, I., and Werman, M. 2004. Color lines: image specific color representation. In Proc. Computer Vision and Pattern Recognition.

Digital Library

[31]

Ostrovsky, Y., Cavanagh, P., and Sinha, P. 2005. Perceiving illumination inconsistencies in scenes. Perception 34, 11.

[32]

Rubinstein, M., Gutierrez, D., Sorkine, O., and Shamir, A. 2010. A comparative study of image retargeting. ACM Trans. on Graphics (SIGGRAPH Asia) 29, 6.

Digital Library

[33]

Shen, L., and Yeo, C. 2011. Intrinsic images decomposition using a local and global sparse representation of reflectance. In Proc. Computer Vision and Pattern Recognition.

Digital Library

[34]

Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In Proc. Computer Vision and Pattern Recognition.

[35]

Shen, J., Yang, X., Jia, Y., and Li, X. 2011. Intrinsic images using optimization. In Proc. Computer Vision and Pattern Recognition.

Digital Library

[36]

Tappen, M., Freeman, W., and Adelson, E. 2005. Recovering intrinsic images from a single image. IEEE Trans. on Pattern Analysis and Machine Intelligence.

Digital Library

[37]

Tappen, M. F., Adelson, E. H., and Freeman, W. T. 2006. Estimating intrinsic component images using non-linear regression. In Proc. Computer Vision and Pattern Recognition.

Digital Library

[38]

van Wijk, J. J., and Nuij, W. A. 2003. Smooth and efficient zooming and panning. In Proc. IEEE Conference on Information Visualization (INFOVIS).

Digital Library

[39]

Weiss, Y. 2001. Deriving intrinsic images from image sequences. In Proc. International Conference on Computer Vision.

[40]

Welinder, P., Branson, S., Belongie, S., and Perona, P. 2010. The multidimensional wisdom of crowds. In Neural Information Processing Systems.

[41]

Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., and Lin, S. 2012. A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. on Pattern Analysis and Machine Intelligence 34, 7.

Digital Library

Cited By

Ulucan DUlucan OEbner M(2025)Challenges and Applications of Intrinsic Image Decomposition: A Short ReviewSN Computer Science10.1007/s42979-025-03659-16:2Online publication date: 30-Jan-2025
https://doi.org/10.1007/s42979-025-03659-1
Guan YLécué FChen JLi RPan JWooldridge MDy JNatarajan S(2024)Knowledge-aware neuron interpretation for scene classificationProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i3.27965(1950-1958)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i3.27965
Careaga CAksoy Y(2024)Colorful Diffuse Intrinsic Image Decomposition in the WildACM Transactions on Graphics10.1145/368798443:6(1-12)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687984
Show More Cited By

Index Terms

Intrinsic images in the wild
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding

Recommendations

An L₁ image transform for edge-preserving smoothing and scene-level intrinsic decomposition

Identifying sparse salient structures from dense pixels is a longstanding problem in visual computing. Solutions to this problem can benefit both image manipulation and understanding. In this paper, we introduce an image transform based on the L₁ norm ...
Multiview Intrinsic Images of Outdoors Scenes with an Application to Relighting

We introduce a method to compute intrinsic images for a multiview set of outdoor photos with cast shadows, taken under the same lighting. We use an automatic 3D reconstruction from these photos and the sun direction as input and decompose each image ...
Recovering Intrinsic Images from a Single Image

Interpreting real-world images requires the ability distinguish the different characteristics of the scene that lead to its final appearance. Two of the most important of these characteristics are the shading and reflectance of each point in the scene. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 4

July 2014

1366 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2601097

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2014

Published in TOG Volume 33, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

307
Total Citations
View Citations
2,018
Total Downloads

Downloads (Last 12 months)114
Downloads (Last 6 weeks)16

Reflects downloads up to 15 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ulucan DUlucan OEbner M(2025)Challenges and Applications of Intrinsic Image Decomposition: A Short ReviewSN Computer Science10.1007/s42979-025-03659-16:2Online publication date: 30-Jan-2025
https://doi.org/10.1007/s42979-025-03659-1
Guan YLécué FChen JLi RPan JWooldridge MDy JNatarajan S(2024)Knowledge-aware neuron interpretation for scene classificationProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i3.27965(1950-1958)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i3.27965
Careaga CAksoy Y(2024)Colorful Diffuse Intrinsic Image Decomposition in the WildACM Transactions on Graphics10.1145/368798443:6(1-12)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687984
Yoon JShu ZRen MZhang CHold-Geoffroy YSingh KZhang H(2024)Generative Portrait Shadow RemovalACM Transactions on Graphics10.1145/368790343:6(1-13)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687903
Luo JCeylan DYoon JZhao NPhilip JFrühstück ALi WRichardt CWang T(2024)IntrinsicDiffusion: Joint Intrinsic Layers from Latent Diffusion ModelsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657472(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657472
Zeng ZDeschaintre VGeorgiev IHold-Geoffroy YHu YLuan FYan LHašan M(2024)RGB↔X: Image decomposition and synthesis using material- and lighting-aware diffusion modelsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657445(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657445
Xu RZhang LZhang F(2024)Intrinsic Omnidirectional Image Decomposition With Illumination Pre-ExtractionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.336634330:7(4416-4428)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3366343
Zhan YWang TShao TZhou K(2024)Pattern Guided UV Recovery for Realistic Video Garment TexturingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.335472730:12(7531-7543)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3354727
Luo JZhao NLi WRichardt C(2024)CRefNet: Learning Consistent Reflectance Estimation With a Decoder-Sharing TransformerIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.333787030:9(6407-6420)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3337870
Xie WLiu TGu Y(2024)Intrinsic Hyperspectral Image Recovery for UAV Strips StitchingIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.343302462(1-13)Online publication date: 2024
https://doi.org/10.1109/TGRS.2024.3433024
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents