research-article

Learning to Remove Soft Shadows

Authors:

Gabriel J. BrostowAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 34, Issue 5

Article No.: 153, Pages 1 - 15

https://doi.org/10.1145/2732407

Published: 03 November 2015 Publication History

Abstract

Manipulated images lose believability if the user's edits fail to account for shadows. We propose a method that makes removal and editing of soft shadows easy. Soft shadows are ubiquitous, but remain notoriously difficult to extract and manipulate. We posit that soft shadows can be segmented, and therefore edited, by learning a mapping function for image patches that generates shadow mattes. We validate this premise by removing soft shadows from photographs with only a small amount of user input.

Given only broad user brush strokes that indicate the region to be processed, our new supervised regression algorithm automatically unshadows an image, removing the umbra and penumbra. The resulting lit image is frequently perceived as a believable shadow-free version of the scene. We tested the approach on a large set of soft shadow images, and performed a user study that compared our method to the state-of-the-art and to real lit scenes. Our results are more difficult to identify as being altered and are perceived as preferable compared to prior work.

Supplementary Material

gryka (gryka.zip)

Supplemental movie, appendix, image and software files for, Learning to Remove Soft Shadows

Download
55.51 MB

MP4 File (a153.mp4)

Download
14.57 MB

References

[1]

E. Arbel and H. Hel-Or. 2011. Shadow removal using intensity surfaces and texture anchor points. IEEE Trans. Pattern Anal. Mach. Intell. 33, 6.

Digital Library

[2]

C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. 2009. PatternMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3.

Digital Library

[3]

J. T. Barron and J. Malik. 2012. Color constancy, intrinsic images, and shape estimation. In Proceedings of the European Conference on Computer Vision (ECCV'12).

Digital Library

[4]

H. Barrow and J. Tenenbaum. 1978. Recovering intrinsic scene characteristics from images. Comput. Vis. Syst. 157.

[5]

A. Bousseau, S. Paris, and F. Durand. 2009. User assisted intrinsic images. ACM Trans. Graph. 28, 5.

Digital Library

[6]

I. Boyadzhiev, S. Paris, and K. Bala. 2013. User-assisted image compositing for photographic lighting. In Proceedings of the Annual ACM Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'13). ACM Press, New York.

Digital Library

[7]

Y. Boykov, O. Veksler, and R. Zabih. 2001. Efficient approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 20, 12.

Digital Library

[8]

L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen. 1984. Classification and Regression Trees. Chapman & Hall/CRC.

[9]

Y. Y. Chuang, D. B. Goldman, B. Curless, D. H. Salesin, and R. Szeliski. 2003. Shadow matting and compositing. ACM Trans. Graph. 22, 3.

Digital Library

[10]

A. Criminisi, P. Péerez, and K. Toyama. 2003. Object removal by exemplar-based inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'03).

[11]

A. Criminisi, D. Robertson, E. Konukoglu, J. Shotton, S. Pathak, S. White, and K. Siddiqui. 2003. Regression forests for efficient anatomy detection and localization in computed tomography scans. Med. Image Anal. 17, 8.

[12]

C. Farabet, C. Couprie, L. Najman, and Y. LeCun. 2013. Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35, 8, 1915--1929.

Digital Library

[13]

G. Finlayson, M. Drew, and C. Lu. 2009. Entropy minimization for shadow removal. Int. J. Comput. Vis. 85, 1.

Digital Library

[14]

D. Glasner, S. Bagon, and M. Irani. 2009. Super-resolution from a single image. In Proceedings of the International Conference on Computer Vision (ICCV'09).

[15]

G. Griffin, A. Holub, and P. Perona. 2007. Caltech-256 object category dataset. Tech. rep. 7694, California Institute of Technology. http://authors. library.caltech.edu/7694/1/CNS-TR-2007-001.pdf.

[16]

R. Grosse, M. K. Johnson, E. H. Adelson, and W. T. Freeman. 2009. Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In Proceedings of the International Conference on Computer Vision (ICCV'09).

[17]

R. Guo, Q. Dai, and D. Hoiem. 2012. Paired regions for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 35, 12.

Digital Library

[18]

Y. HaCohen, E. Schechtman, D. Goldman, and D. Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Graph. 30, 4, 70:1--70:9.

Digital Library

[19]

J. Hays and A. A. Efros. 2007. Scene completion using millions of photographs. ACM Trans. Graph. 26, 3.

Digital Library

[20]

E. Kee, J. F. O'Brien, and H. Farid. 2013. Exposing photo manipulation with inconsistent shadows. ACM Trans. Graph. 32, 4.

Digital Library

[21]

J. M. Kennedy. 1974. A Psychology of Picture Perception. Jossey-Bass.

[22]

V. Kolmogorov. 2006. Convergent tree-reweighted message passing for energy minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28, 10.

Digital Library

[23]

J. Kopf, W. Kienzle, S. Drucker, and S. B. Kang. 2012. Quality prediction for image completion. ACM Trans. Graph. 31, 6.

Digital Library

[24]

J. K. Kruschke. 2011. Doing Bayesian Data Analysis. Academic Press.

Digital Library

[25]

P. Laffont, A. Bousseau, and G. Drettakis. 2013. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. Visual. Comut. Graph. 19, 2.

Digital Library

[26]

E. H. Land and J. J. McCann. 1971. Lightness and retinex theory. J. Optical Soc. Amer. 61, 1.

[27]

A. Levin, D. Lischinski, and Y. Weiss. 2008. A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2.

Digital Library

[28]

Y. D. Lockerman, S. Xue, J. Dorsey, and H. Rushmeier. 2013. Creating texture examplars from unconstrained images. In Proceedings of the International Conference on Computer-Aided Design and Computer Graphics (CAD/Graphics'13).

Digital Library

[29]

O. Mac Aodha, G. J. Brostow, and M. Pollefeys. 2010. Segmenting video into classes of algorithm-suitability. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'10).

[30]

A. Mohan, J. Tumblin, and P. Choudhury. 2007. Editing soft shadows in a digital photograph. IEEE Comput. Graph. Appl. 27, 2.

Digital Library

[31]

Y. Pritch, E. Kav-Venaki, and S. Peleg. 2009. Shift-map image editing. In Proceedings of the International Conference on Computer Vision (ICCV'09).

[32]

P. Rademacher, J. Lengyel, E. Cutrell, and T. Whitted. 2001. Measuring the perception of visual realism in images. In Proceedings of the Eurographics Workshop on Rendering Techniques (EGWR'01). Eurographics Association.

Digital Library

[33]

M. Reynolds, J. Doboš, L. Peel, T. Weyrich, and G. J. Brostow. 2011. Capturing time-of-flight data with confidence. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'11).

Digital Library

[34]

Y. Shih, S. Paris, F. Durand, and W. Freeman. 2013. Data-driven hallucination for different times of day from a single outdoor photo. In Proceedings of the Annual ACM Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'13).

[35]

Y. Shor and D. Lischinski. 2008. The shadow meets the mask: Pyramid-based shadow removal. Comput. Graph. Forum 27, 2.

[36]

J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio, R. Moore, P. Kohli, A. Criminisi, A. Kipman, and A. Blake. 2012. Efficient human pose estimation from single depth images. IEEE Trans. Pattern Anal. Mach. Intell. 35, 12.

Digital Library

[37]

P. Sinha and E. Adelson. 1993. Recovering reflectance and illumination in a world of painted polyhedra. In Proceedings of the International Conference on Computer Vision (ICCV'93).

[38]

J. Sun, L. Yuan, J. Jia, and H. Shum. 2005. Image completion with structure propagation. ACM Trans. Graph. 24, 3.

Digital Library

[39]

K. Tang, J. Yang, and J. Wang. 2014. Investigating haze-relevant features in a learning framework for image dehazing. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'14).

Digital Library

[40]

M. F. Tappen, W. T. Freeman, and E. H. Adelson. 2005. Recovering intrinsic images for a single image. IEEE Trans. Pattern Anal. Mach. Intell. 27, 9.

Digital Library

[41]

J. Wang, M. Agrawala, and M. F. Cohen. 2007. Soft scissors: An interactive tool for realtime high quality matting. ACM Trans. Graph. 26, 3.

Digital Library

[42]

Y. Weiss. 2001. Deriving intrinsic images from image sequences. In Proceedings of the International Conference on Computer Vision (ICCV'01).

[43]

T. Wu, C. Tang, M. S. Brown, and H. Shum. 2007. Natural shadow matting. ACM Trans. Graph. 26, 8.

Digital Library

Cited By

Yoon JShu ZRen MZhang CHold-Geoffroy YSingh KZhang H(2024)Generative Portrait Shadow RemovalACM Transactions on Graphics10.1145/368790343:6(1-13)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687903
Chen ZWan LXiao YZhu LFu H(2024)Learning Physical-Spatio-Temporal Features for Video Shadow RemovalIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.336991034:7(5830-5842)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3369910
Xue JHuang HZhou ZXu SChen A(2024)LRGAN: Learnable Weighted Recurrent Generative Adversarial Network for End-to-End Shadow Generation2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650634(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650634
Show More Cited By

Index Terms

Learning to Remove Soft Shadows
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding

Recommendations

Shadows and Soft Shadows with Participating Media Using Splatting

This paper describes an efficient algorithm to model the light attenuation due to a participating media with low albedo. Here, we consider the light attenuation along a ray, as well as the light attenuation emanating from a surface. The light ...
Distributed rendering of interactive soft shadows
Parallel graphics and visualisation

Recently several distributed rendering systems have been developed that exploit a cluster of commodity computers by connecting host graphics cards over a fast network to form a compositing pipeline. This paper introduces an algorithm for one such system ...
Real-time soft shadows in dynamic scenes using spherical harmonic exponentiation

Previous methods for soft shadows numerically integrate over many light directions at each receiver point, testing blocker visibility in each direction. We introduce a method for real-time soft shadows in dynamic scenes illuminated by large, low-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 34, Issue 5

October 2015

188 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2843519

Editor:
Kavita Bala
Cornell University

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2015

Accepted: 01 January 2015

Revised: 01 January 2015

Received: 01 November 2013

Published in TOG Volume 34, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

UK EPSRC-funded Eng. Doctorate Centre in Virtual Environments, Imaging, and Visualisation
Leverhulme Visiting Professor
EU project http://www.cr-play.eu/
Anthropics Technology Ltd
UK EPSRC

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

105
Total Citations
View Citations
787
Total Downloads

Downloads (Last 12 months)49
Downloads (Last 6 weeks)7

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yoon JShu ZRen MZhang CHold-Geoffroy YSingh KZhang H(2024)Generative Portrait Shadow RemovalACM Transactions on Graphics10.1145/368790343:6(1-13)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687903
Chen ZWan LXiao YZhu LFu H(2024)Learning Physical-Spatio-Temporal Features for Video Shadow RemovalIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.336991034:7(5830-5842)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3369910
Xue JHuang HZhou ZXu SChen A(2024)LRGAN: Learnable Weighted Recurrent Generative Adversarial Network for End-to-End Shadow Generation2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650634(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650634
Wang CYu YGuo LWen B(2024)Benchmarking Adversarial Robustness of Image Shadow Removal with Shadow-Adaptive AttacksICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10446953(13126-13130)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10446953
Vasluianu FSeizinger TZhou ZWu ZChen CTimofte RDong WZhou HTian YChen JFu XLu XZhu YWang XLi DXiao JZhang YZha ZZhang ZZhao SWang BLuo YWei YZhao ZSun LYang TPan JDong JTang JBenjdira BNassif MKoubaa AElhayek AAli ATokoro KKawai KYokoyama KSeno TKondo YUkita NLi CYang BWu ZChen GYu YChen SZhang KYe TZou WLin YXing ZBai JChai WZhu LMaheshwari RVerma RTekchandani RHambarde PTazi SVipparthi SMurala SLee JKim SA SKhujaev NTsoy RGao FYan WShao WZhang DChen BZhang SQian YChen YZhou YTong TWei RSun RLiu YAkalwadi NJoshi AMalagi SDesai CTabib RMudenagudi UMurtaza AKhairuddin UFaudzi ADukre ADeshmukh VPhutke SKulkarni AVipparthi SGonde AMurala SK AN MPriya SHao WYan XFu M(2024)NTIRE 2024 Image Shadow Removal Challenge Report2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00654(6547-6570)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00654
Lu XZhu YWang XLi DXiao JZhang YFu XZha Z(2024)HirFormer: Dynamic High Resolution Transformer for Large-Scale Image Shadow Removal2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00651(6513-6523)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00651
Li DLu XZhu YWang XXiao JZhang YFu XZha Z(2024)Shadow Removal via Global Residual Free Unet and Shadow Generation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00634(6307-6316)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00634
Dong WZhou HTian YSun JLiu XZhai GChen J(2024)ShadowRefiner: Towards Mask-free Shadow Removal via Fast Fourier Transformer2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00625(6208-6217)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00625
Li CYang BWu ZChen GYu YZhou S(2024)Shadow Removal based on Diffusion, Segmentation and Super-resolution Models2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00611(6045-6054)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00611
Kubiak NMustafa APhillipson GJolly SHadfield S(2024)S3R-Net: A Single-Stage Approach to Self-Supervised Shadow Removal2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00597(5898-5908)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00597
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents