article

Natural shadow matting

Authors:

Chi-Keung Tang,

Michael S. Brown,

Heung-Yeung ShumAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 26, Issue 2

Pages 8 - es

https://doi.org/10.1145/1243980.1243982

Published: 01 June 2007 Publication History

Abstract

This article addresses the problem of natural shadow matting, the removal or extraction of natural shadows from a single image. Because textures are maintained in the shadowless image after the extraction process, our approach produces some of the best results to date among shadow removal techniques. Using the image formation equation typical of computer vision, we advocate a new model for shadow formation where shadow effect is understood as light attenuation instead of a mixture of two colors governed by the conventional matting equation. This leads to a new shadow equation with fewer unknowns to solve, where a three-channel shadow matte and a shadowless image are considered in our optimization. Our problem is formulated as one of energy minimization guided by user-supplied hints in the form of a quadmap which can be specified easily by the user. This formulation allows for robust shadow matte extraction while maintaining texture in the shadowed region by considering color transfer, texture gradient, and shadow smoothness. We demonstrate the usefulness of our approach in shadow removal, image matting, and compositing.

References

[1]

Baba, M., Mukunoki, M., and Asada, N. 2004. Shadow removal from a real image based on shadow density. In Proceedings of the SIGGRAPH 2004 Sketches and Applications.

Digital Library

[2]

Barrow, H. and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. In Symposium of Center for Visual Science (CVS'78). 3--26.

[3]

Bertalmio, M., Sapiro, G., Ballester, C., and Caselles, V. 2000. Image inpainting. SIGGRAPH 2000, 417--424.

Digital Library

[4]

Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'01). Vol. 2, 264--271.

[5]

Chuang, Y.-Y., Goldman, D. B., Curless, B., Salesin, D. H., and Szeliski, R. 2003. Shadow matting and compositing. ACM Trans. Graph. 22, 3, 494--500.

Digital Library

[6]

Drori, I., Cohen-Or, D., and Yeshurun, H. 2003. Fragment-based image completion. ACM Trans Graph. 22, 303--312.

Digital Library

[7]

Duda, R., Hart, P., and Stork, D. 2001. Pattern Classification. John Wiley.

Digital Library

[8]

Efros, A. and Leung, T. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the International Conference on Computer Vision (ICCV'99). 1033--1038.

Digital Library

[9]

Finlayson, G., Drew, M., and Lu, C. 2004. Intrinsic images by entropy minimization. In European Conference on Computer Vision (ECCV'04). Vol 3, 582--595.

[10]

Finlayson, G. D., Hordley, S. D., and Drew, M. S. 2002. Removing shadows from images. Proceedings of European Conference on Computer Vision (ECCV'02), Vol. 4, 823--836.

Digital Library

[11]

Jia, J. and Tang, C.-K. 2004. Inference of segmented color and texture description by tensor voting. Trans. Patt. Analy. and Mach. Intell. 26, 6 (June), 771--786.

Digital Library

[12]

Land, E. and McCann, J. 1971. Lightness and the retinex theory. J. Opt. Soc. Am, 1--11.

[13]

Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Trans. Graph. 23, 3, 303--308.

Digital Library

[14]

Mitchell, T. 1997. Machine Learning. McGraw Hill.

Digital Library

[15]

Mortensen, E. N. and Barrett, W. A. 1995. Intelligent scissors for image composition. In SIGGRAPH '1995. 191--198.

Digital Library

[16]

Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In SIGGRAPH 2001. 433--442.

Digital Library

[17]

Porter, T. and Duff, T. 1984. Compositing digital images. Proceedings of SIGGRAPH 2004, 253--259.

Digital Library

[18]

Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comput. Graph. Applic. 21, 34--41.

Digital Library

[19]

Rother, C., Kolmogorov, V., and Blake, A. 2004. “grabcut”: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23, 3, 309--314.

Digital Library

[20]

Sun, J., Jia, J., Tang, C.-K., and Shum, H.-Y. 2004. Poisson matting. ACM Trans. Graph. 23, 3, 351--321.

Digital Library

[21]

Tappen, M., Freeman, W., and Adelson, E. 2002. Recovering intrinsic images from a single image. In MIT Advances in Modeling (AIM).

[22]

Weiss, Y. 2001. Deriving intrinsic images from image sequences. In International Conference on Computer Vision (ICCV), Vol. 2, 68--75.

[23]

Welsh, T., Ashikhmin, M., and Mueller, K. 2002. Transferring color to greyscale images. ACM Trans. Graph. 21, 277--280.

Digital Library

[24]

Wright, S. 2001. Digital Compositing for Film and Video. Focal Press.

Digital Library

[25]

Wu, T.-P. and Tang, C.-K. 2005. A Bayesian approach for shadow extraction from a single image. In Proceedings of the International Conference on Computer Vision (ICCV'05). IEEE, 480--487.

Digital Library

Cited By

Vasluianu FSeizinger TZhou ZWu ZChen CTimofte RDong WZhou HTian YChen JFu XLu XZhu YWang XLi DXiao JZhang YZha ZZhang ZZhao SWang BLuo YWei YZhao ZSun LYang TPan JDong JTang JBenjdira BNassif MKoubaa AElhayek AAli ATokoro KKawai KYokoyama KSeno TKondo YUkita NLi CYang BWu ZChen GYu YChen SZhang KYe TZou WLin YXing ZBai JChai WZhu LMaheshwari RVerma RTekchandani RHambarde PTazi SVipparthi SMurala SLee JKim SA SKhujaev NTsoy RGao FYan WShao WZhang DChen BZhang SQian YChen YZhou YTong TWei RSun RLiu YAkalwadi NJoshi AMalagi SDesai CTabib RMudenagudi UMurtaza AKhairuddin UFaudzi ADukre ADeshmukh VPhutke SKulkarni AVipparthi SGonde AMurala SK AN MPriya SHao WYan XFu M(2024)NTIRE 2024 Image Shadow Removal Challenge Report2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00654(6547-6570)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00654
Zhu XChow CChuah J(2024)From darkness to clarity: A comprehensive review of contemporary image shadow removal research (2017–2023)Image and Vision Computing10.1016/j.imavis.2024.105100(105100)Online publication date: Jun-2024
https://doi.org/10.1016/j.imavis.2024.105100
Zhang LLong CZhang XXiao C(2023)Exploiting Residual and Illumination with GANs for Shadow Detection and Shadow RemovalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/357174519:3(1-22)Online publication date: 25-Feb-2023
https://dl.acm.org/doi/10.1145/3571745
Show More Cited By

Index Terms

Natural shadow matting
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Learning to Remove Soft Shadows

Manipulated images lose believability if the user's edits fail to account for shadows. We propose a method that makes removal and editing of soft shadows easy. Soft shadows are ubiquitous, but remain notoriously difficult to extract and manipulate. We ...
Shadow Remover: Image Shadow Removal Based on Illumination Recovering Optimization
In this paper, we present a novel shadow removal system for single natural images as well as color aerial images using an illumination recovering optimization method. We first adaptively decompose the input image into overlapped patches according to the ...
Shadow matting and compositing
SIGGRAPH '03: ACM SIGGRAPH 2003 Papers

In this paper, we describe a method for extracting shadows from one natural scene and inserting them into another. We develop physically-based shadow matting and compositing equations and use these to pull a shadow matte from a source scene in which the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 26, Issue 2

June 07

106 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1243980

Issue’s Table of Contents

Copyright © 2007 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2007

Published in TOG Volume 26, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

98
Total Citations
View Citations
1,481
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)6

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vasluianu FSeizinger TZhou ZWu ZChen CTimofte RDong WZhou HTian YChen JFu XLu XZhu YWang XLi DXiao JZhang YZha ZZhang ZZhao SWang BLuo YWei YZhao ZSun LYang TPan JDong JTang JBenjdira BNassif MKoubaa AElhayek AAli ATokoro KKawai KYokoyama KSeno TKondo YUkita NLi CYang BWu ZChen GYu YChen SZhang KYe TZou WLin YXing ZBai JChai WZhu LMaheshwari RVerma RTekchandani RHambarde PTazi SVipparthi SMurala SLee JKim SA SKhujaev NTsoy RGao FYan WShao WZhang DChen BZhang SQian YChen YZhou YTong TWei RSun RLiu YAkalwadi NJoshi AMalagi SDesai CTabib RMudenagudi UMurtaza AKhairuddin UFaudzi ADukre ADeshmukh VPhutke SKulkarni AVipparthi SGonde AMurala SK AN MPriya SHao WYan XFu M(2024)NTIRE 2024 Image Shadow Removal Challenge Report2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00654(6547-6570)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00654
Zhu XChow CChuah J(2024)From darkness to clarity: A comprehensive review of contemporary image shadow removal research (2017–2023)Image and Vision Computing10.1016/j.imavis.2024.105100(105100)Online publication date: Jun-2024
https://doi.org/10.1016/j.imavis.2024.105100
Zhang LLong CZhang XXiao C(2023)Exploiting Residual and Illumination with GANs for Shadow Detection and Shadow RemovalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/357174519:3(1-22)Online publication date: 25-Feb-2023
https://dl.acm.org/doi/10.1145/3571745
Zhang LChen BLiu ZXiao C(2023)Facial Image Shadow Removal via Graph‐based Feature FusionComputer Graphics Forum10.1111/cgf.1494442:7Online publication date: 30-Oct-2023
https://doi.org/10.1111/cgf.14944
Fu YGai ZZhao HZhang SShan YWu YTang J(2023)Depth‐Aware Shadow RemovalComputer Graphics Forum10.1111/cgf.1469141:7(455-464)Online publication date: 20-Mar-2023
https://doi.org/10.1111/cgf.14691
Kerim Yucel MDimaridou VManganelli BOzay MDrosou ASaa-Garriga A(2023)LRA&LDRA: Rethinking Residual Predictions for Efficient Shadow Detection and Removal2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00490(4914-4924)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00490
Liu JWang QFan HLi WQu LTang Y(2023)A Decoupled Multi-Task Network for Shadow RemovalIEEE Transactions on Multimedia10.1109/TMM.2023.325227125(9449-9463)Online publication date: 2023
https://doi.org/10.1109/TMM.2023.3252271
Luo WXie XDeng KYang LLai J(2023)Learning Shadow Removal From Unpaired Samples via Reciprocal LearningIEEE Transactions on Image Processing10.1109/TIP.2023.328543932(3455-3464)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3285439
Vasluianu FSeizinger TTimofte R(2023)WSRD: A Novel Benchmark for High Resolution Image Shadow Removal2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW59228.2023.00181(1826-1835)Online publication date: Jun-2023
https://doi.org/10.1109/CVPRW59228.2023.00181
Zhang LHe YZhang QLiu ZZhang XXiao C(2023)Document Image Shadow Removal Guided by Color-Aware Background2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00181(1818-1827)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00181
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents