Abstract
Shadow removal is an important topic in image restoration, and it can benefit many computer vision tasks. State-of-the-art shadow-removal methods typically employ deep learning by minimizing a pixel-level difference between the de-shadowed region and their corresponding (pseudo) shadow-free version. After shadow removal, the shadow and non-shadow regions may exhibit inconsistent appearance, leading to a visually disharmonious image. To address this problem, we propose a style-guided shadow removal network (SG-ShadowNet) for better image-style consistency after shadow removal. In SG-ShadowNet, we first learn the style representation of the non-shadow region via a simple region style estimator. Then we propose a novel effective normalization strategy with the region-level style to adjust the coarsely re-covered shadow region to be more harmonized with the rest of the image. Extensive experiments show that our proposed SG-ShadowNet outperforms all the existing competitive models and achieves a new state-of-the-art performance on ISTD+, SRD, and Video Shadow Removal benchmark datasets. Code is available at: https://github.com/jinwan1994/SG-ShadowNet.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The RMSE is actually calculated by the mean absolute error (MAE) as [22].
References
Arbel, E., Hel-Or, H.: Shadow removal using intensity surfaces and texture anchor points. IEEE Trans. Pattern Anal. Mach. Intell. 33(6), 1202–1216 (2011)
Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
Caelles, S., et al.: One-shot video object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognitio, pp. 5320–5329 (2017)
Chen, Z., Long, C., Zhang, L., Xiao, C.: Canet: A context-aware network for shadow removal. In: International Conference on Computer Vision, pp. 4743–4752 (Oct 2021)
Cun, X., Pun, C.M., Shi, C.: Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan. In: AAAI, pp. 10680–10687 (2020)
Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. arXiv preprint arXiv:1610.07629 (2017)
Fu, L., et al.: Auto-exposure fusion for single-image shadow removal. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 10571–10580 (June 2021)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (June 2014)
Girshick, R.B.: Fast rcnn. In: International Conference on Computer Vision, pp. 1440–1448 (2015)
Gong, H., Cosker, D.: Interactive shadow removal and ground truth for variable scene categories. In: British Machine Vision Conference (2014)
Guo, C., et al.: Zero-reference deep curve estimation for low-light image enhancement. In: Conference on Computer Vision and Pattern Recognition, pp. 1777–1786 (2020)
Guo, R., Dai, Q., Hoiem, D.: Paired regions for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2956–2967 (2012)
Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1712–1722 (2019)
He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask r-cnn. In: International Conference on Computer Vision (Oct 2017)
Hu, X., Fu, C.W., Zhu, L., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. (2019)
Hu, X., Jiang, Y., Fu, C.W., Heng, P.A.: Mask-shadowgan: Learning to remove shadows from unpaired data. In: International Conference on Computer Vision (2019)
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: International Conference on Computer Vision (2017)
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the International Conference on Machine Learning, vol. 37, pp. 448–456 (2015)
Jin, Y., Sharma, A., Tan, R.T.: Dc-shadownet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network. In: International Conference on Computer Vision, pp. 5027–5036 (2021)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2015)
Le, H., Goncalves, B., Samaras, D., Lynch, H.: Weakly labeling the antarctic: The penguin colony case. In: IEEE Conference on Computer Vision and Pattern Recognition of Workshop, pp. 18–25 (2019)
Le, H., Samaras, D.: Shadow removal via shadow image decomposition. In: International Conference on Computer Vision (2019)
Le, H., Samaras, D.: From shadow segmentation to shadow removal. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 264–281. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_16
Le, H., Samaras, D.: Physics-based shadow image decomposition for shadow removal. IEEE Trans. Pattern Anal. Mach. Intell. (2021)
Ling, J., Xue, H., Song, L., Xie, R., Gu, X.: Region-aware adaptive instance normalization for image harmonization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9361–9370 (June 2021)
Liu, F., Gleicher, M.: Texture-consistent shadow removal. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5305, pp. 437–450. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88693-8_32
Liu, H., Wan, Z., Huang, W., Song, Y., Han, X., Liao, J.: Pd-gan: Probabilistic diverse gan for image inpainting. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9371–9381 (June 2021)
Liu, Z., Yin, H., Mi, Y., Pu, M., Wang, S.: Shadow removal by a lightness-guided network with training on unpaired data. IEEE Trans. Image Process. 30, 1853–1865 (2021)
Liu, Z., Yin, H., Wu, X., Wu, Z., Mi, Y., Wang, S.: From shadow generation to shadow removal. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. Syst. 9(1), 62–66 (1979)
Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2332–2341 (2019)
Qu, L., Tian, J., He, S., Tang, Y., Lau, R.W.: Deshadownet: A multi-context embedding deep network for shadow removal. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
Shechtman, E., Sunkavalli, L.-Q., Kalyan, S.-M., Wang, J.: Appearance harmonization for single image shadow removal. Euro. Assoc. Comput. Graph. 35(7), 189–197 (2016)
Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 640–651 (2017)
Shor, Y., Lischinski, D.: The shadow meets the mask: Pyramid-based shadow removal. Comput. Graph. Forum 27, 577–586 (04 2008)
Singh, S., Krishnan, S.: Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (June 2020)
Ulyanov, D., Vedaldi, A., Lempitsky, V.S.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
Vicente, T.F.Y., Hou, L., Yu, C.-P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 816–832. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_49
de Vries, H., Strub, F., Mary, J., Larochelle, H., Pietquin, O., Courville, A.C.: Modulating early visual processing by language. In: Advances in Neural Information Processing Systems (2017)
Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
Wang, P., Li, Y., Vasconcelos, N.: Rethinking and improving the robustness of image style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 124–133 (June 2021)
Wang, T., Hu, X., Wang, Q., Heng, P.A., Fu, C.W.: Instance shadow detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1880–1889 (2020)
Wu, Y., He, K.: Group normalization. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11217, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01261-8_1
Wu, Z., Wu, X., Zhang, X., Wang, S., Ju, L.: Semantic stereo matching with pyramid cost volumes. In: International Conference on Computer Vision, pp. 7483–7492 (2019)
Xie, S., Tu, Z.: Holistically-nested edge detection. Int. J. Comput. Vis. 125, 3–18 (2015)
Yang, Q., Tan, K.H., Ahuja, N.: Shadow removal using bilateral filtering. IEEE Trans. Image Process. 21(10), 4361–4368 (2012)
Yu, T., et al.: Region normalization for image inpainting. In: AAAI, pp. 12733–12740 (2020)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Zhang, S., Liang, R., Wang, M.: Shadowgan: Shadow synthesis for virtual objects with conditional adversarial networks. Comput. Visual Media 5(1), 105–115 (2019)
Zhu, L., et al.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11210, pp. 122–137. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_8
Acknowledgement
This work was supported by the Fundamental Research Funds for the Central Universities (2020YJS031), National Nature Science Foundation of China (51827813, 61472029, U1803264), and Research and Development Program of Beijing Municipal Education Commission (KJZD20191000402).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wan, J., Yin, H., Wu, Z., Wu, X., Liu, Y., Wang, S. (2022). Style-Guided Shadow Removal. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13679. Springer, Cham. https://doi.org/10.1007/978-3-031-19800-7_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-19800-7_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19799-4
Online ISBN: 978-3-031-19800-7
eBook Packages: Computer ScienceComputer Science (R0)