Shadow-aware image colorization
Authors: 
Xin Duan, Yu Cao, Renjie Zhang, Xin Wang, Ping LiAuthors Info & Claims
Abstract
Significant advancements have been made in colorization in recent years, especially with the introduction of deep learning technology. However, challenges remain in accurately colorizing images under certain lighting conditions, such as shadow. Shadows often cause distortions and inaccuracies in object recognition and visual data interpretation, impacting the reliability and effectiveness of colorization techniques. These problems often lead to unsaturated colors in shadowed images and incorrect colorization of shadows as objects. Our research proposes the first shadow-aware image colorization method, addressing two key challenges that previous studies have overlooked: integrating shadow information with general semantic understanding and preserving saturated colors while accurately colorizing shadow areas. To tackle these challenges, we develop a dual-branch shadow-aware colorization network. Additionally, we introduce our shadow-aware block, an innovative mechanism that seamlessly integrates shadow-specific information into the colorization process, distinguishing between shadow and non-shadow areas. This research significantly improves the accuracy and realism of image colorization, particularly in shadow scenarios, thereby enhancing the practical application of colorization in real-world scenarios.
References
[1]
Akita K, Morimoto Y, and Tsuruno R Colorization of line drawings with empty pupils Comput. Graph. Forum 2020 39 7 601-610
[2]
Assarsson U and Akenine-Möller T A geometry-based soft shadow volume algorithm using graphics hardware ACM Trans. Graph. 2003 22 3 511-520
[3]
Baba, M., Mukunoki, M., Asada, N.: Shadow removal from a real image based on shadow density. In: ACM SIGGRAPH Posters, p. 60 (2004)
[4]
Bahng, H., Yoo, S., Cho, W., Park, D.K., Wu, Z., Ma, X., Choo, J.: Coloring with words: guiding image colorization through text-based palette generation. In: European Conference on Computer Vision, pp. 443–459 (2018)
[5]
Caesar, H., Uijlings, J., Ferrari, V.: COCO-Stuff: thing and stuff classes in context. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1209–1218 (2018)
[6]
Cao Y, Meng X, Mok PY, Lee TY, Liu X, and Li P AnimeDiffusion: anime diffusion colorization IEEE Trans. Vis. Comput. Graph. 2024
[7]
Cao, Y., Tian, H., Mok, P.Y.: Attention-aware anime line drawing colorization. In: IEEE International Conference on Multimedia and Expo, pp. 1637–1642 (2023)
[8]
Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: European Conference on Computer Vision, pp. 126–139 (2008)
[9]
Chen, J., Shen, Y., Gao, J., Liu, J., Liu, X.: Language-based image editing with recurrent attentive models. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8721–8729 (2018)
[10]
Chen, Z., Wan, L., Zhu, L., Shen, J., Fu, H., Liu, W., Qin, J.: Triple-cooperative video shadow detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2714–2723 (2021)
[11]
Chen, Z., Zhu, L., Wan, L., Wang, S., Feng, W., Heng, P.A.: A multi-task mean teacher for semi-supervised shadow detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5610–5619 (2020)
[12]
Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: IEEE International Conference on Computer Vision, pp. 415–423 (2015)
[13]
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.F.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
[14]
Deshpande, A., Lu, J., Yeh, M.C., Chong, M.J., Forsyth, D.: Learning diverse image colorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2877–2885 (2017)
[15]
Deshpande, A., Rock, J., Forsyth, D.: Learning large-scale automatic image colorization. In: IEEE International Conference on Computer Vision, pp. 567–575 (2015)
[16]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations, pp. 1–21 (2021)
[17]
Fang F, Wang T, Zeng T, and Zhang G A superpixel-based variational model for image colorization IEEE Trans. Vis. Comput. Graph. 2020 26 10 2931-2943
[18]
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: International Conference on Artificial Intelligence and Statistics, vol. 9, pp. 249–256 (2010)
[19]
Gupta, R.K., Chia, A.Y.S., Rajan, D., Ng, E.S., Huang, Z.: Image colorization using similar images. In: ACM International Conference on Multimedia, pp. 369–378 (2012)
[20]
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
[21]
He M, Chen D, Liao J, Sander P.V, and Yuan L Deep exemplar-based colorization ACM Trans. Graph. 2018 37 4 47:1-47:16
[22]
Hou L, Vicente TFY, Hoai M, and Samaras D Large scale shadow annotation and detection using lazy annotation and stacked CNNs IEEE Trans. Pattern Anal. Mach. Intell. 2021 43 4 1337-1351
[23]
Hou, Y., Zheng, L.: Multiview detection with shadow transformer (and view-coherent data augmentation). In: ACM International Conference on Multimedia, pp. 1673–1682 (2021)
[24]
Hu X, Wang T, Fu CW, Jiang Y, Wang Q, and Heng PA Revisiting shadow detection: a new benchmark dataset for complex world IEEE Trans. Image Process. 2021 30 1925-1934
[25]
Huang Z, Zhao N, and Liao J UniColor: a unified framework for multi-modal colorization with transformer ACM Trans. Graph. 2022 41 6 205:1-205:16
[26]
Iizuka S, Simo-Serra E, and Ishikawa H Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification ACM Trans. Graph. 2016 35 4 110:1-110:11
[27]
Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Eurographics Symposium on Rendering, pp. 201–210 (2005)
[28]
Ji, X., Jiang, B., Luo, D., Tao, G., Chu, W., Xie, Z., Wang, C., Tai, Y.: ColorFormer: image colorization via color memory assisted hybrid-attention transformer. In: European Conference on Computer Vision, pp. 20–36 (2022)
[29]
Kang, X., Yang, T., Ouyang, W., Ren, P., Li, L., Xie, X.: DDColor: towards photo-realistic image colorization via dual decoders. In: IEEE International Conference on Computer Vision, pp. 328–338 (2023)
[30]
Kim, E., Lee, S., Park, J., Choi, S., Seo, C., Choo, J.: Deep edge-aware interactive colorization against color-bleeding effects. In: IEEE International Conference on Computer Vision, pp. 14,647–14,656 (2021)
[31]
Kim, G., Kang, K., Kim, S., Lee, H., Kim, S., Kim, J., Baek, S.H., Cho, S.: BigColor: colorization using a generative color prior for natural images. In: European Conference on Computer Vision, pp. 350–366 (2022)
[32]
Kumar, M., Weissenborn, D., Kalchbrenner, N.: Colorization transformer. In: International Conference on Learning Representations, pp. 1–24 (2021)
[33]
Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: European Conference on Computer Vision, pp. 577–593 (2016)
[34]
Levin A, Lischinski D, and Weiss Y Colorization using optimization ACM Trans. Graph. 2004 23 3 689-694
[35]
Liu, L., Prost, J., Zhu, L., Papadakis, N., Liò, P., Schönlieb, C.B., Aviles-Rivero, A.I.: Scotch and soda: a transformer video shadow detection framework. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 10,449–10,458 (2023)
[36]
Liu X, Wan L, Qu Y, Wong TT, Lin S, Leung CS, and Heng PA Intrinsic colorization ACM Trans. Graph. 2008 27 5 152:1-152:9
[37]
Lu, X., Cao, Y., Liu, S., Long, C., Chen, Z., Zhou, X., Yang, Y., Xiao, C.: Video shadow detection via spatio-temporal interpolation consistency training. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3106–3115 (2022)
[38]
Messaoud, S., Forsyth, D., Schwing, A.G.: Structural consistency and controllability for diverse colorization. In: European Conference on Computer Vision, pp. 603–619 (2018)
[39]
Qu Y, Wong TT, and Heng PA Manga colorization ACM Trans. Graph. 2006 25 3 1214-1220
[40]
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations, pp. 1–14 (2015)
[41]
Su, J.W., Chu, H.K., Huang, J.B.: Instance-aware image colorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7965–7974 (2020)
[42]
Sýkora D, Dingliana J, and Collins S Lazybrush: flexible painting tool for hand-drawn cartoons Comput. Graph. Forum 2009 28 2 599-608
[43]
Vasluianu, F.A., Seizinger, T., Timofte, R.: WSRD: a novel benchmark for high resolution image shadow removal. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1826–1835 (2023)
[44]
Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1788–1797 (2018)
[45]
Wang, Y., Xia, M., Qi, L., Shao, J., Qiao, Y.: PalGAN: image colorization with palette generative adversarial networks. In: European Conference on Computer Vision, pp. 271–288 (2022)
[46]
Welsh T, Ashikhmin M, and Mueller K Transferring color to greyscale images ACM Trans. Graph. 2002 21 3 277-280
[47]
Weng, S., Sun, J., Li, Y., Li, S., Shi, B.: : colorization transformer via color tokens. In: European Conference on Computer Vision, pp. 1–16 (2022)
[48]
Wu, Y., Wang, X., Li, Y., Zhang, H., Zhao, X., Shan, Y.: Towards vivid and diverse image colorization with generative color prior. In: IEEE International Conference on Computer Vision, pp. 14,357–14,366 (2021)
[49]
Xia M, Hu W, Wong TT, and Wang J Disentangled image colorization via global anchors ACM Trans. Graph. 2022 41 6 204:1-204:13
[50]
Xiao C, Han C, Zhang Z, Qin J, Wong TT, Han G, and He S Example-based colourization via dense encoding pyramids Comput. Graph. Forum 2020 39 1 20-33
[51]
Xiao Y, Wu J, Zhang J, Zhou P, Zheng Y, Leung CS, and Kavan L Interactive deep colorization and its application for image compression IEEE Trans. Vis. Comput. Graph. 2022 28 3 1557-1572
[52]
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: SegFormer: simple and efficient design for semantic segmentation with transformers. In: Advances in Neural Information Processing Systems, vol. 34, pp. 12,077–12,090 (2021)
[53]
Xie, X., Zhou, P., Li, H., Lin, Z., Yan, S.: Adan: adaptive nesterov momentum algorithm for faster optimizing deep models. In: Advances in Neural Information Processing Systems Workshop, pp. 1–8 (2022)
[54]
Zhang L, Li C, Wong TT, Ji Y, and Liu C Two-stage sketch colorization ACM Trans. Graph. 2018 37 6 261:1-261:14
[55]
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: European Conference on Computer Vision, pp. 649–666 (2016)
[56]
Zhang R, Zhu JY, Isola P, Geng X, Lin AS, Yu T, and Efros AA Real-time user-guided image colorization with learned deep priors ACM Trans. Graph. 2017 36 4 119:1-119:11
[57]
Zhao J, Han J, Shao L, and Snoek CGM Pixelated semantic colorization Int. J. Comput. Vis. 2020 128 818-834
[58]
Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.H.: Distraction-aware shadow detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5162–5171 (2019)
[59]
Zhu, J., Samuel, K.G.G., Masood, S.Z., Tappen, M.F.: Learning to recognize shadows in monochromatic natural images. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 223–230 (2010)
[60]
Zou C, Mo H, Gao C, Du R, and Fu H Language-based colorization of scene sketches ACM Trans. Graph. 2019 38 6 233:1-233:16
Recommendations
Intrinsic colorization
In this paper, we present an example-based colorization technique robust to illumination differences between grayscale target and color reference images. To achieve this goal, our method performs color transfer in an illumination-independent domain that ...
Intrinsic colorization
SIGGRAPH Asia '08: ACM SIGGRAPH Asia 2008 papersIn this paper, we present an example-based colorization technique robust to illumination differences between grayscale target and color reference images. To achieve this goal, our method performs color transfer in an illumination-independent domain that ...
A Novel Approach for Colorization of a Grayscale Image using Soft Computing Techniques
Colorization of grayscale image is a process to convert a grayscale image into a color one. Few research works reported in literature on this but there is hardly any generalized method that successfully colorizes all types of grayscale image. This study ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s) 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 04 June 2024
Accepted: 16 May 2024
Author Tags
Qualifiers
- Research-article
Funding Sources
- Hong Kong Polytechnic University
- Hong Kong Polytechnic University
- Hong Kong Polytechnic University
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
View options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in