research-article

StyLit: illumination-guided example-based stylization of 3D renderings

Authors:

Ondřej Jamriška,

Michal Lukáč,

Daniel SýkoraAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 35, Issue 4

Article No.: 92, Pages 1 - 11

https://doi.org/10.1145/2897824.2925948

Published: 11 July 2016 Publication History

Abstract

We present an approach to example-based stylization of 3D renderings that better preserves the rich expressiveness of hand-created artwork. Unlike previous techniques, which are mainly guided by colors and normals, our approach is based on light propagation in the scene. This novel type of guidance can distinguish among context-dependent illumination effects, for which artists typically use different stylization techniques, and delivers a look closer to realistic artwork. In addition, we demonstrate that the current state of the art in guided texture synthesis produces artifacts that can significantly decrease the fidelity of the synthesized imagery, and propose an improved algorithm that alleviates them. Finally, we demonstrate our method's effectiveness on a variety of scenes and styles, in applications like interactive shading study or autocompletion.

Supplementary Material

ZIP File (a92-fiser-supp.zip)

Supplemental files.

Download
253.24 MB

MP4 File (a92.mp4)

Download
344.93 MB

References

[1]

Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3, 24.

Digital Library

[2]

Barnes, C., Shechtman, E., Goldman, D. B., and Finkelstein, A. 2010. The generalized PatchMatch correspondence algorithm. In Proceedings of European Conference on Computer Vision, 29--43.

Digital Library

[3]

Barnes, C., Zhang, F.-L., Lou, L., Wu, X., and Hu, S.-M. 2015. PatchTable: Efficient patch queries for large datasets and applications. ACM Transactions on Graphics 34, 4, 97.

Digital Library

[4]

Bénard, P., Lagae, A., Vangorp, P., Lefebvre, S., Drettakis, G., and Thollot, J. 2010. A dynamic noise primitive for coherent stylization. Computer Graphics Forum 29, 4, 1497--1506.

Digital Library

[5]

Bénard, P., Cole, F., Kass, M., Mordatch, I., Hegarty, J., Senn, M. S., Fleischer, K., Pesare, D., and Breeden, K. 2013. Stylizing animation by example. ACM Transactions on Graphics 32, 4, 119.

Digital Library

[6]

Blinn, J. F., and Newell, M. E. 1976. Texture and reflection in computer generated images. Communications of the ACM 19, 10, 542--547.

Digital Library

[7]

Bousseau, A., Kaplan, M., Thollot, J., and Sillion, F. 2006. Interactive watercolor rendering with temporal coherence and abstraction. In International Symposium on Non-Photorealistic Animation and Rendering, 141--149.

Digital Library

[8]

Chen, J., and Wang, B. 2010. High quality solid texture synthesis using position and index histogram matching. The Visual Computer 26, 4, 253--262.

Digital Library

[9]

Curtis, C. J., Anderson, S. E., Seims, J. E., Fleischer, K. W., and Salesin, D. H. 1997. Computer-generated watercolor. In SIGGRAPH Conference Proceedings, 421--430.

Digital Library

[10]

Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image Melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics 31, 4, 82.

Digital Library

[11]

Diamanti, O., Barnes, C., Paris, S., Shechtman, E., and Sorkine-Hornung, O. 2015. Synthesis of complex image appearance from limited exemplars. ACM Transactions on Graphics 34, 2, 22.

Digital Library

[12]

Fišer, J., Lukáč, M., Jamriška, O., Č adík, M., Gingold, Y., Asente, P., and Sýkora, D. 2014. Color Me Noisy: Example-based rendering of hand-colored animations with temporal noise control. Computer Graphics Forum 33, 4, 1--10.

[13]

Gatys, L. A., Ecker, A. S., and Bethge, M. 2015. A neural algorithm of artistic style. CoRR abs/1508.06576.

[14]

Haeberli, P. 1990. Paint by numbers: Abstract image representations. SIGGRAPH Computer Graphics 24, 4, 207--214.

Digital Library

[15]

Haevre, W. V., Laerhoven, T. V., Fiore, F. D., and Reeth, F. V. 2007. From Dust Till Drawn: A real-time bidirectional pastel simulation. The Visual Computer 23, 9-11, 925--934.

Digital Library

[16]

Han, J., Zhou, K., Wei, L.-Y., Gong, M., Bao, H., Zhang, X., and Guo, B. 2006. Fast example-based surface texture synthesis via discrete optimization. The Visual Computer 22, 9-11, 918--925.

Digital Library

[17]

Hashimoto, R., Johan, H., and Nishita, T. 2003. Creating various styles of animations using example-based filtering. In Proceedings of Computer Graphics International, 312--317.

[18]

Heckbert, P. S. 1990. Adaptive radiosity textures for bidirectional ray tracing. SIGGRAPH Computer Graphics 24, 4, 145--154.

Digital Library

[19]

Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. In SIGGRAPH Conference Proceedings, 327--340.

Digital Library

[20]

Jamriška, O., Fišer, J., Asente, P., Lu, J., Shechtman, E., and Sýkora, D. 2015. LazyFluids: Appearance transfer for fluid animations. ACM Transactions on Graphics 34, 4, 92.

Digital Library

[21]

Kajiya, J. T. 1986. The rendering equation. SIGGRAPH Computer Graphics 20, 4, 143--150.

Digital Library

[22]

Kaspar, A., Neubert, B., Lischinski, D., Pauly, M., and Kopf, J. 2015. Self tuning texture optimization. Computer Graphics Forum 34, 2, 349--360.

Digital Library

[23]

Kopf, J., Fu, C.-W., Cohen-Or, D., Deussen, O., Lischinski, D., and Wong, T.-T. 2007. Solid texture synthesis from 2D exemplars. ACM Transactions on Graphics 26, 3, 2.

Digital Library

[24]

Kwatra, V., Essa, I. A., Bobick, A. F., and Kwatra, N. 2005. Texture optimization for example-based synthesis. ACM Transactions on Graphics 24, 3, 795--802.

Digital Library

[25]

Kyprianidis, J. E., Collomosse, J., Wang, T., and Isenberg, T. 2013. State of the "art": A taxonomy of artistic stylization techniques for images and video. IEEE Transactions on Visualization and Computer Graphics 19, 5, 866--885.

Digital Library

[26]

Lee, H., Seo, S., Ryoo, S., and Yoon, K. 2010. Directional texture transfer. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 43--48.

Digital Library

[27]

Lefebvre, S., and Hoppe, H. 2006. Appearance-space texture synthesis. ACM Transactions on Graphics 25, 3, 541--548.

Digital Library

[28]

Lu, C., Xu, L., and Jia, J. 2012. Combining sketch and tone for pencil drawing production. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 65--73.

Digital Library

[29]

Newson, A., Almansa, A., Fradet, M., Gousseau, Y., and Pérez, P. 2014. Video inpainting of complex scenes. SIAM Journal of Imaging Science 7, 4, 1993--2019.

[30]

Rong, G., and Tan, T.-S. 2006. Jump flooding in GPU with applications to Voronoi diagram and distance transform. In Proceedings of Symposium on Interactive 3D Graphics and Games, 109--116.

Digital Library

[31]

Rosenberger, A., Cohen-Or, D., and Lischinski, D. 2009. Layered Shape Synthesis: Automatic generation of control maps for non-stationary textures. ACM Transactions on Graphics 28, 5, 107.

Digital Library

[32]

Salisbury, M. P., Wong, M. T., Hughes, J. F., and Salesin, D. H. 1997. Orientable textures for image-based pen-and-ink illustration. In SIGGRAPH Conference Proceedings, 401--406.

Digital Library

[33]

Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[34]

Simonyan, K., and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556.

[35]

Sloan, P.-P. J., Martin, W., Gooch, A., and Gooch, B. 2001. The Lit Sphere: A model for capturing NPR shading from art. In Proceedings of Graphics Interface, 143--150.

Digital Library

[36]

Tu, Z., Chen, X., Yuille, A. L., and Zhu, S.-c. 2005. Image Parsing: Unifying segmentation, detection, and recognition. International Journal of Computer Vision 63, 2, 113--140.

Digital Library

[37]

Wang, B., Wang, W., Yang, H., and Sun, J. 2004. Efficient example-based painting and synthesis of 2D directional texture. IEEE Transactions on Visualization and Computer Graphics 10, 3, 266--277.

Digital Library

[38]

Wei, L.-Y., Han, J., Zhou, K., Bao, H., Guo, B., and Shum, H.-Y. 2008. Inverse texture synthesis. ACM Transactions on Graphics 27, 3.

Digital Library

[39]

Wexler, Y., Shechtman, E., and Irani, M. 2007. Space-time completion of video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3, 463--476.

Digital Library

[40]

Winnemöller, H., Kyprianidis, J. E., and Olsen, S. C. 2012. XDoG: An extended difference-of-gaussians compendium including advanced image stylization. Computers & Graphics 36, 6, 740--753.

Digital Library

[41]

Xing, J., Chen, H.-T., and Wei, L.-Y. 2014. Autocomplete painting repetitions. ACM Transactions on Graphics 33, 6, 172.

Digital Library

[42]

Xing, J., Wei, L.-Y., Shiratori, T., and Yatani, K. 2015. Autocomplete hand-drawn animations. ACM Transactions on Graphics 34, 6, 169.

Digital Library

[43]

Zeng, K., Zhao, M., Xiong, C., and Zhu, S.-C. 2009. From image parsing to painterly rendering. ACM Transactions on Graphics 29, 1, 2.

Digital Library

[44]

Zeng, Q., Chen, W., Wang, H., Tu, C., Cohen-Or, D., Lischinski, D., and Chen, B. 2015. Hallucinating stereoscopy from a single image. Computer Graphics Forum 34, 2, 1--12.

Digital Library

[45]

Zhao, M., and Zhu, S.-C. 2011. Portrait painting using active templates. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 117--124.

Digital Library

Cited By

Todo HKoyama YSakai KKomiya AKato J(2024)A Practical Style Transfer Pipeline for 3D Animation: Insights from Production R&DSIGGRAPH Asia 2024 Technical Communications10.1145/3681758.3698000(1-4)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3681758.3698000
Hwang HLee SBae S(2024)ValueSphere: A Portable Widget for Quick and Easy Shading in Digital DrawingsAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686350(1-2)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686350
Zheng YCun XXia MPun C(2024)Sketch Video SynthesisComputer Graphics Forum10.1111/cgf.1504443:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15044
Show More Cited By

Index Terms

StyLit: illumination-guided example-based stylization of 3D renderings
1. Computing methodologies
  1. Computer graphics
    1. Rendering
      1. Non-photorealistic rendering

Recommendations

Splatting indirect illumination
I3D '06: Proceedings of the 2006 symposium on Interactive 3D graphics and games

In this paper we present a novel method for plausible real-time rendering of indirect illumination effects for diffuse and non-diffuse surfaces. The scene geometry causing indirect illumination is captured by an extended shadow map, as proposed in ...
Precomputed illuminance composition for real-time global illumination
I3D '16: Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games

In this paper we present a new real-time approach for indirect global illumination under dynamic lighting conditions. We use surfels to gather a sampling of the local illumination and propagate the light through the scene using a hierarchy and a set of ...
Interactive relighting with dynamic BRDFs

We present a technique for interactive relighting in which source radiance, viewing direction, and BRDFs can all be changed on the fly. In handling dynamic BRDFs, our method efficiently accounts for the effects of BRDF modification on the reflectance ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 35, Issue 4

July 2016

1396 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2897824

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2016

Published in TOG Volume 35, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Grid Infrastructure MetaCentrum
Technology Agency of the Czech Republic
Grant Agency of the Czech Technical University in Prague

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

99
Total Citations
View Citations
1,581
Total Downloads

Downloads (Last 12 months)104
Downloads (Last 6 weeks)14

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Todo HKoyama YSakai KKomiya AKato J(2024)A Practical Style Transfer Pipeline for 3D Animation: Insights from Production R&DSIGGRAPH Asia 2024 Technical Communications10.1145/3681758.3698000(1-4)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3681758.3698000
Hwang HLee SBae S(2024)ValueSphere: A Portable Widget for Quick and Easy Shading in Digital DrawingsAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686350(1-2)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686350
Zheng YCun XXia MPun C(2024)Sketch Video SynthesisComputer Graphics Forum10.1111/cgf.1504443:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15044
Li XCao ZWu YWang KXian KWang ZLin G(2024)S-DyRF: Reference-Based Stylized Radiance Fields for Dynamic Scenes2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01900(20102-20112)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01900
Fischer MLi ZNguyen-Phuoc TBožič ADong ZMarshall CRitschel T(2024)NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00444(4640-4650)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00444
Mumuni AMumuni FGerrar N(2024)A Survey of Synthetic Data Augmentation Methods in Machine VisionMachine Intelligence Research10.1007/s11633-022-1411-721:5(831-869)Online publication date: 20-Mar-2024
https://doi.org/10.1007/s11633-022-1411-7
Chen YHuang HVu TShum KYeung S(2024)StyleCity: Large-Scale 3D Urban Scenes StylizationComputer Vision – ECCV 202410.1007/978-3-031-73202-7_23(395-413)Online publication date: 21-Nov-2024
https://doi.org/10.1007/978-3-031-73202-7_23
Ma YZhao CHuang BLi XBasu A(2023)RAST: Restorable Arbitrary Style TransferACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363877020:5(1-21)Online publication date: 30-Dec-2023
https://dl.acm.org/doi/10.1145/3638770
Zhang YTang FDong WHuang HMa CLee TXu C(2023)A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive LearningACM Transactions on Graphics10.1145/360554842:5(1-16)Online publication date: 28-Jul-2023
https://dl.acm.org/doi/10.1145/3605548
Šubrtová ALukáč MČech JFutschik DShechtman ESýkora D(2023)Diffusion Image AnalogiesACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591558(1-10)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591558
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents