HyperNST: Hyper-Networks for Neural Style Transfer
Authors: 
Dan Ruta, Andrew Gilbert, Saeid Motiian, Baldo Faieta, Zhe Lin, John CollomosseAuthors Info & Claims
Pages 201 - 217
Abstract
We present HyperNST; a neural style transfer (NST) technique for the artistic stylization of images, based on Hyper-networks and the StyleGAN2 architecture. Our contribution is a novel method for inducing style transfer parameterized by a metric space, pre-trained for style-based visual search (SBVS). We show for the first time that such space may be used to drive NST, enabling the application and interpolation of styles from an SBVS system. The technical contribution is a hyper-network that predicts weight updates to a StyleGAN2 pre-trained over a diverse gamut of artistic content (portraits), tailoring the style parameterization on a per-region basis using a semantic map of the facial regions. We show HyperNST to exceed state of the art in content preservation for our stylized content while retaining good style transfer performance.
References
[1]
Alaluf, Y., Patashnik, O., Cohen-Or, D.: Restyle: a residual-based stylegan encoder via iterative refinement. CoRR, abs/2104.02699 (2021)
[2]
Alaluf, Y., Tov, O., Mokady, R., Gal, R., Bermano, A.H.: Hyperstyle: Stylegan inversion with hypernetworks for real image editing. CoRR, abs/2111.15666 (2021)
[3]
An, J., Huang, S., Song, Y., Dou, D., Liu, W., Luo, J.: Artflow: unbiased image style transfer via reversible neural flows. CoRR, abs/2103.16877 (2021)
[4]
Chen, H., et al.: Artistic style transfer with internal-external learning and contrastive learning. In: Beygelzimer, A., Dauphin, Y., Liang, P., Wortman Vaughan, J. (eds.) Advances in Neural Information Processing Systems (2021)
[5]
Gal, R., Patashnik, O., Maron, H., Chechik, G., Cohen-Or, D.: Stylegan-nada: Clip-guided domain adaptation of image generators. CoRR, abs/2108.00946 (2021)
[6]
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of CVPR, pp. 2414–2423 (2016)
[7]
Ha, D., Dai, A.M., Le, Q.N.: Hypernetworks. CoRR, abs/1609.09106 (2016)
[8]
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of ICCV (2017)
[9]
Huang X, Liu M-Y, Belongie S, and Kautz J Ferrari V, Hebert M, Sminchisescu C, and Weiss Y Multimodal unsupervised image-to-image translation Computer Vision – ECCV 2018 2018 Cham Springer 179-196
[10]
Karras, T., et al.: Alias-free generative adversarial networks. In: Proceedings of NeurIPS (2021)
[11]
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. CoRR, abs/1812.04948 (2018)
[12]
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of CVPR (2019)
[13]
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. CoRR, abs/1912.04958 (2019)
[14]
Kolkin, N., Kucera, M., Paris, S., Shechtman, E., Shakhnarovich, G.: Neural neighbor style transfer (2022)
[15]
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, Vol. 1, pp. 1097–1105, Red Hook, NY, USA, 2012. Curran Associates Inc. (2012
[16]
Liu, M., Li, Q., Qin, Z., Zhang, G., Wan, P., Zheng, W.: Blendgan: implicitly GAN blending for arbitrary stylized face generation. CoRR, abs/2110.11728 (2021)
[17]
Luo, X., Han, Z., Yang, L., Zhang, L.: Consistent style transfer. CoRR, abs/2201.02233 (2022)
[18]
Park, D.Y., Lee, K.H.: Arbitrary style transfer with style-attentional networks. CoRR, abs/1812.02342 (2018)
[19]
Park, T., et al.: Swapping autoencoder for deep image manipulation. CoRR, abs/2007.00653 (2020)
[20]
Radford, A., et al.: Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020(2021)
[21]
Ruta, D., et al.: Artistic style tagging and captioning (2022)
[22]
Dan Ruta, S., et al.: All layer adaptive instance normalization for fine-grained style similarity. arXiv preprint arXiv:2103.09776 (2021)
[23]
Shaham, T.R., Dekel, T., Michaeli, T.: SinGan: learning a generative model from a single natural image. CoRR, abs/1905.01164 (2019)
[24]
Shen, Y., Yang, C., Tang, X., Bolei Zhou, X.:InterFaceGAN: interpreting the disentangled face representation learned by gans. CoRR, abs/2005.09635 (2020)
[25]
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014)
[26]
Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for stylegan image manipulation. CoRR, abs/2102.02766 (2021)
[27]
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. CoRR, abs/1801.03924 (2018)
[28]
Zheng, Y., et al.: General facial representation learning in a visual-linguistic manner. CoRR, abs/2112.03109 (2021)
Recommendations
StyleBabel: Artistic Style Tagging and Captioning
Computer Vision – ECCV 2022AbstractWe present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and ...
Image neural style transfer: A review
AbstractTraditional methods of style transfer emphasize primarily the transfer of artistic styles. In recent years, style transfer has expanded beyond the realm of artistic expression to encompass fields such as medicine, industry, and literature. ...
Style Transfer Via Texture Synthesis
Style transfer is a process of migrating a style from a given image to the content of another, synthesizing a new image, which is an artistic mixture of the two. Recent work on this problem adopting convolutional neural-networks (CNN) ignited a renewed ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Oct 2022
783 pages
ISBN:978-3-031-25055-2
DOI:10.1007/978-3-031-25056-9
- Editors:
- Leonid Karlinsky,
- Tomer Michaeli,
- Ko Nishino
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 15 February 2023
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 30 Sep 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in