article

Stylistic scene enhancement GAN: mixed stylistic enhancement generation for 3D indoor scenes

Authors:

Matthias Zwicker,

Hui ZhangAuthors Info & Claims

The Visual Computer: International Journal of Computer Graphics, Volume 35, Issue 6-8

Pages 1157 - 1169

https://doi.org/10.1007/s00371-019-01691-w

Published: 01 June 2019 Publication History

Abstract

In this paper, we present stylistic scene enhancement GAN, SSE-GAN, a conditional Wasserstein GAN-based approach to automatic generation of mixed stylistic enhancements for 3D indoor scenes. An enhancement indicates factors that can influence the style of an indoor scene such as furniture colors and occurrence of small objects. To facilitate network training, we propose a novel enhancement feature encoding method, which represents an enhancement by a multi-one-hot vector, and effectively accommodates different enhancement factors. A Gumbel-Softmax module is introduced in the generator network to enable the generation of high fidelity enhancement features that can better confuse the discriminator. Experiments show that our approach is superior to the other baseline methods and successfully models the relationship between the style distribution and scene enhancements. Thus, although only trained with a dataset of room images in single styles, the trained generator can generate mixed stylistic enhancements by specifying multiple styles as the condition. Our approach is the first to apply a Gumbel-Softmax module in conditional Wasserstein GANs, as well as the first to explore the application of GAN-based models in the scene enhancement field.

References

[1]

Trimble 3D warehouse (2019). URL https://3dwarehouse.sketchup.com/. Accessed 15 Sept 2018

[2]

Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene completion from a single depth image. arXiv preprint arXiv:1611.08974 (2016)

[3]

Wang, K., Savva, M., Chang, A.X., Ritchie, D.: Deep convolutional priors for indoor scene synthesis. ACM Trans. Graph. (TOG) 37(4), 70:1---70:14 (2018)

Digital Library

[4]

Chen, G., Li, G., Nie, Y., Xian, C., Mao, A.: Stylistic indoor colour design via Bayesian network. Comput. Graph. 60, 34---45 (2016)

[5]

Chen, K., Xu, K., Yu, Y., Wang, T.Y., Hu, S.M.: Magic decorator: automatic material suggestion for indoor digital scenes. ACM Trans. Graph. (TOG) 34(6), 232:1---232:11 (2015)

Digital Library

[6]

Zhang, S., Han, Z., Martin, R.R., Zhang, H.: Semantic 3D indoor scene enhancement using guide words. Vis. Comput. 33(6---8), 925---935 (2017)

Digital Library

[7]

Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)

[8]

Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text to image synthesis. arXiv preprint arXiv:1605.05396 (2016)

Digital Library

[9]

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: NIPS, 5767---5777 (2017)

Digital Library

[10]

Chen, X., Li, J., Li, Q., Gao, B., Zou, D., Zhao, Q.: Image2scene: transforming style of 3D room. In: Proceedings of the ACM International Conference on Multimedia, 321---330 (2015)

Digital Library

[11]

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS, 2672---2680 (2014)

Digital Library

[12]

Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

[13]

Chen, Y., Lai, Y.K., Liu, Y.J.: CartoonGAN: generative adversarial networks for photo cartoonization. In: IEEE CVPR, 9465---9474 (2018)

[14]

Wu, H., Zheng, S., Zhang, J., Huang, K.: GP-GAN: Towards realistic high-resolution image blending. arXiv preprint arXiv:1703.07195 (2017)

[15]

Wu, J., Zhang, C., Xue, T., Freeman, B., Tenenbaum, J.: Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In: NIPS, pp. 82---90 (2016)

Digital Library

[16]

Liu, J., Yu, F., Funkhouser, T.: Interactive 3D modeling with a generative adversarial network. In: International Conference on 3D Vision (3DV), 126---134. IEEE (2017)

[17]

Chen, K., Choy, C.B., Savva, M., Chang, A.X., Funkhouser, T., Savarese, S.: Text2Shape: Generating shapes from natural language by learning joint embeddings. arXiv preprint arXiv:1803.08495 (2018)

[18]

Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017)

[19]

Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE CVPR, 5967---5976 (2017)

[20]

Donahue, C., McAuley, J., Puckette, M.: Adversarial audio synthesis. arXiv preprint arXiv:1802.04208 (2018)

[21]

Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016)

[22]

Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: a continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)

[23]

Camino, R., Hammerschmidt, C., State, R.: Generating multi-categorical samples with generative adversarial networks. arXiv preprint arXiv:1807.01202 (2018)

[24]

Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders. arXiv preprint arXiv:1511.05644 (2015)

[25]

Gumbel, E.J.: Statistical theory of extreme values and some practical applications. NBS Applied Mathematics Series 33, (1954)

[26]

Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980 (2014)

[27]

Xu, Q., Huang, G., Yuan, Y., Guo, C., Sun, Y., Wu, F., Weinberger, K.: An empirical study on evaluation metrics of generative adversarial networks. arXiv preprint arXiv:1806.07755 (2018)

[28]

Lopez-Paz, D., Oquab, M.: Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545 (2016)

[29]

Bounliphone, W., Belilovsky, E., Blaschko, M.B., Antonoglou, I., Gretton, A.: A test of relative similarity for model selection in generative models. arXiv preprint arXiv:1511.04581 (2015)

[30]

Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for maximizing submodular set functions-i. Math. Program. 14(1), 265---294 (1978)

Digital Library

[31]

Chen, D.Y., Tian, X.P., Shen, Y.T., Ouhyoung, M.: On visual similarity based 3D model retrieval. Comput. Graph. Forum 22(3), 223---232 (2003)

[32]

Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multi-scale signature based on heat diffusion. Comput. Graph. Forum 28(5), 1383---1392 (2009)

Digital Library

[33]

Zhang, Z., Yang, Z., Ma, C., Luo, L., Huth, A., Vouga, E., Huang, Q.: Deep generative modeling for scene synthesis via hybrid representations. arXiv preprint arXiv:1808.02084 (2018)

[34]

Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

Digital Library

Cited By

Wang XLi QYu DLi QXu G(2024)Counterfactual Explanation for Fairness in RecommendationACM Transactions on Information Systems10.1145/364367042:4(1-30)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3643670
Hu FMa YZhong WYe LYang XFang LZhang Q(2024)A Dataset and Benchmark for 3D Scene Plausibility AssessmentIEEE Transactions on Multimedia10.1109/TMM.2024.335345626(6529-6541)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3353456
Wang XLi QYu DLi QXu G(2024)Reinforced Path Reasoning for Counterfactual Explainable RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.335407736:7(3443-3459)Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1109/TKDE.2024.3354077
Show More Cited By

Index Terms

Stylistic scene enhancement GAN: mixed stylistic enhancement generation for 3D indoor scenes
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Semantic 3D indoor scene enhancement using guide words

We propose a novel framework for semantically enhancing a 3D indoor scene in agreement with a user-provided guide word. To do so, we make changes to furniture colors and place small objects in the scene. The relevance of specific furniture colors and ...
Colour Preference in Teenage Boys' Bedrooms
BCS-HCI '14: Proceedings of the 28th International BCS Human Computer Interaction Conference on HCI 2014 - Sand, Sea and Sky - Holiday HCI

Colour in spaces plays an important role on inhabitants' mood. In interior design colour has been used to decorate spaces in order to lift and change mood in people. It has been already proved that certain colours relax people while other colours have ...
Low-light image enhancement based on GAN with attention mechanism and color Constancy
Abstract
Images captured in low-light often suffer from severe quality degraded problems, such as low contrast and color distortion, which make it intractable for further computer vision tasks. To solve the problems above, we proposed a trainable parallel ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image The Visual Computer: International Journal of Computer Graphics

The Visual Computer: International Journal of Computer Graphics Volume 35, Issue 6-8

June 2019

415 pages

ISSN:0178-2789

Issue’s Table of Contents

Copyright © Copyright © 2019 Springer-Verlag GmbH Germany, part of Springer Nature.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 June 2019

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang XLi QYu DLi QXu G(2024)Counterfactual Explanation for Fairness in RecommendationACM Transactions on Information Systems10.1145/364367042:4(1-30)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3643670
Hu FMa YZhong WYe LYang XFang LZhang Q(2024)A Dataset and Benchmark for 3D Scene Plausibility AssessmentIEEE Transactions on Multimedia10.1109/TMM.2024.335345626(6529-6541)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3353456
Wang XLi QYu DLi QXu G(2024)Reinforced Path Reasoning for Counterfactual Explainable RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.335407736:7(3443-3459)Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1109/TKDE.2024.3354077
Jang SLee GOh JLee JKoo B(2024)Automated detailing of exterior walls using NADIAAdvanced Engineering Informatics10.1016/j.aei.2024.10253261:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.aei.2024.102532
Xuan YSong CJin JYang B(2024)CVAE-LAYOUT: automatic furniture layout with constraintsThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-03204-240:11(7731-7745)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s00371-023-03204-2
Fan RWang LLiu XIm SLam C(2024)Real-scene-constrained virtual scene layout synthesis for mixed realityThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-03167-440:9(6319-6339)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s00371-023-03167-4
Wang XLi QYu DLi QXu G(2023)Constrained Off-policy Learning over Heterogeneous Information for Fairness-aware RecommendationACM Transactions on Recommender Systems10.1145/36291722:4(1-27)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3629172
Wang XLi QYu DCui PWang ZXu G(2023)Causal Disentanglement for Semantic-Aware Intent Learning in RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.315980235:10(9836-9849)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1109/TKDE.2022.3159802
Lan JYe FYe ZXu PLing WHuang G(2023)Unsupervised style-guided cross-domain adaptation for few-shot stylized face translationThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02719-439:12(6167-6181)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s00371-022-02719-4
Wang SGao ZLiu D(2023)Swin-GAN: generative adversarial network based on shifted windows transformer architecture for image generationThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02714-939:12(6085-6095)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s00371-022-02714-9
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents