research-article

Photo Stand-Out: Photography with Virtual Character

Authors:

Wei LiangAuthors Info & Claims

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Pages 781 - 788

https://doi.org/10.1145/3394171.3413957

Published: 12 October 2020 Publication History

Abstract

In this paper, we propose a novel optimization framework to synthesize an aesthetic pose for the virtual character with respect to the presented user's pose. Our approach applies aesthetic evaluation that exploits fully connected neural networks trained on example images. The aesthetic pose of the virtual character is obtained by optimizing a cost function that guides the rotation of each body joint angles. In our experiments, we demonstrate the proposed approach can synthesize poses for virtual characters according to user pose inputs. We also conducted objective and subjective experiments of the synthesized results to validate the efficacy of our approach.

Supplementary Material

MP4 File (3394171.3413957.mp4)

Extended augmented reality techniques and applications of virtual characters provide a diverse and immersive experience in the real world. In different scenes, the virtual character should be posed naturally with the user, to obtain photography with the personalized posed virtual character rather than that with the immutable pre-designed pose. In our research, we propose a novel optimization framework to synthesize an aesthetic pose for the virtual character with respect to the presented user's pose. We apply aesthetic evaluation that exploits fully connected neural networks trained on example images. The aesthetic pose of the virtual character is obtained by optimizing a cost function that guides the rotation of each body joint angles. In our experiments, we demonstrate the proposed approach can synthesize poses for virtual characters according to user pose inputs. We also conducted objective and subjective experiments of the synthesized results to validate the efficacy of our approach.

Download
88.29 MB

References

[1]

Patricia C Albers and William R James. 1988. Travel photography: A methodological approach. Annals of tourism research, Vol. 15, 1 (1988), 134--158.

[2]

Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag. 2018. Synthesizing images of humans in unseen poses. In CVPR. 8340--8348.

[3]

Subhabrata Bhattacharya, Rahul Sukthankar, and Mubarak Shah. 2010. A framework for photo-quality assessment and enhancement based on visual aesthetics. In Proceedings of the 18th ACM international conference on Multimedia. 271--280.

Digital Library

[4]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR. 7291--7299.

[5]

Chunhua Chen, Wen Chen, and Jeffrey A Bloom. 2011. A universal reference-free blurriness measure. In Image Quality and System Performance VIII, Vol. 7867. International Society for Optics and Photonics, 78670B.

[6]

Bin Cheng, Bingbing Ni, Shuicheng Yan, and Qi Tian. 2010. Learning to photograph. In Proceedings of the 18th ACM international conference on Multimedia. 291--300.

Digital Library

[7]

Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu. 2014. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 37, 3 (2014), 569--582.

Digital Library

[8]

Cynthia Freeland. 2007. Portraits in painting and photography. Philosophical Studies, Vol. 135, 1 (2007), 95--109.

[9]

Hongbo Fu, Xiaoguang Han, and Quoc Huy Phan. 2013. Data-driven suggestions for portrait posing. In SIGGRAPH Asia 2013 Technical Briefs. ACM, 29.

[10]

Oran Gafni and Lior Wolf. 2020. Wish You Were Here: Context-Aware Human Generation. In CVPR. 7840--7849.

[11]

W Keith Hastings. 1970. Monte Carlo sampling methods using Markov chains and their applications. (1970).

[12]

Xin Jin, Le Wu, Xiaodong Li, Siyu Chen, Siwei Peng, Jingying Chi, Shiming Ge, Chenggen Song, and Geng Zhao. 2018. Predicting aesthetic score distribution through cumulative jensen-shannon divergence. In AAAI Conference on Artificial Intelligence .

[13]

Xin Jin, Le Wu, Geng Zhao, Xiaodong Li, Xiaokun Zhang, Shiming Ge, Dongqing Zou, Bin Zhou, and Xinghui Zhou. 2019. Aesthetic Attributes Assessment of Images. In Proceedings of the 27th ACM International Conference on Multimedia. 311--319.

Digital Library

[14]

Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. 2016. Photo aesthetics ranking network with attributes and content adaptation. In ECCV. Springer, 662--679.

[15]

Bert Krages. 2012. Photography: the art of composition .Simon and Schuster.

[16]

Yining Lang, Wei Liang, Yujia Wang, and Lap-Fai Yu. 2019 b. 3d face synthesis driven by personality impression. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 1707--1714.

[17]

Yining Lang, Wei Liang, and Lap-Fai Yu. 2019 a. Virtual agent positioning driven by scene semantics in mixed reality. In IEEE VR. 767--775.

[18]

Yining Li, Chen Huang, and Chen Change Loy. 2019. Dense intrinsic appearance flow for human pose transfer. In CVPR. 3693--3702.

[19]

Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz. 2019. Planercnn: 3d plane detection and reconstruction from a single image. In CVPR. 4450--4459.

[20]

Ligang Liu, Renjie Chen, Lior Wolf, and Daniel Cohen-Or. 2010. Optimizing photo composition. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 469--478.

Digital Library

[21]

Wei Luo, Xiaogang Wang, and Xiaoou Tang. 2011. Content-based photo quality assessment. In ICCV. IEEE, 2206--2213.

[22]

Shuang Ma, Yangyu Fan, and Chang Wen Chen. 2014. Pose maker: A pose recommendation system for person in the landscape photographing. In Proceedings of the 22nd ACM international conference on Multimedia. 1053--1056.

Digital Library

[23]

Natalia Neverova, Riza Alp Guler, and Iasonas Kokkinos. 2018. Dense pose transfer. In ECCV. 123--138.

[24]

Masashi Nishiyama, Takahiro Okabe, Imari Sato, and Yoichi Sato. 2011. Aesthetic quality classification of photographs based on color harmony. In CVPR. IEEE, 33--40.

[25]

Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In SIGGRAPH. ACM, 497--500.

[26]

Norbert Schneider. 1994. The art of the portrait: masterpieces of European portrait-painting, 1420--1670 .Taschen.

[27]

Yujia Wang, Wei Liang, Jianbing Shen, Yunde Jia, and Lap-Fai Yu. 2019 a. A deep Coarse-to-Fine network for head pose estimation from synthetic data. Pattern Recognition, Vol. 94 (2019), 196--206.

[28]

Yujia Wang, Wenguan Wang, Wei Liang, and Lap-Fai Yu. 2019 b. Comic-guided speech synthesis. TOG, Vol. 38, 6 (2019), 1--14.

[29]

Wenyuan Yin, Tao Mei, Chang Wen Chen, and Shipeng Li. 2013. Socialized mobile photography: Learning to photograph with social context via mobile devices. IEEE Transactions on Multimedia, Vol. 16, 1 (2013), 184--200.

[30]

Lap-Fai Yu, Sai Kit Yeung, Demetri Terzopoulos, and Tony F. Chan. 2012. DressUp!: Outfit Synthesis Through Automatic Optimization. TOG, Vol. 31, 6 (2012), 134:1--134:14.

[31]

Yanhao Zhang, Xiaoshuai Sun, Hongxun Yao, Lei Qin, and Qingming Huang. 2012. Aesthetic composition represetation for portrait photographing recommendation. In Proceedings of the 19th IEEE International Conference on Image Processing. IEEE, 2753--2756.

Cited By

Lee SLee GHan SJeong SLee J(2023)A Simulcast System for Live Streaming and Virtual Avatar ConcertsJournal of the Korea Computer Graphics Society10.15701/kcgs.2023.29.2.2129:2(21-30)Online publication date: 1-Jun-2023
https://doi.org/10.15701/kcgs.2023.29.2.21
Shi ZHou QSheng GKe YWang KJia Y(2023)Where2Stand: Toward a Framework for Portrait Position Recommendation in PhotographyIEEE Access10.1109/ACCESS.2023.332236311(108864-108875)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3322363
Kono MMorimoto NKaku R(2022)VCPoser: Interactive Pose Generation of Virtual Characters Corresponding to Human Pose InputProceedings of the 28th ACM Symposium on Virtual Reality Software and Technology10.1145/3562939.3565640(1-10)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3562939.3565640
Show More Cited By

Index Terms

Photo Stand-Out: Photography with Virtual Character
1. Human-centered computing
  1. Human computer interaction (HCI)
  2. Interaction design

Recommendations

A survey on Arabic character segmentation

Arabic character segmentation is a necessary step in Arabic Optical Character Recognition (OCR). The cursive nature of Arabic script poses challenging problems in Arabic character recognition; however, incorrectly segmented characters will cause ...
Odia character recognition: a directional review

Character recognition is one of the challenging tasks of pattern recognition and machine learning arena. Though a level of saturation has been obtained in machine printed character recognition, there still remains a void while recognizing handwritten ...
3D human pose estimation from a single image via exemplar augmentation
Graphical abstract

Display Omitted
Highlights
- A novel exemplar-based algorithm is proposed to implicitly augment the exemplar set.
Abstract
3D human pose estimation from a single image is a challenging problem due to occlusion, viewpoint variance, and the ill-posed nature of back projection. We follow a standard two-step pipeline which first detects 2D joint locations and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

October 2020

4889 pages

ISBN:9781450379885

DOI:10.1145/3394171

General Chairs:
Chang Wen Chen
Chinese University of Hong Kong, Shenzhen, China
,
Rita Cucchiara
UNIMORE, Italy
,
Xian-Sheng Hua
Alibaba Group, China
,
Program Chairs:
Guo-Jun Qi
Futurewei Technologies, USA
,
Elisa Ricci
UNITN & Fondazione Bruno Kessler, Italy
,
Zhengyou Zhang
Tencent, China
,
Roger Zimmermann
National University of Singapore, Singapore

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

MM '20

Sponsor:

SIGMM

MM '20: The 28th ACM International Conference on Multimedia

October 12 - 16, 2020

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
278
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)5

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lee SLee GHan SJeong SLee J(2023)A Simulcast System for Live Streaming and Virtual Avatar ConcertsJournal of the Korea Computer Graphics Society10.15701/kcgs.2023.29.2.2129:2(21-30)Online publication date: 1-Jun-2023
https://doi.org/10.15701/kcgs.2023.29.2.21
Shi ZHou QSheng GKe YWang KJia Y(2023)Where2Stand: Toward a Framework for Portrait Position Recommendation in PhotographyIEEE Access10.1109/ACCESS.2023.332236311(108864-108875)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3322363
Kono MMorimoto NKaku R(2022)VCPoser: Interactive Pose Generation of Virtual Characters Corresponding to Human Pose InputProceedings of the 28th ACM Symposium on Virtual Reality Software and Technology10.1145/3562939.3565640(1-10)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3562939.3565640
Hou SWang YNing BLiang W(2021)Climaxing VR Character with Scene-Aware Aesthetic Dress Synthesis2021 IEEE Virtual Reality and 3D User Interfaces (VR)10.1109/VR50410.2021.00026(57-64)Online publication date: Mar-2021
https://doi.org/10.1109/VR50410.2021.00026

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents