research-article

Open access

CALM: Conditional Adversarial Latent Models for Directable Virtual Characters

Authors:

Xue Bin PengAuthors Info & Claims

SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

Article No.: 37, Pages 1 - 9

https://doi.org/10.1145/3588432.3591541

Published: 23 July 2023 Publication History

All formats PDF

Abstract

In this work, we present Conditional Adversarial Latent Models (CALM), an approach for generating diverse and directable behaviors for user-controlled interactive virtual characters. Using imitation learning, CALM learns a representation of movement that captures the complexity and diversity of human motion, and enables direct control over character movements. The approach jointly learns a control policy and a motion encoder that reconstructs key characteristics of a given motion without merely replicating it. The results show that CALM learns a semantic motion representation, enabling control over the generated motions and style-conditioning for higher-level task training. Once trained, the character can be controlled using intuitive interfaces, akin to those found in video games.

Supplemental Material

MP4 File

Presentation video and paper appendix (additional results and explanations).

Download
77.45 MB

MP4 File

presentation

Download
426.53 MB

PDF File

Presentation video and paper appendix (additional results and explanations).

Download
6.94 MB

References

[1]

Christopher M Bishop 1995. Neural networks for pattern recognition. Oxford university press.

[2]

Piotr Bojanowski and Armand Joulin. 2017. Unsupervised learning by predicting noise. In International Conference on Machine Learning. PMLR, 517–526.

[3]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.

[4]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139–144.

Digital Library

[5]

Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. Advances in neural information processing systems 29 (2016).

[6]

Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. 2022. PADL: Language-Directed Physics-Based Character Control. In SIGGRAPH Asia 2022 Conference Papers (Daegu, Republic of Korea) (SA ’22). Association for Computing Machinery, New York, NY, USA, Article 19, 9 pages. https://doi.org/10.1145/3550469.3555391

Digital Library

[7]

Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).

[8]

Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. 2020. Character controllers using motion vaes. ACM Transactions on Graphics (TOG) 39, 4 (2020), 40–1.

Digital Library

[9]

Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. 2021. Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=fgFBtYgJQX_

[10]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018).

[11]

Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and Jehee Lee. 2019. Learning predict-and-simulate policies from unorganized human motion data. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–11.

Digital Library

[12]

Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. 2015. Deep face recognition. (2015).

[13]

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37, 4 (2018), 1–14.

Digital Library

[14]

Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. 2022. ASE: Large-scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters. ACM Trans. Graph. 41, 4, Article 94 (July 2022).

Digital Library

[15]

Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. 2021. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–20.

Digital Library

[16]

Reallusion. 2022. 3D Animation and 2D Cartoons Made Simple. (2022). http://www.reallusion.com

[17]

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. Advances in neural information processing systems 29 (2016).

[18]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).

[19]

Richard S Sutton, Doina Precup, and Satinder Singh. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence 112, 1-2 (1999), 181–211.

[20]

Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. 2017. A deep hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31.

[21]

Faraz Torabi, Garrett Warnell, and Peter Stone. 2018. Generative Adversarial Imitation from Observation. CoRR abs/1807.06158 (2018). http://arxiv.org/abs/1807.06158

[22]

Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. 2017. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia. 1041–1049.

Digital Library

[23]

Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. PMLR, 9929–9939.

[24]

Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2022. Physics-based character controllers using conditional VAEs. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–12.

Digital Library

[25]

Jiacheng Xu and Greg Durrett. 2018. Spherical Latent Spaces for Stable Variational Autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 4503–4513.

Cited By

Kang KGu TKwon T(2025)Learning Climbing Controllers for Physics‐Based CharactersComputer Graphics Forum10.1111/cgf.15284Online publication date: 30-Jan-2025
https://doi.org/10.1111/cgf.15284
Liu XWu JXue YQi CXin GGao F(2025)Skill Latent Space Based Multigait Learning for a Legged RobotIEEE Transactions on Industrial Electronics10.1109/TIE.2024.342957672:2(1743-1752)Online publication date: Feb-2025
https://doi.org/10.1109/TIE.2024.3429576
Kang KKwon T(2024)Climbing Motion Synthesis using Reinforcement LearningJournal of the Korea Computer Graphics Society10.15701/kcgs.2024.30.2.2130:2(21-29)Online publication date: 1-Jun-2024
https://doi.org/10.15701/kcgs.2024.30.2.21
Show More Cited By

Index Terms

CALM: Conditional Adversarial Latent Models for Directable Virtual Characters
1. Computing methodologies
  1. Computer graphics
    1. Animation
      1. Procedural animation
  2. Machine learning
    1. Learning paradigms
      1. Reinforcement learning
        Adversarial learning

Recommendations

MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting

Crafting a single, versatile physics-based controller that can breathe life into interactive characters across a wide spectrum of scenarios represents an exciting frontier in character animation. An ideal controller should support diverse control ...
Keyframe animation of virtual humans via motion data learning

This paper proposes a method for creating human animations by utilizing motion capture data in traditional keyframe-based motion editing. This automatically generates complicated sequential movements of the whole body from multiple constraints of end-...
A novel automatic motion capture data recognition method based on statistics learning and subspace

In this paper, we propose a motion recognition method based on motion capture data. To recognise motion type, a generalised Isomap non-linear dimension reduction based on Radius Basis Function (RBF) networks and feature extraction is used to project ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

July 2023

911 pages

ISBN:9798400701597

DOI:10.1145/3588432

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SIGGRAPH '23

Sponsor:

SIGGRAPH

SIGGRAPH '23: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 6 - 10, 2023

CA, Los Angeles, USA

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
1,245
Total Downloads

Downloads (Last 12 months)779
Downloads (Last 6 weeks)74

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kang KGu TKwon T(2025)Learning Climbing Controllers for Physics‐Based CharactersComputer Graphics Forum10.1111/cgf.15284Online publication date: 30-Jan-2025
https://doi.org/10.1111/cgf.15284
Liu XWu JXue YQi CXin GGao F(2025)Skill Latent Space Based Multigait Learning for a Legged RobotIEEE Transactions on Industrial Electronics10.1109/TIE.2024.342957672:2(1743-1752)Online publication date: Feb-2025
https://doi.org/10.1109/TIE.2024.3429576
Kang KKwon T(2024)Climbing Motion Synthesis using Reinforcement LearningJournal of the Korea Computer Graphics Society10.15701/kcgs.2024.30.2.2130:2(21-29)Online publication date: 1-Jun-2024
https://doi.org/10.15701/kcgs.2024.30.2.21
Tessler CGuo YNabati OChechik GPeng X(2024)MaskedMimic: Unified Physics-Based Character Control Through Masked Motion InpaintingACM Transactions on Graphics10.1145/368795143:6(1-21)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687951
Wu YDou ZIshiwaka YOgawa SLou YWang WLiu LKomura T(2024)CBIL: Collective Behavior Imitation Learning for Fish from Real VideosACM Transactions on Graphics10.1145/368790443:6(1-17)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687904
Serifi AGrandia RKnoop EGross MBächer M(2024)Robot Motion Diffusion Model: Motion Generation for Robotic CharactersSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687626(1-9)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687626
Boursin PKedadry YZordan VKry PCani M(2024)ReGAIL: Toward Agile Character Control From a Single Reference MotionProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696330(1-10)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3677388.3696330
Yao HSong ZZhou YAo TChen BLiu L(2024)MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete RepresentationsACM Transactions on Graphics10.1145/365813743:4(1-21)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658137
Hu HYi XCao ZYong JXu F(2024)Hand-Object Interaction Controller (HOIC): Deep Reinforcement Learning for Reconstructing Interactions with PhysicsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657505(1-10)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657505
Chen RShi MHuang STan PKomura TChen X(2024)Taming Diffusion Probabilistic Models for Character ControlACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657440(1-10)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657440
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten