research-article

Mode-adaptive neural networks for quadruped motion control

Authors:

Sebastian Starke,

Jun SaitoAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 4

Article No.: 145, Pages 1 - 11

https://doi.org/10.1145/3197517.3201366

Published: 30 July 2018 Publication History

Abstract

Quadruped motion includes a wide variation of gaits such as walk, pace, trot and canter, and actions such as jumping, sitting, turning and idling. Applying existing data-driven character control frameworks to such data requires a significant amount of data preprocessing such as motion labeling and alignment. In this paper, we propose a novel neural network architecture called Mode-Adaptive Neural Networks for controlling quadruped characters. The system is composed of the motion prediction network and the gating network. At each frame, the motion prediction network computes the character state in the current frame given the state in the previous frame and the user-provided control signals. The gating network dynamically updates the weights of the motion prediction network by selecting and blending what we call the expert weights, each of which specializes in a particular movement. Due to the increased flexibility, the system can learn consistent expert weights across a wide range of non-periodic/periodic actions, from unstructured motion capture data, in an end-to-end fashion. In addition, the users are released from performing complex labeling of phases in different gaits. We show that this architecture is suitable for encoding the multi-modality of quadruped locomotion and synthesizing responsive motion in real-time.

Supplementary Material

MP4 File (145-431.mp4)

Download
351.53 MB

MP4 File (a145-zhang.mp4)

Download
283.31 MB

References

[1]

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-scale Machine Learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI'16). USENIX Association, Berkeley, CA, USA, 265--283. http://dl.acm.org/citation.cfm?id=3026877.3026899

Digital Library

[2]

Mazen Al Borno, Martin De Lasa, and Aaron Hertzmann. 2013. Trajectory optimization for full-body movements with complex contacts. IEEE Trans on Vis and Comp Graph 19, 8 (2013), 1405--1414.

Digital Library

[3]

Okan Arikan and David A Forsyth. 2002. Interactive motion generation from examples. ACM Trans on Graph 21, 3 (2002), 483--490.

Digital Library

[4]

Emad Barsoum, John Kender, and Zicheng Liu. 2017. HP-GAN: Probabilistic 3D human motion prediction via GAN. CoRR abs/1711.09561 (2017). arXiv:1711.09561 http://arxiv.org/abs/1711.09561

[5]

Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. 2015. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297 (2015). https://arxiv.org/abs/1511.06297

[6]

Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. 2016. Learning feed-forward one-shot learners. In Proc. NIPS. 523--531.

Digital Library

[7]

Jinxiang Chai and Jessica K Hodgins. 2005. Performance Animation from Low-dimensional Control Signals. ACM Trans on Graph 24, 3 (2005), 686--696.

Digital Library

[8]

Xiaobin Chang, Timothy M Hospedales, and Tao Xiang. 2018. Multi-level factorisation net for person re-identification. arXiv preprint arXiv:1803.09132 (2018). https://arxiv.org/abs/1803.09132v2

[9]

Simon Clavet. 2016. Motion Matching and The Road to Next-Gen Animation. In Proc. of GDC 2016.

[10]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). CoRR abs/1511.07289 (2015). http://arxiv.org/abs/1511.07289

[11]

Stelian Coros, Andrej Karpathy, Ben Jones, Lionel Reveret, and Michiel Van De Panne. 2011. Locomotion skills for simulated quadrupeds. ACM Trans on Graph 30, 4 (2011), 59.

Digital Library

[12]

David Eigen, Marc' Aurelio Ranzato, and Ilya Sutskever. 2013. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314 (2013).

[13]

Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In Proc. ICCV. 4346--4354.

Digital Library

[14]

Keith Grochow, Steven L Martin, Aaron Hertzmann, and Zoran Popović. 2004. Style-based inverse kinematics. ACM Trans on Graph 23, 3 (2004), 522--531.

Digital Library

[15]

Chris Hecker, Bernd Raabe, Ryan W Enslow, John DeWeese, Jordan Maynard, and Kees van Prooijen. 2008. Real-time motion retargeting to highly varied user-created morphologies. ACM Trans on Graph 27, 3 (2008), 27.

Digital Library

[16]

Daniel Holden, Taku Komura, and Jun Saito. 2017. Phase-functioned neural networks for character control. ACM Trans on Graph 36, 4 (2017), 42.

Digital Library

[17]

Daniel Holden, Jun Saito, and Taku Komura. 2016. A deep learning framework for character motion synthesis and editing. ACM Trans on Graph 35, 4 (2016).

Digital Library

[18]

Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015. Learning Motion Manifolds with Convolutional Autoencoders. In SIGGRAPH Asia 2015 Technical Briefs. Article 18, 4 pages.

Digital Library

[19]

Eugene Hsu, Kari Pulli, and Jovan Popovic. 2005. Style Translation for Human Motion. ACM Trans on Graph 24, 3 (2005), 1082--1089.

Digital Library

[20]

Ting-Chieh Huang, Yi-Jheng Huang, and Wen-Chieh Lin. 2013. Real-time horse gait synthesis. Computer Animation and Virtual Worlds 24, 2 (2013), 87--95.

[21]

Leslie Ikemoto, Okan Arikan, and David Forsyth. 2009. Generalizing motion edits with gaussian processes. ACM Trans on Graph 28, 1 (2009), 1.

Digital Library

[22]

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural Computation 3, 1 (1991), 79--87.

[23]

Michael I Jordan and Robert A Jacobs. 1994. Hierarchical mixtures of experts and the EM algorithm. Neural Computation 6, 2 (1994), 181--214.

Digital Library

[24]

Ahmad Abdul Karim, Thibaut Gaudin, Alexandre Meyer, Axel Buendia, and Saida Bouakaz. 2013. Procedural locomotion of multilegged characters in dynamic environments. Computer Animation and Virtual Worlds 24, 1 (2013), 3--15.

[25]

Lucas Kovar and Michael Gleicher. 2004. Automated Extraction and Parameterization of Motions in Large Data Sets. ACM Trans on Graph 23, 3 (2004), 559--568.

Digital Library

[26]

Lucas Kovar, Michael Gleicher, and Frédéric Pighin. 2002. Motion graphs. ACM Trans on Graph 21, 3 (2002), 473--482.

Digital Library

[27]

Paul G Kry, Lionel Revéret, François Faure, and M-P Cani. 2009. Modal locomotion: Animating virtual characters with natural vibrations. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 289--298.

[28]

Manfred Lau and James J Kuffher. 2005. Behavior planning for character animation. In Proc. SCA. 271--280.

Digital Library

[29]

Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, and Nancy S Pollard. 2002. Interactive control of avatars animated with human motion data. ACM Trans on Graph 21, 3 (2002), 491--500.

Digital Library

[30]

Yongjoon Lee, Kevin Wampler, Gilbert Bernstein, Jovan Popović, and Zoran Popović. 2010. Motion fields for interactive character locomotion. ACM Trans on Graph 29, 6 (2010), 138.

Digital Library

[31]

Sergey Levine and Jovan Popović. 2012. Physically Plausible Simulation for Character Animation. In Proc. SCA. 221--230. http://dl.acm.org/citation.cfm?id=2422356.2422388

Digital Library

[32]

Sergey Levine, Jack M Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun. 2012. Continuous character control with low-dimensional embeddings. ACM Trans on Graph 31, 4 (2012), 28.

Digital Library

[33]

Zimo Li, Yi Zhou, Shuangjiu Xiao, Chong He, and Hao Li. 2017. Auto-Conditioned LSTM Network for Extended Complex Human Motion Synthesis. arXiv preprint arXiv:1707.05363 (2017). https://arxiv.org/abs/1707.05363

[34]

C. Karen Liu, Aaron Hertzmann, and Zoran Popović;. 2005. Learning physics-based motion style with nonlinear inverse optimization. ACM Trans on Graph 24, 3 (2005), 1071--1081.

Digital Library

[35]

C. Karen Liu and Zoran Popović. 2002. Synthesis of complex dynamic character motion from simple animations. ACM Trans on Graph 21, 3 (2002), 408--416.

Digital Library

[36]

Libin Liu and Jessica Hodgins. 2017. Learning to schedule control fragments for physics-based characters using deep q-learning. ACM Trans on Graph 36, 3 (2017), 29.

Digital Library

[37]

Ilya Loshchilov and Frank Hutter. 2017. Fixing Weight Decay Regularization in Adam. CoRR abs/1711.05101 (2017). arXiv:1711.05101 http://arxiv.org/abs/1711.05101

[38]

Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. 2017a. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201 (2017). https://arxiv.org/abs/1707.02201

[39]

Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. 2017b. Learning human behaviors from motion capture by adversarial imitation. CoRR abs/1707.02201 (2017). arXiv:1707.02201 http://arxiv.org/abs/1707.02201

[40]

Jianyuan Min and Jinxiang Chai. 2012. Motion graphs++: a compact generative model for semantic motion analysis and synthesis. ACM Trans on Graph 31, 6 (2012), 153.

Digital Library

[41]

Tomohiko Mukai and Shigeru Kuriyama. 2005. Geostatistical motion interpolation. ACM Trans on Graph 24, 3 (2005), 1062--1070.

Digital Library

[42]

Xue Bin Peng, Glen Berseth, and Michiel van de Panne. 2015. Dynamic Terrain Traversal Skills Using Reinforcement Learning. ACM Trans on Graph 34, 4, Article 80 (2015), 80:1--80:11 pages.

Digital Library

[43]

Xue Bin Peng, Glen Berseth, and Michiel van de Panne. 2016. Terrain-Adaptive Locomotion Skills Using Deep Reinforcement Learning. ACM Trans on Graph 35, 4 (2016).

Digital Library

[44]

Marc H. Raibert and Jessica K. Hodgins. 1991. Animation of dynamic legged locomotion. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1991, Providence, RI, USA, April 27--30, 1991. 349--358.

Digital Library

[45]

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. arXiv preprint arXiv:1705.08045 (2017). https://arxiv.org/abs/1705.08045

[46]

Charles Rose, Michael F. Cohen, and Bobby Bodenheimer. 1998. Verbs and Adverbs: Multidimensional Motion Interpolation. IEEE Computer Graphics and Applications 18, 5 (1998), 32--40.

Digital Library

[47]

Alla Safonova and Jessica K Hodgins. 2007. Construction and optimal search of interpolated motion graphs. ACM Trans on Graph 26, 3 (2007), 106.

Digital Library

[48]

Alla Safonova, Jessica K Hodgins, and Nancy S Pollard. 2004. Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. ACM Trans on Graph 23, 3 (2004), 514--521.

Digital Library

[49]

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. CoRR abs/1701.06538 (2017). arXiv:1701.06538 http://arxiv.org/abs/1701.06538

[50]

Jochen Tautges, Arno Zinke, Björn Krüger, Jan Baumann, Andreas Weber, Thomas Helfen, Meinard Müller, Hans-Peter Seidel, and Bernd Eberhardt. 2011. Motion reconstruction using sparse accelerometer data. ACM Trans on Graph 30, 3 (2011), 18.

Digital Library

[51]

Graham W Taylor and Geoffrey E Hinton. 2009. Factored conditional restricted Boltzmann machines for modeling motion style. In Proc. ICML. ACM, 1025--1032.

Digital Library

[52]

Graham W Taylor, Geoffrey E Hinton, and Sam T Roweis. 2011. Two distributed-state models for generating high-dimensional time series. The Journal of Machine Learning Research 12 (2011), 1025--1068.

Digital Library

[53]

Michiel van de Panne. 1996. Parameterized gait synthesis. IEEE Computer Graphics and Applications 16, 2 (1996), 40--49.

Digital Library

[54]

Kevin Wampler and Zoran Popović. 2009. Optimal gait and form for animal locomotion. ACM Trans on Graph 28, 3 (2009), 60.

Digital Library

[55]

Kevin Wampler, Zoran Popović, and Jovan Popović. 2014. Generalizing locomotion style to new animals with inverse optimal regression. ACM Trans on Graph 33, 4 (2014), 49.

Digital Library

[56]

J.M. Wang, D.J. Fleet, and A. Hertzmann. 2008. Gaussian Process Dynamical Models for Human Motion. Pattern Analysis and Machine Intelligence, IEEE Trans. on 30, 2 (Feb 2008), 283--298. D0I

Digital Library

[57]

Shihong Xia, Congyi Wang, Jinxiang Chai, and Jessica Hodgins. 2015. Realtime Style Transfer for Unlabeled Heterogeneous Human Motion. ACM Trans on Graph 34, 4 (2015), 119:1--119:10.

Digital Library

[58]

Yuting Ye and C Karen Liu. 2012. Synthesis of detailed hand manipulations using contact sampling. ACM Trans on Graph 31, 4 (2012), 41.

Digital Library

[59]

KangKang Yin, Kevin Loken, and Michiel Van de Panne. 2007. Simbicon: Simple biped locomotion control. ACM Trans on Graph 26, 3 (2007), 105.

Digital Library

[60]

Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. 2012. Twenty years of mixture of experts. IEEE Trans on Neural Networks and Learning Systems 23, 8 (2012), 1177--1193.

Cited By

Dima MDaylamani-Zad D(2024)Intelligent Immersion: The current landscape of AI tools in the Mixed Reality development pipeline for creative experiencesRevista de la Asociación Española de Investigación de la Comunicación10.24137/raeic.11.e.1(raeic11e01)Online publication date: 31-Jul-2024
https://doi.org/10.24137/raeic.11.e.1
Shao YJin YHuang ZWang HYang W(2024)A learning-based control pipeline for generic motor skills for quadruped robots基于学习的四足机器人通用技能控制方法Journal of Zhejiang University-SCIENCE A10.1631/jzus.A230012825:6(443-454)Online publication date: 12-Feb-2024
https://doi.org/10.1631/jzus.A2300128
Si ZGu TKwon T(2024)An Auto Obstacle Collision Avoidance System using Reinforcement Learning and Motion VAEJournal of the Korea Computer Graphics Society10.15701/kcgs.2024.30.4.130:4(1-10)Online publication date: 1-Sep-2024
https://doi.org/10.15701/kcgs.2024.30.4.1
Show More Cited By

Index Terms

Mode-adaptive neural networks for quadruped motion control
1. Computing methodologies
  1. Computer graphics
    1. Animation
      1. Motion capture
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Neural animation layering for synthesizing martial arts movements

Interactively synthesizing novel combinations and variations of character movements from different motion skills is a key problem in computer animation. In this paper, we propose a deep learning framework to produce a large variety of martial arts ...
A deep learning framework for character motion synthesis and editing

We present a framework to synthesize character movements based on high level parameters, such that the produced movements respect the manifold of human motion, trained on a large motion capture dataset. The learned motion manifold, which is represented ...
Phase-functioned neural networks for character control

We present a real-time character control mechanism using a novel neural network architecture called a Phase-Functioned Neural Network. In this network structure, the weights are computed via a cyclic function which uses the phase as an input. Along with ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 4

August 2018

1670 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3197517

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2018

Published in TOG Volume 37, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Google
Adobe Systems

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

194
Total Citations
View Citations
3,139
Total Downloads

Downloads (Last 12 months)448
Downloads (Last 6 weeks)92

Reflects downloads up to 27 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dima MDaylamani-Zad D(2024)Intelligent Immersion: The current landscape of AI tools in the Mixed Reality development pipeline for creative experiencesRevista de la Asociación Española de Investigación de la Comunicación10.24137/raeic.11.e.1(raeic11e01)Online publication date: 31-Jul-2024
https://doi.org/10.24137/raeic.11.e.1
Shao YJin YHuang ZWang HYang W(2024)A learning-based control pipeline for generic motor skills for quadruped robots基于学习的四足机器人通用技能控制方法Journal of Zhejiang University-SCIENCE A10.1631/jzus.A230012825:6(443-454)Online publication date: 12-Feb-2024
https://doi.org/10.1631/jzus.A2300128
Si ZGu TKwon T(2024)An Auto Obstacle Collision Avoidance System using Reinforcement Learning and Motion VAEJournal of the Korea Computer Graphics Society10.15701/kcgs.2024.30.4.130:4(1-10)Online publication date: 1-Sep-2024
https://doi.org/10.15701/kcgs.2024.30.4.1
Tao DRuizhen HLibin LLi YHao Z(2024)Research progress in human-like indoor scene interactionJournal of Image and Graphics10.11834/jig.24000429:6(1575-1606)Online publication date: 2024
https://doi.org/10.11834/jig.240004
Egan DJovane ASzkaradek JFletcher GCosker DMcDonnell R(2024)Dog Code: Human to Quadruped Embodiment using Shared CodebooksProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696339(1-11)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3677388.3696339
Starke SStarke PHe NKomura TYe Y(2024)Categorical Codebook Matching for Embodied Character ControllersACM Transactions on Graphics10.1145/365820943:4(1-14)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658209
Kucherenko TWolfert PYoon YViegas CNikolov TTsakov MHenter G(2024)Evaluating Gesture Generation in a Large-scale Open Challenge: The GENEA Challenge 2022ACM Transactions on Graphics10.1145/365637443:3(1-28)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3656374
Sun HZheng RHuang HMa CHuang HHu R(2024)LGTM: Local-to-Global Text-Driven Human Motion Diffusion ModelACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657422(1-9)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657422
Hong SKim HCho KNoh J(2024)Long‐term Motion In‐betweening via Keyframe PredictionComputer Graphics Forum10.1111/cgf.15171Online publication date: 9-Oct-2024
https://doi.org/10.1111/cgf.15171
Zhao QLi PYifan WOlga SWetzstein G(2024)Pose‐to‐Motion: Cross‐Domain Motion Retargeting with Pose PriorComputer Graphics Forum10.1111/cgf.15170Online publication date: 9-Oct-2024
https://doi.org/10.1111/cgf.15170
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents