research-article

Generative GaitNet

Authors:

Phil Sik Chang,

Moon Seok Park,

Jehee LeeAuthors Info & Claims

SIGGRAPH '22: ACM SIGGRAPH 2022 Conference Proceedings

Article No.: 22, Pages 1 - 9

https://doi.org/10.1145/3528233.3530717

Published: 24 July 2022 Publication History

Abstract

Understanding the relation between anatomy and gait is key to successful predictive gait simulation. In this paper, we present Generative GaitNet, which is a novel network architecture based on deep reinforcement learning for controlling a comprehensive, full-body, musculoskeletal model with 304 Hill-type musculotendons. The Generative GaitNet is a pre-trained, integrated system of artificial neural networks learned in a 618-dimensional continuous domain of anatomy conditions (e.g., mass distribution, body proportion, bone deformity, and muscle deficits) and gait conditions (e.g., stride and cadence). The pre-trained GaitNet takes anatomy and gait conditions as input and generates a series of gait cycles appropriate to the conditions through physics-based simulation. We will demonstrate the efficacy and expressive power of Generative GaitNet to generate a variety of healthy and pathological human gaits in real-time physics-based simulation.

Supplementary Material

MP4 File (video.mp4)

Supplemental video

Download
245.87 MB

References

[1]

Farzad Abdolhosseini, Hung Yu Ling, Zhaoming Xie, Xue Bin Peng, and Michiel van de Panne. 2019. On learning symmetric locomotion. In Motion, Interaction and Games. 1–10.

[2]

Rinat Abdrashitov, Seungbae Bang, David IW Levin, Karan Singh, and Alec Jacobson. 2021. Interactive Modelling of Volumetric Musculoskeletal Anatomy. arXiv preprint arXiv:2106.05161(2021).

[3]

Mazen Al Borno, Martin De Lasa, and Aaron Hertzmann. 2012. Trajectory optimization for full-body movements with complex contacts. IEEE transactions on visualization and computer graphics 19, 8(2012), 1405–1414.

[4]

Akhil S Anand, Guoping Zhao, Hubert Roth, and Andre Seyfarth. 2019. A deep reinforcement learning based approach towards generating human walking behavior with a neuromuscular model. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids). IEEE, 537–543.

Digital Library

[5]

Frank C Anderson and Marcus G Pandy. 2001. Dynamic optimization of human walking. J. Biomech. Eng. 123, 5 (2001), 381–390.

[6]

Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. 2019. DReCon: data-driven responsive control of physics-based characters. ACM Transactions On Graphics (TOG) 38, 6 (2019), 1–11.

Digital Library

[7]

Anil Bhave, Dror Paley, and John E Herzenberg. 1999. Improvement in gait parameters after lengthening for the treatment of limb-length discrepancy. JBJS 81, 4 (1999), 529–34.

[8]

Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. 2010. Generalized biped walking control. ACM Transactions On Graphics (TOG) 29, 4 (2010), 1–9.

Digital Library

[9]

Scott L Delp, Frank C Anderson, Allison S Arnold, Peter Loan, Ayman Habib, Chand T John, Eran Guendelman, and Darryl G Thelen. 2007. OpenSim: open-source software to create and analyze dynamic simulations of movement. IEEE transactions on biomedical engineering 54, 11 (2007), 1940–1950.

[10]

Christopher L Dembia, Nicholas A Bianco, Antoine Falisse, Jennifer L Hicks, and Scott L Delp. 2020. Opensim moco: musculoskeletal optimal control. PLOS Computational Biology 16, 12 (2020), e1008493.

[11]

Antoine Falisse, Gil Serrancolí, Christopher L Dembia, Joris Gillis, Ilse Jonkers, and Friedl De Groote. 2019. Rapid predictive simulations with complex musculoskeletal models suggest that diverse healthy and pathological human gaits can emerge from similar control strategies. Journal of the Royal Society Interface 16, 157 (2019), 20190402.

[12]

Levi Fussell, Kevin Bergamin, and Daniel Holden. 2021. SuperTrack: motion tracking for physically simulated characters using supervised learning. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1–13.

Digital Library

[13]

Thomas Geijtenbeek, Michiel Van De Panne, and A Frank Van Der Stappen. 2013. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1–11.

Digital Library

[14]

Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286(2017).

[15]

Jessica K Hodgins, Wayne L Wooten, David C Brogan, and James F O’Brien. 1995. Animating human athletics. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 71–78.

Digital Library

[16]

Yifeng Jiang, Tom Van Wouwe, Friedl De Groote, and C Karen Liu. 2019. Synthesis of biologically realistic human motion using joint torque actuation. ACM Transactions On Graphics (TOG) 38, 4 (2019), 1–12.

Digital Library

[17]

Eunjung Ju, Jungdam Won, Jehee Lee, Byungkuk Choi, Junyong Noh, and Min Gyu Choi. 2013. Data-driven control of flapping flight. ACM Transactions on Graphics (TOG) 32, 5 (2013), 1–12.

Digital Library

[18]

Shuuji Kajita, Fumio Kanehiro, Kenji Kaneko, Kiyoshi Fujiwara, Kensuke Harada, Kazuhito Yokoi, and Hirohisa Hirukawa. 2003. Biped walking pattern generation by using preview control of zero-moment point. In 2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422), Vol. 2. IEEE, 1620–1626.

[19]

Łukasz Kidziński, Sharada Prasanna Mohanty, Carmichael F Ong, Zhewei Huang, Shuchang Zhou, Anton Pechenko, Adam Stelmaszczyk, Piotr Jarosik, Mikhail Pavlov, Sergey Kolesnikov, 2018. Learning to run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments. In The NIPS’17 Competition: Building Intelligent Systems. Springer, 121–153.

[20]

Łukasz Kidziński, Carmichael Ong, Sharada Prasanna Mohanty, Jennifer Hicks, Sean Carroll, Bo Zhou, Hongsheng Zeng, Fan Wang, Rongzhong Lian, Hao Tian, 2020. Artificial intelligence for prosthetics: Challenge solutions. In The NeurIPS’18 Competition. Springer, 69–128.

[21]

Taesoo Kwon and Jessica K Hodgins. 2017. Momentum-mapped inverted pendulum models for controlling dynamic human motions. ACM Transactions on Graphics (TOG) 36, 1 (2017), 1–14.

Digital Library

[22]

Taesoo Kwon, Yoonsang Lee, and Michiel Van De Panne. 2020. Fast and flexible multilegged locomotion using learned centroidal dynamics. ACM Transactions on Graphics (TOG) 39, 4 (2020), 46–1.

Digital Library

[23]

Jehee Lee. 2008. Representing Rotations and Orientations in Geometric Computing. IEEE Computer Graphics and Applications 28, 2 (2008), 75–83.

Digital Library

[24]

Jeongseok Lee, Michael X Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S Srinivasa, Mike Stilman, and C Karen Liu. 2018a. DART: Dynamic animation and robotics toolkit. The Journal of Open Source Software 3, 22 (2018), 500.

[25]

Seyoung Lee, Sunmin Lee, Yongwoo Lee, and Jehee Lee. 2021. Learning a family of motor skills from a single motion clip. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–13.

Digital Library

[26]

Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. 2019. Scalable muscle-actuated human simulation and control. ACM Transactions On Graphics (TOG) 38, 4 (2019), 1–13.

Digital Library

[27]

Seunghwan Lee, Ri Yu, Jungnam Park, Mridul Aanjaneya, Eftychios Sifakis, and Jehee Lee. 2018b. Dexterous manipulation and control with volumetric muscles. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–13.

Digital Library

[28]

Sung-Hee Lee, Eftychios Sifakis, and Demetri Terzopoulos. 2009. Comprehensive biomechanical modeling and simulation of the upper body. ACM Transactions on Graphics (TOG) 28, 4 (2009), 1–17.

Digital Library

[29]

Yoonsang Lee, Sungeun Kim, and Jehee Lee. 2010. Data-driven biped control. In ACM SIGGRAPH 2010 papers. 1–8.

Digital Library

[30]

Yoonsang Lee, Moon Seok Park, Taesoo Kwon, and Jehee Lee. 2014. Locomotion control for many-muscle humanoids. ACM Transactions on Graphics (TOG) 33, 6 (2014), 1–11.

Digital Library

[31]

Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2018. RLlib: Abstractions for Distributed Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 3053–3062.

[32]

Libin Liu and Jessica Hodgins. 2018. Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–14.

Digital Library

[33]

Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. 2020. Catch & Carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG) 39, 4 (2020), 39–1.

Digital Library

[34]

Sehee Min, Jungdam Won, Seunghwan Lee, Jungnam Park, and Jehee Lee. 2019. Softcon: Simulation and control of soft-bodied animals with biomimetic actuators. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–12.

Digital Library

[35]

Igor Mordatch, Martin De Lasa, and Aaron Hertzmann. 2010. Robust physics-based locomotion using low-dimensional planning. In ACM SIGGRAPH 2010 papers. 1–8.

Digital Library

[36]

Igor Mordatch and Emo Todorov. 2014. Combining the benefits of function approximation and trajectory optimization. In Robotics: Science and Systems, Vol. 4.

[37]

Carmichael F Ong, Thomas Geijtenbeek, Jennifer L Hicks, and Scott L Delp. 2019. Predicting gait adaptations due to ankle plantarflexor muscle weakness and contracture using physics-based musculoskeletal simulations. PLoS computational biology 15, 10 (2019), e1006993.

[38]

Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and Jehee Lee. 2019. Learning predict-and-simulate policies from unorganized human motion data. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–11.

Digital Library

[39]

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–14.

Digital Library

[40]

Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–13.

Digital Library

[41]

Xue Bin Peng and Michiel van de Panne. 2017. Learning locomotion skills using deeprl: Does the choice of action space matter?. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 1–13.

Digital Library

[42]

Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. 2020. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning. PMLR, 835–853.

[43]

Hoseok Ryu, Minseok Kim, Seungwhan Lee, Moon Seok Park, Kyoungmin Lee, and Jehee Lee. 2021. Functionality-Driven Musculature Retargeting. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 341–356.

[44]

Kwang Won Sok, Manmyung Kim, and Jehee Lee. 2007. Simulating biped behaviors from human motion data. In ACM SIGGRAPH 2007 papers. 107–es.

Digital Library

[45]

Seungmoon Song and Hartmut Geyer. 2018. Predictive neuromechanical simulations indicate why walking performance declines with ageing. The Journal of physiology 596, 7 (2018), 1199–1210.

[46]

Seungmoon Song, Łukasz Kidziński, Xue Bin Peng, Carmichael Ong, Jennifer L Hicks, Serge Levine, Christopher Atkeson, and Scot Delp. 2020. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. bioRxiv (2020).

[47]

Yuval Tassa, Tom Erez, and Emanuel Todorov. 2012. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 4906–4913.

[48]

Nitish Thatte and Hartmut Geyer. 2015. Toward balance recovery with leg prostheses using neuromuscular model control. IEEE Transactions on Biomedical Engineering 63, 5 (2015), 904–913.

[49]

Jianpeng Wang, Wenhu Qin, and Libo Sun. 2019. Terrain adaptive walking of biped neuromuscular virtual human using deep reinforcement learning. IEEE Access 7(2019), 92465–92475.

[50]

Jack M Wang, David J Fleet, and Aaron Hertzmann. 2009. Optimizing walking controllers. In ACM SIGGRAPH Asia 2009 papers. 1–8.

[51]

Jack M Wang, Samuel R Hamner, Scott L Delp, and Vladlen Koltun. 2012. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1–11.

Digital Library

[52]

NFJ Waterval, K Veerkamp, T Geijtenbeek, J Harlaar, F Nollet, MA Brehm, and MM van der Krogt. 2021. Validation of forward simulations to predict the effects of bilateral plantarflexor weakness on gait. Gait & Posture 87(2021), 33–42.

[53]

Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2020. A scalable approach to control diverse behaviors for physically simulated characters. ACM Transactions on Graphics (TOG) 39, 4 (2020), 33–1.

Digital Library

[54]

Jungdam Won and Jehee Lee. 2019. Learning body shape variation in physics-based characters. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–12.

Digital Library

[55]

KangKang Yin, Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. 2008. Continuation methods for adapting simulated skills. In ACM SIGGRAPH 2008 papers. 1–7.

Digital Library

[56]

KangKang Yin, Kevin Loken, and Michiel Van de Panne. 2007. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics (TOG) 26, 3 (2007), 105–es.

Digital Library

[57]

Wenhao Yu, Greg Turk, and C Karen Liu. 2018. Learning symmetric and low-energy locomotion. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–12.

Digital Library

Cited By

Loi IZacharaki EMoustakas K(2024)Machine Learning Approaches for 3D Motion Synthesis and Musculoskeletal Dynamics Estimation: A SurveyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.330875330:8(5810-5829)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3308753
Jeon JKwon T(2023)Reinforcement Learning of Bipedal Walking with Musculoskeletal Models and Reference MotionsJournal of the Korea Computer Graphics Society10.15701/kcgs.2022.29.1.2329:1(23-29)Online publication date: 1-Mar-2023
https://doi.org/10.15701/kcgs.2022.29.1.23
YAN YCHENG YCHEN ZPENG YWU SZHANG WLI JLI YGAO JZHANG WZHAI GYANG X(2023)A survey on generative 3D digital humans based on neural networks: representation, rendering, and learningSCIENTIA SINICA Informationis10.1360/SSI-2022-031953:10(1858)Online publication date: 13-Oct-2023
https://doi.org/10.1360/SSI-2022-0319
Show More Cited By

Recommendations

Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions
SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

We present a novel generative model, called Bidirectional GaitNet, that learns the relationship between human anatomy and its gait. The simulation model of human anatomy is a comprehensive, full-body, simulation-ready, musculoskeletal model with 304 ...
That Which Does Not Stabilize, Will Only Make Us Stronger

Many places in the world are too rugged or enclosed for vehicles to access. Even today, material transport to such areas is limited to manual labor and beasts of burden. Modern advancements in wearable robotics may make ...
Towards a diagnostic tool for neurological gait disorders in childhood combining 3D gait kinematics and deep learning
Abstract
Gait abnormalities are frequent in children and can be caused by different pathologies, such as cerebral palsy, neuromuscular disease, toe walker syndrome, etc. Analysis of the “gait pattern” (i.e., the way the person walks) using 3D analysis ...
Highlights
- Data-driven diagnostic models require no human intervention or expert knowledge.
- The diagnostic tool distinguished different etiologies of pathological gait.
- The diagnostic tool determined the onset time of stroke.
- Diagnostic ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '22: ACM SIGGRAPH 2022 Conference Proceedings

July 2022

553 pages

ISBN:9781450393379

DOI:10.1145/3528233

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Samsung Research Funding Center

Conference

SIGGRAPH '22

Sponsor:

SIGGRAPH

SIGGRAPH '22: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 7 - 11, 2022

BC, Vancouver, Canada

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
553
Total Downloads

Downloads (Last 12 months)164
Downloads (Last 6 weeks)10

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Loi IZacharaki EMoustakas K(2024)Machine Learning Approaches for 3D Motion Synthesis and Musculoskeletal Dynamics Estimation: A SurveyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.330875330:8(5810-5829)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3308753
Jeon JKwon T(2023)Reinforcement Learning of Bipedal Walking with Musculoskeletal Models and Reference MotionsJournal of the Korea Computer Graphics Society10.15701/kcgs.2022.29.1.2329:1(23-29)Online publication date: 1-Mar-2023
https://doi.org/10.15701/kcgs.2022.29.1.23
YAN YCHENG YCHEN ZPENG YWU SZHANG WLI JLI YGAO JZHANG WZHAI GYANG X(2023)A survey on generative 3D digital humans based on neural networks: representation, rendering, and learningSCIENTIA SINICA Informationis10.1360/SSI-2022-031953:10(1858)Online publication date: 13-Oct-2023
https://doi.org/10.1360/SSI-2022-0319
Feng YXu XLiu L(2023)MuscleVAE: Model-Based Controllers of Muscle-Actuated CharactersSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618137(1-11)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618137
Kim CEom HYoo JChoi SNoh J(2023)Interactive Locomotion Style Control for a Human Character based on Gait Cycle FeaturesComputer Graphics Forum10.1111/cgf.1498843:1Online publication date: 18-Oct-2023
https://doi.org/10.1111/cgf.14988
Hodossy BFarina D(2023)Shared Autonomy Locomotion Synthesis With a Virtual Powered Prosthetic AnkleIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2023.333671331(4738-4748)Online publication date: 2023
https://doi.org/10.1109/TNSRE.2023.3336713

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents