research-article

Agent-based cooperative animation for box-manipulation using reinforcement learning

Authors:

Hsiang-Yu Yang,

Sai-Keung WongAuthors Info & Claims

Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 2, Issue 1

Article No.: 2, Pages 1 - 18

https://doi.org/10.1145/3320283

Published: 03 June 2019 Publication History

Abstract

This paper presents an approach to assist the generation of agent-based cooperative animation using reinforcement learning. We focus on manipulation skills for box-shaped objects, including pushing, pulling, and moving objects in a relay way. There are a learning process and an animation process. In the learning process, different kinds of agents are trained using reinforcement learning. Policies are learned to control the agents to perform specific tasks. A physics simulator is adopted to simulate the interaction among objects. In the animation process, users animate agents with the learned policies. We propose several tools to intuitively create cooperative animations. We applied our method to generate several animations in which agents work together to finish tasks. A user study indicates that by using our tools, diverse cooperative animations can be easily created.

References

[1]

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866 (2017).

[2]

Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35, 8 (2013), 1798--1828.

Digital Library

[3]

Hsuan Chen and Sai-Keung Wong. 2018. Transporting objects by multiagent cooperation in crowd simulation. Computer Animation and Virtual Worlds 29, 3-4 (2018), e1826.

[4]

Alexander Clegg, Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. 2018. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Transactions on Graphics 37, 6 (2018).

Digital Library

[5]

Hongliang Guo and Yan Meng. 2010. Distributed reinforcement learning for coordinate multi-robot foraging. Journal of intelligent & robotic systems 60, 3-4 (2010), 531--551.

Digital Library

[6]

Shihui Guo, Meili Wang, Gabriel Notman, Jian Chang, Jianjun Zhang, and Minghong Liao. 2017. Simulating collective transport of virtual ants. Computer Animation and Virtual Worlds 28, 3-4 (2017), e1779.

[7]

Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017).

[8]

Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. 2018. Unity: A General Platform for Intelligent Agents. arXiv preprint arXiv:1809.02627 (2018).

[9]

Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. 2016. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In Computational Intelligence and Games, 2016 IEEE Conference on. 1--8.

[10]

Guan-Wen Lin and Sai-Keung Wong. 2018. Evacuation simulation with consideration of obstacle removal and using game theory. Phys. Rev. E 97 (Jun 2018), 062303. Issue 6.

[11]

Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha. 2018. Fluid directed rigid body control using deep reinforcement learning. ACM Transactions on Graphics 37, 4 (2018), 96.

Digital Library

[12]

Tekin Meriçli, Manuela Veloso, and H Levent Akın. 2015. Push-manipulation of complex passive mobile objects using experimentally acquired motion models. Autonomous Robots 38, 3 (2015), 317--329.

Digital Library

[13]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529.

[14]

Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. 2016. Control of memory, active perception, and action in minecraft. arXiv preprint arXiv:1605.09128 (2016).

Digital Library

[15]

Liviu Panait and Sean Luke. 2005. Cooperative multi-agent learning: The state of the art. Autonomous agents and multi-agent systems 11, 3 (2005), 387--434.

Digital Library

[16]

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. arXiv preprint arXiv:1804.02717 (2018).

[17]

Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics) 36, 4 (2017), 41.

Digital Library

[18]

Martin Riedmiller. 2005. Neural fitted Q iteration--first experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning. Springer, 317--328.

Digital Library

[19]

Samuel Rodriguez, Marco Morales, and Nancy M Amato. 2016. Multi-agent push behaviors for large sets of passive objects. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on. IEEE, 4437--4442.

[20]

Eric Rohmer, Surya PN Singh, and Marc Freese. 2013. V-REP: A versatile and scalable robot simulation framework. In Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on. IEEE, 1321--1326.

[21]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).

[22]

Peter Stone, Richard S Sutton, and Gregory Kuhlmann. 2005. Reinforcement learning for robocup soccer keepaway. Adaptive Behavior 13, 3 (2005), 165--188.

[23]

Elio Tuci, Muhanad Hayder Mohammed Alkilabi, and Otar Akanyeti. 2018. Cooperative object transport in multi-robot systems: A review of the state-of-the-art. Frontiers in Robotics and AI 5 (2018), Article:59.

[24]

Kai Wang, Manolis Savva, Angel X Chang, and Daniel Ritchie. 2018. Deep convolutional priors for indoor scene synthesis. ACM Transactions on Graphics (TOG) 37, 4 (2018), 70.

Digital Library

[25]

Jungdam Won, Jongho Park, Kwanyu Kim, and Jehee Lee. 2017. How to train your dragon: example-guided control of flapping flight. ACM Transactions on Graphics 36, 6 (2017), 198.

Digital Library

[26]

Sai-Keung Wong, Yi-Hung Chou, and Hsiang-Yu Yang. 2018. A framework for simulating agent-based cooperative tasks in crowd simulation. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, 11.

Digital Library

[27]

Sai-Keung Wong, Yu-Shuen Wang, Pao-Kun Tang, and Tsung-Yu Tsai. 2017. Optimized evacuation route based on crowd simulation. Computational Visual Media 3, 3 (2017), 243--261.

[28]

Wei Xiang, Jiaping Ren, Kuan Wang, Zhigang Deng, and Xiaogang Jin. 2018. Biologically inspired ant colony simulation. Computer Animation and Virtual Worlds (2018), e1867.

[29]

Jun Xing, Rubaiat Habib Kazi, Tovi Grossman, Li-Yi Wei, Jos Stam, and George Fitzmaurice. 2016. Energy-brushes: Interactive tools for illustrating stylized elemental dynamics. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 755--766.

Digital Library

[30]

Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Upcroft, and Peter Corke. 2015. Towards vision-based deep reinforcement learning for robotic motion control. arXiv preprint arXiv: 1511.03791 (2015).

[31]

Xinyi Zhang and Michiel van de Panne. 2018. Data-driven autocompletion for keyframe animation. In Proceedings of the 11th Annual International Conference on Motion, Interaction, and Games. ACM, 10.

Digital Library

Cited By

Cen RRen B(2025)Layer-Based Simulation for Three-Dimensional Fluid Flow in Spherical CoordinatesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.336052131:2(1435-1447)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1109/TVCG.2024.3360521
Li YNumerow LThomaszewski BCoros S(2024)Differentiable Geodesic Distance for Intrinsic Minimization on Triangle MeshesACM Transactions on Graphics10.1145/365812243:4(1-14)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658122
An BLi JLi ZLi R(2024)The Mechanism of Aluminum Deposition and Dendrite Growth on a Copper Substrate in Anode-Free Aluminum BatteriesACS Applied Materials & Interfaces10.1021/acsami.4c1268816:46(63553-63559)Online publication date: 6-Nov-2024
https://doi.org/10.1021/acsami.4c12688
Show More Cited By

Index Terms

Agent-based cooperative animation for box-manipulation using reinforcement learning
1. Computing methodologies
  1. Computer graphics
    1. Animation
    2. Graphics systems and interfaces
      1. Virtual reality

Recommendations

Distributed Reinforcement Learning for Cooperative Multi-Robot Object Manipulation
AAMAS '20: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems

We consider solving a cooperative multi-robot object manipulation task using reinforcement learning (RL). We propose two distributed multi-agent RL approaches: distributed approximate RL (DA-RL), where each agent applies Q-learning with individual ...
Object manipulation system based on image-based reinforcement learning
Abstract
Advances in reinforcement learning algorithms allow robots to learn complex tasks such as object manipulation. However, most of these tasks have been implemented only in simulations. In addition, it is difficult to apply reinforcement learning in ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques

Proceedings of the ACM on Computer Graphics and Interactive Techniques Volume 2, Issue 1

May 2019

132 pages

EISSN:2577-6193

DOI:10.1145/3339245

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2019

Published in PACMCGIT Volume 2, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

The Ministry of Science and Technology of the ROC

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
316
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)6

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cen RRen B(2025)Layer-Based Simulation for Three-Dimensional Fluid Flow in Spherical CoordinatesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.336052131:2(1435-1447)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1109/TVCG.2024.3360521
Li YNumerow LThomaszewski BCoros S(2024)Differentiable Geodesic Distance for Intrinsic Minimization on Triangle MeshesACM Transactions on Graphics10.1145/365812243:4(1-14)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658122
An BLi JLi ZLi R(2024)The Mechanism of Aluminum Deposition and Dendrite Growth on a Copper Substrate in Anode-Free Aluminum BatteriesACS Applied Materials & Interfaces10.1021/acsami.4c1268816:46(63553-63559)Online publication date: 6-Nov-2024
https://doi.org/10.1021/acsami.4c12688
Liu GWong S(2024)Mastering broom‐like tools for object transportation animation using deep reinforcement learningComputer Animation and Virtual Worlds10.1002/cav.225535:3Online publication date: 14-Jun-2024
https://doi.org/10.1002/cav.2255
Shang XXu TKaramouzas IKallmann M(2023)Constraint‐based multi‐agent reinforcement learning for collaborative tasksComputer Animation and Virtual Worlds10.1002/cav.218234:3-4Online publication date: 23-May-2023
https://doi.org/10.1002/cav.2182
Wong SWei X(2023)Animation generation for object transportation with a rope using deep reinforcement learningComputer Animation and Virtual Worlds10.1002/cav.216834:3-4Online publication date: 11-May-2023
https://doi.org/10.1002/cav.2168
Deng YWang MKong XXiong SXian ZZhu B(2022)A moving eulerian-lagrangian particle method for thin film and foam simulationACM Transactions on Graphics10.1145/3528223.353017441:4(1-17)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530174
Cardoso JKerbl BYang LUralsky YWimmer M(2022)Training and Predicting Visual Error for Real-Time ApplicationsProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/35226255:1(1-17)Online publication date: 4-May-2022
https://dl.acm.org/doi/10.1145/3522625
Wong CCai CTsai HLiu GWong S(2022)Generation of cart‐pulling animation in a multiagent environment using deep learningComputer Animation and Virtual Worlds10.1002/cav.208133:3-4Online publication date: 3-Jun-2022
https://doi.org/10.1002/cav.2081
Cui QLanglois TSen PKim T(2021)Spiral-spectral fluid simulationACM Transactions on Graphics10.1145/3478513.348053640:6(1-16)Online publication date: 10-Dec-2021
https://dl.acm.org/doi/10.1145/3478513.3480536
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents