Abstract
Reinforcement learning (RL) has proven successful in games, but suffers from long training times when compared to other forms of machine learning. Curriculum learning, an optimisation technique that improves a model’s ability to learn by presenting training samples in a meaningful order, known as curricula, could offer a solution. Curricula are usually designed manually, due to limitations involved with automating curricula generation. However, as there is a lack of research into effective design of curricula, researchers often rely on intuition and the resulting performance can vary. In this paper, we explore different ways of manually designing curricula for RL in real-time strategy game StarCraft II. We propose four generalised methods of manually creating curricula and verify their effectiveness through experiments. Our results show that all four of our proposed methods can improve a RL agent’s learning process when used correctly. We demonstrate that using subtasks, or modifying the state space of the tasks, is the most effective way to create training samples for StarCraft II. We found that utilising subtasks during training consistently accelerated the learning process of the agent and improved the agent’s final performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Adamsson, M.: Curriculum learning for increasing the performance of a reinforcement learning agent in a static first-person shooter game. Ph.D. thesis, Kth Royal Institute of Technology, Stockholm, Sweden, October 2018
Adil, K., Jiang, F., Liu, S., Jifara, W., Tian, Z., Fu, Y.: State-of-the-art and open challenges in RTS game-AI and starcraft. Int. J. Adv. Comput. Sci. Appl. 8(12) (2017). https://doi.org/10.14569/IJACSA.2017.081203
Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48. ICML 2009, Association for Computing Machinery, New York (2009). https://doi.org/10.1145/1553374.1553380
Cerny Green, M., Sergent, B., Shandilya, P., Kumar, V.: Evolutionarily-curated curriculum learning for deep reinforcement learning agents. arXiv preprint arXiv:1901.05431 (2019)
Florensa, C., Held, D., Geng, X., Abbeel, P.: Automatic goal generation for reinforcement learning agents. In: International Conference on Machine Learning, pp. 1515–1528 (2018)
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., Abbeel, P.: Reverse curriculum generation for reinforcement learning. In: Levine, S., Vanhoucke, V., Goldberg, K. (eds.) Proceedings of the 1st Annual Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 78, pp. 482–495. PMLR, November 2017
Justesen, N., Risi, S.: Learning macromanagement in starcraft from replays using deep learning. In: 2017 IEEE Conference on Computational Intelligence and Games (CIG), pp. 162–169, August 2017. https://doi.org/10.1109/CIG.2017.8080430
Justesen, N., Risi, S.: Continual online evolutionary planning for in-game build order adaptation in starcraft. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 187–194. GECCO 2017. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3071178.3071210
Krueger, K.A., Dayan, P.: Flexible shaping: how learning in small steps helps. Cognition 110(3), 380–394 (2009). https://doi.org/10.1016/j.cognition.2008.11.014
Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937 (2016)
Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Narvekar, S., Sinapov, J., Leonetti, M., Stone, P.: Source task creation for curriculum learning. In: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, AAMAS 2016, pp. 566–574. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2016)
Ontañón, S., Synnaeve, G., Uriarte, A., Richoux, F., Churchill, D., Preuss, M.: A survey of real-time strategy game AI research and competition in starcraft. IEEE Trans. Comput. Intell. AI Games 5(4), 293–311 (2013). https://doi.org/10.1109/TCIAIG.2013.2286295
Pang, Z.J., Liu, R.Z., Meng, Z.Y., Zhang, Y., Yu, Y., Lu, T.: On reinforcement learning for full-length game of starcraft. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4691–4698 (2019)
Pekaalto: sc2aibot (2017) https://github.com/pekaalto/sc2aibot
Peng, B., MacGlashan, J., Loftin, R., Littman, M.L., Roberts, D.L., Taylor, M.E.: An empirical study of non-expert curriculum design for machine learners. In: Proceedings of the Interactive Machine Learning Workshop (at IJCAI 2016) (2016)
Shao, K., Zhu, Y., Zhao, D.: Starcraft micromanagement with reinforcement learning and curriculum transfer learning. IEEE Trans. Emer. Top. Comput. Intell. 3(1), 73–84 (2019). https://doi.org/10.1109/TETCI.2018.2823329
Silva, F.L.D., Costa, A.H.R.: Object-oriented curriculum generation for reinforcement learning. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, pp. 1026–1034. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2018)
Svetlik, M., Leonetti, M., Sinapov, J., Shah, R., Walker, N., Stone, P.: Automatic curriculum graph generation for reinforcement learning agents. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI 2017, pp. 2590–2596. AAAI Press (2017)
Vinyals, O., et al.: Starcraft ii: a new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Hao, D., Sweetser, P., Aitchison, M. (2020). Designing Curriculum for Deep Reinforcement Learning in StarCraft II. In: Gallagher, M., Moustafa, N., Lakshika, E. (eds) AI 2020: Advances in Artificial Intelligence. AI 2020. Lecture Notes in Computer Science(), vol 12576. Springer, Cham. https://doi.org/10.1007/978-3-030-64984-5_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-64984-5_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64983-8
Online ISBN: 978-3-030-64984-5
eBook Packages: Computer ScienceComputer Science (R0)