research-article

Using an agent-based approach for robust automated testing of computer games

Authors:

Samira Shirzadehhajimahmood,

I. S. W. B. Prasetya,

Gabriele KellerAuthors Info & Claims

A-TEST 2021: Proceedings of the 12th International Workshop on Automating TEST Case Design, Selection, and Evaluation

Pages 1 - 8

https://doi.org/10.1145/3472672.3473952

Published: 23 August 2021 Publication History

Abstract

Modern computer games typically have a huge interaction spaces and non-deterministic environments. Automation in testing can provide a vital boost in development and it further improves the overall software's reliability and efficiency. Moreover, layout and game logic may regularly change during development or consecutive releases which makes it difficult to test because the usage of the system continuously changes. To deal with the latter, tests also need to be robust. Unfortunately, existing game testing approaches are not capable of maintaining test robustness. To address these challenges, this paper presents an agent-based approach for robust automated testing based on the reasoning type of AI.

References

[1]

2019-2020. World Quality Report 2019-2020. https://www.capgemini.com/research/world-quality-report-2019/

[2]

Tomás Ahumada and Alexandre Bergel. 2020. Reproducing Bugs in Video Games using Genetic Algorithms. In 2020 IEEE Games, Multimedia, Animation and Multiple Realities Conference (GMAX). 1–6.

[3]

Saswat Anand, Edmund K Burke, and Tsong Yueh Chen. 2013. An orchestrated survey of methodologies for automated software test case generation. Jour. of Sys. and Software, 86, 8 (2013).

[4]

Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2019. Automated Video Game Testing Using Synthetic and Human-Like Agents. IEEE Trans. on Games.

[5]

A Askarunisa, P Prameela, and N Ramraj. 2009. A proposed agent based framework for testing data-centric applications. International Journal of Computational Intelligence Research, 5, 4 (2009), 429–453.

[6]

Xiaoying Bai, Bin Chen, Bo Ma, and Yunzhan Gong. 2011. Design of intelligent agents for collaborative testing of service-based systems. In Proceedings of the 6th International Workshop on Automation of Software Test. 22–28.

Digital Library

[7]

Sebastian Bauersfeld and Tanja EJ Vos. 2014. User interface level testing with TESTAR; what about more sophisticated action specification and selection? In SATToSE. 60–78.

[8]

Joakim Bergdahl, Camilo Gordillo, Konrad Tollmar, and Linus Gisslén. 2020. Augmenting automated game testing with deep reinforcement learning. In 2020 IEEE Conference on Games (CoG). 600–603.

[9]

John-Jules Ch. Meyer. 2007. Agent Technology. Wiley Encyclopedia of Computer Science and Engineering, 1–8.

[10]

Arilo C Dias Neto, Rajesh Subramanyan, Marlon Vieira, and Guilherme H Travassos. 2007. A survey on model-based testing approaches: a systematic review. In Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies.

Digital Library

[11]

Alexander Elyasov, I. S. W. B. Prasetya, and Jurriaan Hage. 2018. Search-Based Test Data Generation for JavaScript Functions that Interact with the DOM. In 29th International Symposium on Software Reliability Engineering (ISSRE).

[12]

Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419.

Digital Library

[13]

Gordon Fraser and Andreas Zeller. 2011. Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering, 38, 2 (2011), 278–292.

Digital Library

[14]

Peter E Hart, Nils J Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4, 2 (1968), 100–107.

[15]

Andreas Herzig, Emiliano Lorini, Laurent Perrussel, and Zhanhao Xiao. 2017. BDI logics for BDI architectures: old problems, new perspectives. KI-Künstliche Intelligenz, 31, 1 (2017), 73–83.

[16]

John Hughes. 2009. Software testing with QuickCheck. In Central European Functional Programming School. 183–223.

[17]

Sidra Iftikhar, Muhammad Zohaib Iqbal, Muhammad Uzair Khan, and Wardah Mahmood. 2015. An automated model based testing approach for platform games. In 2015 ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS). 426–435.

[18]

Phil McMinn. 2004. Search-based software test data generation: a survey. Software testing, Verification and reliability, 14, 2 (2004), 105–156.

[19]

Ian Millington and John Funge. 2019. Artificial intelligence for games, 3rd edition. CRC Press.

[20]

Seyedali Mirjalili. 2019. Genetic algorithm. In Evolutionary algorithms and neural networks. Springer, 43–55.

[21]

Michail Ostrowski and Samir Aroudj. 2013. Automated Regression Testing within Video Game Development. GSTF Journal on Computing, 3, 2 (2013).

[22]

Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In 29th ACM SIGSOFT International Symposium on Software Testing and Analysis.

Digital Library

[23]

Corina S Păsăreanu and Willem Visser. 2009. A survey of new trends in symbolic execution for software testing and analysis. International journal on software tools for technology transfer, 11, 4 (2009), 339.

[24]

Samad Paydar and Mohsen Kahani. 2011. An agent-based framework for automated testing of web-based systems. Journal of Softw. Eng. and Applications, 4, 02 (2011).

[25]

Johannes Pfau, Jan David Smeddinck, and Rainer Malaka. 2017. Automated game testing with icarus: Intelligent completion of adventure riddles via unsupervised solving. In Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play. 153–164.

Digital Library

[26]

ISWB Prasetya, Maurin Voshol, Tom Tanis, Adam Smits, Bram Smit, Jacco van Mourik, Menno Klunder, Frank Hoogmoed, Stijn Hinlopen, and August van Casteren. 2020. Navigation and exploration in 3D-game automated play testing. In Proceedings of the 11th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 3–9.

Digital Library

[27]

Ignatius SWB Prasetya. 2016. Budget-aware random testing with T3: benchmarking at the SBST2016 testing tool contest. In 2016 IEEE/ACM 9th International Workshop on Search-Based Software Testing (SBST). 29–32.

Digital Library

[28]

I. S. W. B. Prasetya, Mehdi Dastani, Rui Prada, Tanja E. J. Vos, Frank Dignum, and Fitsum Kifetew. 2020. Aplib: Tactical Agents for Testing Computer Games. 8th International Workshop on Engineering Multi-Agent Systems (EMAS).

[29]

IS Wishnu B Prasetya. 2013. T3, a combinator-based random testing tool for java: benchmarking. In International Workshop on Future Internet Testing.

[30]

Yu Qi, David Kung, and Eric Wong. 2005. An agent-based testing approach for Web applications. In 29th Int. Comp. Software and Applications Conf. 2.

[31]

Anand S Rao and Michael P Georgeff. 1995. BDI agents: From theory to practice. In ICMAS. 95, 312–319.

[32]

Urko Rueda, Tanja E. J. Vos, and I. S. W. B. Prasetya. 2015. Unit testing tool competition –round three. In Int. Worksh. on Search-Based Software Testing.

[33]

Ronnie ES Santos, Cleyton VC Magalhães, Luiz Fernando Capretz, Jorge S Correia-Neto, Fabio QB da Silva, and Abdelrahman Saher. 2018. Computer games are serious business and so is their quality: particularities of software testing in game development from the perspective of practitioners. In 12th ACM/IEEE Int. Symposium on Empirical Software Engineering and Measurement. 1–10.

Digital Library

[34]

Charles P Schultz and Robert Denton Bryant. 2016. Game testing: All in one. Stylus Publishing, LLC.

[35]

Tom Tervoort and I. S. W. B. Prasetya. 2017. APSL: A light weight testing tool for protocols with complex messages. In Haifa Verification Conference.

[36]

GJ Tretmans and Hendrik Brinksma. 2003. Torx: Automated model-based testing. In First European Conference on Model-Driven Software Engineering. 31–43.

[37]

J Tuovenen, Mourad Oussalah, and Panos Kostakos. 2019. MAuto: Automatic Mobile Game Testing Tool Using Image-Matching Based Approach. The Computer Games Journal, 8, 3 (2019), 215–239.

[38]

Thi Anh Tuyet Vuong and Shingo Takada. 2018. A reinforcement learning based approach to automated testing of android applications. In 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation.

[39]

Yuan Yao, Lavindra de Silva, and Brian Logan. 2016. Reasoning about the executability of goal-plan trees. In International Workshop on Engineering Multi-Agent Systems. 176–191.

[40]

Imants Zarembo. 2019. Analysis of Artificial Intelligence Applications for Automated Testing of Video Games. In Proceedings of the 12th International Scientific and Practical Conference. Volume II. 170, 174.

[41]

Yan Zheng, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, and Changjie Fan. 2019. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 772–784.

Digital Library

Cited By

Lahiri SKalita PChittora AVankudre VRoy SFilkov VRay BZhou M(2024)Program Synthesis Meets Visual What-Comes-Next PuzzlesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695015(418-429)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695015
Coppola RFulcini TManzi SStrada F(2024)How to Measure Game Testing: a Survey of Coverage MetricsProceedings of the ACM/IEEE 8th International Workshop on Games and Software Engineering10.1145/3643658.3643920(15-19)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643658.3643920
Shirzadeh-hajimahmood SPrasteya IDastani MDignum F(2024)Cooperative Multi-agent Approach for Automated Computer Game TestingEngineering Multi-Agent Systems10.1007/978-3-031-71152-7_2(23-41)Online publication date: 16-Sep-2024
https://doi.org/10.1007/978-3-031-71152-7_2
Show More Cited By

Index Terms

Using an agent-based approach for robust automated testing of computer games
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Contextual software domains
      1. Virtual worlds software
        Interactive games

Recommendations

An agent-based approach to automated game testing: an experience report
A-TEST 2022: Proceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation

Computer games are very challenging to handle for traditional automated testing algorithms. In this paper we will look at intelligent agents as a solution. Agents are suitable for testing games, since they are reactive and able to reason about their ...
An online agent-based search approach in automated computer game testing with model construction
A-TEST 2022: Proceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation

The complexity of computer games is ever increasing. In this setup, guiding an automated test algorithm to find a solution to solve a testing task in a game's huge interaction space is very challenging. Having a model of a system to automatically ...
Cooperative Multi-agent Approach for Automated Computer Game Testing
Engineering Multi-Agent Systems
Abstract
Automated testing of computer games is a challenging problem, especially when lengthy scenarios have to be tested. Automating such a scenario boils down to finding the right sequence of interactions given an abstract description of the scenario. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

A-TEST 2021: Proceedings of the 12th International Workshop on Automating TEST Case Design, Selection, and Evaluation

August 2021

40 pages

ISBN:9781450386234

DOI:10.1145/3472672

General Chairs:
Sinem Getir Yaman
Humboldt University of Berlin, Germany / Ege University, Turkey
,
Rui Prada
INESC-ID, Portugal / University of Lisbon, Portugal
,
Program Chairs:
Fitsum Meshesha Kifetew
Fondazione Bruno Kessler, Italy
,
Nicolás Cardozo
Universidad de los Andes, Colombia

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Commission

Conference

ESEC/FSE '21

Sponsor:

SIGPLAN

ESEC/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 23 - 24, 2021

Athens, Greece

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
287
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)10

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lahiri SKalita PChittora AVankudre VRoy SFilkov VRay BZhou M(2024)Program Synthesis Meets Visual What-Comes-Next PuzzlesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695015(418-429)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695015
Coppola RFulcini TManzi SStrada F(2024)How to Measure Game Testing: a Survey of Coverage MetricsProceedings of the ACM/IEEE 8th International Workshop on Games and Software Engineering10.1145/3643658.3643920(15-19)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643658.3643920
Shirzadeh-hajimahmood SPrasteya IDastani MDignum F(2024)Cooperative Multi-agent Approach for Automated Computer Game TestingEngineering Multi-Agent Systems10.1007/978-3-031-71152-7_2(23-41)Online publication date: 16-Sep-2024
https://doi.org/10.1007/978-3-031-71152-7_2
Azizi EZaman L(2023)Automatic Bug Detection in Games using LSTM Networks2023 IEEE Conference on Games (CoG)10.1109/CoG57401.2023.10333253(1-4)Online publication date: 21-Aug-2023
https://doi.org/10.1109/CoG57401.2023.10333253
Pastor Ricós F(2022)Scriptless Testing for Extended Reality SystemsResearch Challenges in Information Science10.1007/978-3-031-05760-1_56(786-794)Online publication date: 14-May-2022
https://doi.org/10.1007/978-3-031-05760-1_56

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents