research-article

What do game developers test in their products?

Authors:

Jussi Kasurinen,

Kari SmolanderAuthors Info & Claims

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 1, Pages 1 - 10

https://doi.org/10.1145/2652524.2652525

Published: 18 September 2014 Publication History

Abstract

Context: Software projects often have four objectives; to produce the required functionality, in budget and in schedule, with acceptable quality. That statement may be true for most of the ordinary software development projects, but are these objectives enough for game development, where creativity and artistic aspects have a major role? Goal: We analyze how game developing organizations test their products, what are their main test objectives and how they perceive themselves in the software business. Method: We interviewed seven game development teams from different companies and studied how they test their products with grounded theory approach. Results: Our results suggest that game developers focus on soft values such as game content or user experience, instead of more traditional objectives such as reliability or efficiency. Conclusions: Game developers have similar, but not fully comparable to software industry, set of priorities in their software testing and quality assurance approaches.

References

[1]

Kaner, C., Falk, J. and Nguyen, H. Q., 1999. Testing Computer Software, second edition, John Wiley & Sons Inc., New York. ISBN: 0-471-35846-0

Digital Library

[2]

Koomen, T. and Pol, M., 1999. Test Process Improvement: A practical step-by-step guide to structured testing, Addison-Wesley, Great Britain. ISBN: 978-0-201-59624-3

Digital Library

[3]

Blow, J., 2004. Game Development: Harder Than You Think, Queue, vol. 1, nro. 10, ss. 28--37, February 2004.

Digital Library

[4]

Kanode, C. M. and Haddad, H. M., 2009. Software Engineering Challenges in Game Development, in Information Technology: New Generations (ITNG '09). Sixth International Conference on, ss. 260--265.

Digital Library

[5]

Kultima, A., 2010. The Organic Nature of Game Ideation: Game Ideas Arise from Solitude and Mature by Bouncing, ACM FuturePlay, 6.-7.5.2010, Vancouver, Canada.

Digital Library

[6]

McGill, M., 2009. Defining the expectation gap: a comparison of industry needs and existing game development curriculum, In Proceedings of the 4th International Conference on Foundations of Digital Games (FDG '09). ACM, New York, NY, USA, 129--136. DOI=10.1145/1536513.1536542

Digital Library

[7]

Murugesan, S., 1994. Attitude towards testing: a key contributor to software quality, Software Testing, Reliability and Quality Assurance, 1994. Conference Proceedings., First International Conference on, vol., no., pp.111--115, 21-22 Dec 1994

[8]

Myers, G. J., 2004, The Art of Software Testing, 2nd edition, John Wiley & Sons, Inc., Hoboken, New Jersey, USA. ISBN: 0-471-46912-2

Digital Library

[9]

Slaughter, S. A., Harter, D. E. and Krishnan, M. S., 1998. Evaluating the cost of software quality, Communications of the ACM, Vol. 41, Issue 8.

Digital Library

[10]

ISO/IEC/IEEE 12207:2008. Systems and software engineering, Software life cycle processes. Second edition, dated 01.02.2008.

[11]

Kasurinen, J., Taipale, O. and Smolander, K., 2011. How Test Organization Adopt New Testing Practices and Methods?, Proceedings of the Testing: Academic & Industrial Conference: Practice and Research Techniques 2011 (TAIC PART) co-located with 4th IEEE International Conference on Software Testing, Verification and Validation (ICST), 25.3.2011, Berlin, Germany.

Digital Library

[12]

Kasurinen, J., Taipale, O., Vanhanen, J., and Smolander, K., 2012. Exploring the Perceived End-Product Quality in Software-Developing Organizations. International Journal of Information System Modeling and Design (IJISMD), 3(2), 1--32.

Digital Library

[13]

Kasurinen, J., Laine, R. and Smolander, K., 2013. How Applicable Is ISO/IEC 29110 in Game Software Development?, Lecture Notes in Computer Science, Springer Berlin Heidelberg, Vol. 7983, pp. 5--19.

[14]

J. Kasurinen, J-P Strandén and K. Smolander, "What do game developers expect from development and design tools?", In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE '13). ACM, New York, NY, USA, 2013. pp. 36--41. DOI=10.1145/2460999.2461004

Digital Library

[15]

Huang, L. and Boehm, B., 2006. How Much Software Quality Investment Is Enough: A Value-Based Approach, IEEE Software, Vol. 23(5), pp. 88--95.

Digital Library

[16]

Peltoniemi, M., 2008. Life-cycle of the Games Industry The Specificities of Creative Industries, Proceedings of the Mindtrek'08, 7.-9.10.2008, Tampere, Finland.

Digital Library

[17]

Kultima, A. and Alha, K., 2010. Hopefully Everything I'm Doing Has to Do with Innovation: Games industry professionals on innovation in 2009, Proc. 2nd International IEEE Consumer Electronics Society's Games Innovation Conference, Hong Kong, China, 2010.

[18]

Callele, D., Neufeld, E. and Schneider, K., 2005. Requirements Engineering and the Creative Process in the Video Game Industry, In Proceedings of the 13th IEEE International Conference on Requirements Engineering (RE '05). IEEE Computer Society, Washington, DC, USA, 240--252. DOI=10.1109/RE.2005.58

Digital Library

[19]

Glaser, B. G., 2002. Constuctivist Grounded Theory?, Forum: Qualitative Social Research (FQS), Vol 3(3).

[20]

Glaser, B. and Strauss, A. L., 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine.

[21]

Strauss, A. and Corbin J., 1990. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. SAGE Publications, Newbury Park, CA, USA.

[22]

Niekerk, van J. C. and Roode, J. D., 2009. Glaserian and Straussian grounded theory: similar or completely different?, Proc. of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, Vanderbijlpark, South Africa.

[23]

Eisenhardt, K. M., 1989. Building Theories from Case Study Research, Academy of Management Review, vol. 14, no. 4, pp. 532--550.

[24]

Paré, G. and Elam, J. J., 1997. Using Case Study Research to Build Theories of IT Implementation, The IFIP TC8 WG International Conference on Information Systems and Qualitative Research, Philadelphia, USA. Chapman & Hall.

Digital Library

[25]

Klein, H. K. and Myers, M. D., 1999. A set of principles for conducting and evaluating interpretive field studies in information systems, MIS Quarterly, vol. 23, pp. 67--94.

Digital Library

[26]

European Comission (EC), 2003. The new SME Definition User guide and model declaration, Enterprise and Industry Publications, European Commission, 2003.

[27]

Seaman, C. B., 1999. Qualitative methods in empirical studies of software engineering, IEEE Transactions on Software Engineering, vol. 25, pp. 557--572.

Digital Library

[28]

Petrillo, F., Pimenta, M., Trindade, F. and Dietrich, C., 2008. Houston, we have a problem...: A survey of Actual Problems in Computer Games Development, Proceedings of SAC'08, 16.-20.3.2008, Fortaleza, Brazil.

Digital Library

[29]

Kasurinen, J., Taipale, O. and Smolander, K., 2010. Test Case Selection and Prioritization: Risk-based or Design-based?, Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 16.-17.9.2010, Bolzano-Bozen, Italy.

Digital Library

[30]

Petschenik, N. H., 1985. Practical Priorities in System Testing, IEEE Software, Vol. 2(5), pp. 18--23.

Digital Library

[31]

Redmill, F., 2004. Exploring risk-based testing and its implications, Software Testing, Verification and Reliability, Vol. 14(1), pp. 3--15.

Digital Library

[32]

Golafshani, N., 2003. Understanding Reliability and Validity in Qualitative Research, The Qualitative Report, Vol 8(4), December 2003, pages 596--607.

[33]

Robson, C., 2002. Real World Research, Second Edition. Blackwell Publishing.

[34]

Whittemore, R., Chase, S. K. and Mandle, C. L., 2001. Validity in Qualitative Research, Qual Health Res, July 2001, 11: 522--537.

Cited By

Jones SRochat YRonga P(2024) Concevoir un témoignage vidéoludique en temps de pandémie : le développement de Quatre apparts et un confinement Kinephanos10.7202/1113435ar9:1(145-168)Online publication date: 11-Sep-2024
https://doi.org/10.7202/1113435ar
Duarte YMandelli HDurelli VNardi PEndo A(2024)Exploratory testing for platform video games: strategies and lessons learnedJournal on Interactive Systems10.5753/jis.2024.415615:1(657-669)Online publication date: 5-Jul-2024
https://doi.org/10.5753/jis.2024.4156
Brewer JRomine M(2024)Breaking Barriers in Mobile Game DevelopmentProceedings of the ACM on Human-Computer Interaction10.1145/36770618:CHI PLAY(1-20)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677061
Show More Cited By

Index Terms

What do game developers test in their products?
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. System management
        Quality assurance

Recommendations

A study on agility and testing processes in software organizations
ISSTA '10: Proceedings of the 19th international symposium on Software testing and analysis

In this paper, we studied the differences in testing activities between software organizations which apply agile development methods and organizations which take the traditional plan-driven approach. Our focus was on the concepts which allow the ...
Realizing quality improvement through test driven development: results and experiences of four industrial teams

Test-driven development (TDD) is a software development practice that has been used sporadically for decades. With this practice, a software engineer cycles minute-by-minute between writing failing unit tests and writing implementation code to pass ...
Software test maturity assessment and test process improvement

ContextSoftware testing practices and processes in many companies are far from being mature and are usually conducted in ad-hoc fashions. Such immature practices lead to various negative outcomes, e.g., ineffectiveness of testing practices in detecting ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

September 2014

461 pages

ISBN:9781450327749

DOI:10.1145/2652524

General Chair:
Maurizio Morisio
Politecnico di Torino, Italy
,
Program Chairs:
Tore Dybå
Sintef, Norway
,
Marco Torchiano
Politecnico di Torino, Italy

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Union Regional Development Fund

Conference

ESEM '14

Sponsor:

SIGSOFT

ESEM '14: 2014 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 18 - 19, 2014

Torino, Italy

Acceptance Rates

ESEM '14 Paper Acceptance Rate 23 of 123 submissions, 19%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
578
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)6

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jones SRochat YRonga P(2024) Concevoir un témoignage vidéoludique en temps de pandémie : le développement de Quatre apparts et un confinement Kinephanos10.7202/1113435ar9:1(145-168)Online publication date: 11-Sep-2024
https://doi.org/10.7202/1113435ar
Duarte YMandelli HDurelli VNardi PEndo A(2024)Exploratory testing for platform video games: strategies and lessons learnedJournal on Interactive Systems10.5753/jis.2024.415615:1(657-669)Online publication date: 5-Jul-2024
https://doi.org/10.5753/jis.2024.4156
Brewer JRomine M(2024)Breaking Barriers in Mobile Game DevelopmentProceedings of the ACM on Human-Computer Interaction10.1145/36770618:CHI PLAY(1-20)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677061
Chueca JVerón JFont JPérez FCetina C(2024)The consolidation of game software engineering: A systematic literature review of software engineering for industry-scale computer gamesInformation and Software Technology10.1016/j.infsof.2023.107330165(107330)Online publication date: Jan-2024
https://doi.org/10.1016/j.infsof.2023.107330
Wrześniewska ASkublewska-Paszkowska M(2023)Video game performance analysis on selected operating systemsJournal of Computer Sciences Institute10.35784/jcsi.377229(317-324)Online publication date: 29-Dec-2023
https://doi.org/10.35784/jcsi.3772
Duarte YDurelli VNardi PEndo A(2023)Exploratory testing strategies for video games:an experience reportProceedings of the 22nd Brazilian Symposium on Games and Digital Entertainment10.1145/3631085.3631227(46-55)Online publication date: 6-Nov-2023
https://dl.acm.org/doi/10.1145/3631085.3631227
Cho JAli K(2023)Exploring Quality Assurance Practices and Tools for Indie Games2023 IEEE/ACM 7th International Workshop on Games and Software Engineering (GAS)10.1109/GAS59301.2023.00010(16-24)Online publication date: May-2023
https://doi.org/10.1109/GAS59301.2023.00010
Politowski CPetrillo FGueheneuc Y(2021)A Survey of Video Game Testing2021 IEEE/ACM International Conference on Automation of Software Test (AST)10.1109/AST52587.2021.00018(90-99)Online publication date: May-2021
https://doi.org/10.1109/AST52587.2021.00018
Borg MGarousi VMahmoud AOlsson TStalberg O(2020)Video Game Development in a Rush: A Survey of the Global Game Jam ParticipantsIEEE Transactions on Games10.1109/TG.2019.291024812:3(246-259)Online publication date: Sep-2020
https://doi.org/10.1109/TG.2019.2910248
Savchenko DKasurinen JTaipale O(2019)Smart Tools in Software Engineering: A Systematic Mapping Study2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)10.23919/MIPRO.2019.8756975(1509-1513)Online publication date: May-2019
https://doi.org/10.23919/MIPRO.2019.8756975
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents