research-article

Expert review method in game evaluations: comparison of two playability heuristic sets

Authors:

Hannu Korhonen,

Janne Paavilainen,

Hannamari SaarenpääAuthors Info & Claims

MindTrek '09: Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era

Pages 74 - 81

https://doi.org/10.1145/1621841.1621856

Published: 30 September 2009 Publication History

Abstract

The expert review method is not yet widely adopted in game evaluations, although it has been used successfully in productivity software evaluations for years. In order to use the method effectively, there need to be playability heuristics that take into account the characteristics of the videogames. There are a few playability heuristic sets available, but they have several differences, and they have not been compared to discover their strengths and weaknesses in game evaluations. In this paper, we report on a first study comparing two playability heuristic sets when evaluating the playability of a videogame. The results indicate that the heuristics can assist the evaluators in evaluating both the user interface and the gameplay aspects of the game. However, playability heuristics need to be developed further before they can be utilized by the practitioners. Especially, the clarity and comprehensibility of the heuristics need to be improved, and the optimal number of heuristics is still open.

References

[1]

Adams E., Rollings A. 2007. Game Design and Development: Fundamentals of Game Design, Prentice Hall.

[2]

Desurvire, H., Caplan, M., Toth, J. A. 2004. Using heuristics to evaluate the playability of games. In proceedings of ACM SIGCHI '04 extended abstracts, 1509--1512. DOI=http://doi.acm.org/10.1145/985921.986102

Digital Library

[3]

Egenfield-Nielsen, S., Smith, J. H. and Tosca, S. P. 2008. Understanding Video Games: The Essential Introduction. Routledge.

Digital Library

[4]

Ermi, L., Mäyrä, F. 2005. Fundamental Components of the Gameplay Experience: Analysing Immersion. In proceedings of DIGRA 2005 Conference: Changing Views - Worlds in Play.

[5]

Fabricatore, C., Nussbaum, M., Rosas, R. 2002. Playability in Action Videogames: A Qualitative Design Model. Human-Computer Interaction 17, 311--368. DOI=http://dx.doi.org/10.1207/S15327051HCI1704_1

Digital Library

[6]

Falstein, N. and Barwood, H. The 400 Project, http://theinspiracy.com/400_project.htm.

[7]

Federoff, M. A. 2002. Heuristics and Usability Guidelines for the Creation and Evaluation of Fun in Video Games. Department of Telecommunications, Master of Science. Indiana University, Indiana.

[8]

Galagher K., Foster D., Parsons J. 2001. The Medium Is Not the Message: Advertising Effectiveness and Content Evaluation in Print and on the Web, Journal of Advertising Research 41(4), 57--70.

[9]

Gray, W. D., Salzman, M. C. 1998. Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods. Human-Computer Interaction 13 (3), 203--261. DOI=http://dx.doi.org/10.1207/s15327051hci1303_2

Digital Library

[10]

Jacobsen N. E., Hertzum M., John B. E. 1998. The evaluator Effect in Usability Tests, In Proceedings of ACM SIGCHI, 255--256. DOI=http://doi.acm.org/10.1145/286498.286737

Digital Library

[11]

Johnson D., Wiles J. 2003. Effective Affective User Interface Design in Games, Ergonomics 46 (13/14), 1332--1345. DOI=http://dx.doi.org/10.1080/00140130310001610865

[12]

Järvinen, A., Heliö, S., Mäyrä, F. 2002. Communication and Community in Digital Entertainment Services. University of Tampere.

[13]

Koivisto E. M. I, Korhonen H. 2006. Mobile Game Playability Heuristics, www.forum.nokia.com.

[14]

Korhonen, H., Koivisto, E. M. I. 2006. Playability Heuristics for Mobile Games. In proceedings of MobileHCI'06, 9--16. DOI=http://doi.acm.org/10.1145/1152215.1152218

Digital Library

[15]

Ling, C., Salvendy, G. 2005. Extension of Heuristic Evaluation Method: a Review and Reappraisal. Ergonomia. An International Journal of Ergonomics and Human Factors 27 (3). 179--197.

[16]

Malone, T. W. 1982. Heuristics for designing enjoyable user interfaces: Lessons from computer games. In proceedings of ACM SIGCHI, 63--68. DOI=http://doi.acm.org/10.1145/800049.801756

Digital Library

[17]

Miller, G. A. 1956. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review 63. 81--97.

[18]

Molich, R., Dumas, J. S. 2008. Comparative Usability Evaluation (CUE-4). Behaviour&Information Technology 27, 263--281.

Digital Library

[19]

Nielsen J. 1993. Usability Engineering, Academic Press.

Digital Library

[20]

Nielsen, J., Molich, R. 1990. Heuristic evaluation of user interfaces. In proceedings of ACM SIGCHI, 249--256. DOI=http://doi.acm.org/10.1145/97243.97281

Digital Library

[21]

Pagulayan, R. J., Keeker, K., Wixon, D., Romero, R. L. and Fuller, T. 2008. User-centered Design in Games. In Jacko, J. Sears A., (Eds.). Handbook for Human-Computer Interaction in Interactive Systems, Second Edition, Lawrence Erlbaum Associates, Inc., 741--759.

[22]

Pagulayan R., Steury K. 2004. Beyond Usability in Games, Interactions (11), 5, 70--71. DOI=http://doi.acm.org/10.1145/1015530.1015566

Digital Library

[23]

Pinelle, S., Wong, N., Stach, T. 2008. Heuristic evaluation for games: usability principles for video game design. In proceedings of ACM SIGCHI, 1453--1462. DOI=http://doi.acm.org/10.1145/1357054.1357282

Digital Library

[24]

Snow, B. 2007. Game Usability 101. BusinessWeek, http://www.businessweek.com/innovate/content/oct2007/id20071012_041625.htm.

Cited By

Zacharia KGotsis M(2024) Design and evaluation of Enthralled : A higher education conversation game based on Euripides’ Bacchae Arts and Humanities in Higher Education10.1177/1474022224127616524:1(23-59)Online publication date: 7-Sep-2024
https://doi.org/10.1177/14740222241276165
Froehlich JHorn B(2024)Physically Prototyping Physics for Play: Possibilities and Practicality2024 IEEE Conference on Games (CoG)10.1109/CoG60054.2024.10645608(1-8)Online publication date: 5-Aug-2024
https://doi.org/10.1109/CoG60054.2024.10645608
Iyer RNdulue CMeier SOrji R(2024)Smartphone Games Heuristics (SmGH) – Towards a Standard Set of Platform-Centric Heuristic for Smartphone Games EvaluationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2356911(1-22)Online publication date: 11-Jun-2024
https://doi.org/10.1080/10447318.2024.2356911
Show More Cited By

Index Terms

Expert review method in game evaluations: comparison of two playability heuristic sets
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing design and evaluation methods

Recommendations

Comparison of playtesting and expert review methods in mobile game evaluation
Fun and Games '10: Proceedings of the 3rd International Conference on Fun and Games

Selecting an evaluation method for product evaluations depends on many issues, such as the development stage of the product, time schedule, resources, and money that can be invested on the evaluation. The user testing and expert review methods are ...
Playability heuristics for mobile games
MobileHCI '06: Proceedings of the 8th conference on Human-computer interaction with mobile devices and services

Expert evaluation is a widely used method for evaluating the usability of software products. When evaluating games, traditional usability heuristics lack comprehension and cannot be directly applied. In this paper, we introduce playability heuristics ...
The explanatory power of playability heuristics
ACE '11: Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology

Research of playability heuristics to be used with usability inspection methods in videogame evaluations has been active in recent years, and there are multiple playability heuristic sets available. However, they differ quite a lot from each other, and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MindTrek '09: Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era

September 2009

254 pages

ISBN:9781605586335

DOI:10.1145/1621841

Conference Chairs:
Artur Lugmayr
Tampere University of Technology, Finland
,
Heljä Franssila
University of Tampere, Finland
,
Olli Sotamaa
University of Tampere, Finland
,
Pertti Näränen
Tampere University of Applied Sciences, Finland
,
Jukka Vanhala
Tampere University of Technology, Finland
,
Editors:
Artur Lugmayr,
Heljä Franssila,
Olli Sotamaa,
Pertti Näränen,
Jukka Vanhala

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 September 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MindTrek '09

Sponsor:

MindTrek '09: Academic MindTrek 2009

September 30 - October 2, 2009

Tampere, Finland

Acceptance Rates

Overall Acceptance Rate 110 of 207 submissions, 53%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
1,200
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)6

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zacharia KGotsis M(2024) Design and evaluation of Enthralled : A higher education conversation game based on Euripides’ Bacchae Arts and Humanities in Higher Education10.1177/1474022224127616524:1(23-59)Online publication date: 7-Sep-2024
https://doi.org/10.1177/14740222241276165
Froehlich JHorn B(2024)Physically Prototyping Physics for Play: Possibilities and Practicality2024 IEEE Conference on Games (CoG)10.1109/CoG60054.2024.10645608(1-8)Online publication date: 5-Aug-2024
https://doi.org/10.1109/CoG60054.2024.10645608
Iyer RNdulue CMeier SOrji R(2024)Smartphone Games Heuristics (SmGH) – Towards a Standard Set of Platform-Centric Heuristic for Smartphone Games EvaluationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2356911(1-22)Online publication date: 11-Jun-2024
https://doi.org/10.1080/10447318.2024.2356911
Kumar B(2023)Towards design science methodology for developing mobile learning applicationsInteractive Learning Environments10.1080/10494820.2023.2191258(1-15)Online publication date: 28-Mar-2023
https://doi.org/10.1080/10494820.2023.2191258
Chand SKumar BGoundar MNarayan A(2022)Extended UTAUT Model for Mobile Learning Adoption StudiesInternational Journal of Mobile and Blended Learning10.4018/IJMBL.31257014:1(1-20)Online publication date: 1-Jan-2022
https://doi.org/10.4018/IJMBL.312570
Mokhtar EChe Omar AAziz N(2022)Proposed Design Principles of Malay Sign Language (MSL) Mobile Application for Hearing-Impaired Alpha Generation Based on Nielsen and Molich’s Design Guidelines: Validation through Expert Review Method2022 IEEE 12th Symposium on Computer Applications & Industrial Electronics (ISCAIE)10.1109/ISCAIE54458.2022.9794474(233-240)Online publication date: 21-May-2022
https://doi.org/10.1109/ISCAIE54458.2022.9794474
Paliokas IPatenidis AMitsopoulou ETsita CPehlivanides GKaryati ETsafaras SStathopoulos EKokkalas ADiplaris SMeditskos GVrochidis STasiopoulou ERiggas CVotis KKompatsiaris ITzovaras D(2020)A Gamified Augmented Reality Application for Digital Heritage and TourismApplied Sciences10.3390/app1021786810:21(7868)Online publication date: 6-Nov-2020
https://doi.org/10.3390/app10217868
Sudarmilah EMaharani Utami YYuliana I(2020)3D Interactive Multimedia in English Language LearningJournal of Physics: Conference Series10.1088/1742-6596/1569/2/0220041569(022004)Online publication date: 24-Jul-2020
https://doi.org/10.1088/1742-6596/1569/2/022004
Yanez-Gomez RFont JCascado-Caballero DSevillano J(2019)Heuristic usability evaluation on gamesMultimedia Tools and Applications10.1007/s11042-018-6593-178:4(4937-4964)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s11042-018-6593-1
Musarurwa AFlowerday SCilliers L(2019)The bring‐your‐own‐device unintended administrator: A perspective from ZimbabweTHE ELECTRONIC JOURNAL OF INFORMATION SYSTEMS IN DEVELOPING COUNTRIES10.1002/isd2.1207685:4Online publication date: 29-Jan-2019
https://doi.org/10.1002/isd2.12076
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten