Article

Usability inspection methods after 15 years of research and practice

Authors:

Tasha Hollingsed,

David G. NovickAuthors Info & Claims

SIGDOC '07: Proceedings of the 25th annual ACM international conference on Design of communication

Pages 249 - 255

https://doi.org/10.1145/1297144.1297200

Published: 22 October 2007 Publication History

Abstract

Usability inspection methods, such as heuristic evaluation, the cognitive walkthrough, formal usability inspections, and the pluralistic usability walkthrough, were introduced fifteen years ago. Since then, these methods, analyses of their comparative effectiveness, and their use have evolved in different ways. In this paper, we track the fortunes of the methods and analyses, looking at which led to use and to further research, and which led to relative methodological dead ends. Heuristic evaluation and the cognitive walkthrough appear to be the most actively used and researched techniques. The pluralistic walkthrough remains a recognized technique, although not the subject of significant further study. Formal usability inspections appear to have been incorporated into other techniques or largely abandoned in practice. We conclude with lessons for practitioners and suggestions for future research.

References

[1]

Almeida, L., Amdal, I., Beires, N., Boualem, M., Boves, L., den Os, E., Filoche, P., Gomes, R., Knudsen, J., Kvale, K., Rugelbak, J., Tallec, C., Warakagoda, N. (2002). The MUST guide to Paris: Implementation and expert evaluation of a multimodal tourist guide to Paris, ISCA Tutorial and Research Workshop (IDS'2002), Kloster Irsee, Germany, June, 2002, available at http://citeseer.ist.psu.edu/cache/papers/cs/26307/http:zSzzSzlands.let.kun. nlzSzliteraturezSzboves.2002.1.pdf/almeida02must.pdf, accessed May 26, 2007

[2]

Andreasen, M. S., Nielsen, H., Schrøder, S., and Stage, J. 2007. What happened to remote usability testing?: An empirical study of three methods, Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 2007), San Jose, CA, April 28-May 3, 2007, 1405--1414.

Digital Library

[3]

Bias, R. G. 1994. The pluralistic usability walkthrough: coordinated empathies. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 63--76.

Digital Library

[4]

Blackmon, M., Polson, P., Kitajima, M., and Lewis, C. (2002). Cognitive walkthrough for the Web, Proceedings of the Conference on Human Factors in Computing Systems (CHI 2002), Minneapolis, MN, April 20-25, 2002, 463--470.

Digital Library

[5]

Blatt, L. A. and Knutson, J. F. 1994. Interface design guidance systems. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 351--384.

Digital Library

[6]

Bradford, J. (1994). Evaluating high-level design: synergistic use of inspection and usability methods for evaluating early software designs. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 235--253.

Digital Library

[7]

Brodie, C., Karat, C., and Karat, J. (2006). An empirical study of natural language parsing of privacy policy rules using the SPARCLE policy workbench, Proceedings of the Second Symposium on Usable Privacy and Security (SOUPS '06), Pittsburgh, PA, July 12-14, 2006, 8-19.

Digital Library

[8]

Brooks, P. (1994). Adding value to usability testing. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 255--271.

Digital Library

[9]

Desurvire, H. (1994). Faster, cheaper!! Are usability inspection methods as effective as empirical testing? In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 173--202.

Digital Library

[10]

Desurvire, H., Kondziela, J., and Atwood, M. (1992). What is gained and lost when using evaluation methods other than empirical testing, Proceedings of the Conference on Human Factors in Computing Systems (CHI 92), Monterey, California, May 3-7, 1992, 125--126.

Digital Library

[11]

Dunbar, A. (1997). CADNET design team asks customers: How does it work on paper? The Culpepper eBulletin Newsletter 157, available at http://www.culpepper.com/eBulletin/1997/157cs.asp, accessed May 26, 2007.

[12]

Ebling, M. and John, B. (2000). On the contributions of different empirical data in usability testing, in Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, New York, NY, August 17-19, 2000, 289--286.

Digital Library

[13]

Franzke, M. (1995). Turning research into practice: characteristics of display-based interaction, in Proceedings of the Conference on Human Factors in Computing Systems (CHI 95), Denver, Colorado, May 7-11, 1995, 421--428.

Digital Library

[14]

Galliers, J., Sutcliffe, A., and Minocha, S. (1999). An impact analysis method for safety-critical user interface design, ACM Transactions on Computer-Human Interaction 6(4), 341--369.

Digital Library

[15]

Grasland, I., Kerbiriou, P., Janse, M., Issarny, V., Sacchetti, D., Talamona, A., Leal, A. (2004). User experience report of the three demonstrators and the external application of the Ozone framework, Ozone Project, available at http://www.hitech-projects.com/euprojects/ozone/public_docs/del_oct04/ozone-tho-31aug04-d15d-pk.pdf, accessed May 27, 2007.

[16]

Gunn, C. (1995). An example of formal usability inspections in practice at Hewlett-Packard company, Proceeding of the Conference on Human Factors in Computing System (CHI 95), Denver, Colorado, May 7-11, 1995, 103--104.

Digital Library

[17]

Hovater, J., Krot, M., Kiskis, D. L., Holland, H., & Altman, M. (2002). Usability testing of the Virtual Data Center, Workshop on Usability of Digital Libraries, Second ACM-IEEE Joint Conference on Digital Libraries, Portland, OR, July 14-18, 2002, available at http://www.uclic.ucl.ac.uk/ annb/DLUsability/Hovater7.pdf, accessed May 26, 2007.

[18]

Jeffries, R. (1994). Usability problem reports: helping evaluators communicate effectively with developers. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 273--294.

Digital Library

[19]

Jeffries, R., Miller, J., Wharton, C., and Uyeda, K. (1991). User interface evaluation in the real world: a comparison of four techniques, Proceeding of the Conference on Human Factors in Computing System (CHI 91), New Orleans, LA, April 27-May 2, 1991, 119--124.

Digital Library

[20]

Jeffries, R., and Desurvire, H. (1992). Usability testing vs. heuristic evaluation: was there a contest? ACM SIGCHI Bulletin 24(4), 39--41.

Digital Library

[21]

John B., and Packer, H (1995). Learning and using the cognitive walkthrough method: A case study approach, Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 95), Denver, Colorado, May 7-11, 1995, 429--436.

Digital Library

[22]

Kahn, M., and Prail, A. (1994). Formal usability inspections, in Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 141--171.

Digital Library

[23]

Karat, C.-M. 1994. A comparison of user interface evaluation methods, in Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 203--233.

Digital Library

[24]

Karat, C.-M., Campbell, R. and Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation, Proceedings of the Conference on Human Factors in Computing Systems (CHI 92), Monterey, CA, May 3-7, 1992, 397--404.

Digital Library

[25]

Lewis, C., and Polson, P. (1991). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces (tutorial), Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 91), April 27-May 2, 1991, New Orleans, LA.

[26]

Mack, R. and Montaniz, F. (1994). Observing, predicting, and analyzing usability problems. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 295--339.

Digital Library

[27]

Mack, R., and Nielsen, J. (1993). Usability inspection methods: Report on a workshop held at CHI'92, Monterey, CA, May 3-4, 1992, ACM SIGCHI Bulletin 25(1) 28--33.

Digital Library

[28]

Molich, R., and Nielsen, J. (1990). Improving a human-computer dialogue, Communications of the ACM 33(3), 338--348.

Digital Library

[29]

Muller, M., Dayton, T., and Root, R. (1993). Comparing studies that compare usability assessment methods: an unsuccessful search for stable criteria, INTERACT '93 and CHI '93 Conference Companion on Human Factors in Computing Systems, Amsterdam, April 24-29, 1993, 185--186.

Digital Library

[30]

Naseem, A. (2005). Dynamics of user needs analysis in redesigning an open learning website: A case from Pakistan, Proceedings of ASCLITE 2005, Brisbane, Australia, December 4-7, 2005, 503--507.

[31]

Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces, Proceedings of the Conference on Human Factors in Computing Systems (CHI 90), Seattle, WA, April 1-5, 1990, 249--256.

Digital Library

[32]

Nielsen, J. (1992). Finding usability problems through heuristic evaluation, Proceedings of the Conference on Human Factors in Computing System (CHI 92), Monterey, CA, May 3-7, 1992, 373--380.

Digital Library

[33]

Nielsen, J. and Mack, R. (eds.) (1994), Usability inspection methods, John Wiley & Sons, Inc., New York.

Digital Library

[34]

Nielsen, J. (2005). Ten best intranets of 2005, Jakob Nielsen's Alertbox, February 28, 2005, available at http: //www.useit.com/alertbox/20050228.html, accessed May 26, 2007.

[35]

Nielsen, J., and Phillips, V, (1993). Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared, Proceedings of the Conference on Human Factors in Computing System (CHI 93), Amsterdam, April 24-29, 1993, 214--221.

Digital Library

[36]

Norman, Donald A. (1988). The design of everyday things. Doubleday, New York.

[37]

Novick, D. and Chater, M. (1999). Evaluating the design of human-machine cooperation: The cognitive walkthrough for operating procedures, Proceedings of the Conference on Cognitive Science Approaches to Process Control (CSAPC 99), Villeneuve d'Ascq, FR, September 21-24, 1999.

[38]

Novick, D. (2000). Testing documentation with "low-tech" simulation, Proceedings of IPCC/SIGDOC 2000, Cambridge, MA, September 24-27, 2000, 55--68.

Digital Library

[39]

Perceptive Sciences (2006). Project briefs, available at http: //www.perceptivesciences.com/clients/project_briefs.php, accessed May 27, 2007.

[40]

Rieman, J. (1996). A field study of exploratory learning strategies, ACM Transactions on Computer-Human Interaction 3(3), 189--218.

Digital Library

[41]

Rieman, J., Franzke, M., and Redmiles, D. (1995). Usability evaluation with the cognitive walkthrough, Proceedings of the Conference on Human Factors in Computing Systems (CHI 95), Denver, Colorado, May 7-11, 1995, 387--388.

Digital Library

[42]

Rogers, T., Jagodzinski, A., Phillips, M., and Turley, S. (1997). Interactive drama: A tool for humanistic learning. Proceedings of Computer Aided Learning (CAL) 97, Exeter, UK, March 23-26, 1997, 249--257.

[43]

Sánchez, J., Salinas, A., and Sáenz, M. (2006). Mobile game-based science learning, Proceedings of the Conference on Distance Learning and the Internet (APRU DLI 2006), Tokyo, November 10-8, 2006, 18--30.

[44]

Spagnolli, A., Gamberini, L., Cottone, P., and Mantovani, G. (2004). Ergonomics of virtual environments for clinical use. In Riva, G., Botella, C., Légeron, P., and Optale G., (eds.), Cybertherapy: Internet and Virtual Reality as Assessment and Rehabilitation Tools for Clinical Psychology and Neuroscience, IOS Press, Amsterdam, 217-230.

[45]

Sawyer, P., Flanders, A., and Wixon, D. (1996). Making a difference - the impact of inspections, Proceedings of the Conference on Human Factors in Computing System (CHI 96), Vancouver, BC April 13-18, 1996, 376--382.

Digital Library

[46]

Spencer, R. (2000). The streamlined cognitive walkthrough method, Proceedings of the Conference on Human Factors in Computing System (CHI 2000), The Hague, The Netherlands, April 1-6, 2000, 353--359.

Digital Library

[47]

Tang, Z., Johnson, T., Tindall, R., and Zhang, J. (2006) Applying heuristic evaluation to improve the usability of a telemedicine system, Telemedecine Journal and E-Health 12(1), 24--34.

[48]

Tognazzini, B. (1992). Tog on interface. Addison-Wesley Publishing Company, Reading, MA.

Digital Library

[49]

Usability Professionals Association (undated). Methods: Pluralistic usability walkthrough, available at http://www.usabilitybok.org/methods/p2049, accessed May 26, 2007.

[50]

User-Centered Web Effective Business Solutions (2007). Heuristic usability evaluation, available at http://www.u-cwebs.com/usability/web-site-usability.htm, accessed May 27, 2007.

[51]

Wharton, C., Bradford, J., Jeffries, R., and Franzke, M. (1992). Applying cognitive walkthroughs to more complex user interfaces: Experiences, issues, and recommendations, Proceedings of the Conference on Human Factors in Computing System (CHI 92), Monterey, CA, May 3-7, 1992, 381--388.

Digital Library

[52]

Wharton, C. and Lewis, C. 1994. The role of psychological theory in usability inspection methods. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 341--350

Digital Library

[53]

Wharton, C., Rieman, J., Lewis, C., and Polson, P. (1994). The Cognitive Walkthrough Method: A Practitioner's Guide. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 1994, 105--140.

Digital Library

[54]

Wixon, D., Jones, S., Tse, L., and Casaday, G. (1994). Inspections and Design Reviews: Framework, History and Reflection. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 1994, 77--103.

Digital Library

[55]

Zhang, Z., Basili, V., Shneiderman, B. (1998). An empirical study of perspective-based usability inspection, Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, Santa Monica, CA, October 5-9, 1998, 1346--1350.

Cited By

Svarre TSkov M(2024)The Online presence of the Danish public sector from 2010 to 2022: Generating an archived web corpusExploring the Archived Web during a Highly Transformative Age10.36253/979-12-215-0413-2.17Online publication date: 2024
https://doi.org/10.36253/979-12-215-0413-2.17
Pascual Almenara ASayago Barrantes S(2024)Towards Designing a Set of Usability and Accessibility Heuristics Focused on Cognitive Diversity: An Exploratory Case Study with Generative Artificial IntelligenceInformation10.3390/info1512076915:12(769)Online publication date: 2-Dec-2024
https://doi.org/10.3390/info15120769
Omar KFakhouri HZraqou JMarx Gómez J(2024)Usability Heuristics for MetaverseComputers10.3390/computers1309022213:9(222)Online publication date: 6-Sep-2024
https://doi.org/10.3390/computers13090222
Show More Cited By

Index Terms

Usability inspection methods after 15 years of research and practice
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Usability inspection methods: report on a workshop held at CHI'92, Monterey, CA, May 3–4, 1992

Usability inspection methods, based on informed intuition s about interface design quality, hold promise of providing faster, more cost-effective ways to generate usability evaluations, compared to empirical user evaluation methods . Examples of ...
Metaphors of human thinking for usability inspection and design

Usability inspection techniques are widely used, but few focus on users' thinking and many are appropriate only for particular devices and use contexts. We present a new technique (MOT) that guides inspection by metaphors of human thinking. The metaphors ...
Design and Evaluation of Intelligent Menu Interface through Cognitive Walkthrough Procedure and Automated Logging for Management Information System
Computer Supported Cooperative Work in Design IV

One of the critcal issues in Managment Information System (MIS) is the information overload from complicated hierachical menu. Novice users easily get lost when an important and frequently used menu item is rooted and hidden in a deeper level of a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGDOC '07: Proceedings of the 25th annual ACM international conference on Design of communication

October 2007

286 pages

ISBN:9781595935885

DOI:10.1145/1297144

General Chair:
David Novick
The University of Texas at El Paso, TX
,
Program Chair:
Clay Spinuzzi
The University of Texas at Austin, TX

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGDOC07

Sponsor:

SIGDOC07: The 25th ACM International Conference on Design of Communication

October 22 - 24, 2007

Texas, El Paso, USA

Acceptance Rates

Overall Acceptance Rate 355 of 582 submissions, 61%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

100
Total Citations
View Citations
6,734
Total Downloads

Downloads (Last 12 months)139
Downloads (Last 6 weeks)9

Reflects downloads up to 09 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Svarre TSkov M(2024)The Online presence of the Danish public sector from 2010 to 2022: Generating an archived web corpusExploring the Archived Web during a Highly Transformative Age10.36253/979-12-215-0413-2.17Online publication date: 2024
https://doi.org/10.36253/979-12-215-0413-2.17
Pascual Almenara ASayago Barrantes S(2024)Towards Designing a Set of Usability and Accessibility Heuristics Focused on Cognitive Diversity: An Exploratory Case Study with Generative Artificial IntelligenceInformation10.3390/info1512076915:12(769)Online publication date: 2-Dec-2024
https://doi.org/10.3390/info15120769
Omar KFakhouri HZraqou JMarx Gómez J(2024)Usability Heuristics for MetaverseComputers10.3390/computers1309022213:9(222)Online publication date: 6-Sep-2024
https://doi.org/10.3390/computers13090222
Galavi ZNorouzi SKhajouei R(2024)Heuristics used for evaluating the usability of mobile health applications: A systematic literature reviewDIGITAL HEALTH10.1177/2055207624125353910Online publication date: 15-May-2024
https://doi.org/10.1177/20552076241253539
Malki LKaleva IPatel DWarner MAbu-Salma R(2024)Exploring Privacy Practices of Female mHealth Apps in a Post-Roe WorldProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642521(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642521
Fischer KVelentza ALucas GWilliams D(2024)Seeing Eye to Eye with Robots: An Experimental Study Predicting Trust in Social Robots for Domestic Use2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN)10.1109/RO-MAN60168.2024.10731371(2162-2168)Online publication date: 26-Aug-2024
https://doi.org/10.1109/RO-MAN60168.2024.10731371
Brackley COuf MO'brien W(2024)Bridging the gap between theory and adoption: A critical review of socio-technical and human-computer interaction studies of fault detection and diagnosis in commercial buildingsScience and Technology for the Built Environment10.1080/23744731.2024.2378674(1-14)Online publication date: 31-Jul-2024
https://doi.org/10.1080/23744731.2024.2378674
Maqbool BHerold S(2024)Potential effectiveness and efficiency issues in usability evaluation within digital health: A systematic literature reviewJournal of Systems and Software10.1016/j.jss.2023.111881208(111881)Online publication date: Feb-2024
https://doi.org/10.1016/j.jss.2023.111881
Piirisild APerandrés Gómez ATaveter K(2024)A New Usability Inspection Method: Experience-Based AnalysisRequirements Engineering: Foundation for Software Quality10.1007/978-3-031-57327-9_5(74-91)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57327-9_5
Bigdeli SSoleimani Movahed MSandars J(2024)Technology-Enhanced Learning in Medical EducationGlobal Medical Education in Normal and Challenging Times10.1007/978-3-031-51244-5_20(183-189)Online publication date: 30-May-2024
https://doi.org/10.1007/978-3-031-51244-5_20
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents