article

The state of the art in automating usability evaluation of user interfaces

Authors:

Melody Y. Ivory,

Marti A HearstAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 33, Issue 4

Pages 470 - 516

https://doi.org/10.1145/503112.503114

Published: 01 December 2001 Publication History

Get Access

Abstract

Usability evaluation is an increasingly important part of the user interface design process. However, usability evaluation can be expensive in terms of time and human resources, and automation is therefore a promising way to augment existing approaches. This article presents an extensive survey of usability evaluation methods, organized according to a new taxonomy that emphasizes the role of automation. The survey analyzes existing techniques, identifies which aspects of usability evaluation automation are likely to be of use in future research, and suggests new ways to expand existing approaches to better support usability evaluation.

References

[1]

ABELOW, D. 1993. Automating feedback on software product use. CASE Trends December, 15- 17.

Google Scholar

[2]

ADDY & ASSOCIATES. 2000. Dr. Watson version 4.0. Available at http://watson.addy.com/.

Google Scholar

[3]

AHLBERG,C.AND SHNEIDERMAN, B. 1994. Visual information seeking: Tight coupling of dynamic query filters with starfield displays. In Proceedings of the Conference on Human Factors in Computing Systems (Boston, MA, April), pp. 313- 317. New York, NY: ACM Press.

Digital Library

Google Scholar

[4]

AL-QAIMARI,G.AND MCROSTIE, D. 1999. KALDI: A computer-aided usability engineering tool for supporting testing and analysis of humancomputer interaction. In J. Vanerdonckt and A. Puerta, Eds., Proceedings of the Third International Conference on Computer-Aided Design of User Interfaces (Louvain-la-Neuve, Belgium, October). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Digital Library

Google Scholar

[5]

ANDERSON, J. 1993. Rules of the Mind. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Google Scholar

[6]

ANDERSON, J. R. 1990. The Adaptive Character of Thought. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Google Scholar

[7]

BACHELDOR, B. 1999. Push for performance. Information Week September 20, 18-20.

Google Scholar

[8]

BALBO, S. 1995. Automatic evaluation of user interface usability: Dream or reality. In S. Balbo, Ed., Proceedings of the Queensland Computer- Human Interaction Symposium (Queensland, Australia, August). Bond University.

Google Scholar

[9]

BALBO, S. 1996. EMA: Automatic analysis mechanism for the ergonomic evaluation of user interfaces. Tech. Rep. 96/44 (August), CSIRO Division of Information Technology. Available at http: // www.cmis.csiro.au / sandrine.balbo / Ema/ ema tr/ema-tr.doc.

Google Scholar

[10]

BARNARD, P. J. 1987. Cognitive resources and the learning of human-computer dialogs. In J. M. Carroll, Ed., Interfacing Thought: Cognitive Aspects of Human-Computer Interaction, pp. 112- 158. Cambridge, MA: MIT Press.

Digital Library

Google Scholar

[11]

BARNARD,P.J.AND TEASDALE, J. D. 1991. Interacting cognitive subsystems: A systemic approach to cognitive-affective interaction and change. Cognition and Emotion 5, 1-39.

Crossref

Google Scholar

[12]

BAUMEISTER, L. K., JOHN,B.E.,AND BYRNE,M.D. 2000. A comparison of tools for building GOMS models. In Proceedings of the Conference on Human Factors in Computing Systems, Volume 1 (The Hague, The Netherlands, April), pp. 502- 509. New York, NY: ACM Press.

Digital Library

Google Scholar

[13]

BEARD,D.V.,SMITH,D.K.,AND DENELSBECK,K.M. 1996. Quick and dirty GOMS: A case study of computed tomography interpretation. Human- Computer Interaction 11, 2, 157-180.

Digital Library

Google Scholar

[14]

BELLOTTI, V. 1988. Implications of current design practice for the use of HCI techniques. In Proceedings of the HCI Conference on People and Computers IV (Manchester, United Kingdom, September), pp. 13-34. Amsterdam, The Netherlands: Elsevier Science Publishers.

Digital Library

Google Scholar

[15]

BODART, F., HENNEBERT, A. M., LEHEUREUX,J.M.,AND VANDERDONCKT, J. 1994. Towards a dynamic strategy for computer-aided visual placement. In T. Catarci, M. Costabile, M. Levialdi, and G. Santucci, Eds., Proceedings of the International Conference on Advanced Visual Interfaces (Bari, Italy), pp. 78-87. New York, NY: ACM Press.

Digital Library

Google Scholar

[16]

BORGES, J. A., MORALES, I., AND RODRIGUEZ, N. J. 1996. Guidelines for designing usable world wide web pages. In Proceedings of the Conference on Human Factors in Computing Systems, Volume 2 (Vancouver, Canada, April), pp. 277-278. New York, NY: ACM Press.

Digital Library

Google Scholar

[17]

BOWERS, N. 1996. Weblint: Quality assurance for the World Wide Web. In Proceedings of the Fifth International World Wide Web Conference (Paris, France, May). Amsterdam, The Netherlands: Elsevier Science Publishers. Available at http://www5conf.inria.fr/fich html/papers/P34/ Overview.html.

Digital Library

Google Scholar

[18]

BRAJNIK, G. 2000. Automatic web usability evaluation: Where is the limit? In Proceedings of the Sixth Conference on Human Factors & the Web (Austin, TX, June). Available at http://www.tri.sbc.com/hfweb/brajnik/hfwebbrajnik. html.

Google Scholar

[19]

BYRNE, M. D., JOHN, B. E., WEHRLE,N.S.,AND CROW, D. C. 1999. The tangled web we wove: A taxonomy of WWW use. In Proceedings of the Conference on Human Factors in Computing Systems, Volume 1 (Pittsburgh, PA, May), pp. 544- 551, New York, NY: ACM Press.

Digital Library

Google Scholar

[20]

BYRNE,M.D.,WOOD,S.D.,SUKAVIRIYA,P.N.,FOLEY, J. D., AND KIERAS, D. 1994. Automating interface evaluation. In Proceedings of the Conference on Human Factors in Computing Systems, Volume (April), pp. 232-237. New York, NY: ACM Press.

Digital Library

Google Scholar

[21]

CARD, S. K., MORAN,T.P.,AND NEWELL, A. 1983. The Psychology of Human-Computer Interaction. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Digital Library

Google Scholar

[22]

CENTERLINE. 1999. QC/Replay. Available at http:// www.centerline.com/productline/qcreplay/qcre-play. html.

Google Scholar

[23]

CHAK, A. 2000. Usability tools: A useful start. Web Techniques (2000), August, 18-20. Available at http://www.webtechniques.com/archives/2000 /08/stratrevu/.

Google Scholar

[24]

CHI, E. H., PIROLLI,P.,AND PITKOW, J. 2000. The scent of a site: A system for analyzing and predicting information scent, usage, and usability of a web site. In Proceedings of the Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April), pp. 161-168. New York, NY: ACM Press.

Digital Library

Google Scholar

[25]

CLARK,D.AND DARDAILLER, D. 1999. Accessibility on the web: Evaluation & repair tools to make it possible. In Proceedings of the CSUN Technology and Persons with Disabilities Conference (Los Angeles, CA, March). Available at http://www.dinf.org/csun 99/session0030.html.

Google Scholar

[26]

COMBER, T. 1995. Building usable web pages: An HCI perspective. In R. Debreceny and A. Ellis, Eds., Proceedings of the First Australian World Wide Web Conference (Ballina, Australia, April), pp. 119-124. Ballina, Australia: Norsearch. Available at http://www. scu.edu.au/sponsored/ausweb/ausweb95/papers/ hypertext/comber/.

Google Scholar

[27]

COOPER, M. 1999. Universal design of a web site. In Proceedings of the CSUN Technology and Persons with Disabilities Conference (Los Angeles, CA, March). Available at http://www.dinf.org/csun 99/session0030.html.

Google Scholar

[28]

COUTAZ, J. 1995. Evaluation techniques: Exploring the intersection of HCI and software engineering. In R. N. Taylor and J. Coutaz, Eds., Software Engineering and Human-Computer Interaction, Lecture Notes in Computer Science, pp. 35-48. Heidelberg, Germany: Springer- Verlag.

Digital Library

Google Scholar

[29]

CUGINI,J.AND SCHOLTZ, J. 1999. VISVIP: 3D visualization of paths through web sites. In Proceedings of the International Workshop on Web-Based Information Visualization (Florence, Italy, September), pp. 259-263. Institute of Electrical and Electronics Engineers.

Digital Library

Google Scholar

[30]

DE HAAN,G.,VAN DER VEER,G.C.,AND VAN VLIET,J.C. 1992. Formal modelling techniques in humancomputerF interaction. In G. C. van der Veer, S. Bagnara, and G. A. M. Kempen, Eds., Cognitive Ergonomics: Contributions from Experimental Psychology, Theoretical Issues, pp. 27-67. Amsterdam, The Netherlands: Elsevier Science Publishers.

Google Scholar

[31]

DE SOUZA,F.AND BEVAN, N. 1990. The use of guidelines in menu interface design: Evaluation of a draft standard. In G. Cockton, D. Diaper, and B. Shackel, Eds., Proceedings of the Third IFIP TC13 Conference on Human-Computer Interaction (Cambridge, UK, August), pp. 435-440. Amsterdam, The Netherlands: Elsevier Science Publishers.

Digital Library

Google Scholar

[32]

DETWEILER,M.C.AND OMANSON, R. C. 1996. Ameritech web page user interface standards and design guidelines. Ameritech Corporation, Chicago, IL. Available at http:// www.ameritech.com/corporate/testtown/library/ standard/web guidelines/index.html.

Google Scholar

[33]

DIX, A., FINLAY, J., ABOWD,G.,AND BEALE,R. 1998. Human-Computer Interaction (second ed.). Prentice Hall, Upper Saddle River, NJ.

Digital Library

Google Scholar

[34]

DROTT, M. C. 1998. Using web server logs to improve site design. In Proceedings of the 16th International Conference on Systems Documentation (Quebec, Canada, September), pp. 43-50. New York, NY: ACM Press.

Digital Library

Google Scholar

[35]

ETGEN,M.AND CANTOR, J. 1999. What does getting WET (web event-logging tool) mean for web usability. In Proceedings of the Fifth Conference on Human Factors & the Web (Gaithersburg, MD, June). Available at http://www.nist.gov/itl/div894/vvrg/hfweb/ proceedings/etgen-cantor/index.html.

Google Scholar

[36]

FARADAY, P. 2000. Visually critiquing web pages. In Proceedings of the Sixth Conference on Human Factors & the Web (Austin, TX, June). Available at http://www.tri.sbc.com/hfweb/faraday/ faraday.htm.

Crossref

Google Scholar

[37]

FARADAY,P.AND SUTCLIFFE, A. 1998. Providing advice for multimedia designers. In Proceedings of the Conference on Human Factors in Computing Systems, Volume 1 (Los Angeles, CA, April), pp. 124-131. New York, NY: ACM Press.

Digital Library

Google Scholar

[38]

FARENC,C.AND PALANQUE, P. 1999. A generic framework based on ergonomics rules for computer aided design of user interface. In J. Vanderdonckt and A. Puerta, Eds., Proceedings of the Third International Conference on Computer-Aided Design of User Interfaces (Louvain-la-Neuve, Belgium, October). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Digital Library

Google Scholar

[39]

FARENC, C., LIBERATI,V.,AND BARTHET, M.-F. 1999. Automatic ergonomic evaluation: What are the limits. In Proceedings of the Third International Conference on Computer-Aided Design of User Interfaces (Louvain-la-Neuve, Belgium, October). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Google Scholar

[40]

FOGG, B. J. 1999. What variables affect web credibility? Available at http://www. webcredibility.org/variables files/v3 document. htm.

Google Scholar

[41]

FOGG,B.J.,MARSHALL, J., OSIPOVICH, A., VARMA, C., LARAKI, O., FANG, N., PAVI, J., RANGNEKAR, A., SHON, J., SWANI,P.,AND TREINEN, M. 2000. Elements that affect web credibility: Early results from a self-report study. In Proceedings of the Conference on Human Factors in Computing Systems, number Extended Abstracts, 287-288. The Hague, The Netherlands. New York, NY: ACM Press. Available at http://www.webcredibility.org/webCredEarly- Results. ppt.

Digital Library

Google Scholar

[42]

FULLER,R.AND DE GRAAFF, J. J. 1996. Measuring user motivation from server log files. In Proceedings of the Second Conference on Human Factors & the Web (Redmond, WA, October). Available at http://www.microsoft.com/usability/ webconf/ fuller/fuller.htm.

Google Scholar

[43]

GLENN, F. A., SCHWARTZ,S.M.,AND ROSS, L. V. 1992. Development of a human operator simulator version v (HOS-V): Design and implementation. U.S. Army Research Institute for the Behavioral and Social Sciences, PERI-POX, Alexandria, VA.

Google Scholar

[44]

GUZDIAL, M., SANTOS, P., BADRE, A., HUDSON,S., AND GRAY, M. 1994. Analyzing and visualizing log files: A computational science of usability. GVU Center TR GIT-GVU-94-8, Georgia Institute of Technology. Available at http:// www.cc.gatech.edu/gvu/reports/1994/abstracts/ 94-08.html.

Google Scholar

[45]

HAMMONTREE, M. L., HENDRICKSON,J.J.,AND HENSLEY, B. W. 1992. Integrated data capture and analysis tools for research and testing on graphical user interfaces. In Proceedings of the Conference on Human Factors in Computing Systems (Monterey, CA, May), pp. 431-432. New York, NY: ACM Press.

Digital Library

Google Scholar

[46]

HARPER,B.AND NORMAN, K. 1993. Improving user satisfaction: The questionnaire for user satisfaction interaction version 5.5. In Proceedings of the First Annual Mid-Atlantic Human Factors Conference (Virginia Beach, VA), pp. 224-228.

Google Scholar

[47]

HARTSON, H. R., CASTILLO,J.C.,KELSA,J.,AND NEALE, W. C. 1996. Remote evaluation: The network as an extension of the usability laboratory. In M. J. Tauber, V. Bellotti, R. Jeffries, J. D. Mackinlay, and J. Nielsen, Eds., Proceedings of the Conference on Human Factors in Computing Systems (Vancouver, Canada, April), pp. 228- 235. New York, NY: ACM Press.

Digital Library

Google Scholar

[48]

HELFRICH,B.AND LANDAY, J. A. 1999. QUIP: quantitative user interface profiling. Unpublished manuscript. Available at http://home. earthlink.net/ >> bhelfrich/quip/index.html.

Google Scholar

[49]

HOCHHEISER,H.AND SHNEIDERMAN, B. 2001. Using interactive visualizations of WWW log data to characterize access patterns and inform site design. Journal of American Society for Information Science and Technology 52, 4 (February), 331-343.

Digital Library

Google Scholar

[50]

HOM, J. 1998. The usability methods toolbox. Available at http://www.best.com/ >> jthom/usability/ usable.htm.

Google Scholar

[51]

HUDSON, S. E., JOHN, B. E., KNUDSEN, K., AND BYRNE, M. D. 1999. A tool for creating predictive performance models from user interface demonstrations. In Proceedings of the 12th Annual ACMSymposium on User Interface Software and Technology (Asheville, NC, November), pp. 92- 103. New York, NY: ACM Press.

Digital Library

Google Scholar

[52]

HUMAN FACTORS ENGINEERING. 1999. Usability evaluation methods. Available at http://www. cs.umd.edu/ zzj/UsabilityHome.html.

Google Scholar

[53]

INTERNATIONAL STANDARDS ORGANIZATION. 1999. Ergonomic requirements for office work with visual display terminals, part 11: Guidance on usability. Available at http://www.iso.ch/iso/ en/catalogueDetailPage.catalogueDetail? CS-Number D 16883&ICS1 D 13&ICS2 D 180&ICS3 D IVORY, M. Y. 2001.

Google Scholar

[54]

IVORY, M. Y., SINHA,R.R.,AND HEARST,M.A. 2000. Preliminary findings on quantitative measures for distinguishing highly rated information-centric web pages. In Proceedings of the Sixth Conference on Human Factors & the Web (Austin, TX, June). Available at http://www.tri.sbc.com/hfweb/ivory/paper.html.

Google Scholar

[55]

IVORY,M.Y.,SINHA,R.R.,AND HEARST, M. A. 2001. Empirically validated web page design metrics. In Proceedings of the Conference on Human Factors in Computing Systems (Seattle, WA, March), pp. 53-60. New York, NY: ACM Press.

Digital Library

Google Scholar

[56]

JAIN, R. 1991. Human-Computer Interaction. Wiley-Interscience, New York, NY.

Google Scholar

[57]

JEFFRIES, R., MILLER, J. R., WHARTON,C.,AND UYEDA, K. M. 1991. User interface evaluation in the real world: A comparison of four techniques. In Proceedings of the Conference on Human Factors in Computing Systems (New Orleans, LA, April), pp. 119-124. New York, NY: ACM Press.

Digital Library

Google Scholar

[58]

JIANG, J., MURPHY, E., AND CARTER, L. 1993. Computer-human interaction models (CHIMES): Revision 3. Tech. Rep. DSTL-94- 008 (May), National Aeronautics and Space Administration.

Google Scholar

[59]

JOHN,B.E.AND KIERAS, D. E. 1996. The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Transactions on Computer-Human Interaction 3, 4, 320- 351.

Digital Library

Google Scholar

[60]

KASIK,D.J.AND GEORGE, H. G. 1996. Toward automatic generation of novice user test scripts. In Proceedings of the Conference on Human Factors in Computing Systems, Volume 1 (Vancouver, Canada, April), pp. 244-251. New York, NY: ACM Press.

Digital Library

Google Scholar

[61]

KIERAS,D.AND POLSON, P. G. 1985. An approach to the formal analysis of user complexity. International Journal of Man-Machine Studies 22,4, 365-394.

Crossref

Google Scholar

[62]

KIERAS, D. E., WOOD,S.D.,ABOTEL, K., AND HORNOF, A. 1995. GLEAN: A computer-based tool for rapid GOMS model usability evaluation of user interface designs. In Proceedings of the Eighth ACMSymposium on User Interface Software and Technology (Pittsburgh, PA, November), pp. 91- 100. New York, NY: ACM Press.

Digital Library

Google Scholar

[63]

KIERAS, D. E., WOOD,S.D.,AND MEYER,D.E. 1997. Predictive engineering models based on the EPIC architecture for a multimodal highperformance human-computer interaction task. ACM Transactions on Computer-Human Interaction 4, 3 (September), 230-275.

Digital Library

Google Scholar

[64]

KIM,W.C.AND FOLEY, J. D. 1993. Providing highlevel control and expert assistance in the user interface presentation design. In S. Ashlund, K. Mullet, A. Henderson, E. Hollnagel, and T. White, Eds., Proceedings of the Conference on Human Factors in Computing Systems (Amsterdam, The Netherlands, April), pp. 430-437. New York, NY: ACM Press.

Digital Library

Google Scholar

[65]

KIRAKOWSKI,J.AND CLARIDGE, N. 1998. Human centered measures of success in web site design. In Proceedings of the Fourth Conference on Human Factors & the Web (Basking Ridge, NJ, June). Available at http:// www. research.att. com/conf/hfweb/proceedings/ kirakowski/index.html.

Google Scholar

[66]

LAIRD,J.E.AND ROSENBLOOM, P. 1996. The evolution of the Soar cognitive architecture. In D. M. Steier and T. M. Mitchell, Eds., Mind Matters: A Tribute to Allen Newell, pp. 1-50. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Google Scholar

[67]

LARSON,K.AND CZERWINSKI, M. 1998. Web page design: Implications of memory, structure and scent for information retrieval. In Proceedings of the Conference on Human Factors in Computing Systems, Volume 1 (Los Angeles, CA, April), pp. 25-32. New York, NY: ACM Press.

Digital Library

Google Scholar

[68]

LECEROF,A.AND PATERN` O, F. 1998. Automatic support for usability evaluation. IEEE Transactions on Software Engineering 24, 10 (October), 863- 888.

Digital Library

Google Scholar

[69]

LEE, K. 1997. Motif FAQ. Available at http://wwwbioeng. ucsd.edu/ >> fvetter/misc/Motif-FAQ.txt.

Google Scholar

[70]

LEVINE, R. 1996. Guide to web style. Sun Microsytems. Available at http://www.sun.com/ styleguide/.

Google Scholar

[71]

LEWIS, C., POLSON,P.G.,WHARTON,C.,AND RIEMAN, J. 1990. Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. In Proceedings of the Conference on Human Factors in Computing Systems (Seattle, WA, April), pp. 235-242. New York, NY: ACM Press.

Digital Library

Google Scholar

[72]

LOWGREN,J.AND NORDQVIST, T. 1992. Knowledgebased evaluation as design support for graphical user interfaces. In Proceedings of the Conference on Human Factors in Computing Systems (Monterey, CA, May), pp. 181-188. New York, NY: ACM Press.

Digital Library

Google Scholar

[73]

LYNCH, G., PALMITER,S.,AND TILT, C. 1999. The max model: A standard web site user model. In Proceedings of the Fifth Conference on Human Factors & the Web (Gaithersburg, MD, June). Available at http://www.nist.gov/itl/div894/ vvrg/hfweb/ proceedings/lynch/index.html.

Google Scholar

[74]

LYNCH,P.J.AND HORTON, S. 1999. Web Style Guide: Basic Design Principles for Creating Web Sites. Yale University Press. Available at http://info.med.yale.edu/caim/manual/.

Digital Library

Google Scholar

[75]

MACLEOD,M.AND RENGGER, R. 1993. The development of DRUM: A software tool for videoassisted usability evaluation. In Proceedings of the HCI Conference on People and Computers VIII (Loughborough, UK, September), pp. 293- 309. Cambridge University Press, Cambridge, UK.

Google Scholar

[76]

MAHAJAN,R.AND SHNEIDERMAN, B. 1997. Visual & textual consistency checking tools for graphical user interfaces. IEEE Transactions on Software Engineering 23, 11 (November), 722-735.

Digital Library

Google Scholar

[77]

MAY,J.AND BARNARD, P. J. 1994. Supportive evaluation of interface design. In C. Stary, Ed., Proceedings of the First Interdisciplinary Workshop on Cognitive Modeling and User Interface Design (Vienna, Austria, December).

Google Scholar

[78]

MERCURY INTERACTIVE. 2000. Winrunner. Available at http://www-svca.mercuryinteractive.com/ products/winrunner/.

Google Scholar

[79]

MOLICH, R., BEVAN, N., BUTLER, S., CURSON, I., KIND- LUND, E., KIRAKOWSKI,J.,AND MILLER, D. 1998. Comparative evaluation of usability tests. In Proceedings of the UPA Conference (Washington, DC, June), pp. 189-200. Usability Professionals' Association, Chicago, IL.

Google Scholar

[80]

MOLICH, R., THOMSEN,A.D.,KARYUKINA, B., SCHMIDT, L., EDE, M., VAN OEL,W.,AND ARCURI,M. 1999. Comparative evaluation of usability tests. In Proceedings of the Conference on Human Factors in Computing Systems (Pittsburgh, PA, May), pp. 83-86. New York, NY: ACM Press.

Digital Library

Google Scholar

[81]

MORAN, T. P. 1981. The command language grammar: Arepresentation for the user interface of interactive computer systems. International Journal of Man-Machine Studies 15, 1, 3-50.

Crossref

Google Scholar

[82]

MORAN, T. P. 1983. Getting into a system: External-internal task mapping analysis. In Proceedings of the Conference on Human Factors in Computing Systems (Boston, MA, December), pp. 45-49. New York, NY: ACM Press.

Digital Library

Google Scholar

[83]

NETRAKER. 2000. The NetRaker suite. Available at http://www.netraker.com/info/applications/ index.asp.

Google Scholar

[84]

NIELSEN, J. 1993. Usability Engineering. Boston, MA: Academic Press.

Digital Library

Google Scholar

[85]

OLSEN,JR., D. R. 1992. User Interface Management Systems: Models and Algorithms. San Mateo, CA: Morgan Kaufman Publishers, Inc.

Digital Library

Google Scholar

[86]

OLSEN,JR., D. R. AND HALVERSEN, B. W. 1988. Interface usage measurements in a user interface management system. In Proceedings of the ACM SIGGRAPH Symposium on User Interface Software (Alberta, Canada, October), pp. 102-108. New York, NY: ACM Press.

Digital Library

Google Scholar

[87]

OPEN SOFTWARE FOUNDATION. 1991. OSF/Motif Style Guide. Number Revision 1.1 (for OSF/Motif release 1.1). Englewood Cliffs, NJ: Prentice Hall.

Digital Library

Google Scholar

[88]

PALANQUE, P., FARENC,C.,AND BASTIDE, R. 1999. Embedding ergonomic rules as generic requirements in the development process of interactive software. In A. Sasse and C. Johnson, Eds., Proceedings of IFIP TC13 Seventh International Conference on Human-Computer Interaction (Edinburgh, Scotland, August). Amsterdam, The Netherlands: IOS Press.

Google Scholar

[89]

PARUSH, A., NADIR, R., AND SHTUB, A. 1998. Evaluating the layout of graphical user interface screens: Validation of a numerical, computerized model. International Journal of Human Com-puter Interaction 10, 4, 343-360.

Crossref

Google Scholar

[90]

PATERN O,F.AND BALLARDIN, G. 1999. Modelaided remote usability evaluation. In A. Sasse and C. Johnson, Eds., Proceedings of the IFIP TC13 Seventh International Conference on Human- Computer Interaction (Edinburgh, Scotland, August), pp. 434-442. Amsterdam, The Netherlands: IOS Press.

Google Scholar

[91]

PATERN O, F., MANCINI,C.,AND MENICONI, S. 1997. ConcurTaskTrees: Diagrammatic notation for specifying task models. In Proceedings of the IFIP TC13 Sixth International Conference on Human-Computer Interaction (Sydney, Australia, July), pp. 362-369. Sydney, Australia: Chapman and Hall.

Digital Library

Google Scholar

[92]

PAYNE,S.J.AND GREEN, T. R. G. 1986. Taskaction grammars: A model of the mental representation of task languages. Human-Computer Interaction 2, 93-133.

Digital Library

Google Scholar

[93]

PECK,V.A.AND JOHN, B. E. 1992. Browsersoar: A computational model of a highly interactive task. In Proceedings of the Conference on Human Factors in Computing Systems (Monterey, CA, May), pp. 165-172. New York, NY: ACM Press.

Digital Library

Google Scholar

[94]

PETRI, C. A. 1973. Concepts of net theory. In Mathematical Foundations of Computer Science: Proceedings of the Symposium and Summer School (High Tatras, Czechoslovakia, September), pp. 137-146. Mathematical Institute of the Slovak Academy of Sciences.

Google Scholar

[95]

PEW,R.W.AND MAVOR,A.S.,EDS. 1998. Modeling Human and Organizational Behavior: Application to Military Simulations. Washington, DC: National Academy Press. Available at http://books.nap.edu/html/model.

Google Scholar

[96]

POLK,T.A.AND ROSENBLOOM, P. S. 1994. Taskindependent constraints on a unified theory of cognition. In F. Boller and J. Grafman, Eds., Handbook of Neuropsychology, Volume 9. Amsterdam, The Netherlands: Elsevier Science Publishers.

Google Scholar

[97]

RATNER, J., GROSE,E.M.,AND FORSYTHE, C. 1996. Characterization and assessment of HTML style guides. In Proceedings of the Conference on Human Factors in Computing Systems, Volume 2 (Vancouver, Canada, April), pp. 115-116. New York, NY: ACM Press.

Digital Library

Google Scholar

[98]

RAUTERBERG, M. 1995. From novice to expert decision behaviour: A qualitative modeling approach with Petri nets. In Y. Anzai, K. Ogawa, and H. Mori, Eds., Symbiosis of Human and Artifact: Human and Social Aspects of Human-Computer Interaction, Volume 20B of Advances in Human Factors/Ergonomics (1995), pp. 449-454. Amsterdam, The Netherlands: Elsevier Science Publishers.

Google Scholar

[99]

RAUTERBERG, M. 1996a. How to measure and to quantify usability of user interfaces. In A. Ozok and G. Salvendy, Eds., Advances in Applied Ergonomics (1996), pp. 429-432. West Lafayette, IN: USA Publishing.

Google Scholar

[100]

RAUTERBERG, M. 1996b. A Petri net based analyzing and modeling tool kit for logfiles in humancomputer interaction. In Proceedings of Cognitive Systems Engineering in Process Control (Kyoto, Japan, November), pp. 268-275. Kyoto University: Graduate School of Energy Science.

Google Scholar

[101]

RAUTERBERG,M.AND AEPPILI, R. 1995. Learning in man-machine systems: the measurement of behavioural and cognitive complexity. In Proceedings of the IEEE Conference on Systems, Man and Cybernetics (Vancouver, BC, October), pp. 4685-4690. Institute of Electrical and Electronics Engineers.

Crossref

Google Scholar

[102]

REISNER, P. 1984. Formal grammar as a tool for analyzing ease of use: Some fundamental concepts. In J. C. Thomas and M. L. Schneider, Eds., Human Factors in Computer Systems, pp. 53-78. Norwood, NJ: Ablex Publishing Corp.

Google Scholar

[103]

REITERER, H. 1994. A user interface design assistant approach. In K. Brunnstein and E. Raubold, Eds., Proceedings of the IFIP 13th World Computer Congress, Volume 2 (Hamburg, Germany, August), pp. 180-187. Amsterdam, The Netherlands: Elsevier Science Publishers.

Google Scholar

[104]

RIEMAN, J., DAVIES, S., HAIR,D.C.,ESEMPLARE, M., POLSON,P.,AND LEWIS, C. 1991. An automated cognitive walkthrough. In Proceedings of the Conference on Human Factors in Computing Systems (New Orleans, LA, April), pp. 427-428. New York, NY: ACM Press.

Digital Library

Google Scholar

[105]

SCAPIN, D., LEULIER, C., VANDERDONCKT, J., MARIAGE, C., BASTIEN, C., FARENC, C., PALANQUE,P.,AND BASTIDE, R. 2000. A framework for organizing web usability guidelines. In Proceedings of the Sixth Conference on Human Factors & the Web (Austin, TX, June). Available at http:// www.tri.sbc.com/hfweb/scapin/Scapin.html.

Google Scholar

[106]

SCHOLTZ,J.AND LASKOWSKI, S. 1998. Developing usability tools and techniques for designing and testing web sites. In Proceedings of the Fourth Conference on Human Factors & the Web (Basking Ridge, NJ, June). Available at http://www.research.att.com/conf/hfweb/ proceedings/ scholtz/index.html.

Google Scholar

[107]

SCHWARTZ, M. 2000. Web site makeover. Computerworld January 31. Available at http:// www.computerworld.com/home/print.nsf/all/ 000126e3e2.

Google Scholar

[108]

SEARS, A. 1995. AIDE: A step toward metricbased interface development tools. In Proceedings of the Eighth ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, November), pp. 101-110. New York, NY: ACM Press.

Digital Library

Google Scholar

[109]

SERVICE METRICS. 1999. Service metrics solutions. Available at http://www.servicemetrics.com/solutions/ solutionsmain.asp.

Google Scholar

[110]

SHNEIDERMAN, B. 1998. Designing the User Interface: Strategies for Effective Human- Computer Interaction (third ed.). Reading, MA: Addison- Wesley.

Digital Library

Google Scholar

[111]

SIOCHI,A.C.AND HIX, D. 1991. A study of computer-supported user interface evaluation using maximal repeating pattern analysis. In Proceedings of the Conference on Human Factors in Computing Systems (New Orleans, LA, April), pp. 301-305. New York, NY: ACM Press.

Digital Library

Google Scholar

[112]

SMITH, S. L. 1986. Standards versus guidelines for designing user interface software. Behaviour and Information Technology 5, 1, 47-61.

Crossref

Google Scholar

[113]

SMITH,S.L.AND MOSIER, J. N. 1986. Guidelines for designing user interface software. Tech. Rep. ESD-TR-86-278, The MITRE Corporation, Bedford, MA 01730.

Google Scholar

[114]

STEIN, L. D. 1997. The rating game. Available at http://stein.cshl.org/ lstein/rater/.

Google Scholar

[115]

STREVELER,D.J.AND WASSERMAN, A. I. 1984. Quantitative measures of the spatial properties of screen designs. In B. Shackel, Ed., Proceedings of the IFIP TC13 First International Conference on Human-Computer Interaction (London, UK, September), pp. 81-89. Amsterdam, The Netherlands: North-Holland.

Google Scholar

[116]

SULLIVAN, T. 1997. Reading reader reaction: A proposal for inferential analysis of web server log files. In Proceedings of the Third Conference on Human Factors & the Web (Boulder, CO, June). Available at http://www.research.att.com/ conf/hfweb/conferences/denver3.zip.

Google Scholar

[117]

TAUBER, M. J. 1990. ETAG: Extended task action grammar-A language for the description of the user's task language. In G. Cockton, D. Diaper, and B. Shackel, Eds., Proceedings of the IFIP TC13 Third International Conference on Human-Computer Interaction (Cambridge, UK, August), pp. 163-168. Amsterdam, The Netherlands: Elsevier Science Publishers.

Digital Library

Google Scholar

[118]

THENG,Y.L.AND MARSDEN, G. 1998. Authoring tools: Towards continuous usability testing of web documents. In Proceedings of the First International Workshop on Hypermedia Development (Pittsburgh, PA, June). Available at http://www.eng.uts.edu.au/ >> dbl/HypDev/ht98w/ YinLeng/HT98 YinLeng.html.

Google Scholar

[119]

THIMBLEBY, H. 1997. Gentler: A tool for systematic web authoring. International Journal of Human- Computer Studies 47, 1, 139-168.

Digital Library

Google Scholar

[120]

TULLIS, T. S. 1983. The formatting of alphanumeric displays: A review and analysis. Human Factors 25, 657-682.

Crossref

Google Scholar

[121]

UEHLING,D.L.AND WOLF, K. 1995. User action graphing effort (UsAGE). In Proceedings of the Conference on Human Factors in Computing Systems (Denver, CO, May). New York, NY: ACM Press.

Digital Library

Google Scholar

[122]

USABLE NET. 2000. LIFT online. Available at http://www.usablenet.com.

Google Scholar

[123]

VANDERDONCKT,J.AND GILLO, X. 1994. Visual techniques for traditional and multimedia layouts. In T. Catarci, M. Costabile, M. Levialdi, and G. Santucci, Eds., Proceedings of the International Conference on Advanced Visual Interfaces (Bari, Italy, June), pp. 95-104. New York, NY: ACM Press.

Digital Library

Google Scholar

[124]

WEB ACCESSIBILITY INITIATIVE. 1999. Web content accessibility guidelines 1.0. World Wide Web Consortium, Geneva, Switzerland. Available at http://www.w3.org/TR/WAI-WEBCON-TENT/.

Google Scholar

[125]

WEB CRITERIA. 1999. Max, and the objective measurement of web sites. Available at http://www.webcriteria.com.

Google Scholar

[126]

WEBTRENDS CORPORATION. 2000. Webtrends live. Available at http://www.webtrendslive.com/default. htm.

Google Scholar

[127]

WHITEFIELD, A., WILSON,F.,AND DOWELL, J. 1991. A framework for human factors evaluation. Behaviour and Information Technology 10, 1, 65- 79.

Crossref

Google Scholar

[128]

WILLIAMS, K. E. 1993. Automating the cognitive task modeling process: An extension to GOMS for HCI. In Proceedings of the Conference on Human Factors in Computing Systems, Volume 3 (Amsterdam, The Netherlands, April), pp. 182. New York, NY: ACM Press.

Digital Library

Google Scholar

[129]

WORLD WIDE WEB CONSORTIUM. 2000. HTML valiation service. Available at http://validator.w3. org/.

Google Scholar

[130]

YOUNG, R. M., GREEN,T.R.G.,AND SIMON, T. 1989. Programmable user models for predictive evaluation of interface designs. In Proceedings of the Conference on Human Factors in Computing Systems (Austin, TX, April), pp. 15-19. New York, NY: ACM Press.

Digital Library

Google Scholar

[131]

ZACHARY, W., MENTEC, J.-C. L., AND RYDER,J. 1996. Interface agents in complex systems. In C. N. Ntuen and E. H. Park, Eds., Human Interaction With Complex Systems: Conceptual Principles and Design Practice. Dordrecht, The Netherlands: Kluwer Academic Publishers.

Google Scholar

[132]

ZAPHIRIS,P.AND MTEI, L. 1997. Depth vs. breadth in the arrangement of Web links. Available at http://www.otal.umd.edu/SHORE/bs04.

Google Scholar

[133]

ZETTLEMOYER, L. S., ST.AMANT,R.S.,AND DULBERG, M. S. 1999. IBOTS: Agent control through the user interface. In Proceedings of the International Conference on Intelligent User Interfaces (Redondo Beach, CA, Jan.), pp. 31-37. New York, NY: ACM Press.

Digital Library

Google Scholar

Cited By

View all

Haider ARomanus TTang RGerling KAbeele V(2024)Elevating Game User Research with a Guided Interface for Data AnalysisProceedings of the 19th International Conference on the Foundations of Digital Games10.1145/3649921.3650015(1-10)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3649921.3650015
Deng XWang YSu JLi Z(2024)A Chinese Version of the Aesthetics Scale: Cultural Adaptation and Psychological MeasurementInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2352214(1-15)Online publication date: 16-May-2024
https://doi.org/10.1080/10447318.2024.2352214
Olayiwola JYusuf AAkanmu AGonsalves NAbraham Y(2024)Efficacy of Annotated Video-Based Learning Environment for Drawing Students’ Attention to Construction Practice ConceptsJournal of Construction Engineering and Management10.1061/JCEMD4.COENG-13778150:1Online publication date: Jan-2024
https://doi.org/10.1061/JCEMD4.COENG-13778
Show More Cited By

Index Terms

The state of the art in automating usability evaluation of user interfaces
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. Interaction paradigms
      1. Graphical user interfaces

Recommendations

Automating UI guidelines verification by leveraging pattern based UI and model based development
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

In large enterprises different teams work on different parts of a big software application. Therefore, retaining user interaction paradigms and concepts becomes important. However, during the development of a large software product, these principles and ...
A Dialog-Oriented User Interface Generation Mechanism
APSEC '96: Proceedings of the Third Asia-Pacific Software Engineering Conference

Nowadays, for GUI application development, there are a number of interface builders, which make it possible for the user to create user interfaces easily, and UIMSs, which help to specify and design user interfaces. However, interface builders lack the ...
Usability of user interfaces: from monomodal to multimodal
BCS-HCI '07: Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI...but not as we know it - Volume 2

This workshop is aimed at reviewing and comparing existing Usability Evaluation Methods (UEMs) which are applicable to monomodal and multimodal applications, whether they are web-oriented or not. It addresses the problem on how to assess the usability ...

Reviews

Reviewer: Alan M Arnfeld

Ivory and Hearst have brought together an excellent and comprehensive review of automated usability evaluation techniques. Usability evaluation can be expensive, and the ability to support this dimension of business and product development with other cost effective solutions is extremely valuable. The authors stress that automated methods should be used in conjunction with traditional usability methods, and should not replace them. This paper analyzes a wide range of usability methods, and presents a taxonomy of 132 methods, split by applicability to either WIMP (windows, icons, pointer, and mouse) interfaces or Web interfaces. It also indicates which can be used for both. The methods are structured into five categories: testing, inspection, inquiry, analytical modeling, and simulation. One of the valuable contributions of this paper is bringing information on all these different methods together. This is especially true for simulation methods, which are often kept quite separate. For each area, the authors consider the current state of the art, and discuss the extent to which automation has been achieved. A full set of paper references is provided, allowing the interested reader to follow up with the developers of each technique. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ACM Computing Surveys Volume 33, Issue 4

December 2001

132 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/503112

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2001

Published in CSUR Volume 33, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

566
Total Citations
View Citations
21,922
Total Downloads

Downloads (Last 12 months)234
Downloads (Last 6 weeks)30

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Haider ARomanus TTang RGerling KAbeele V(2024)Elevating Game User Research with a Guided Interface for Data AnalysisProceedings of the 19th International Conference on the Foundations of Digital Games10.1145/3649921.3650015(1-10)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3649921.3650015
Deng XWang YSu JLi Z(2024)A Chinese Version of the Aesthetics Scale: Cultural Adaptation and Psychological MeasurementInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2352214(1-15)Online publication date: 16-May-2024
https://doi.org/10.1080/10447318.2024.2352214
Olayiwola JYusuf AAkanmu AGonsalves NAbraham Y(2024)Efficacy of Annotated Video-Based Learning Environment for Drawing Students’ Attention to Construction Practice ConceptsJournal of Construction Engineering and Management10.1061/JCEMD4.COENG-13778150:1Online publication date: Jan-2024
https://doi.org/10.1061/JCEMD4.COENG-13778
(2024)BibliographyHuman-Computer Interaction10.1016/B978-0-44-314096-9.00015-4(391-429)Online publication date: 2024
https://doi.org/10.1016/B978-0-44-314096-9.00015-4
MacKenzie I(2024)Scientific foundationsHuman-Computer Interaction10.1016/B978-0-44-314096-9.00010-5(161-201)Online publication date: 2024
https://doi.org/10.1016/B978-0-44-314096-9.00010-5
Mosqueira-Rey EHernández-Pereira EBobes-Bascarán JAlonso-Ríos DPérez-Sánchez AFernández-Leal ÁMoret-Bonillo VVidal-Ínsua YVázquez-Rivera F(2024)Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approachNeural Computing and Applications10.1007/s00521-023-09197-236:5(2597-2616)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s00521-023-09197-2
Ciancarini PNakatsu RRauterberg M(2024)Entertainment Computing: Past, Present, and FutureHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_34-1(1-21)Online publication date: 2-Apr-2024
https://doi.org/10.1007/978-3-319-27648-9_34-1
Tao YNandigam J(2024)Discovering Unanticipated Uses of Interactive Applications to Improve Usability and Enhance FunctionalityDesign, User Experience, and Usability10.1007/978-3-031-61362-3_15(193-202)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61362-3_15
Venter Ade Vries M(2024)Evaluating the Usability of Online Tools During Participatory Enterprise Modelling, Using the Business Model CanvasHCI in Business, Government and Organizations10.1007/978-3-031-61318-0_8(96-114)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61318-0_8
Klein ADeutschländer JKölln KRauschenberger MEscalona M(2024)Exploring the context of use for voice user interfacesJournal of Software: Evolution and Process10.1002/smr.261836:7Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1002/smr.2618
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Automating UI guidelines verification by leveraging pattern based UI and model based development

A Dialog-Oriented User Interface Generation Mechanism

Usability of user interfaces: from monomodal to multimodal

Reviews

Access critical reviews of Computing literature here