Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/347642.347766acmconferencesArticle/Chapter ViewAbstractPublication PagesdisConference Proceedingsconference-collections
Article
Free access

On the contributions of different empirical data in usability testing

Published: 01 August 2000 Publication History

Abstract

Many sources of empirical data can be used to evaluate an interface (e.g., time to learn, time to perform benchmark tasks, number of errors on benchmark tasks, answers on questionnaires, comments made in verbal protocols). This paper examines the relative contributions of both quanti?ta?tive and qualitative data gathered during a usability study. For each usability problem uncovered by this study, we trace each contributing piece of evidence back to its empirical source. For this usability study, the verbal protocol provided the sole source of evidence for more than one third of the most severe problems and more than two thirds of the less severe problems. Thus, although the verbal protocol provided the bulk of the evidence, other sources of data contributed disproportionately to the more critical problems. This work suggests that further research is required to determine the relative value of different forms of empirical evidence.

References

[1]
Card, S. K., T. P. Moran, and A. Newell. "The Keystroke- Level Model for User Performance Time with Interactive Systems." Communications of the ACM 23,396-410 (1980).]]
[2]
Cuomo, D. L. and C. D. Bowen. "Understanding Usability Issues Address by Three User-system Interface Evaluation Techniques." Interacting with Computers, 6(1), 86-108 (1994).]]
[3]
Desurvire, H. W. "Faster! Cheaper!! Are usability inspection methods as efficient as empirical testing?" In Jakob Nielsen and Robert L. Mack, eds., Usability Inspection Methods. John Wiley, NY (1994).]]
[4]
Dumas, J. R. and J. C. Redish. A Practical Guide to Usability Testing. Ablex Publishing Corporation, Norwood NJ (1993).]]
[5]
Ebling, M. R. "Translucent Cache Management for Mobile Computing." Ph.D. Dissertation. Carnegie Mellon University (1998).]]
[6]
Ericsson, K. A. and H. Simon. "Verbal Reports as Data." Psychological Review 87(3), 215-51 (May 1980).]]
[7]
Fink, A. and J. Kosecoff. How to Conduct Surveys: A Stepby-Step Guide. Sage Publications, Newbury Park, CA (1985).]]
[8]
Gray, W. D. and M. C. Salzman. "Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods." Human-Computer Interaction 13(3), 203-261 (1998).]]
[9]
Jacobsen, N. E., M. Hertzum, B. E. John. "The Evaluator Effect in Usability Studies: Problem Detection and Severity Judgements." Proceedings of the Human Factors and Ergonomics Society 42na Annual Meeting. HFES, 1336-1340 (1998).]]
[10]
Jeffries, R., J. R. Miller, C. Wharton, and K. M. Uyeda. "User interface evaluation in the real world: A comparison of four techniques", Proceedings of CHI, (1991).]]
[11]
John, B. E. and S. J. Marks. "Tracking the Effectiveness of Usability Evaluation Methods." Behaviour & Information Technology 16(4/5), 188-202 (1997).]]
[12]
John, B. E., and M. M. Mashyna. "Evaluating a Multimedia Authoring Tool with Cognitive Walkthrough and Think- Aloud User Studies." Journal of the American Society of Information Science. 48(9), (1997).]]
[13]
Jorgensen, A. H. "Using the thinking-aloud method in system development." In Salvendy, G., and Smith, M. J. (Eds.), Designing and Using Human-Computer Interfaces and Knowledge Based Systems. Elsevier Science Publishers, Amsterdam, 743-750 (1989).]]
[14]
Karat, C.-M., R. Campbell, and T. Fiegel. "Comparison of empirical testing and walkthrough methods in user interface evaluation." In Proceedings of CHI, 397-404 (May 1992).]]
[15]
Kistler, J. and M. Satyanarayanan. "Disconnected Operation in the Coda File System." ACM Transactions on Computers Systems. 10(1), 3-25 (February 1992).]]
[16]
Lewis, C. "Using the 'thinking-aloud' method in cognitive interface design." Research Report RC9265, IBM T. J. Watson Research Center, Yorktown Heights, NY (1982).]]
[17]
Mummert, L. B., M. R. Ebling, M. Satyanarayanan. "Exploiting Weak Connectivity for Mobile File Access." In Proceedings of the Fifteenth A CM Symposium on Operating Systems Principles, 143-155 (December 1995).]]
[18]
Newell, A. and H. A. Simon. Human Problem Solving. Prentice-Hall, Englewood Cliffs, NJ (1972).]]
[19]
Nielsen, J. "Evaluating the thinking aloud technique for use by computer scientists." In Hartson, H. R., and Hix, D. (Eds.), Advances in Human-Computer Interaction, Vol. 3, Ablex, Norwood, NJ, 69-82 (1992).]]
[20]
Nielsen, J. Usability Engineering. AP Professional, NY (1993).]]
[21]
Nielsen, J. and V. L. Phillips. "Estimating the relative usability of two interfaces: Heuristic, formal, and empirical methods compared." In Proceedings of lNTERCHI'93, 214- 221 (1993).]]
[22]
Satyanarayanan, M., J. J. Kistler, P. Kumar, M. E. Okasaki, E. H. Siegel, and D. C. Steere. "Coda: A Highly Available File System for a Distributed Workstation Environment." IEEE Transactions on Computers. 39(4), 447-59 (April 1990).]]

Cited By

View all
  • (2020)Comparing Two Methods of Usability Testing in Saudi Arabia: Concurrent Think-Aloud vs. Co-DiscoveryInternational Journal of Human–Computer Interaction10.1080/10447318.2020.180915237:2(118-130)Online publication date: 26-Aug-2020
  • (2020)A Comparative Usability Study of Blackboard and Desire2Learn: Students’ PerspectiveLearning and Collaboration Technologies. Designing, Developing and Deploying Learning Experiences10.1007/978-3-030-50513-4_1(3-19)Online publication date: 10-Jul-2020
  • (2018)Rethinking Thinking AloudProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173618(1-12)Online publication date: 21-Apr-2018
  • Show More Cited By

Index Terms

  1. On the contributions of different empirical data in usability testing

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    DIS '00: Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques
    August 2000
    456 pages
    ISBN:1581132190
    DOI:10.1145/347642
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 01 August 2000

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. empirical data
    2. usability testing
    3. verbal protocol

    Qualifiers

    • Article

    Conference

    DIS00
    Sponsor:
    DIS00: Designing Interactive Systems 2000
    August 17 - 19, 2000
    New York, New York City, USA

    Acceptance Rates

    DIS '00 Paper Acceptance Rate 48 of 127 submissions, 38%;
    Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)39
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 10 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2020)Comparing Two Methods of Usability Testing in Saudi Arabia: Concurrent Think-Aloud vs. Co-DiscoveryInternational Journal of Human–Computer Interaction10.1080/10447318.2020.180915237:2(118-130)Online publication date: 26-Aug-2020
    • (2020)A Comparative Usability Study of Blackboard and Desire2Learn: Students’ PerspectiveLearning and Collaboration Technologies. Designing, Developing and Deploying Learning Experiences10.1007/978-3-030-50513-4_1(3-19)Online publication date: 10-Jul-2020
    • (2018)Rethinking Thinking AloudProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173618(1-12)Online publication date: 21-Apr-2018
    • (2017)The ABCD of usability testingProceedings of the 31st British Computer Society Human Computer Interaction Conference10.14236/ewic/HCI2017.3(1-4)Online publication date: 3-Jul-2017
    • (2017)Further on down the digital road: Narrative design and reading pleasure in five New Media Writing Prize narrativesConvergence: The International Journal of Research into New Media Technologies10.1177/135485651772660326:1(35-54)Online publication date: 29-Aug-2017
    • (2015)Developing a Self-Report-Based Sequential Analysis Method for Educational Technology SystemsJournal of Educational Computing Research10.1177/073563311559077253:1(95-123)Online publication date: 23-Jul-2015
    • (2015)Look Who's Talking: Evaluating the Utility of Interventions During an Interactive Think-AloudInteracting with Computers10.1093/iwc/iwv01428:3(387-403)Online publication date: 8-Jun-2015
    • (2014)The state of user experience evaluation practiceProceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational10.1145/2639189.2641208(93-102)Online publication date: 26-Oct-2014
    • (2011)Identifying Usability Issues for Personalization During Formative Evaluations: A Comparison of Three MethodsInternational Journal of Human-Computer Interaction10.1080/10447318.2011.55530427:7(670-698)Online publication date: Jul-2011
    • (2010)Where Do We Go From Here? Readers' Responses to Interactive FictionConvergence: The International Journal of Research into New Media Technologies10.1177/135485650934877416:1(75-94)Online publication date: 19-Feb-2010
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media