Article

Supporting cognitive walkthrough with video data: a mobile learning evaluation study

Authors:

Silvia Gabrielli,

Valeria Mirabella,

Stephen Kimani,

Tiziana CatarciAuthors Info & Claims

MobileHCI '05: Proceedings of the 7th international conference on Human computer interaction with mobile devices & services

Pages 77 - 82

https://doi.org/10.1145/1085777.1085791

Published: 19 September 2005 Publication History

Abstract

Although expert-based evaluation techniques such as heuristic evaluation and cognitive walkthrough are often inexpensive and quick to apply, they have not proved to be effective in capturing contextual factors that arise in real-world settings. It is no trivial issue to understand how such evaluation techniques could be modified or differently applied so as to better take into account context, without loosing the advantages inherent in those techniques. This paper explores a possible way of addressing the trade-off between application of cognitive walkthrough and low cost improvements of its contextual validity. In particular, we propose and investigate the benefits of supporting cognitive walkthrough with video data about user interaction with an eLearning course on mobile device. Initial results from this study indicated that video data provided evaluators with a more detailed understanding of user characteristics and interaction contexts, leading to an improvement of their assessments in terms of the total number of system's flaws detected. Video data was regarded by experts as both relevant and useful, especially for tuning the evaluation focus on types of difficulties they would normally not have experienced because of differences in terms of abilities, knowledge and background with those of the target user group.

References

[1]

Abowd, G. and Mynatt, E. Charting past, present and future research in ubiquitous computting. ACM Transactions on Computer-Human Interaction, Vol.7(1), 2000, 29--58.

Digital Library

[2]

Benyon, D., Turner, P. and Turner, S. Designing Interactive Systems . London: Addison Wesley, 2005.

[3]

Buisson, M. and Jestin, Y. In M D Dunlop and S A Brewster (eds): Proceedings of Mobile HCI 2001: 3rd Int. Workshop on IHM-HCI 2001. Design issues in distributed interaction supporting tools: mobile devices in an ATC Working Position., Lille, France, Sept. 2001, http://www.tls.cena.fr/divisions/PII/Rapports/NR01-764.pdf

[4]

Cockton, G., Lavery, D., Woolrych, A. In J. A. Jacko & A. Sears (Eds.), The HCI Handbook, Lawrence Erl. Assoc., 2003. Inspection-based Evaluations, 2003, 1118--1138.

Digital Library

[5]

Dimitrova, M., Sharp H. and Wilson, S.: Are Experts Able to Predict Learner Problems during Usability Evaluations? World Conference on. Educational Multimedia, Hypermedia and Telecommunications, 2001(1), 419--424.

[6]

Dimitrova, M., Sharp H. and Wilson, S. Educational Multimedia Cognitive Walkthrough: Supporting Experts to Predict Valid User Problems. In Proceedings of Human Computer Interaction 2002, September 2-6, London, vol. 2 2002, 26--29.

[7]

Johnson, P. Usability and mobility; interaction on the move. Proc. First Workshop on Human-Computer Interaction with Mobile Devices, Glasgow, UK, 1998.

[8]

Kjeldskov, J. and Graham, C. A review of MobileHCI research methods. Proc. Mobile HCI 2003, Udine, Italy, 2003.

[9]

Mackay, W.E., Ratzer, A.V. and Janecek, P.: Video artefacts for design: Bridging the gap between abstraction and detail. In Symposium on Designing Interactive Systems, 2000, 72--82.

Digital Library

[10]

Magoulas, G. D., Chen, S. Y., and Papanikolaou, K. A: Integrating Layered and Heuristic Evaluation for Adaptive Learning Environments. In Proc. of the 2nd Workshop on Empirical Evaluation of Adaptive Systems, 9th International Conference on User Modeling, 2003, University of Pittsburgh, 2003.

[11]

Nielsen, J. Heuristic Evaluation. In Nielsen and Mack (eds.), Usability Inspection Methods. Wiley, 1994, 25--62.

Digital Library

[12]

Novick, D.: Using the cognitive walkthrough for operating procedures, Interactions, 6(3), 1999, 31--37.

Digital Library

[13]

Po, S., Howard, S., Vetere, F., Skov M.B. In S. Brewster and M. Dunlop (eds.) Proc. of Mobile HCI 2004. Heuristic Evaluation and Mobile Usability: Bridging the Realism Gap, 2004, 49--60.

[14]

Spencer, R.: The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company. CHI Letters, 2(1), 2000, 353--359.

Digital Library

[15]

Suchman, L.A. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge: Cambridge Press, 1987.

Digital Library

[16]

Wharton, C., Rieman, J., Lewis, C. & Polson, P. In: J. Nielsen & R. Mack (Eds.), Usability inspection methods New York: John Wiley, 1994. The cognitive walkthrough method: A practitioner's guide., 1994, 105--140.

Digital Library

Cited By

Robertson IMunson BStephenson JAmick R(2022)Developing and Validating Measurement Scales During Pandemic Conditions: A Case Study with the Scale for Habitat UsabilityProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/107118132266142066:1(677-681)Online publication date: 27-Oct-2022
https://doi.org/10.1177/1071181322661420
Shanklin RKortum PRobertson I(2022)Rating Product Satisfaction by Watching Others Using VideoProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/107118132266139266:1(1003-1007)Online publication date: 27-Oct-2022
https://doi.org/10.1177/1071181322661392
Robertson IKortum P(2020)Validity of three discount methods for measuring perceived usabilityJournal of Usability Studies10.5555/3532747.353275016:1(13-28)Online publication date: 1-Nov-2020
https://dl.acm.org/doi/10.5555/3532747.3532750
Show More Cited By

Index Terms

Supporting cognitive walkthrough with video data: a mobile learning evaluation study
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Applying cognitive walkthroughs to more complex user interfaces: experiences, issues, and recommendations
CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

The Cognitive Walkthrough methodology was developed in an effort to bring cognitive theory closer to practice; to enhance the design and evaluation of use interfaces in industrial settings. For the first time, small teams of professional developers have ...
Cognitive walkthrough for the web
CHI '02: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

This paper proposes a transformation of the Cognitive Walkthrough (CW), a theory-based usability inspection method that has proven useful in designing applications that support use by exploration. The new Cognitive Walkthrough for the Web (CWW) is ...
Experimenting on the cognitive walkthrough with users
MobileHCI '14: Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services

This paper presents a case study aiming to investigate which variant of the Think-Aloud Protocol (i.e., the Concurrent Think-Aloud and the Retrospective Think-Aloud) better integrates with the Cognitive Walkthrough with Users. To this end we performed a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

MobileHCI '05: Proceedings of the 7th international conference on Human computer interaction with mobile devices & services

September 2005

400 pages

ISBN:1595930892

DOI:10.1145/1085777

General Chair:
Manfred Tscheligi
University of Salzburg, Austria
,
Program Chairs:
Regina Bernhaupt
University of Salzburg, Austria
,
Kristijan Mihalic
University of Salzburg, Austria

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 September 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

MobileHCI05

MobileHCI05: 7th International Conference on Human-Computer Interaction with Mobile Devices and Services

September 19 - 22, 2005

Salzburg, Austria

Acceptance Rates

Overall Acceptance Rate 202 of 906 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
1,405
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Robertson IMunson BStephenson JAmick R(2022)Developing and Validating Measurement Scales During Pandemic Conditions: A Case Study with the Scale for Habitat UsabilityProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/107118132266142066:1(677-681)Online publication date: 27-Oct-2022
https://doi.org/10.1177/1071181322661420
Shanklin RKortum PRobertson I(2022)Rating Product Satisfaction by Watching Others Using VideoProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/107118132266139266:1(1003-1007)Online publication date: 27-Oct-2022
https://doi.org/10.1177/1071181322661392
Robertson IKortum P(2020)Validity of three discount methods for measuring perceived usabilityJournal of Usability Studies10.5555/3532747.353275016:1(13-28)Online publication date: 1-Nov-2020
https://dl.acm.org/doi/10.5555/3532747.3532750
Owda MOwda AGasir F(2020)A Comprehensive Methodology for Evaluating Conversation-Based Interfaces to Relational Databases (C-BIRDs)Intelligent Systems and Applications10.1007/978-3-030-55187-2_17(196-208)Online publication date: 25-Aug-2020
https://doi.org/10.1007/978-3-030-55187-2_17
Billi MBurzagli LCatarci TSantucci GBertini EGabbanini FPalchetti E(2018)A unified methodology for the evaluation of accessibility and usability of mobile applicationsUniversal Access in the Information Society10.1007/s10209-009-0180-19:4(337-356)Online publication date: 20-Dec-2018
https://dl.acm.org/doi/10.1007/s10209-009-0180-1
Leimeister JLeimeister J(2014)Designvalidierung (Design Validation)Collaboration Engineering10.1007/978-3-662-46066-5_11(283-312)Online publication date: 15-Apr-2014
https://doi.org/10.1007/978-3-662-46066-5_11
Leimeister JLeimeister J(2014)Designvalidierung (Design Validation)Collaboration Engineering10.1007/978-3-642-20891-1_11(283-312)Online publication date: 15-Apr-2014
https://doi.org/10.1007/978-3-642-20891-1_11
Cole I(2013)Usability of Online Virtual Learning EnvironmentsStudent Usability in Educational Software and Games10.4018/978-1-4666-1987-6.ch002(41-58)Online publication date: 2013
https://doi.org/10.4018/978-1-4666-1987-6.ch002
Catarci TDix AKimani SSantucci G(2010)User-Centered Data ManagementSynthesis Lectures on Data Management10.2200/S00285ED1V01Y201006DTM0062:1(1-106)Online publication date: Jan-2010
https://doi.org/10.2200/S00285ED1V01Y201006DTM006
Cabrall CMartin LLee PJobe K(2009)Exploration of human factors issues within a future separation assurance concept2009 IEEE/AIAA 28th Digital Avionics Systems Conference10.1109/DASC.2009.5347469(5.B.3-1-5.B.3-12)Online publication date: Oct-2009
https://doi.org/10.1109/DASC.2009.5347469

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents