research-article

How relevant is an expert evaluation of user experience based on a psychological needs-driven approach?

Authors:

Carine Lallemand,

Vincent Koenig,

Guillaume GronierAuthors Info & Claims

NordiCHI '14: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational

Pages 11 - 20

https://doi.org/10.1145/2639189.2639214

Published: 26 October 2014 Publication History

Abstract

Many methods and tools have been proposed to assess the User Experience (UX) of interactive systems. However, while researchers have empirically studied the relevance and validity of several UX evaluation methods, few studies only have explored expert-based evaluation methods for the assessment of UX. If experts are able to assess something as complex and inherently subjective as UX, how they conduct such an evaluation and what criteria they rely on, thus remain open questions. In the present paper we report on 33 UX experts performing a UX evaluation on 4 interactive systems. We provided the experts with UX Cards, a tool based on a psychological-needs driven approach, developed to support UX Design and Evaluation. Results are encouraging and show that UX experts encountered no major issues to conduct a UX evaluation. However, significant differences exist between individual elements that experts have reported on and the overall assessment they made of the systems.

References

[1]

Arhippainen, L. (2013). A Tutorial of Ten User Experience Heuristics. In Proc. AcademicMindTrek '13, Tampere, Finland.

Digital Library

[2]

Bach, C. (2004). Elaboration et validation de critères ergonomiques pour les interactions homme-environnements virtuels. Thèse de Doctorat. Université Paul Verlaine, Metz.

[3]

Cockton, G. and Woolrych, A. (2001). Understanding inspection methods: lessons from an assessment of heuristic evaluation. In: A. Blandford, J. Vanderdonckt, and P. D. Gray, eds. Proceedings of people and computers XV: joint proceedings of HCI 2001 and IHM 2001. Berlin: Springer-Verlag, 171--192

[4]

Colombo, L. and Pasch, M. (2012). 10 Heuristics for an Optimal User Experience. Proc. CHI2012 Altchi. ACM Press.

[5]

Csikszentmihalyi, M. (1990). Flow -- The Psychology of Optimal Experience. Steps Toward Enhancing the Quality of Life. New York, Harper Perennial.

[6]

Deci, E. L., & Ryan, R. M. (2000). The 'what' and 'why' of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11, 227--268.

[7]

Dillon, A. (2001) Usability evaluation. In W. Karwowski (ed.) Encyclopedia of Human Factors and Ergonomics, London: Taylor and Francis.

[8]

Dix, A., Finlay, J., Abowd, G. & Beale, R. (2004). Human-Computer Interaction. Prentice Hall.

Digital Library

[9]

Hassenzahl, M., Eckoldt, K., Diefenbach, S., Laschke, M., Lenz, E., & Kim, J. (2013). Designing moments of meaning and pleasure. Experience design and happiness. International Journal of Design, 7(3), 21--31

[10]

Hassenzahl, M. (2013). Experiences Before Things: a Primer for the (Yet) Unconvinced. Proc. Of CHI '13 Extended Abstracts, 2059--2068.

Digital Library

[11]

Hassenzahl, M., Diefenbach, S., & Göritz, A. (2010). Needs, affect, and interactive products -- Facets of user experience. Interacting with Computers, 22(5), 353--362.

Digital Library

[12]

Hassenzahl, M. (2010) Experience Design: Technology for All the Right Reasons. Synthesis Lectures on Human-Centered Informatics, 3, 1, 1--95.

[13]

Hassenzahl, M. (2008) User Experience (UX): Towards an experiential perspective on product quality. Proceedings of IHM'08, Metz, France.

Digital Library

[14]

Hertzum, M., & Jacobsen, N. E. (2001). The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. International Journal of Human Computer Interaction, 13(4), 421--443.

[15]

Hornbæk, K. (2010). Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology, 29: 1, 97 --- 111

Digital Library

[16]

Hvannberg, E. B., Law, E. & Larusdottir, M. K. (2007). Heuristic evaluation: Comparing ways of finding and reporting usability problems. Interacting with Computers, 19, 225--240.

Digital Library

[17]

Jordan, P. (2000). Designing pleasurable products: An introduction to the new human factors. CRC Press.

[18]

Kim, J., Park, S., Hassenzahl, M., & Eckoldt, K. (2011). The Essence of Enjoyable Experiences: The Human Needs - A Psychological Needs-Driven Experience Design Approach. In Proc. DUXU'2011.

[19]

Korhonen, H. and Koivisto, E. (2006). Playability heuristics for mobile games. Proc. MobileHCI'06, ACM Press, NY, USA, 9--16.

Digital Library

[20]

Malinen, S. & Ojala, J. (2011). Applying the Heuristic Evaluation Method in the Evaluation of Social Aspects of an Exercise Community. Proceedings of DPPI 2011.

Digital Library

[21]

Molich, R. & Jeffries, R. (2003). Comparative experts reviews. Proc. ACM CHI'EA 2003 Conference.

Digital Library

[22]

Nielsen, J. (1993). Usability Engineering. New York: AP Professional.

[23]

Nielsen, J. (1989). Usability engineering at a discount. In Salvendy, G., and Smith, M. J. (Eds), Designing and Using Human-Computer Interfaces and Knowledge Based Systems. Elsevier Science Publishers, Amsterdam, 394--401.

Digital Library

[24]

Nielsen, J. & Molich, R. (1990). Heuristic evaluation of user interfaces. Proc. ACM CHI'90 Conf., 249--156.

Digital Library

[25]

Roto, V., Obrist, M., Väänänen-Vainio-Mattila, K. User Experience Evaluation Methods in Academic and Industrial Contexts. In User Experience Evaluation Methods in Product Development (UXEM'09). Workshop in Interact'09, Sweden.

[26]

Pine, B. J. & Gilmore, J. H. (1998). Welcome to the Experience Economy: work is theatre & every business a stage. Havard Business Review, July-August 1998.

[27]

Sheldon, K. M. (2011). Integrating behavioral-motive and experiential-requirement perspectives on psychological needs: A two process model. Psychological Review, 118(4), 552--569.

[28]

Sheldon, K. M., Elliot, A. J., Kim, Y., & Kasser, T. (2001). What is satisfying about satisfying events? Testing 10 candidate psychological needs. Journal of Personality and Social Psychology, 89, 325--339.

[29]

Sheldon, K. M., Ryan, R. M., & Reis, H. T. (1996). What makes for a good day? Competence and autonomy in the day and in the person. Personality and Social Psychology Bulletin, 22, 1270--1279

[30]

Squires, D. & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11(5), 467--483

[31]

Tuch, A. N., Trusell, R. & Hornbæk, K. (2013). Analyzing Users' Narratives to Understand Experience with Interactive Products. In Proceedings CHI '13.

Digital Library

[32]

Väänänen-Vainio-Mattila, K., Wäljas, M. (2009). Developing an Expert Evaluation Method for User eXperience of Cross-Platform Web Services. Proceedings of Mindtrek'09, ACM

Digital Library

[33]

Väänänen-Vainio-Mattila, K., Wäljas, M. (2009). Development of Evaluation Heuristics for Web Service User Experience. A Work-in-Progress paper in Proceedings (Extended Abstracts) of CHI'09, ACM

Digital Library

[34]

Väänänen-Vainio-Mattila, K., Roto, V., Hassenzahl, M. Towards Practical User Experience Evaluation Methods. Proceedings of the 5th COST294-MAUSE Open Workshop on Meaningful Measures: VUUM 2008, Reykjavik, Iceland.

[35]

Vermeeren, A, Lai-Chong Law, E., Roto, V., Obrist, M., Hoonhout, J., Väänänen-Vainio-Mattila, K. User Experience Evaluation Methods: Current State and Development Needs. In Proc. of NordiCHI'10, ACM.

Digital Library

[36]

Wharton, C. Rieman, J. Lewis, C. and Polson, P. (1994) The Cognitive Walkthrough Method: A Practitioner's Guide. In J. Nielsen and R. Mack (eds.) Usability Inspection Methods (New York: Wiley) 105--140.

Digital Library

[37]

Wilson, C. (2011). Perspective-Based Inspection. 100 User Experience (UX) Design and Evaluation Methods for Your Toolkit. Available on http://dux.typepad.com/ (last verified 19 April 2013).

[38]

Yarosh, S., Matthews, T., and Zhou, M. (2012). Asking the Right Person: Supporting Expertise Selection in the Enterprise. Proc. Of ACM CHI 2012.

Digital Library

Cited By

Tyack AMekler E(2024)Self-Determination Theory and HCI Games Research: Unfulfilled Promises and Unquestioned ParadigmsACM Transactions on Computer-Human Interaction10.1145/367323031:3(1-74)Online publication date: 15-Jun-2024
https://dl.acm.org/doi/10.1145/3673230
Moradbakhti LLeichtmann BMara M(2024)Development and Validation of a Basic Psychological Needs Scale for Technology UsePsychological Test Adaptation and Development10.1027/2698-1866/a0000625:1(26-45)Online publication date: 1-Jan-2024
https://doi.org/10.1027/2698-1866/a000062
Chalhoub GKraemer MFlechais I(2024)Useful shortcutsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103177182:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103177
Show More Cited By

Index Terms

How relevant is an expert evaluation of user experience based on a psychological needs-driven approach?
1. Human-centered computing

Recommendations

Developing an expert evaluation method for user eXperience of cross-platform web services
MindTrek '09: Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era

Positive user experience (UX), including its pragmatic and hedonic aspects, is a central design target for interactive products and services. Increasingly, new types of Web services are developed for both PCs and mobile terminals to support user needs ...
Research on User Experience Evaluation System of Popular Science Games Based on Analytic Hierarchy Process
Design, User Experience, and Usability
Abstract
The main purpose of this study is to establish a systematic evaluation index system and conduct research on popular science games from the perspective of user experience. Firstly, the user experience index system of popular science games is ...
The ‘Angstfabriek’ Experience: Factoring Fear into Transformative Interactive Narrative Design
Interactive Storytelling
Abstract
Interactive Narratives in the form of Interactive Theater have the potential to offer a transformational learning experience on societal and political topics. The purposive interactive installation Angstfabriek (Dutch for fear factory) lets ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

NordiCHI '14: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational

October 2014

361 pages

ISBN:9781450325424

DOI:10.1145/2639189

General Chairs:
Virpi Roto,
Jonna Häkkilä,
Program Chairs:
Kaisa Väänänen-Vainio-Mattila,
Oskar Juhlin,
Publications Chairs:
Thomas Olsson,
Ebba Hvannberg

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

NordiCHI '14

NordiCHI '14: The 8th Nordic Conference on Human-Computer Interaction

October 26 - 30, 2014

Helsinki, Finland

Acceptance Rates

NordiCHI '14 Paper Acceptance Rate 89 of 361 submissions, 25%;

Overall Acceptance Rate 379 of 1,572 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
471
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)3

Reflects downloads up to 14 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tyack AMekler E(2024)Self-Determination Theory and HCI Games Research: Unfulfilled Promises and Unquestioned ParadigmsACM Transactions on Computer-Human Interaction10.1145/367323031:3(1-74)Online publication date: 15-Jun-2024
https://dl.acm.org/doi/10.1145/3673230
Moradbakhti LLeichtmann BMara M(2024)Development and Validation of a Basic Psychological Needs Scale for Technology UsePsychological Test Adaptation and Development10.1027/2698-1866/a0000625:1(26-45)Online publication date: 1-Jan-2024
https://doi.org/10.1027/2698-1866/a000062
Chalhoub GKraemer MFlechais I(2024)Useful shortcutsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103177182:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103177
Nakamura WMarques LRedmiles Dde Oliveira EConte T(2022)Investigating the Influence of Different Factors on the UX Evaluation of a Mobile ApplicationInternational Journal of Human–Computer Interaction10.1080/10447318.2022.210865839:20(3948-3968)Online publication date: 16-Aug-2022
https://doi.org/10.1080/10447318.2022.2108658
Lallemand C(2022)UX Needs Cards – A Pragmatic Tool to Support Experience Design Through Psychological Needs[ ] With Design: Reinventing Design Modes10.1007/978-981-19-4472-7_123(1892-1909)Online publication date: 6-Nov-2022
https://doi.org/10.1007/978-981-19-4472-7_123
Kaimara POikonomou ADeliyannis IPapadopoulou AMiliotis GFokides EFloros A(2021)Waking up in the morning (WUIM): A transmedia project for daily living skills trainingTechnology and Disability10.3233/TAD-20032633:2(137-161)Online publication date: 1-May-2021
https://doi.org/10.3233/TAD-200326
Meiners AKollmorgen JSchrepp MThomaschewski J(2021)Which UX Aspects Are Important for a Software Product?Proceedings of Mensch und Computer 202110.1145/3473856.3473997(136-139)Online publication date: 5-Sep-2021
https://dl.acm.org/doi/10.1145/3473856.3473997
Brombacher Hvan Koningsbruggen RMarkopoulos P(2021)How Students with different levels of Design Experience use PLEX Cards within the Brainstorming ProcessProceedings of the 32nd European Conference on Cognitive Ergonomics10.1145/3452853.3452854(1-6)Online publication date: 26-Apr-2021
https://dl.acm.org/doi/10.1145/3452853.3452854
Tyack AMekler EBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Self-Determination Theory in HCI Games Research: Current Uses and Open QuestionsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376723(1-22)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376723
Lockton DLallemand CBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Meeting Designers Where They Are: Using Industry Events as a Research Venue for HCI and Design Methods DevelopmentProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376206(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376206
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten