research-article

Development of a framework for UX KPIs in Industry - a case study: ABSTRACT

Authors:

Lars Bo Larsen,

Lucca Julie Nellemann,

Jędrzej CzaplaAuthors Info & Claims

OzCHI '20: Proceedings of the 32nd Australian Conference on Human-Computer Interaction

Pages 141 - 147

https://doi.org/10.1145/3441000.3441042

Published: 15 February 2021 Publication History

Abstract

This work addresses the needs companies face when assessing and tracking the UX quality of their products, which is necessary to ensure that the desired level of quality is met, clarify potential areas of improvement, and compare competitor products via benchmarking.

We present a framework for UX KPI assessment at Bang & Olufsen, a luxury audio product manufacturer. The framework is inspired by well-known UX scales such as the UEQ and AttrakDiff as well as more business-oriented measures such as NPS and CES scores. The resulting UX KPI framework comprises the test procedures and a scale containing these and ten unique features such as "Enjoyable”, “Exciting”, “Sound experience".

The paper presents and discusses the UX KPI framework and the rationales behind it. Results of more than 200 user tests are presented and discussed. Factor analysis has been employed to analyse the assumptions behind the UX KPI and identify where improvements can be made.

References

[1]

Bang and Olufsen: https://www.bang-olufsen.com (retrieved August 2020)

[2]

Google Voice Assistant https://assistant.google.com/ (retrieved August 2020)

[3]

Alexa Voice Assistant https://developer.amazon.com/en-US/alexa/alexa-voice-service (retrieved August 2020)

[4]

B&O App. Apple IOS version: https://apps.apple.com/us/app/bang-olufsen/id1203736217 (retrieved August 2020)

[5]

B&O App. Android version: https://play.google.com/store/apps/details?id=com.bang_olufsen.OneApp&hl=en (retrieved August 2020)

[6]

UEQ: Website: https://www.ueq-online.org/ (retrieved August 2020)

[7]

Laugwitz B., Held T., Schrepp M. (2008) Construction and Evaluation of a User Experience Questionnaire. In: Holzinger A. (eds) HCI and Usability for Education and Work. USAB 2008. Lecture Notes in Computer Science, vol 5298. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-89350-9_6

[8]

Schrepp, M., Hinderks, A., and Thomaschewski, J., 2017. Construction of a Benchmark for the User Experience Questionnaire (UEQ). International Journal of Interactive Multimedia and Artificial Intelligence, 4(4), pp. 40–44. ISSN 1989-1660.

[9]

SUS: Brooke, J., 1996. SUS - A quick and dirty usability scale. Usability evaluation in industry, 09, 189(194), pp. 4–7.

[10]

Attrakdiff: Website: http://www.attrakdiff.de/index-en.html (retrieved August 2020)

[11]

Hassenzahl, M., Burmester, M., and Koller, F., 2008. On the track of user experience. Available at: <https://www.uid.com/en/publications/attrakdiff>. (retrieved August 2020)

[12]

ISO 9241-11:2018. Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. Available at: https://www.iso.org/standard/63500.html (retrieved August 2020)

[13]

Reichheld, Frederick F. "One Number You Need to Grow". Harvard Business Review December 2003. Available at: https://hbr.org/2003/12/the-one-number-you-need-to-grow (retrieved August 2020).

[14]

Reichheld, Fred; Markey, Rob (2011). The Ultimate Question 2.0: How Net Promoter Companies Thrive in a Customer-Driven World. Boston, Mass.: Harvard Business Review Press. p. 52- ISBN 978-1-4221-7335-0. Available at: https://archive.org/details/ultimatequestion00reic_0/page/52 (retrieved August 2020)

[15]

Net Promoter Score: https://www.netpromoter.com (retrieved August 2020)

[16]

Faulkner, L. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers 35, 379–383 (2003). https://doi.org/10.3758/BF03195514

[17]

Nielsen, J., Landauer, K. ”A mathematical model of the finding of usability problems” in CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems. May 1993 Pages 206–213.

Digital Library

[18]

Matthew Dixon, Karen Freeman and Nicholas Toman. Stop Trying to Delight Your Customers, Harvard Business Review Press. July–August 2010. Available from: https://hbr.org/2010/07/stop-trying-to-delight-your-customers (retrieved August 2020).

[19]

Farris, Paul W.; Neil T. Bendle; Phillip E. Pfeifer; David J. Reibstein (2010). Marketing Metrics: The Definitive Guide to Measuring Marketing Performance. Upper Saddle River, New Jersey: Pearson Education, Inc. ISBN 0-13-705829-2.

Digital Library

[20]

Mittal, Vikas and Frennea, Carly, Customer Satisfaction: A Strategic Review and Guidelines for Managers (2010). MSI Fast Forward Series, Marketing Science Institute, Cambridge, MA, 2010, Available at SSRN: https://ssrn.com/abstract=2345469 (retrieved August 2020)

[21]

Tabachick, Barbara and Fidell Using Multivariate Statistics. 4 th editon. Allyn and Bacon, 2001. ISBN 0-321-05677-9

[22]

R Core Team (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org/ (retrieved August 2020)

[23]

James Dean Brown: “Choosing the Right Type of Rotation in PCA and EFA”, in Shiken: JALT Testing & Evaluation SIG Newsletter. 13 (3) November 2009 (p. 20 - 25)

[24]

Grant, S.; Aitchison, T.; Henderson, E.; Christie, J.; Zare, S.; McMurray, J.; Dargie, H. (1999). "A comparison of the reproducibility and the sensitivity to change of visual analogue scales, Borg scales, and Likert scales in normal subjects during submaximal exercise". Chest. 116 (5): 1208–17.

Cited By

Almeida FCanedo Ede Oliveira Albuquerque Rde Deus FSandoval Orozco AGarcía Villalba L(2022)A Model for the Definition, Prioritization and Optimization of IndicatorsElectronics10.3390/electronics1106096711:6(967)Online publication date: 21-Mar-2022
https://doi.org/10.3390/electronics11060967

Recommendations

UX trek: a post-interaction journey from immersive experience logs
IHC '20: Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems

User Experience (UX) is a holistic attribute focused on the quality of user-product interaction. However, the technology evolution in different contexts has not imposed limits on user-product interactions. For instance, in immersive technologies, the ...
Determining and validating smart TV UX factors: A multiple-study approach
Highlights
- We identified a comprehensive set of smart TV UX factors via think-aloud and diary studies.
Abstract
Television (TV) remains one of the most important media channels in our daily lives. Smart TVs support highly interactive functions between the TV and users, offer Internet connections, and run various applications. Compared with ...
The Development of a Framework for Understanding the UX of Subtitles
ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility

Approximately 10% of the television audience use subtitles (captioning) to support their viewing experience. Subtitles enable viewers to participate in an experience that is often taken for granted by the general audience. However, when reviewing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

OzCHI '20: Proceedings of the 32nd Australian Conference on Human-Computer Interaction

December 2020

764 pages

ISBN:9781450389754

DOI:10.1145/3441000

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

OzCHI '20

OzCHI '20: 32nd Australian Conference on Human-Computer-Interaction

December 2 - 4, 2020

NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 362 of 729 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
190
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)2

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Almeida FCanedo Ede Oliveira Albuquerque Rde Deus FSandoval Orozco AGarcía Villalba L(2022)A Model for the Definition, Prioritization and Optimization of IndicatorsElectronics10.3390/electronics1106096711:6(967)Online publication date: 21-Mar-2022
https://doi.org/10.3390/electronics11060967

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents