A Case Study Exploring the Applicability of Heuristic Evaluation in Smart Home Systems
Authors: 
Mengyu Chen, Shuang He, Xinyi FuAuthors Info & Claims
Article No.: 498, Pages 1 - 7
Abstract
Smart home systems, integrating multiple devices and interaction methods, give rise to complex usability challenges. While heuristic evaluation has been a preferred method for evaluating usability across various domains, its applicability to smart home systems remains underexplored. This case study investigates the strengths and constraints of heuristic evaluation applied to smart home systems. Our results suggests that while heuristic evaluation effectively identifies GUI issues and guides expert solutions, it falls short in addressing technical and hardware intricacies unique to hybrid systems like smart homes. This study not only presents recommendations for enhancing heuristic evaluations in this evolving domains, but also offers practical insights specifically for the evaluation of hybrid systems.
Supplemental Material
MP4 File
Talk Video
- Download
- 122.69 MB
- Transcript
MP4 File - Video Preview
Video Preview
- Download
- 9.66 MB
- Transcript
References
[1]
Yasemin Afacan and Çiğdem Erbuğ. 2008. An interdisciplinary heuristic evaluation method for universal building design. Applied ergonomics 40 (10 2008), 731–44. https://doi.org/10.1016/j.apergo.2008.07.002
[2]
Asmaa Alsumait and Asma Al-Osaimi. 2009. Usability heuristics evaluation for child e-learning applications. In Proceedings of the 11th international conference on information integration and web-based applications & services. 425–430.
[3]
Federico Botella, Eloy Alarcon, and Antonio Peñalver. 2014. How to Classify to Experts in Usability Evaluation. In Proceedings of the XV International Conference on Human Computer Interaction (Puerto de la Cruz, Tenerife, Spain) (Interacción ’14). Association for Computing Machinery, New York, NY, USA, Article 25, 4 pages. https://doi.org/10.1145/2662253.2662278
[4]
Richard E Boyatzis. 1998. Transforming qualitative information: Thematic analysis and code development. sage.
[5]
Michael Brown. 2008. Evaluating computer game usability: Developing heuristics based on user experience. In Proceedings of IHCI conference. 16–21.
[6]
A. Brush, Bongshin Lee, Ratul Mahajan, Sharad Agarwal, Stefan Saroiu, and Colin Dixon. 2011. Home Automation in the Wild: Challenges and Opportunities. CHI ’11 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2115–2124. https://doi.org/10.1145/1978942.1979249
[7]
Tayana Conte, Jobson Silva, Emilia Mendes, and Guilherme Travassos. 2007. Usability Evaluation Based on Web Design Perspectives, In First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007). Proceedings - 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 2007, 146–155. https://doi.org/10.1109/ESEM.2007.30
[8]
Statista Research Department. 2023. Number of users of smart homes worldwide from 2018 to 2027. Retrieved August 17, 2023 from https://www.statista.com/forecasts/887613/number-of-smart-homes-in-the-smart-home-market-in-the-world
[9]
Dean Dykstra. 1993. A comparison of heuristic evaluation and usability testing : the efficacy of a domain-specific heuristic checklist /. (01 1993).
[10]
Nicole Fink, Richard Pak, and Dina Battisto. 2010. Developing a Usability Evaluation Tool To Assess the Patient Room Bathroom. HERD 3 (04 2010), 22–41. https://doi.org/10.1177/193758671000300305
[11]
Xinyi Fu, He Zhang, Cheng Xue, Xinyang Li, Zhe Sun, and Yingqing Xu. 2022. Design Research and Application Practice of Integrated Experimental Platform for Smart Home. Packaging Engineering 43, 50-58+108 (2022). https://doi.org/10.19554/j.cnki.1001-3563.2022.16.005
[12]
Xinyi Fu, He Zhang, Cheng Xue, and Tongxin Sun. 2023. A Review of the Frontier Research on Future Smart Home. Science and Technology Review 41, 8, Article 36 (2023), 16 pages. https://doi.org/10.3981/j.issn.1000-7857.2023.08.004
[13]
Robin Jeffries, James R. Miller, Cathleen Wharton, and Kathy Uyeda. 1991. User Interface Evaluation in the Real World: A Comparison of Four Techniques. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New Orleans, Louisiana, USA) (CHI ’91). Association for Computing Machinery, New York, NY, USA, 119–124. https://doi.org/10.1145/108844.108862
[14]
Raina Langevin, Ross J Lordon, Thi Avrahami, Benjamin R. Cowan, Tad Hirsch, and Gary Hsieh. 2021. Heuristic Evaluation of Conversational Agents. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (, Yokohama, Japan, ) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 632, 15 pages. https://doi.org/10.1145/3411764.3445312
[15]
Yi Hsuan Liao and Chun-Yi Shen. 2012. Heuristic evaluation of digital game based learning: a case study. In 2012 IEEE Fourth international conference on digital game and intelligent toy enhanced learning. IEEE, 192–196.
[16]
Jie Liu, Dan Luo, Xinyi Fu, Qi Lu, and Karen Yixin Kang. 2022. Design Strategy of Multimodal Perception System for Smart Environment. In Internet of Things for Smart Environments. Springer, 93–115.
[17]
Xiaoyi Liu, Yingtian Shi, Chun Yu, Cheng Gao, Tianao Yang, Chen Liang, and Yuanchun Shi. 2023. Understanding In-Situ Programming for Smart Home Automation. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7 (06 2023), 1–31. https://doi.org/10.1145/3596254
[18]
Jeremy Lopez, Claire Textor, W Braxton Hicks, Makenzie Pryor, Anne Collins McLaughlin, and Richard Pak. 2019. An aging-focused heuristic evaluation of home automation controls. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 63. SAGE Publications Sage CA: Los Angeles, CA, 6–10.
[19]
Adriana Marques da Silva, Hande Ayanoglu, and Bruno MC Silva. 2020. An age-friendly system design for smart home: findings from heuristic evaluation. In HCI International 2020–Late Breaking Papers: Universal Access and Inclusive Design: 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings 22. Springer, 643–659.
[20]
Ana Isabel Martins, Ana Carolina Oliveira Lima, and Nelson Pacheco Rocha. 2021. Heuristic Evaluation Method Applied to the Usability Assessment of Smart Homes Applications. In World Conference on Information Systems and Technologies. Springer, 445–454.
[21]
Jakob Nielsen. 1992. Finding Usability Problems through Heuristic Evaluation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Monterey, California, USA) (CHI ’92). Association for Computing Machinery, New York, NY, USA, 373–380. https://doi.org/10.1145/142750.142834
[22]
Jakob Nielsen. 1992. Reliability of Severity Estimates for Usability Problems Found by Heuristic Evaluation. In Posters and Short Talks of the 1992 SIGCHI Conference on Human Factors in Computing Systems (Monterey, California) (CHI ’92). Association for Computing Machinery, New York, NY, USA, 129–130. https://doi.org/10.1145/1125021.1125117
[23]
Jakob Nielsen. 1994. 10 Usability Heuristics for User Interface Design. Retrieved November 15, 2020 from https://www.nngroup.com/articles/ten-usability-heuristics/
[24]
Jakob Nielsen. 1994. Usability Inspection Methods. In Conference Companion on Human Factors in Computing Systems (Boston, Massachusetts, USA) (CHI ’94). Association for Computing Machinery, New York, NY, USA, 413–414. https://doi.org/10.1145/259963.260531
[25]
Jakob Nielsen and Rolf Molich. 1990. Heuristic Evaluation of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’90). Association for Computing Machinery, New York, NY, USA, 249–256. https://doi.org/10.1145/97243.97281
[26]
Costin Pribeanu. 2009. A usability assistant for the heuristic evaluation of interactive systems. Studies in informatics and control 18, 4 (2009), 355–362.
[27]
Aiswarya R, Vaishali Anil, Jayasree Raveendran, Gijs Wijk, and Shantanu Talukdar. 2021. Living as a Service (LaaS): An Experiment to Evaluate Living Experience in Connected Homes in the Netherlands. 1–6. https://doi.org/10.1145/3411763.3451684
[28]
Nelson Pacheco Rocha. 2021. Heuristic Evaluation of the Usability of Smart Home Applications. Comprehensible Science: ICCS 315 (2021), 483.
[29]
Raymond Scupin. 1997. The KJ Method: A Technique for Analyzing Data Derived from Japanese Ethnology. Human Organization 56 (06 1997). https://doi.org/10.17730/humo.56.2.x335923511444655
[30]
Gavin Sim, Janet Read, and Phil Holifield. 2008. Heuristics for evaluating the usability of CAA applications. Computer Assisted Assessment Conference Proceedings, Loughborough, England (01 2008).
[31]
A. Sivaji, Azween Abdullah, and Alan Downe. 2011. Usability Testing Methodology: Effectiveness of Heuristic Evaluation in E-Government Website Development. In 2011 fifth Asia modelling symposium. 68 – 72. https://doi.org/10.1109/AMS.2011.24
[32]
Sarmad Soomro, Wan Fatimah Wan Ahmad, and Suziah Sulaiman. 2012. A preliminary study on heuristics for mobile games. In 2012 International Conference on Computer & Information Science (ICCIS), Vol. 2. IEEE, 1030–1035.
[33]
Benjamin K. Sovacool and Dylan D. Furszyfer Del Rio. [n. d.]. Smart home technologies in Europe: A critical review of concepts, benefits, risks and policies. 120 ([n. d.]), 109663. https://doi.org/10.1016/j.rser.2019.109663
[34]
OM-IE Sucar, SH Aviles, and C Miranda-Palma. 2003. From HCI to HRI-usability inspection in multimodal human-robot interactions. In The 12th IEEE International Workshop on Robot and Human Interactive Communication, 2003. Proceedings. ROMAN 2003. IEEE, 37–41.
[35]
KM Tsui, K Abu-Zahra, R Casipe, J M’Sadoques, and JL Drury. 2009. A process for developing specialized heuristics: Case study in assistive robotics (Technical Report No. 2009-11). Lowell: University of Massachusetts Lowell (2009).
[36]
Karl Ulrich. 2003. KJ diagrams. Opim. Wharton. Upenn. Edu (2003).
[37]
Blase Ur, Elyse McManus, Melwyn Pak Yong Ho, and Michael L. Littman. 2014. Practical Trigger-Action Programming in the Smart Home. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 803–812. https://doi.org/10.1145/2556288.2557420
[38]
Edvar Vilar Neto and Fábio FC Campos. 2014. Evaluating the usability on multimodal interfaces: a case study on tablets applications. In Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience: Third International Conference, DUXU 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014, Proceedings, Part I 3. Springer, 484–495.
[39]
Jong-bum Woo and Youn-kyung Lim. 2015. User Experience in Do-It-Yourself-Style Smart Homes. In Proceedings of the 2015 ACM International Joint Conference on Pervasiv e and Ubiquitous Computing (Osaka, Japan) (UbiComp ’15). Association for Computing Machinery, 779–790. https://doi.org/10.1145/2750858.2806063
Index Terms
- A Case Study Exploring the Applicability of Heuristic Evaluation in Smart Home Systems
Recommendations
Evolution towards smart home environments: empirical evaluation of three user interfaces
Smart home environments have evolved to the point where everyday objects and devices at home can be networked to give the inhabitants new means to control them. Familiar information appliances can be used as user interfaces (UIs) to home functions to ...
The scenario‐based usability checklist development for home appliance design: A case study
In the past decade, home appliances have been rapidly developed to satisfy the various requirements of users. Thus, there are increases in requirements to explore appropriate evaluation methods that reflect the usability of home appliances quickly and ...
Comparing heuristic evaluation and MALTU model in interaction evaluation of ubiquitous systems
IHC '20: Proceedings of the 19th Brazilian Symposium on Human Factors in Computing SystemsUbiquitous systems are made up of sensors and smart devices that communicate with each other to achieve certain goals. They have specific characteristics, such as transparency, attention, calmness, mobility, and context-awareness. Given that the Human-...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Extended-abstract
- Research
- Refereed limited
Data Availability
Video Preview: Video Previewhttps://dl.acm.org/doi/10.1145/3613905.3637131#3613905.3637131-video-preview.mp4
Funding Sources
- the National Natural Science Foundation of China
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 220Total Downloads
- Downloads (Last 12 months)220
- Downloads (Last 6 weeks)64
Reflects downloads up to 24 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format