Abstract
The advent of ubiquitous systems places even more focus on users, since these systems must support their daily activities in such a transparent way that does not disturb them. Thus, much more attention should be provided to human–computer interaction (HCI) and, as a consequence, to its quality. Dealing with quality issues implies first the identification of the quality characteristics that should be achieved and, then, which software measures should be used to evaluate them in a target system. Therefore, this work aims to identify what quality characteristics and measures have been used for the HCI evaluation of ubiquitous systems. In order to achieve our goal, we performed a large literature review, using a systematic mapping study, and we present our results in this paper. We identified 41 pertinent papers that were deeply analyzed to extract quality characteristics and software measures. We found 186 quality characteristics, but since there were divergences on their definitions and duplicated characteristics, an analysis of synonyms by peer review based on the equivalence of definitions was also done. This analysis allowed us to define a final suitable set composed of 27 quality characteristics, where 21 are generic to any system but are particularized for ubiquitous applications and 6 are specific for this domain. We also found 218 citations of measures associated with the characteristics, although the majority of them are simple definitions with no detail about their measurement functions. Our results provide not only an overview of this area to guide researchers in directing their efforts but also it can help practitioners in evaluating ubiquitous systems using these measures.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Some papers did not have the abstract registered in the database. For those papers we downloaded the complete paper to start the review (next step screening the papers).
The sum of the numbers in the graphic exceeds the total number of measures because there are duplicate measures among the papers.
ISO/IEC 25000 2014 presents the following definitions:
Quality measure element (QME): measure defined in terms of a property and the measurement method for quantifying it.
Property to quantify: property of a target entity that is related to a quality measure element and which can be quantified by a measurement method
Quality measure (QM): derived measure that is defined as a measurement function of two or more values of quality measure elements.
Measurement function: algorithm or calculation performed to combine two or more quality measure elements.
References
Abi-Char, P. E., Mhamed, A., El-Hassan, B., & Mokhtari, M. (2010). A flexible privacy and trust based context-aware secure framework. In Proceedings of the aging friendly technology for health and independence, and 8th international conference on smart homes and health telematics, ICOST’10. Springer. http://dl.acm.org/citation.cfm?id=1894439.1894443.
Ammar, L., Trabelsi, A., & Mahfoudhi, A. (2015). A model-driven approach for usability engineering of interactive systems. Software Quality Journal, 1–35.
Bezerra, C. I. M., Oliveira, K. M., Andrade, R. M. C., et al. (2014). Challenges for usability testing in ubiquitous system. In l’Interaction Homme-Machine.
Cappiello, I., Puglia, S., & Vitaletti, A. (2009). Design and initial evaluation of a ubiquitous touch-based remote grocery shopping process. In First international workshop on near field communication, 9–14.
Carvalho, R. M., Andrade, R. M. C., & Oliveira, K. M. (2015). Using the GQM method to evaluate calmness in ubiquitous applications*. In HCI international.
Chalmers, D., & Sloman, M. (1999). A survey of quality of service in mobile computing environments. IEEE Communications Surveys & Tutorials, 2(2), 2–10.
Chang, Y.-H., & Lin, B.-S. (2011). An inquiry-based ubiquitous tour system. In International conference on complex, intelligent and software intensive systems.
Da Silva, C. M. R., Da Silva, J. L. C., Rodrigues, R. B., Do Nascimento, L. M., & Garcia, V. C. (2013). Systematic mapping study on security threats in cloud computing. arXiv preprint arXiv:1303.6782,
Damián-Reyes, P., Favela, J., & Contreras-Castillo, J. (2011). Uncertainty management in context-aware applications: Increasing usability and user trust. Wireless Personal Communications, 56(1), 37–53.
De Moor, K., Ketyko, I., Joseph, W., et al. (2010). Proposed framework for evaluating quality of experience in a mobile, testbed-oriented living lab setting. Mobile Networks and Applications, 15(3), 378–391.
Dey, A. K. (2001). Understanding and using context. Personal and Ubiquitous Computing, 5(1), 4–7.
Evers, V., Cramer, H., Van Someren, M., & Wielinga, B. (2010). Interacting with adaptive systems. In Interactive collaborative information systems (pp. 299–325). Springer.
Evers, C., Kniewel, R., Geihs, K., & Schmidt, L. (2014). The user in the loop: Enabling user participation for self-adaptive applications. Future Generation Computer Systems, 34, 110–123.
Fenton, N., & Pfleeger, S. (1997). Software metrics: A rigorous and practical approach. PWS Pub.
Haapalainen, E., Kim, S., Forlizzi, J. F., & Dey, A. K. (2010). Psycho-physiological measures for assessing cognitive load. In Proceedings of the 12th ACM international conference on ubiquitous computing, Ubicomp’10. ACM. http://doi.acm.org/10.1145/1864349.1864395.
Hernandes, E., Zamboni, A., Fabbri, S., & Thommazo, A. Di. (2012). Using GQM and TAM to evaluate StArt—A tool that supports systematic review. CLEI Electronic Journal, 15, 3.
Iqbal, R., Sturm, J., Kulyk, O., Wang, J., & Terken, J. (2005). User-centred design and evaluation of ubiquitous services. In Proceedings of the 23rd international conference on design of communication—documenting and designing for pervasive information. http://www.scopus.com/inward/record.url?eid=2-s2.0-32044471829&partnerID=40&md5=9a711a41b59e2b258fd13669a3826bc7.
ISO 9241-11. (1998). Ergonomic requirements for office work with visual display terminals. In The International Organization for Standardization
ISO/IEC 14598. (1999). Information Technology—Software Product Evaluation—Part 1.
ISO/IEC 9126. (2001). Software engineering—Product Quality—Part 1.
ISO/IEC 25000. (2014). Software Engineering—Software Product Quality Requirements and Evaluation (SQuaRE)—Guide to SQuaRE.
ISO/IEC 25010. (2011). ISO/IEC 25010. Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)—System and software quality models, v. 2011.
Jafari, S., Mtenzi, F., O’Driscoll, C., Fitzpatrick, R., & O’Shea, B. (2010). Privacy metrics in ubiquitous computing applications. In International conference for internet technology and secured transactions.
Jafari, S., Mtenzi, F., O’Driscoll, C., Fitzpatrick, R., & O’Shea, B. (2011). Measuring privacy in ubiquitous computing applications. International Journal of Digital Society, 2(3), 547–550.
Jia, L., Collins, M., & Nixon, P. (2009). Evaluating trust-based access control for social interaction. In 3rd international conference on mobile ubiquitous computing, systems, services, and technologies. http://www.scopus.com/inward/record.url?eid=2-s2.0-77951440915&partnerID=40&md5=241ed20a5c57e71b94e32fcaec543a9e.
Karaiskos, D., Kourouthanassis, P., & Giaglis, G. M. (2009). Towards a validated construct for information systems pervasiveness: An exploratory assessment. In BLED 2009 proceedings, p. Paper 12.
Karvonen, H., & Kujala, T. (2014). Designing and evaluating ubicomp characteristics of intelligent in-car systems. In 5th International conference on applied human factors and ergonomics.
Kemp, E. A., Thompson, A.-J., & Johnson, R. S. (2008). Interface evaluation for invisibility and ubiquity: An example from E-learning. In Proceedings of the 9th ACM SIGCHI New Zealand Chapter’s international conference on human–computer interaction: Design Centered HCI, CHINZ’08. ACM. http://doi.acm.org/10.1145/1496976.1496981.
Kim, H. J., Choi, J. K., & Ji, Y. (2008). Usability evaluation framework for ubiquitous computing device. In Proceedings—3rd international conference on convergence and hybrid information technology, ICCIT 2008. http://www.scopus.com/inward/record.url?eid=2-s2.0-57849159822&partnerID=40&md5=2965dfe3adcf63a35f173ca1fd4cb7e9.
Kim, Y., & Lee, K. (2006). A quality measurement method of context information in ubiquitous environments. In 2006 International conference on hybrid information technology. http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4021269.
Kitchenham, B. A., Budgen, D., & Brereton, P. (2010). The value of mapping studies—A participant-observer case study. In Proceedings of evaluation and assessment of software engineering, EASE.
Kitchenham, B. A., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. In Technical report. EBSE-2007-01, Keele University.
Ko, I.-Y., Koo, H.-M., & Jimenez-Molina, A. (2010). User-centric web services for ubiquitous computing. In Advanced techniques in web intelligence-I.
Kourouthanassis, P. E., Giaglis, G. M., & Karaiskos, D. C. (2008). Delineating the degree of “pervasiveness” in pervasive information systems: An assessment framework and design implications. In Pan-Hellenic conference on informatics, PCI.
Kryvinska, N., Strausss, C., & Zinterhof, P. (2011). “Variated availability” approach to the services manageable delivering. In Fifth international conference on innovative mobile and internet services in ubiquitous computing (IMIS).
Lee, J., & Yun, M. H. (2012). Usability assessment for ubiquitous services: Quantification of the interactivity in inter-personal services. In IEEE international conference on management of innovation & technology.
Lee, J., Song, J., Kim, H., Choi, J., & Yun, M. H. (2008). A user-centered approach for ubiquitous service evaluation: An evaluation metrics focused on human–system interaction capability. In Asia-Pacific conference, APCHI.
Liampotis, N., Roussaki, I., Papadopoulou, E., et al. (2009). A privacy framework for personal self-improving smart spaces. In International conference on computational science and engineering.
Montagud, S., Abrahão, S., & Insfran, E. (2012). A systematic review of quality attributes and measures for software product lines. In Software Quality Journa, v. 20.
Nielsen, J. (1994). Usability engineering. Access Online via Elsevier.
Novais, R. L., Torres, A., Mendes, T. S., Mendonça, M., & Zazworka, N. (2013). Software evolution visualization: A systematic mapping study. Information and Software Technology, 55(11), 1860–1883.
Oriol, M., Marco, J., & Franch, X. (2014). Quality models for web services: A systematic mapping. Information and Software Technology, 56(10), 1167–1182.
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008). Systematic mapping studies in software engineering. In Proceedings of the 12th international conference on evaluation and assessment in software engineering, EASE’08. British Computer Society. http://dl.acm.org/citation.cfm?id=2227115.2227123.
Petersen, K., & Gencel, C. (2013). Worldviews, research methods, and their relationship to validity in empirical software engineering research. In Proceedings—Joint conference of the 23rd international workshop on software measurement and the 8th international conference on software process and product measurement, IWSM-MENSURA 2013 (pp. 81–89).
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, 64, 1–18.
Poppe, R., Rienks, R., & Dijk, B. Van (2007). Evaluating the future of HCI: Challenges for the evaluation of emerging applications. AI for Human Computing, 234–250.
Ranganathan, A., Al-Muhtadi, J., Biehl, J., et al. (2005). Towards a pervasive computing benchmark. In International conference on pervasive computing and communications workshops.
Reis, R. A. C. (2015). Usability evaluation approaches for (ubiquitous) mobile applications: A systematic mapping study usability evaluation approaches for (ubiquitous) mobile applications: A systematic mapping study. n. September.
Rocha, L. S., Ferreira, J. B., Lima, F. F. P., et al. (2011). Ubiquitous software engineering: Achievements, challenges and beyond. In Brazilian symposium on software engineering (in Portuguese)
Ross, T., & Burnett, G. (2001). Evaluating the human–machine interface to vehicle navigation systems as an example of ubiquitous computing. In International Journal of Human–Computer Studies,
Rubio, J. M. L., & Bozo, J. P. (2007). Approach to a quality process for the ubiquitous software development. In Electronics, robotics and automotive mechanics conference.
Ryu, H., Hong, G. Y., & James, H. (2006). Quality assessment technique for ubiquitous software and middleware. Research Letters in the Information and Mathematical Sciences, 9, 13–87.
Sanchez-pi, N., & Carb, J. (2012). An evaluation method for context—Aware systems in U-Health. In 3rd international symposium on ambient intelligence (ISAmI 2012).
Santos, R. M., Oliveira, K. M., Andrade, R. M. C., Santos, I. S., & Lima, E. R. R. (2013). A quality model for human–computer interaction evaluation in ubiquitous systems. In Latin American conference on human computer interaction.
Schalkwyk, J., Beeferman, D., Beaufays, F., et al. (2010). Your word is my command: Google Search by Voice: A case study. Advances in Speech Recognition.
Scholtz, J., & Consolvo, S. (2004). Toward a framework for evaluating ubiquitous computing applications. IEEE Pervasive Computing,
Sears, A., & Jacko, J. A. (2009). Human–computer interaction: Development process. Boca Raton: CRC Press.
Silveira, P. A. M., Machado, I. C., McGregor, J. D., Santana, E., & Meira, S. R. L. (2011). A systematic mapping study of software product lines testing. In Information and Software Technology
Song, J., Park, K. R., Kwon, S., Lee, J. H. J. H., & Yun, M. H. (2009). The development of human-system interactivity metrics for ubiquitous service applying user-centered design methodology. In World Congress on Services.
Sousa, B., Pentikousis, K., & Curado, M. (2011). UEF: Ubiquity evaluation framework. Wired/Wireless Internet Communications
Spínola, R. O., & Travassos, G. H. (2012). Towards a framework to characterize ubiquitous software projects. Information and Software Technology
Sun, T., & Denko, M. K. (2008). Performance evaluation of trust management in pervasive computing. In International conference on advanced information networking and applications, AINA.
Tahir, T., & Jafar, A. (2011). A systematic review on software measurement programs. In Frontiers of Information Technology (FIT), 2011 (Vol. 73, pp. 39–44).
Thompson, S. G., & Azvine, B. (2004). No pervasive computing without intelligent systems. In BT technology journal
Toch, E. (2011). Super-Ego: A framework for privacy-sensitive bounded context-awareness. In ACM international workshop on context-awareness for self-managing systems.
Viana, J. R. M., Viana, N. P., Trinta, F. A. M., & Carvalho, W. V. De (2014). A systematic review on software engineering in pervasive games development. In Brazilian symposium on computer games and digital entertainment. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=7000032.
Wagner, S., Toftegaard, T., & Bertelsen, O. (2012). Requirements for an evaluation infrastructure for reliable pervasive healthcare research. In International conference on pervasive computing technologies for healthcare, IEEE.
Waibel, A., Stiefelhagen, R., Carlson, R., et al. (2010). Computers in the human interaction loop. Handbook of Ambient Intelligence and Smart Environments.
Weihong-Guo, A., Blythe, P., Olivier, P., Singh, P., & Nam Ha, H. (2008). Using immersive video to evaluate future traveller information systems. In IET intelligent transport systems,
Weiser, M. (1991). The computer for the 21st century. In Scientific American.
Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2005). Requirements engineering paper classification and evaluation criteria: A proposal and a discussion. In Requirements Engineering.
Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In 18th international conference on evaluation and assessment in software engineering (EASE 2014) (pp. 1–10).
Wohlin, C., Runeson, P., Da Mota Silveira Neto, P. A., et al. (2013). On the reliability of mapping studies in software engineering. Journal of Systems and Software, 86(10), 2594–2610.
Wu, C. L., & Fu, L. C. (2012). Design and realization of a framework for human–system interaction in smart homes. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 42(1), 15–31.
Zhang, Y., Zhang, S., Tong, H., & Yong Zhang, S. Z. (2006). Adaptive service delivery for mobile users in ubiquitous computing environments. In International conference on Ubiquitous Intelligence and Computing.
Acknowledgments
We thank FUNCAP (Ceará State Foundation for Support of Scientific and Technological Development, Brazil) and CNRS (Centre National de la Recherche Scientifique, France) for the financial support of this work, which is a result of the Maximum Project—A Measurement-based Approach for the Quality Evaluation of Human–Computer Interaction in Ubiquitous Systems, under grant number INC-0064-00012.01.00/12. We also thank CAPES for sponsoring Rainara Maia Carvalho and Ismayle de Sousa Santos with PhD scholarships, and CNPq for sponsoring Rossana Maria de Castro Andrade with a Researcher Scholarship - DT Level 2.
Author information
Authors and Affiliations
Corresponding author
Appendix: Software Measures
Appendix: Software Measures
This appendix presents the 219 extracted software measures from the systematic mapping presented in this paper. These measures are classified and organized in tables according to the quality characteristics that they are aimed at evaluating (Tables 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27).
Rights and permissions
About this article
Cite this article
Carvalho, R.M., de Castro Andrade, R.M., de Oliveira, K.M. et al. Quality characteristics and measures for human–computer interaction evaluation in ubiquitous systems. Software Qual J 25, 743–795 (2017). https://doi.org/10.1007/s11219-016-9320-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-016-9320-z