Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

Explainability as a non-functional requirement: challenges and recommendations

Published: 01 December 2020 Publication History

Abstract

Software systems are becoming increasingly complex. Their ubiquitous presence makes users more dependent on their correctness in many aspects of daily life. As a result, there is a growing need to make software systems and their decisions more comprehensible, with more transparency in software-based decision making. Transparency is therefore becoming increasingly important as a non-functional requirement. However, the abstract quality aspect of transparency needs to be better understood and related to mechanisms that can foster it. The integration of explanations into software has often been discussed as a solution to mitigate system opacity. Yet, an important first step is to understand user requirements in terms of explainable software behavior: Are users really interested in software transparency and are explanations considered an appropriate way to achieve it? We conducted a survey with 107 end users to assess their opinion on the current level of transparency in software systems and what they consider to be the main advantages and disadvantages of embedded explanations. We assess the relationship between explanations and transparency and analyze its potential impact on software quality. As explainability has become an important issue, researchers and professionals have been discussing how to deal with it in practice. While there are differences of opinion on the need for built-in explanations, understanding this concept and its impact on software is a key step for requirements engineering. Based on our research results and on the study of existing literature, we offer recommendations for the elicitation and analysis of explainability and discuss strategies for the practice.

References

[1]
2019 cigi-ipsos global survey on internet security and trust. https://www.cigionline.org/internet-survey-2019 (2019). Accessed 30 Nov 2019
[2]
Abdollahi B, Nasraoui O (2018) Transparency in fair machine learning: the case of explainable recommender systems. In: Human and machine learning, pp 21–35. Springer
[3]
Alexander IF and Beus-Dukic L Discovering requirements: how to specify products and services 2009 Hoboken Wiley
[4]
Ameller D, Ayala C, Cabot J, Franch X (2012) How do software architects consider non-functional requirements: an exploratory study. In: 2012 20th IEEE international requirements engineering conference (RE), pp 41–50
[5]
Anderson N, Herriot P, and Hodgkinson GP The practitioner-researcher divide in industrial, work and organizational (IWO) psychology: where are we now, and where do we go from here? J Occup Org Psychol 2001 74 4 391-411
[6]
Aronson E, Wilson TD, and Brewer MB Experimentation in social psychology Handb Soc Psychol 1998 1 99-142
[7]
Berander P, Damm LO, Eriksson J, Gorschek T, Henningsson K, Jönsson P, Kågström S, Milicic D, Mårtensson F, Rönkkö K, et al. Software quality attributes and trade-offs 2005 Karlskrona Blekinge Institute of Technology
[8]
Beyer H and Holtzblatt K Contextual design: defining customer-centered systems 1997 Amsterdam Elsevier
[9]
Boehm B and In H Identifying quality-requirement conflicts IEEE Softw 1996 13 2 25-35
[10]
Bohlender D, Köhl MA (2019) Towards a characterization of explainable systems. arXiv:1902.03096
[11]
Breaux TD, Vail MW, Anton AI (2006) Towards regulatory compliance: extracting rights and obligations to align requirements with regulations. In: 14th IEEE international requirements engineering conference (RE’06), pp 49–58. 10.1109/RE.2006.68
[12]
Bunt A, Lount M, Lauzon C (2012) Are explanations always important?: a study of deployed, low-cost intelligent interactive systems. In: Proceedings of the 2012 ACM international conference on intelligent user interfaces, pp 169–178. ACM
[13]
Bussone A, Stumpf S, O’Sullivan D (2015) The role of explanations on trust and reliance in clinical decision support systems. In: 2015 international conference on healthcare informatics, pp 160–169. IEEE
[14]
Carvalho RM (2017) Dealing with conflicts between non-functional requirements of UbiComp and IoT applications. In: 2017 IEEE 25th international requirements engineering conference (RE), pp 544–549. IEEE
[15]
Carvallo J.P, Franch X, Quer C (2006) Managing non-technical requirements in cots components selection. In: 14th IEEE international requirements engineering conference (RE’06), pp 323–326. IEEE
[16]
Charmaz K Constructing grounded theory: a practical guide through qualitative analysis 2006 Thousand Oaks Sage
[17]
Chazette L (2019) Survey data - perception of end-users regarding the need for explanations in software systems. 10.5281/zenodo.3261127
[18]
Chazette L, Karras O, Schneider K (2019) Do end-users want explanations? Analyzing the role of explainability as an emerging aspect of non-functional requirements. In: 2019 IEEE 27th international requirements engineering conference (RE). IEEE
[19]
Choe JM The consideration of cultural differences in the design of information systems Inf Manag 2004 41 5 669-684
[20]
Chromik M, Eiband M, Völkel ST, Buschek D (2019) Dark patterns of explainability, transparency, and user control for intelligent systems. In: IUI workshops
[21]
Chung L, Nixon BA (1995) Dealing with non-functional requirements: three experimental studies of a process-oriented approach. In: 1995 17th international conference on software engineering, pp 25–25. IEEE
[22]
Chung L, do Prado Leite JCS (2009) On non-functional requirements in software engineering. In: Conceptual modeling: foundations and applications, pp 363–379. Springer
[23]
Cohen J Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit Psychol Bull 1968 70 4 213-220
[24]
Cooper A, Reimann R, and Cronin D About face 3: the essentials of interaction design 2007 Hoboken Wiley
[25]
Cranor LF and Garfinkel S Guest editors’ introduction: secure or usable? IEEE Secur Priv 2004 2 5 16-18
[26]
Cysneiros LM, do Prado Leite JCS, and Neto JDMSA framework for integrating non-functional requirements into conceptual modelsRequir Eng20016297-1150972.68722
[27]
Cysneiros LM, Raffi M, do Prado Leite JCS (2018) Software transparency as a key requirement for self-driving cars. In: 2018 IEEE 26th international requirements engineering conference (RE), pp 382–387. IEEE
[28]
Cysneiros LM, Werneck VM, Kushniruk A (2005) Reusable knowledge for satisficing usability requirements. In: 13th IEEE international conference on requirements engineering (RE’05), pp 463–464. IEEE
[29]
Cysneiros LM, Werneck VMB (2009) An initial analysis on how software transparency and trust influence each other. In: WER
[30]
Damian DE, Zowghi D (2003) An insight into the interplay between culture, conflict and distance in globally distributed requirements negotiations. In: Proceedings of the 36th annual Hawaii international conference on system sciences, 2003, pp 10. IEEE
[31]
De La Vara JL, Wnuk K, Berntsson-Svensson R, Sánchez J, Regnell B (2011) An empirical study on the importance of quality requirements in industry. In: SEKE, pp 438–443
[32]
Doerr J, Kerkow D, Koenig T, Olsson T, Suzuki T (2005) Non-functional requirements in industry-three case studies adopting an experience-based NFR method. In: 13th IEEE international conference on requirements engineering (RE’05), pp 373–382. IEEE
[33]
Doran D, Schulz S, Besold TR (2017) What does explainable ai really mean? A new conceptualization of perspectives. arXiv:1710.00794
[34]
do Prado Leite JCS and Cappelli C Software transparency Bus Inf Syst Eng 2010 2 3 127-139
[35]
Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpretable machine learning. arXiv:1702.08608
[36]
Doshi-Velez F, Kortz M, Budish R, Bavitz C, Gershman S, O’Brien D, Schieber S, Waldo J, Weinberger D, Wood A (2017) Accountability of ai under the law: the role of explanation. arXiv:1711.01134
[37]
Ethics guidelines for trustworthy ai. (2019) https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai. Accessed 30 Nov 2019
[38]
Flavián C, Guinalíu M, and Gurrea R The role played by perceived usability, satisfaction and consumer trust on website loyalty Inf Manag 2006 43 1 1-14
[39]
Fotrousi F, Fricker SA, Fiedler M (2014) Quality requirements elicitation based on inquiry of quality-impact relationships. In: 2014 IEEE 22nd international requirements engineering conference (RE), pp 303–312. IEEE
[40]
Frese M (1987) A theory of control and complexity: Implications for software design and integration of computer systems into the work place. In: Psychological issues of human computer interaction in the work place. North-Holland Publishing Co., NLD, pp 313–337
[41]
Glinz M (2007) On non-functional requirements. In: 15th IEEE international requirements engineering conference (RE 2007), pp 21–26. IEEE
[42]
Goodman B and Flaxman S European union regulations on algorithmic decision-making and a “right to explanation” AI Mag 2017 38 3 50-57
[43]
Groen EC, Kopczyńska S, Hauer MP, Krafft TD, Doerr J (2017) Users—the hidden software product quality experts? A study on how app users report quality aspects in online reviews. In: 2017 IEEE 25th international requirements engineering conference (RE), pp 80–89. IEEE
[44]
Gulliksen J, Göransson B, Boivie I, Blomkvist S, Persson J, and Cajander Å Key principles for user-centred systems design Behav Inf Technol 2003 22 6 397-409
[45]
Gutmann P and Grigg I Security usability IEEE Secur Priv 2005 3 4 56-58
[46]
Hamel L (2006) Visualization of support vector machines with unsupervised learning. In: Proceedings of 2006 IEEE symposium on computational intelligence in bioinformatics and computational biology
[47]
Hartson R and Pyla PS The UX book: process and guidelines for ensuring a quality user experience 2012 Amsterdam Elsevier
[48]
Hehn J, Uebernickel F (2018) The use of design thinking for requirements engineering: an ongoing case study in the field of innovative software-intensive systems. In: 2018 IEEE 26th international requirements engineering conference (RE), pp 400–405. IEEE
[49]
Herlocker JL, Konstan JA, Riedl J (2000) Explaining collaborative filtering recommendations. In: Proceedings of the 2000 ACM conference on computer supported cooperative work, pp 241–250. ACM
[50]
Hoffmann CP, Lutz C, and Meckel M Digital natives or digital immigrants? The impact of user characteristics on online trust J Manag Inf Syst 2014 31 3 138-171
[51]
Hosseini M, Shahri A, Phalp K, Ali R (2016) Foundations for transparency requirements engineering. In: International working conference on requirements engineering: foundation for software quality, pp 225–231. Springer
[52]
ISO 9241-210:2019 Ergonomics of human-system interaction—Part 210: human-centred design for interactive systems. Standard, International Organization for Standardization, Geneva CH (2019)
[53]
ISO/IEC 25010:2011 Systems and Software Engineering-Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models. Standard, International Organization for Standardization, Geneva CH (2011)
[54]
Jakulin A, Možina M, Demšar J, Bratko I, Zupan B (2005) Nomograms for visualizing support vector machines. In: Proceedings of the eleventh ACM SIGKDD international conference on knowledge discovery in data mining, KDD ’05, pp 108–117. ACM, New York, NY, USA
[55]
Jerome B and Kazman R Seffah A, Gulliksen J, and Desmarais MC Surveying the solitudes: an investigation into the relationships between human computer interaction and software engineering in practice Human-centered software engineering—integrating usability in the software development lifecycle 2005 Netherlands Springer 59-70
[56]
Johnson-Laird PN Mental models: towards a cognitive science of language, inference, and consciousness, 6 1983 Cambridge Harvard University Press
[57]
Yee Ka-Ping Aligning security and usability IEEE Secur Priv 2004 2 5 48-55
[58]
Kieras DE and Bovair S The role of a mental model in learning to operate a device Cognit Sci 1984 8 3 255-273
[59]
Kim B, Glassman E, Johnson B, and Shah J ibcm: Interactive Bayesian case model empowering humans via intuitive interaction 2015 Cambridge, MA Massachusetts Institute of Technology
[60]
Koh PW, Liang P (2017) Understanding black-box predictions via influence functions. arXiv:1703.04730
[61]
Konstan JA and Riedl J Recommender systems: from algorithms to user experience User Model User-Adapt Interact 2012 22 1–2 101-123
[62]
Kulesza T, Stumpf S, Burnett M, Yang S, Kwan I, Wong WK (2013) Too much, too little, or just right? Ways explanations impact end users’ mental models. In: 2013 IEEE symposium on visual languages and human centric computing, pp 3–10. IEEE
[63]
Kummer TF, Leimeister JM, and Bick M On the importance of national culture for the design of information systems Bus Inf Syst Eng 2012 4 6 317-330
[64]
Landes D, Studer R (1995) The treatment of non-functional requirements in mike. In: European software engineering conference, pp 294–306. Springer
[65]
Landis JR and Koch GGThe measurement of observer agreement for categorical dataBiometrics1977331159-1740351.62039
[66]
Lepri B, Oliver N, Letouzé E, Pentland A, and Vinck P Fair, transparent, and accountable algorithmic decision-making processes Philos Technol 2018 31 4 611-627
[67]
Leßmann H (2017) Durchführung einer umfrage-studie zur nutzung von code-reviews in der praxis. Master’s thesis, Leibniz Universität Hannover, Fachgebiet Software Engineering
[68]
Lim BY, Dey AK (2009) Assessing demand for intelligibility in context-aware applications. In: Proceedings of the 11th international conference on Ubiquitous computing, pp 195–204. ACM
[69]
Lipton ZC The mythos of model interpretability Commun ACM 2018 61 10 36-43
[70]
Mairiza D, Zowghi D (2010) Constructing a catalogue of conflicts among non-functional requirements. In: International conference on evaluation of novel approaches to software engineering, pp 31–44. Springer
[71]
Mao JY, Vredenburg K, Smith PW, and Carey T The state of user-centered design practice Commun. ACM 2005 48 3 105-109
[72]
Mayhew DJ and Mayhew D The usability engineering lifecycle: a practitioner’s handbook for user interface design 1999 Burlington Morgan Kaufmann
[73]
Miles MB and Huberman AM Qualitative data analysis: an expanded sourcebook 1994 Thousand Oaks Sage
[74]
Možina M, Demšar J, Kattan M, and Zupan B Boulicaut JF, Esposito F, Giannotti F, and Pedreschi D Nomograms for visualization of naive Bayesian classifier Knowledge discovery in databases: PKDD 2004 2004 Berlin Springer 337-348
[75]
Nielsen J Usability engineering 1994 Amsterdam Elsevier
[76]
Norman DA Some observations on mental models Human-computer interaction: a multidisciplinary approach 1987 San Francisco, CA, USA Morgan Kaufmann Publishers Inc. 241-244
[77]
Norman KL The psychology of menu selection: designing cognitive control at the human/computer interface 1991 Norwood, NJ Ablex Pub. Corp
[78]
Otto PN, Antón AI (2007) Addressing legal requirements in requirements engineering. In: 15th IEEE international requirements engineering conference (RE 2007), pp 5–14. IEEE
[79]
O’Neil C Weapons of math destruction: how big data produces inequality and threatens democracy 2016 New York Penguin Randomhouse Ltd
[80]
Pacey A The culture of technology 1983 Cambridge MIT Press
[81]
Paech B, Kerkow D (2004) Non-functional requirements engineering-quality is essential. In: 10th international workshop on requirements engineering foundation for software quality
[82]
Papadimitriou A, Symeonidis P, and Manolopoulos Y A generalized taxonomy of explanations styles for traditional and social recommender systems Data Min Knowl Discov 2012 24 3 555-583
[83]
Prensky M Digital natives, digital immigrants On Horiz 2001 9 5 1-6
[84]
Pynadath DV, Barnes MJ, Wang N, Chen JY (2018) Transparency communication for machine learning in human-automation interaction. In: Human and machine learning, pp 75–90. Springer
[85]
Ribeiro MT, Singh S, Guestrin C (2016) Why should i trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. Association for Computing Machinery, New York, NY, USA, pp 1135–1144. 10.1145/2939672.2939778
[86]
Regulation (eu) (2016) 2016/679 of the European parliament and of the council of 27 april 2016 (general data protection regulation). https://eur-lex.europa.eu/eli/reg/2016/679/oj
[87]
Saldaña J The coding manual for qualitative researchers 2015 Thousand Oaks Sage
[88]
Seffah A, Gulliksen J, and Desmarais MC Seffah A, Gulliksen J, and Desmarais MC An introduction to human-centered software engineering Human-centered software engineering—integrating usability in the software development lifecycle 2005 Dordrecht Springer Netherlands 3-14
[89]
Seffah A and Metzker E The obstacles and myths of usability and software engineering Commun ACM 2004 47 12 71-76
[90]
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626
[91]
Siena A, Mylopoulos J, Perini A, Susi A (2009) Designing law-compliant software requirements. In: International conference on conceptual modeling, pp 472–486. Springer
[92]
Tapscott D Grown up digital: how the net generation is changing your world Int J Mark Res 2010 52 1 139
[93]
Thomsen S Corporate values and corporate governance Corp Gov Int J Bus Soc 2004 4 29-46
[94]
Tintarev N and Masthoff J Evaluating the effectiveness of explanations for recommender systems User Model User-Adapt Interact 2012 22 4–5 399-439
[95]
Tomsett R, Braines D, Harborne D, Preece AD, Chakraborty S (2018) Interpretable to whom? A role-based model for analyzing interpretable machine learning systems. CoRR arXiv:1806.07552
[96]
Vanwelkenhuysen J (1996) Quality requirements analysis in customer-centered software development. In: Proceedings of the second international conference on requirements engineering, pp 117–124. IEEE
[97]
Whittle J Is your software valueless? IEEE Softw 2019 36 3 112-115
[98]
Winkler J.P, Vogelsang A (2017) What does my classifier learn? A visual approach to understanding natural language text classifiers. In: International conference on applications of natural language to information systems, pp 468–479. Springer
[99]
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, and Wesslén A Experimentation in software engineering 2012 Berlin Heidelberg Springer
[100]
Young I Mental models: aligning design strategy with human behavior 2008 New York Rosenfeld Media
[101]
Zinovatna O, Cysneiros LM (2015) Reusing knowledge on delivering privacy and transparency together. In: 2015 IEEE fifth international workshop on requirements patterns (RePa), pp 17–24. IEEE

Cited By

View all
  • (2025)How do ML practitioners perceive explainability? an interview study of practices and challengesEmpirical Software Engineering10.1007/s10664-024-10565-230:1Online publication date: 1-Feb-2025
  • (2024)The X Factor: On the Relationship between User eXperience and eXplainabilityProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685352(1-12)Online publication date: 13-Oct-2024
  • (2024)Towards Personal Explanations for Recommender Systems: A Study on the Impact of Familiarity and UrgencyAdjunct Proceedings of the 2024 Nordic Conference on Human-Computer Interaction10.1145/3677045.3685430(1-8)Online publication date: 13-Oct-2024
  • Show More Cited By

Index Terms

  1. Explainability as a non-functional requirement: challenges and recommendations
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Please enable JavaScript to view thecomments powered by Disqus.

          Information & Contributors

          Information

          Published In

          cover image Requirements Engineering
          Requirements Engineering  Volume 25, Issue 4
          Dec 2020
          100 pages

          Publisher

          Springer-Verlag

          Berlin, Heidelberg

          Publication History

          Published: 01 December 2020
          Accepted: 19 May 2020
          Received: 01 December 2019

          Author Tags

          1. Explainability
          2. Software transparency
          3. Non-functional requirements
          4. Software quality

          Qualifiers

          • Research-article

          Funding Sources

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)0
          • Downloads (Last 6 weeks)0
          Reflects downloads up to 13 Nov 2024

          Other Metrics

          Citations

          Cited By

          View all
          • (2025)How do ML practitioners perceive explainability? an interview study of practices and challengesEmpirical Software Engineering10.1007/s10664-024-10565-230:1Online publication date: 1-Feb-2025
          • (2024)The X Factor: On the Relationship between User eXperience and eXplainabilityProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685352(1-12)Online publication date: 13-Oct-2024
          • (2024)Towards Personal Explanations for Recommender Systems: A Study on the Impact of Familiarity and UrgencyAdjunct Proceedings of the 2024 Nordic Conference on Human-Computer Interaction10.1145/3677045.3685430(1-8)Online publication date: 13-Oct-2024
          • (2024)Decomposition of Reliability Requirements for Self-Adaptive Systems Using the NFR FrameworkProceedings of the 20th Brazilian Symposium on Information Systems10.1145/3658271.3658325(1-10)Online publication date: 20-May-2024
          • (2024)Novel Contract-based Runtime Explainability Framework for End-to-End Ensemble Machine Learning ServingProceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI10.1145/3644815.3644964(234-244)Online publication date: 14-Apr-2024
          • (2024)Explaining the Unexplainable: The Impact of Misleading Explanations on Trust in Unreliable Predictions for Hardly Assessable TasksProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659573(36-46)Online publication date: 22-Jun-2024
          • (2024)AI is Entering Regulated Territory: Understanding the Supervisors' Perspective for Model Justifiability in Financial Crime DetectionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642326(1-21)Online publication date: 11-May-2024
          • (2024)Analyzing and Debugging Normative Requirements via Satisfiability CheckingProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639093(1-12)Online publication date: 20-May-2024
          • (2024)Explainable Artificial Intelligence (XAI) 2.0Information Fusion10.1016/j.inffus.2024.102301106:COnline publication date: 1-Jun-2024
          • (2024)Prioritizing user requirements for digital products using explainable artificial intelligenceFuture Generation Computer Systems10.1016/j.future.2024.04.037158:C(167-182)Online publication date: 1-Sep-2024
          • Show More Cited By

          View Options

          View options

          Get Access

          Login options

          Media

          Figures

          Other

          Tables

          Share

          Share

          Share this Publication link

          Share on social media