research-article

Open access

Human-Centered AI Design in Reality: A Study of Developer Companies’ Practices : A study of Developer Companies’ Practices

Authors:

Maria Hartikainen,

Kaisa Väänänen,

Saara Ala-Luopa,

Thomas OlssonAuthors Info & Claims

NordiCHI '22: Nordic Human-Computer Interaction Conference

Article No.: 55, Pages 1 - 11

https://doi.org/10.1145/3546155.3546677

Published: 08 October 2022 Publication History

All formats PDF

Abstract

Human-Centered AI (HCAI) advocates the development of AI applications that are trustworthy, usable, and based on human needs. While the conceptual foundations of HCAI are extensively discussed in recent literature, the industry practices and methods appear to lag behind. To advance HCAI method development, current practices of AI developer companies need to be understood. To understand how HCAI principles manifest in the current practices of AI development, we conducted an interview study of practitioners from 12 AI developer companies in Finland, focusing on the early stages of AI application development. Our thematic analysis reveals current development practices and identifies four main challenges: (i) detachment of HCAI work from technical development, (ii) clients’ central role as the source of user requirements, (iii) uncertain nature of AI, and (iv) lack of value-based understanding of AI. The findings inform the development of HCAI methods and implementation of HCAI principles in AI application development.

References

[1]

Amershi, Saleema, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice. ACM, Montreal, Canada, 291-300

Digital Library

[2]

Amershi, Saleema, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow, 1-13.

Digital Library

[3]

Arrieta, Alejandro Barredo, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities, and challenges toward responsible AI. Information Fusion 58 (2020): 82-115.

Digital Library

[4]

Auernhammer, Jan. 2020. Human-centered AI: The role of Human-centered Design Research in the development of AI. In Synergy - DRS International Conference 2020. Held online.

[5]

Balaji, S., and M. Sundararajan Murugaiyan. 2012. Waterfall vs. V-Model vs. Agile: A comparative study on SDLC. International Journal of Information Technology and Business Management 2, 1 (2012), 26-30.

[6]

van Berkel, Niels, Tag, Benjamin, Goncalves, Jorge, and Hosio, Simo. 2020. Human-centred artificial intelligence: a contextual morality perspective. Behaviour & Information Technology. 1-17.

[7]

Braun, Virginia, and Clarke, Victoria. 2012. Thematic analysis. APA handbook of research methods in psychology, Vol. 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. American Psychological Association, Washington.

[8]

Bond, Raymond R., Mulvenna, Maurice D., Wan, Hui, Finlay, Dewar D., Wong, Alexander, Ansgar, Koene, Brisk, Rob, Boger, Jennifer and Adel, Tameem. 2019. Human Centered Artificial Intelligence: Weaving UX into Algorithmic Decision Making. In Proceedings of 16th International Conference on Human-Computer Interaction. Bucharest, 2-9.

[9]

Bogers, Sander, Joep Frens, Janne Van Kollenburg, Eva Deckers, and Caroline Hummels. 2016. Connected Baby Bottle: A Design Case Study Towards a Framework for Data-Enabled Design. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 301-311.

Digital Library

[10]

Cantamessa, Marco, Montagna, Francesca, Altavilla, Stefania, and Casagrande-Seretti, Alessandro. 2020. Data-Driven Design: The New Challenges of Digitalization on Product Design and Development. Design Science, 6.

[11]

Cerejo, Joana. 2021. The design process of human-centered AI. Retrieved Feb 8, 2022, from https://bootcamp.uxdesign.cc/human-centered-ai-design-process-part-1-8cf7e3ce00

[12]

Chandra, Yanto, and Shang, Liang. 2019. Inductive coding. In Qualitative research using R: A systematic approach. Springer, Singapore.

[13]

Dellermann, Dominik, Adrian Calma, Nikolaus Lipusch, Thorsten Weber, Sascha Weigel, and Philipp Ebel. 2021. The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems. arXiv preprint arXiv:2105.03354.

[14]

Dignum, Virginia. 2019. HumanE AI. Project report. Retrieved Oct 8, 2021, from https://www.humane-ai.eu/wp-content/uploads/2019/11/D13-HumaneAI-framework-report.pdf

[15]

Dove, Graham, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX design innovation: Challenges for working with machine learning as a design material. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Neew York, 278-288.

Digital Library

[16]

Feuerriegel, Stefan, Mateusz Dolata, and Gerhard Schwabe. 2020. Fair AI: Challenges and Opportunities. Business & information systems engineering 62, 4 (2020): 379-384.

[17]

Eiband, M., Völkel, S. T., Buschek, D., Cook, S., & Hussmann, H. (2019, March). When people and algorithms meet: User-reported problems in intelligent everyday applications. In Proceedings of the 24th international conference on intelligent user interfaces (pp. 96-106).

[18]

Ghallab, Malik. 2019. Responsible AI: requirements and challenges. AI Perspectives, 1:1. 1-7.

[19]

Giacomin, Joseph. 2014. What is human centred design? The Design Journal 17, 4 (2014): 606-623.

[20]

Girardin, Fabien, and Neal Lathia. 2017. When user experience designers partner with data scientists. In AAAI Spring Symposium Series on Designing the User Experience of Machine Learning Systems. The IEEE press, Palo Alto.

[21]

Guidotti, Riccardo, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys 51, 5 (2018). 1-42.

[22]

Hagendorff, Thilo. 2020. The ethics of AI ethics: An evaluation of guidelines. Minds and Machines 30, 1 (2020). 99-120.

Digital Library

[23]

Harper, Richard. 2019. The Role of HCI in the Age of AI. International Journal of Human–Computer Interaction, 35, 15 (2019). 1331-1344.

[24]

HAX. 2021. Microsoft Human-centered AI eXperience Toolkit. Retrieved Feb 8, 2022, from https://www.microsoft.com/en-us/haxtoolkit/

[25]

Herbsleb, James, David Zubrow, Dennis Goldenson, Will Hayes, and Mark Paulk. 1997. Software quality and the capability maturity model. Communications of the ACM 40, 6 (1997). 30-40.

[26]

Hesenius, Marc, Nils Schwenzfeier, Ole Meyer, Wilhelm Koop, and Volker Gruhn. 2019. Towards a software engineering process for developing data-driven applications. In Proceedings of 7th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering. ACM, Montreal, 35-41.

Digital Library

[27]

Hill, Charles, Bellamy, Rachel, Erickson, Thomas, and Burnett, Margaret. 2016. Trials and Tribulations of Developers of Intelligent Systems: A Field Study. In 2016 IEEE Symposium on Visual Languages and Human-Centric Computing.162-170.

[28]

Holmquist, Lars Erik. 2017. Intelligence on tap: artificial intelligence as a new design material. Interactions 24, no. 4 (2017). 28-33.

Digital Library

[29]

IBM Design for AI. 2019. Retrieved Oct 8, 2021 from https://www.ibm.com/design/ai/

[30]

Jin, Xiaoneng, Evans, Mark, Dong, Hua, and Yao, Anqi. 2021. Design Heuristics for Artificial Intelligence: Inspirational Design Stimuli for Supporting UX Designers in Generating AI-powered Ideas. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1-8.

Digital Library

[31]

Lu, Jiahao, Ortega, Alejandra. G., Gonçalves, Milene, and Bourgeois, Jacky. 2021. The impact of data on the role of designers and their process. Proceedings of the Design Society, 1. 3021-3030.

[32]

Maguire, Martin. 2001. Methods to support human-centred design. International journal of human-computer studies 55, 4 (2001). 587-634.

Digital Library

[33]

Oppermann, Leif, Alexander Boden, Britta Hofmann, Wolfgang Prinz, and Stefan Decker. 2019. Beyond HCI and CSCW: Challenges and useful practices towards a human-centred vision of AI and IA. In Proceedings of the Halfway to the Future Symposium. ACM, 1-5.

Digital Library

[34]

PAIR with Google. People + AI Research. Retrieved Oct 8 from https://pair.withgoogle.com.

[35]

Paulk, Mark C., Curtis, Bill, Chrissis, Mary Beth, and Weber, Charles V. 1993. Capability maturity model, version 1.1. IEEE software 10, 4 (1993). 18-27.

Digital Library

[36]

Passi, Samir and Jackson, Steven J. 2018. Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects. Proceedings of the ACM on Human-Computer Interaction, 2. 1-28.

Digital Library

[37]

Piorkowski, David, Park, Soya, Wang, April Y., Wang, Dakuo, Muller, Michael, and Portnoy, Felix. 2021. How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study. Proceedings of the ACM on Human-Computer Interaction, 5. 1-25.

Digital Library

[38]

Saldaña, Johnny. 2014. Coding and analysis strategies. In The Oxford handbook of qualitative research.

[39]

Shneiderman, Ben. 2020. Bridging the gap between ethics and practice: Guidelines for reliable, safe, and trustworthy Human-Centered AI systems. ACM Transactions on Interactive Intelligent Systems, 10, 4 (2020). 1-31.

Digital Library

[40]

Shneiderman, Ben. 2020. Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020). 495-504.

[41]

Silberg, Jake and Manyika, James. 2019. Notes from the AI frontier: Tackling bias in AI (and in humans). McKinsey Global Institute. 1-6.

[42]

Steen, Marc. 2011. Tensions in Human-Centred Design. CoDesign 7, 1 (2011). 45-60.

[43]

Subramonyam, Hariharan, Colleen Seifert, and Eytan Adar. 2021. ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In Proceedigns of 26th International Conference on Intelligent User Interfaces. ACM, Virtual, 48–58.

Digital Library

[44]

Verheij, Bart. 2020. Artificial intelligence as law. Artificial Intelligence Law 28, 2 (2020). 181-206.

[45]

Wan, Zhiyuan, Xia, Xin, Lo, David, and Murphy, Gail C. 2019. How Does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering, 47:9. 1857-1871

[46]

Wickramasinghe, Chathurika S., Marino, Daniel L., Grandio, Javier, and Manic, Milos. 2020. Trustworthy AI Development Guidelines for Human System Interaction. In International Conference on Human System Interaction. 130-136.

[47]

Windl, Maximiliane, Feger, Sebastian S., Zijlstra, Lara, Schmidt, Albrecht, and Wozniak, Pawel W. 2022. ‘It Is Not Always Discovery Time’: Four Pragmatic Approaches in Designing AI Systems. In CHI Conference on Human Factors in Computing Systems. 1-12.

Digital Library

[48]

van Wynsberghe, Aimee. 2021. Sustainable AI: AI for sustainability and the sustainability of AI. AI and Ethics, 1:3. 213-218.

[49]

Zhang, Jie M., Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes, and horizons. IEEE Transactions on Software Engineering.

Digital Library

[50]

Xu, Wei. 2019. Toward Human-Centered AI: A Perspective from Human-Computer Interaction. Interactions 26, 4 (2019). 42-46.

Digital Library

[51]

Yang, Qian, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu, 1-13.

Digital Library

Cited By

Kucher KZohrevandi EWestin C(2025)Towards Visual Analytics for Explainable AI in Industrial ApplicationsAnalytics10.3390/analytics40100074:1(7)Online publication date: 12-Feb-2025
https://doi.org/10.3390/analytics4010007
Schmager SPappas IVassilakopoulou P(2025)Understanding Human-Centred AI: a review of its defining elements and a research agendaBehaviour & Information Technology10.1080/0144929X.2024.2448719(1-40)Online publication date: 16-Feb-2025
https://doi.org/10.1080/0144929X.2024.2448719
Bach TKaarstad MSolberg EBabic A(2025)Insights into suggested Responsible AI (RAI) practices in real-world settings: a systematic literature reviewAI and Ethics10.1007/s43681-024-00648-7Online publication date: 8-Jan-2025
https://doi.org/10.1007/s43681-024-00648-7
Show More Cited By

Index Terms

Human-Centered AI Design in Reality: A Study of Developer Companies’ Practices : A study of Developer Companies’ Practices
1. Computing methodologies
  1. Artificial intelligence
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Towards a Human-Centred Artificial Intelligence Maturity Model
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

Artificial intelligence (AI) is becoming a central building block of computational systems. Following the long traditions of human-centered design, Human-Centered AI (HCAI) emphasises the importance of putting humans and various societal considerations ...
Enhancing Explainability in Medical AI: Developing Human-Centered Participatory Design Cards
HCI International 2024 – Late Breaking Papers
Abstract
Explainable artificial intelligence (XAI) aims to develop AI systems that are easy for humans to understand and explain the decision-making process clearly. Especially in high-risk fields such as medical, the explainability of AI systems is ...
Practising Appropriate Trust in Human-Centred AI Design
CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

Appropriate trust, trust which aligns with system trustworthiness, in Artificial Intelligence (AI) systems has become an important area of research. However, there remains debate in the community about how to design for appropriate trust. This debate is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

NordiCHI '22: Nordic Human-Computer Interaction Conference

October 2022

1091 pages

ISBN:9781450396998

DOI:10.1145/3546155

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2022

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

NordiCHI '22

NordiCHI '22: Nordic Human-Computer Interaction Conference

October 8 - 12, 2022

Aarhus, Denmark

Acceptance Rates

Overall Acceptance Rate 379 of 1,572 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
1,719
Total Downloads

Downloads (Last 12 months)831
Downloads (Last 6 weeks)103

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kucher KZohrevandi EWestin C(2025)Towards Visual Analytics for Explainable AI in Industrial ApplicationsAnalytics10.3390/analytics40100074:1(7)Online publication date: 12-Feb-2025
https://doi.org/10.3390/analytics4010007
Schmager SPappas IVassilakopoulou P(2025)Understanding Human-Centred AI: a review of its defining elements and a research agendaBehaviour & Information Technology10.1080/0144929X.2024.2448719(1-40)Online publication date: 16-Feb-2025
https://doi.org/10.1080/0144929X.2024.2448719
Bach TKaarstad MSolberg EBabic A(2025)Insights into suggested Responsible AI (RAI) practices in real-world settings: a systematic literature reviewAI and Ethics10.1007/s43681-024-00648-7Online publication date: 8-Jan-2025
https://doi.org/10.1007/s43681-024-00648-7
Lin HKarusala NOkolo CD'Ignazio CGajos K(2024)"Come to us first": Centering Community Organizations in Artificial Intelligence for Social Good PartnershipsProceedings of the ACM on Human-Computer Interaction10.1145/36870098:CSCW2(1-28)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687009
Dritsa Dvan Renswouw LColombo SVäänänen KBogers SMartinez AHolbrook JBrombacher A(2024)Designing (with) AI for WellbeingExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636282(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636282
Xu WGao Z(2024)Enabling Human-Centered AI: A Methodological Perspective2024 IEEE 4th International Conference on Human-Machine Systems (ICHMS)10.1109/ICHMS59971.2024.10555771(1-6)Online publication date: 15-May-2024
https://doi.org/10.1109/ICHMS59971.2024.10555771
Bach TKristiansen JBabic AJacovi A(2024)Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature ReviewIEEE Access10.1109/ACCESS.2024.343719012(106385-106414)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3437190
Truss MSchmitt M(2024)Human-Centered AI Product Prototyping with No-Code AutoML: Conceptual Framework, Potentials and LimitationsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2425454(1-16)Online publication date: 26-Nov-2024
https://doi.org/10.1080/10447318.2024.2425454
Ala-Luopa SOlsson TVäänänen KHartikainen MMakkonen J(2024)Trusting Intelligent Automation in Expert Work: Accounting Practitioners’ Experiences and PerceptionsComputer Supported Cooperative Work10.1007/s10606-024-09499-633:4(1343-1371)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10606-024-09499-6
Ryan M(2024)We’re only human after all: a critique of human-centred AIAI & SOCIETY10.1007/s00146-024-01976-2Online publication date: 4-Jun-2024
https://doi.org/10.1007/s00146-024-01976-2
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten