Human-Centered AI Design in Reality: A Study of Developer Companies’ Practices
A study of Developer Companies’ Practices

Maria Hartikainen, Faculty of Information Technology and Communication, Tampere University, Finland, maria.hartikainen@tuni.fi

Kaisa Väänänen, Faculty of Information Technology and Communication, Tampere University, Finland, kaisa.vaananen@tuni.fi

Anu Lehtiö, Faculty of Information Technology and Communication, Tampere University, Finland, anu.lehtio@tuni.fi

Saara Ala-Luopa, Faculty of Information Technology and Communication, Tampere University, Finland, saara.ala-luopa@tuni.fi

Thomas Olsson, Faculty of Information Technology and Communication, Tampere University, Finland, thomas.olsson@tuni.fi

DOI: https://doi.org/10.1145/3546155.3546677
NordiCHI '22: Nordic Human-Computer Interaction Conference, Aarhus, Denmark, October 2022

Human-Centered AI (HCAI) advocates the development of AI applications that are trustworthy, usable, and based on human needs. While the conceptual foundations of HCAI are extensively discussed in recent literature, the industry practices and methods appear to lag behind. To advance HCAI method development, current practices of AI developer companies need to be understood. To understand how HCAI principles manifest in the current practices of AI development, we conducted an interview study of practitioners from 12 AI developer companies in Finland, focusing on the early stages of AI application development. Our thematic analysis reveals current development practices and identifies four main challenges: (i) detachment of HCAI work from technical development, (ii) clients’ central role as the source of user requirements, (iii) uncertain nature of AI, and (iv) lack of value-based understanding of AI. The findings inform the development of HCAI methods and implementation of HCAI principles in AI application development.

CCS Concepts: • Computing methodologies → Artificial intelligence; • Human-centered computing → Human computer interaction (HCI);

KEYWORDS: human-centered artificial intelligence (HCAI), human-centered design, AI applications, company practices

ACM Reference Format:
Maria Hartikainen, Kaisa Väänänen, Anu Lehtiö, Saara Ala-Luopa and Thomas Olsson. 2022. Human-Centered AI Design in Reality: A Study of Developer Companies’ Practices : A study of Developer Companies’ Practices. In Nordic Human-Computer Interaction Conference (NordiCHI '22), October 08-12, 2022, Aarhus, Denmark. ACM, New York, NY, USA, 16 Pages. https://doi.org/10.1145/3546155.3546677

1 INTRODUCTION

In recent years, the progress of AI has been driven by the development of new algorithmic approaches and AI has been largely adopted to new types of applications and business opportunities that this technology might enable [4, 43]. Recent studies suggest that AI's role in our everyday lives will only continue to grow as it will be possible to infuse into various systems [29, 35, 55]. New technologies and processes for solving problems or automating decision making bring not only new opportunities, but new challenges as well [39]. AI systems can make unpredictable decisions or predictions that harm the user experience (UX) and can even lead to undesired societal impact like unfair treatment or discrimination of minority groups or privacy violations [43, 48, 56]. In this situation, HCAI underlines the call for more focus on the human-centered and societal considerations of AI, its applications, and human-AI interaction in diverse application areas [43, 48].

Human-Centered Artificial Intelligence (HCAI) is an umbrella term referring to various individual, societal, and ethical considerations related to the development of AI [42, 55]. HCAI is based on the idea of human-centered technology development, ensuring that the interaction and UX are appropriate and delightful for the users [20, 46]. This includes the basic starting points of understanding user needs and the contextual and sociotechnical factors of system design, as well as introduces new ones that are particular to AI as technology [4, 20, 27, 34]. AI-specific issues, such as transparency and fairness, have been actively discussed over the past decade [3, 17, 43]. Designing AI with a human focus is essential for end-users' well-being and for tackling ethical issues that may cause unwanted societal-level consequences [35, 43, 48, 55].

Despite a strong interest in HCAI in academic research, there is little research-based understanding of how the new AI-related requirements and principles manifest in practise in AI development. Our exploratory research is motivated by the goal of forming new knowledge on how the human-centered design practices are realised in the context of developing AI applications. We believe that deeper understanding on the processes that companies have for developing AI applications can advocate the design of appropriate guidance and support to develop human-centered AI applications that users can trust. We examine AI application development practices from the human-centered perspective. We focus on understanding the practices in the early-phase of AI application development because most of the decisions on how the user needs and other human-centered requirements are met, are typically done early in the development process. As the “early-phase” we refer to the three first steps of the Software Development Life Cycle: planning, requirements gathering, and design [5]. In the currently prevailing agile Software Development (SD) processes, also the development phase may include continuous design decisions affecting the user. The research questions we address in this paper are:

RQ1: What are the characteristics of early-phase AI application development practices?RQ2: How is human-centeredness considered in companies’ AI application development practices?We approach this research goal by interviewing 12 IT professionals who are working on AI application development in Finland-based companies, to gain in-depth understanding of their development practices. Participating companies represent a variety of domain areas and company settings. The AI applications these companies develop vary from automated sensor-based applications to recommendation systems.

The contribution of this paper is empirical, and the findings offer novel insights to cross-cutting themes of HCAI. The findings can add to the understanding of gaps between theoretical ideals of HCAI and the reality in companies. As AI's role in our everyday lives continues to grow, this will benefit not only users and developer companies, but society, too.

2 BACKGROUND AND RELATED WORK

The first subsection builds an understanding of HCAI principles. The second discusses recent research on how AI applications are being developed.

2.1 Human-centered AI principles

Several central principles related to HCAI can be extracted from the related literature. Explainable AI (XAI) has been among the most discusses themes of HCAI. Explainability helps users to understand the algorithms and parameters a system uses by, for example, giving a reason for a particular action, increasing trust in AI and supports interaction between the user and AI [3, 15, 22, 55]. A study investigating problems reported by users states that the explainability should concentrate more on the outcome instead of the operating [18]. However, the author of [24] states that users do not really care how the application works as long as it works, hence proper human computer interaction (HCI) design would make the need for explainability obsolete. Transparency of the system aims to increase the ability for people to understand the reasoning behind systems’ inner workings, and that way to build trust between human and AI [6, 51]. Ethical AI is meant to harness the disruptive potentials of new AI technologies, as ethically designed AI is aligned with social values and fundamental rights [23]. Fair AI aims for decision making that is non-discriminatory and fair, since the purpose of fair AI is to prevent harm or benefit to different groups of people affected by AI [17]. Trustworthy AI aims to increase the user trust in AI systems, allowing users utilise AI without fear [51]. Responsible AI refers to efforts to foster AI research and development toward socially beneficial applications, as well as mitigating the human and social risks of AI systems [19]. Sustainable AI focuses not only to the AI applications, but the sociotechnical system of AI as whole, as it is important, that AI applications are both developed and used with ecological integrity and social justice in mind [53].

The topic of HCAI seems interesting both to academy and industry. Academic research in this topic has been resulting in aids to guide and support HCAI design. In [4] the authors offer an overview of different human-centered design (HCD) approaches and their contribution to the development of HCAI, concluding comprehensive or pan-disciplinary design approaches are needed to design, develop, and advance human-centered AI. In [2] and [51] the authors present human-AI interaction design guidelines for design and evaluation of AI applications and systems. To aid the UX designer's ideation processes, [31] present design heuristics and a framework for AI design. The authors of [43], [55], and [56] have suggested frameworks to guide and support HCAI development. Industry has participated to HCAI development by suggesting design approaches and practices for human-centered AI [12, 25, 30, 36]. However, HCAI is not yet a mature discipline as it has no clear definition nor established methodology [39].

2.2 AI application development practices

2.2.1 AI and software development. In related literature one of the biggest mentioned differences between AI applications and non-AI applications is the data-driven nature of AI applications. Measures related to data and AI models require new kinds of roles and expertise, as well as activities that should be integrated to common SD processes to ensure project success [1, 26, 56]. AI software changes its behaviour based on the new data it processes, and this kind of uncertainty causes challenges [27, 52, 56]. Study examining ML application development practices in Microsoft found that their ML development workflows follow pre-existing, agile-like software engineering processes [1]. Study investigating the changes that applying ML to software causes to software development practices found that applying ML significantly affects various aspects of software engineering, like requirements, design, testing, processes, as well as work characteristics [49]. In [28] the authors investigated SD professionals who use ML techniques to develop intelligent systems and present a high-level description of the process the developers follow. They found that the developers face challenges in every phase of this process and that the developers struggle especially to produce repeatable processes.

2.2.2 AI's and HCI. Studies investigating human-AI interaction highlights the potential this new kind of collaboration has – together the human and technology can solve problems in unforeseen ways as they work together [2, 14]. The vast amount of data that is collected for the AI algorithm can be a source of information for the UX design. For example, it can be utilised to determine use cases and user groups. This can be beneficial especially when designing applications that are meant for big user groups.[9, 11] As many AI applications evolve during time, researchers has determined that AI can be used as a material for UX design before, during, and after the implementation [33].

However, previous research shows that designers struggle to understand AI capabilities, meaning that they do not fully understand what AI can and cannot do. AI-powered interactions can adapt to different users and use contexts and they can evolve over time and for this uncertainty designers have challenges in envisioning novel AI solutions for a given problem [33]. Designers also find it difficult to ideate many possible new interactions, as the adaptability of AI means that there could be infinite number of outcomes.[16, 56] Prototypes are a common tool in testing, but research shows that designers struggle to prototype AI systems largely because of AI's dynamic behaviour in response to training data, end-user data, and variations in input data individual users contribute [47]. A study investigating the changes AI causes to the design process shows that the design process of an interactive system is affected when AI is integrated and that design teams adapt their processes to accommodate AI [52].

Recent research stresses the importance of specialists – engineers, data scientists, HCI and UX specialists – working together in order to craft efficient and useful AI applications [9, 21, 27]. As humans are increasingly engaging with AI systems and AI's algorithm-based decision making, it is crucial that UX researchers are involved in order to include end-user values throughout the AI development [8]. However, designers report challenges in collaborating with AI engineers, because first, these two groups do not share workflows, and second, if they do, they seem not to share the same language, and this causes problems in understanding the other party [54].

Many of the prior studies investigate AI application development practices from certain perspective, e.g., ML or SD, and concentrates on how AI changes accustomed processes. In the field of HCI much of the literature concertantes on guiding designers and practitioners on how to work with AI and how to design AI in human-centered way. However, to our knowledge, the topic of how users or the principles of HCAI are considered in real life AI application development in develop companies is under researched. To fill this gap, in this paper we examine AI developer company practices specifically from a human-centered design perspective to find what is the role of the human in AI application development and how the principles of HCAI have been acknowledged in the development. Our work hence contributes to HCAI by formulating insights of AI application development practices and human-centeredness of these practices.

3 METHODOLOGY

In this section we describe participant recruitment, the study procedure, and the analysis method.

3.1 Participants

To recruit relevant participants, we contacted software companies that are developing AI applications in Finland. Due to the regional nature of the research project, we initially searched for suitable companies from a company data base managed by the city of Tampere. This was later complemented with general Internet search to find recently established start-ups, for example. Subsequently, we approached seemingly relevant individuals, such as designers and developers in lead roles, via email invitations to participate in a voluntary interview (no compensation offered). The invitation specified that the interview would focus on the development of AI applications and the human-centered aspect in these practices. We required the participating person to be familiar with the development process of AI applications in their company. Based on this iterative snowball sampling, we managed to recruit 12 participants, each from different companies. Ten of the companies were small and medium-sized enterprises (SMEs, i.e., <250 employees) and two companies were classified as large. To understand the evolving practices in contexts without strong organisational or legal frameworks guiding the development, we focused on SMEs and other companies where the development processes could be shaped primarily by the teams.

The participating companies have from six months to eight years of experience with developing AI applications, with the mean of 2,5 years. All participants also have experience with developing non-AI applications, either currently or previously. Participating companies represented a variety of domain areas. Different AI techniques were represented in this study, ML and Natural Language Processing (NLP) being the most common. Some of the companies concentrate on a certain application domain or AI technique, whereas others offer a wide range of AI solutions. Table 1 shows the background information of the participants and the companies they represent.

Table 1 - Background info of the companies and the participants

Participant	Company's AI application domain	Primary AI technology	Participant's role(s) in the company	Company size
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12	Customer service chatbot Enterprise resource planning system Digital services using multiple AI techniques Digital services using multiple AI techniques Computer vision services for different affordances and domains Internet of Things solutions for different domains Sensor-based solutions for public spaces Digital services using multiple AI techniques ML services for different domains Intelligent lighting solutions for industry Customer service chatbot Customer service solutions	NLP ML Multiple Multiple CV ML ML Multiple ML ML NLP NLP, CV	UX designer CEO Developer, project manager Project manager Project manager Developer CEO and developer Management consultant Data scientist Management consultant Project manager, designer Developer	SME SME SME Large SME SME SME Large SME SME SME SME

The participating companies’ clients are usually organisations that want to integrate AI into their product or service. Usually there is one key person representing the client company that deals with the developer. We refer to this person as the client. The end-user of the developed applications is, for example, a customer or an employee of the client company. We refer to the end-user as the user.

3.2 Interview procedure

To understand AI application development practices, we adopted a qualitative study approach. The aim of the interviews was to gather insights into the development practices of AI applications in companies, AI-related challenges, and human-centered practices relevant to application design. We interviewed AI developers and qualitatively analysed interview transcripts through the HCAI lens. We chose to approach the topic with semi-structured interviews, as the interview research method is well-suited for the exploratory nature of this empirical work. We started all the interviews with demographic questions about the company's experience with AI, and their motives to work with it. We also asked participants details about the participating team. After the warm-up questions we had ten pre-prepared questions to guide the interview that addressed four themes: (i) company's development practices of AI applications, (ii) the impact of AI on the development practices, and (iii) the ways the user is considered in the development. Questions were open-ended to understand the characteristics of the practices. Follow-up questions were additionally asked. This study was conducted during the COVID-19 pandemic, so all the interviews were conducted remotely using Microsoft Teams or regular phone calls, depending on the participant's preferences. The interviews were recorded for analysis purposes with the participant's permission. Each interview lasted about an hour – the shortest being 29 minutes and the longest 74 minutes. In total we had 551 minutes of interview audio. The first author transcribed and anonymised all interviews. We had total 57414 words of interview transcriptions.

3.3 Interview analysis

As this was an exploratory study, we adapted a thematic approach to analyse the interview data to find reoccurring patterns and themes [7]. We had two iteration rounds. First, three of the authors conducted preliminary analysis of the interviews, either with the transcripts or with the recordings. We read all the transcripts to obtain an overview of the similarities and differences found within the dataset. After that, each of three coders annotated each interview transcript following an inductive coding approach where we developed the codes as we analysed the transcripts [13]. We separately coded all the data by tagging specific parts of texts with codes. Then we combined the three sets of codes. After merging the codes, we searched for similarities and reoccurring themes and patterns across interviews, and we recognised six preliminary themes. Second, we conducted an additional round of data analysis based on the recognised themes from the first analysis round. This was supported by analytic memo writing to collect observations and ideas from the data [41]. Then we examined each theme to gain an understanding of the characteristics of the development practices and human-centeredness (or lack thereof). We combined two of the preliminary themes, and that resulted in five main themes of which three were related mainly to RQ1 and two to RQ2. We recognised 12 subthemes, providing insights into the AI application development practices and the human-centered traits of them. Table 2 present the codes, themes, and subthemes.

Table 2 - Codes and recognised themes

Codes	Theme	Subtheme
Process, design, tool, development, training, method, challenge	Design and evaluation practices	AI developers are often responsible for the early phase of design
	Design and evaluation practices	No strict processes but flexible adaptation of practices and workflows
Data, uncertainty, AI specific	Data-driven and uncertain nature of AI	Importance of data – it is all about quality data
		Uncertainty of AI – it must be tested and retested
		Early-phase testing with clients, not with end-users
Client, communication, expectation, domain, requirement	Communication as the key to fully benefit from AI	Client is the king even without AI knowledge
Client, communication, expectation, domain, requirement	Communication as the key to fully benefit from AI	Educating the client while listening to them
User, usability, UI, stakeholder, adaptivity	Bringing user needs into the design	User needs guide the design
		Data as design material
		Adaptivity serves a wide user population
HCAI	HCAI design requirements and opportunities	AI is a tool among other tools, but a powerful one
HCAI	HCAI design requirements and opportunities	Transparency and explainability are needed

4 RESULTS

In this section, we present the results of the thematic analysis under two sections, each presenting themes that address one of the two research questions. When reporting the qualitative findings, the number of interviewees addressing each subtheme is mentioned in parentheses (X/12).

4.1 Current practices in AI application development (RQ1)

This section presents the three identified themes related to RQ1, characterising the current practices of AI application design.

4.1.1 Theme 1: Design and evaluation practices.

4.1.1.1 AI developers are often responsible for the early phase of design. When asked about the team participating in the AI application development, most participants (10/12) described this team to be mostly technical, a combination of project managers, developers, programmers specialised in different AI techniques, and data scientists. Participants reported that, in addition to the AI development, AI teams are responsible for the need assessment, early-phase interaction design, and communications with the client. After the AI algorithm is working and its functionality has been tested and it has been trained, the work of the AI team finishes and the work of the UX team can begin. We found that in many of the participating companies (10/12) there is a clear separation between the AI and UX teams. UX practioners are not considered to be a part of the AI team, nor are they involved in the early-phase development practices. In many cases, these two teams do not work together nor share knowledge. “When the ML part of the project is ready, some software developer enters the picture and does the UI on top of it. At that point, our job (“AI team”) is done, and ML understanding is not needed anymore” (P9). In contrast, in two of the participating companies, UX professionals are part of the AI team. One participant specified that it is the UX people who are directing the AI projects. Another participant explained that in their company, the AI and UX teams are separate, but that the UX team is supported by the AI team, to share insights of the AI functionality.

4.1.1.2 No strict processes but flexible adaptation of practices and workflows. Participants were asked to describe their usual ways of working or to give a precise example of an AI application project. Most participants (8/12) reported that they do not have established processes when designing AI applications: “We do not really have processes, just enough so that it would not be chaos” (P8). Participants reported that this is because of the uncertainty or dynamic nature of AI – the ways of working are so strongly dependent on the data, the product, used AI technique, and the client. The influence of regular SD practices is strong, although most of the participants agreed that designing AI products differs from designing non-AI products (11/12). Participants specified that they take components of familiar SD processes that they know from experience to be good solutions: “We cherry-pick the most suitable parts (from SD experience) and leave out the rest” (P1). In contrast, one company was said to have a process that they modify to AI if necessary. Although most participants (10/12) agreed on not having explicit processes, we found noticeable similarities and reoccurring steps in the ways of working reported by the participants. Based on the reported ways to design AI applications by the participants, we identified four workflow stages: (i) design, (ii) data, (iii) model, and (iv) test and analyse. In the design stage the AI team defines the problem, collects domain knowledge, and decides which user needs they should address and what way. During the data stage, developers collect, investigate, and clean data. Choosing or building a model was reported to be one very specific AI development step. Some companies build their own models (4/12) whereas some of the companies used third-party models (4/12). The test and analyse phase targets the functionality of the AI model with the available data, and the results are analysed concerning the expectation and metrics set by the client.

4.1.1 Theme 2: Data-driven and uncertain nature of AI.

4.1.2.1 Importance of data – it is all about high-quality data. All participants agreed that developing AI applications differs from non-AI applications and data was mentioned as one of the main reasons for this. To make AI application work, large amounts of high-quality data are needed. “The premise of AI is that the solution can only be as good as the data” explains (P11). But as one participant highlights: “There is never enough data” (P3) and that is one of the biggest challenges related to AI application projects. After the data is collected, companies must build or choose a suitable model for processing the data. Most participants (10/12) stressed that this is one of the most AI specific aspects of the development process, because is important to use a model that can achieve the wanted results from the available data. Participants agreed that the availability and the quality of data is one the biggest challenges in the early phase of AI application development. In addition, the lack or low quality of training data may also cause problems because the AI training might be inadequate, it cannot be done, or it does not train the AI to function in the intended way. One participant also reminded that training data suffices for testing, but that real data from the client is required in order to train the AI to make sure that the application works properly in the client's working contexts: “Of course, we can easily create some test data and use it to test basic functionality. But yes, AI requires the real data from a real customer to work properly” (P8).

4.1.2.2 Uncertainty of AI – it must be tested and retested. Uncertainty came up often among the participants to describe AI's specific traits: “The key to AI product development is how you work with the uncertainty (of AI). It is a challenge but not an obstacle” (P3). Uncertainty caused by AI manifests itself in several ways in the early phase of development. First, many participants (7/12) brought up the issue of uncertainty in the AI functionality - it is not certain if the company can produce a working solution to the client's problem, as AI's uncertainty arises because of the data, the model, and their interworkings, or the adaptative nature of AI. One participant highlights this issue: “Even if the solution is perfect, the outcome of AI cannot be perfect, there is always the uncertainty” (P1). Second, uncertainty relates to the project. In a regular SD project, it is clearer to define the problem and ways to solve it, yet AI is different due to its inherent data-driven nature. Before collecting the data, choosing a model, and testing how they work together, you cannot know if you can achieve the desired results. “It is a reality that often these AI projects end, because the data does not offer the needed outcomes” (P9).

4.1.2.3 Early-phase testing with clients, not with end-users. All participants agreed that data and the model are the most specific elements related to AI in the early phase of development. The success of the functionality cannot be deduced directly from these elements or how they work together. Hence, all participants reported early testing to be an important part of their practices: “First we must figure out if this idea really works. Is there enough information in the data? Is it possible to find an accurate solution to the problem? That is the first step to every project, the only common thing in them” (P8). Participants explained that the testing serves a few different purposes. It is done to see what the data has to offer, and to test if the AI model can generate expected results from the available data. In addition, several participants (5/12) explained early testing serving as a user need mapping method, as sometimes it is not clear for the client what AI can do, and that is why they might not be able to specify all the needs until they see how AI works. Several participants (5/12) reported that they are using quick prototypes, visualizations, or simulated examples for the early testing. Two participants reported that simple user interfaces (UI) might be developed for the testing, but this was usually related to demonstration to the client. In some participating companies (5/12) this early-phase testing is done only internally. Other participants (6/12) reported that it is important to include a client in this phase: “The communication with the client is often difficult, so we try to get something to show and give the client to test and see as early as possible, so that we can ensure we are going in the direction the client wants” (P9). In most of the participating companies, users are not included in this testing phase (11/12).

Participants reported various challenges that AI's uncertainty brings to the testing. First, because the operation of AI often changes, it is difficult and time-consuming to test (4/12). In addition, the continuous learning and adaptive nature of AI means continuous testing (1/12). One participant explained this that testing requires a lot of time and work if aiming at a comprehensive set of use cases. Nevertheless, not all scenarios can be predicted nor tested. This means that no one is really sure how AI is going to behave or what it is going to do.

4.1.2 Theme 3: Communication is the key to fully benefit from AI.

4.1.3.1 Educating the client while listening to them. Most participants (11/12) considered communication between the developer and the client to be a critical success factor in the early phase of the development, as AI and its functionality might be difficult for the client to comprehend. Communication is important for gaining the necessary synergy and to get a better picture of the use context. “We focus specifically on understanding the client's operating environment and trying to become part of the client's organisation and help them that way. [...] Then at that point we offer our own knowledge and tell them as much as they find interesting. In a way it's a bit like educating and listening to the client at the same time” (P3). However, most participants (9/12) mentioned the communication gap being a recurrent challenge in AI projects. This was reported to be due two reasons. First is the client's lack of skills or expertise, as participants mentioned that it can be difficult to explain the benefits and opportunities of AI, if the client is not familiar with the basic functions of AI. Another reason mentioned by the participants is that the client may have unrealistic perceptions of AI, obtained from media or even sci-fi movies. This can cause communication difficulties, as well as over-expectations, e.g., it is expected that the chatbot is as evolved as Siri, and the resulting disappointment with the product. One participant explains the issue: “How can we meet the expectations that are inside of someone's head and are based on sci-fi movies?”(P1). Nevertheless, communication is critical for transparency and trust between the client and the developer, and most participants (9/12) felt positive that they can work with this challenge with the help of early prototypes, visualisations, and including the client in the testing. They explained that by conversing and educating the client on how and what AI can and cannot do and involving them in every phase of the development, they can demonstrate how difficult it is for AI to produce even basic answers.

4.1.3.2 The client is the king even without AI knowledge. It emerged clearly in the interviews that in the early phase of AI application design the client is the king. Most participants (10/12) brought up the strong client involvement in AI projects. First, the client is the one bringing the problem that needs to be solved, with a need for a suitable solution. Participants stressed that although they have the technical understanding and skills, the only thing they do is give suggestions to the client and based on those the client makes the decisions about the AI. Second, the client is expected to participate throughout the process with involvement in various phases, e.g., initial planning and testing. Sometimes clients have neither time nor interest to participate, and many participants reported this to be problematic, as participation in the project was seen as a way to share information, too. Many of the AI products require training, and that is something that should be done by the client after the implementation. If the client does not have time for this, it might cause the AI to malfunction. Some participants (3/12) saw the client as representative of the user and were referring to the client when asked about the user.

4.2 Human-centeredness in the AI development practices (RQ2)

The human considerations and HCAI aspects of the design are presented in this section, addressing RQ2.

4.2.1 Theme 4: Bringing user needs into the design.

4.2.1.1 User needs guide design . Most participants (11/12) mentioned user needs to be the basis of the entire design. Usually, the information on the users and their needs is collected by conversing with the client (6/12). In addition, early demonstration and testing the AI functionality are also used for mapping the user needs, because seeing something concrete helps the client realise the ways AI can behave. In some of the participating companies (4/12), user needs are mapped inside the company, because they understand AI and its possibilities, and this is viewed as helpful to map the user needs. Another participant said that they simply guess based on strong self-confidence. A few participants (3/12) perceived acknowledging the human and their needs mostly as a part of UI design, that happens in later phases of the development: ”If we talk about the end-user, UI is the way to answer their needs. It's hard for me to come up with any user-related practices during the early-phase of development” (P9). In most of the participating companies (9/12) it is the technical AI development team who maps the user needs. A few (3/12) companies reported UX designers also being involved. Two of the participating companies included the end-user in this step by observing the ways of work and discussing with them about their needs and the use context. Participants reported resources to be the biggest motive for not including the user to the need assessment. For some (5/12), including the client is enough, like P9 explains: “We have the client there (involving to the project), so user involvement is not needed”. Some of the participants expressed that they had a strong trust in their own ability to map the user needs, so involving the user was not considered necessary. One participant explained this to be because of the uncertainty of AI: ”It's pretty pointless to ask the user at this stage what kind of chatbot you'd like if we're not sure it's technically even possible to make work.”(P8), adding that in these kinds of situations it is better to turn to the data.

4.2.1.2 Data as design material. At the beginning of the AI project, vast amount of data needs to be collected for the AI model to work in wanted way. We found that in most of the participating companies (9/12) this collected historical data was seen as material for the AI team only, and UX practioners did not work with this data. In contrast, one participant explained that AI and especially the data offers a possibility to broader organisational changes. The goal is to change the organisation and its operations, not just to automate existing operating models or processes, but by changing ways of working and ways of managing with the help of its data. Another participant reported that they use historical data as a base for the design, collecting information such as usage data and telemetry. One participant explained, that instead of involving the users to the design, it is better to consult the data and determine the users and the use cases from that. Another participant also mentioned that in their company they are utilising historical data to understand users better.

4.2.1.3 Adaptivity serves a wide user population. Two participants saw AI, especially ML, human-centered because of its adaptive nature. One participant specifies that “regular software you design around the needs of the average user” […] ”The user learns to interact with the application and adapts their ways of working to this. AI adapts to the ways of working of the user” (P8). Two participants highlighted that by adapting to different users, AI applications enable acknowledging various users and their needs. However, this is done with data and means that it requires enough data from different user groups. One participant mentioned that different AI techniques could be utilised for better usability and UIs. They explained that AI can meet the needs of broader user groups or offer possibilities for use that are difficult or sometimes even impossible to implement with regular code, like NLP as part of UIs and as a means to interact with the technology.

4.2.2 Theme 5: HCAI design requirements and opportunities.

4.2.2.1 AI is a tool among other tools, but a powerful one. Most participants (10/12) explained that the main motivation to use AI is to solve a problem or to meet a need. Several participants (7/12) highlighted that AI provides technological solutions that are not possible, or would be much harder to implement, with regular software solutions: ”You cannot program things that AI is capable of doing. You have an electronical brain that recognises, analyses, and learns the things. Programming something like that would take infinite time” (P7). There is an ongoing hype around AI, and that has created a strong demand for AI from the client's side, as P1 explains: ”Often the client assumes that we have a magic jar labelled ‘AI’, we open the jar, and the problem is solved”. However, it was emphasised by the most participants (8/12) that AI is not used if it is not the best solution to the problem: “The business case cannot be found from an AI sticker on the side of the product, but it can be found from a solution that brings concrete value” (P4).

4.2.2.2 Transparency and explainability are needed, but so is hiding complexity. A few participants (3/12) brought up the importance of transparency and explainability in AI applications. Based on the interviews we found no set rules or established practices when it comes to explainability and transparency, and that their use depends on various aspects, like used AI-techniques, use context, or the level of the user control and responsibility. The information provided also depends on the user, e.g., their technical understanding or desire to understand. In many companies these decisions are done based on the opinions of the AI developers. On the contrary, one participant reported that when they consider the use of explainability or transparency, their UX practioners come to collaborate in the design.

Participants were questioning how much is “necessary” to convey about AI to the user. How much are users even interested in this information and at what level? Will there be a blast of information boxes? We found that in many of the participating companies (5/12) AI is intentionally hidden in many contexts. Developers decide what parts of AI should be visible or controllable for the user, and the rest they try to hide from the user. Participants explained that this is done because the user does not want too much information, while many agreed that the user is not really interested in how the system works as long as it works, as one participant explains: “The user is only interested in the results. It doesn't matter how they are produced.” (P2). A few participants also mentioned that it is user friendly to make the application easy-to-use and easy-to-access, and for that it may be better to hide some features of AI from the user. One participant explained that people do not fully understand what AI is and what it can do, and that scares them - they do not want to give the control to AI, because they do not know what the consequences are. For this reason, the participant explained, it is better to hide AI, so that the user can just concentrate on the use, not the fear of AI. Another participant added that it is easier to not to mention AI, because people might have very unrealistic expectations of AI, and if the used AI product does not match with these expectations, the UX might be very disappointing.

5 DISCUSSION AND FUTURE WORK

5.1 Early-phase design practices and values in AI application development

Most of the participants reported that they have no strict processes but general steps that seemed to be similar in most of the companies and between different AI techniques. Aligned with the findings of [1] and [28], we found that early-phase AI design relies strongly on the experience, methods, practices, and tools familiar from regular software development. Relying on SD practices means that developers continue their work as usual, without reconsidering their methods or practices. Adapting to new ways of working was seen more important to the client company, not for the developers. Nevertheless, AI requires new tasks, practices, and expertise. Beyond the questions of early-phase data-driven testing, special characteristics of AI were not given explicit consideration. For example, many companies use training data that comes from outside sources. This seems to be a common practice in the field, but also with such third-party data, the AI application should also be tested for biases [38, 45], but none of the participants reported this being part of their practices.

Regarding the composition of the team participating in the AI design process, it seems that usually the UX people are not considered as part of the development group. Some participants said that they do not know what happens in the product design process after the AI part is technically functioning. This indicates that there is a gap between the people involved in the technical development of AI and the UX designers. This consolidates the findings of [39] and [54]. To fully benefit AI, it is important to understand its functioning, as well as the user needs. Hence, the integrated composition of AI and UX professionals ideally work tightly together. Also, it is important to support HCAI design also in situations when no UX professionals are involved, by offering clear methodological approaches.

The strong role of client's is novel information - in many companies, the client has the last word throughout the development process. However, most of the participants reported that one of the biggest challenges in AI application development is client's lack knowledge of AI. Client-centered thinking is common in the business world in general, but the characteristics of AI introduce a problem – the client is not expected to have too much knowledge of AI and how it works, yet they are responsible for the decisions about AI. Hence, clients implicitly take responsibility for the consequences caused by its functions, for example related to fairness or the ethics of AI. If the client is not aware of this responsibility, this might result in a situation that the design of AI applications leans heavily on the values of the development company. The changes that designing AI in a human-centered way requires, are not only related to aspects of the development process, but also to the company culture.

5.2 Human-centeredness in AI application development

The motives for the use of AI seem to be human-centered - AI is used only when it is the best solution to the problem or has something unique to offer - like means to process data in a meaningful way. This is the premise of HCAI regarding [36] and it contrasts with the current understanding of AI development being technology-centered. User needs guide the development from the beginning; however, AI development practices seem to be more user-concerning than user-including. In most of the participating companies, the considerations regarding the user needs are made internally based on the opinions of the technical team or the client. Including the users in the early-phase design was not considered important, and it was evident that companies trust their own skills and opinions in decisions regarding the user. Also, when referring to a user, many of the participants spoke about the client, even in cases when the client was not the user. It might be problematic to erroneously consider the client as the user. The needs of the client can be considered the user needs, and the design can be based on those, and then the result serves the needs of the clients, not those of the user. Nevertheless, many decisions seemed to be based on this concept.

Companies have strong confidence in their skills for developing AI applications. However, principles of HCAI are not reflected in the ways of working, but values are focused on the technical excellence. From the recognised principles of HCAI, transparency and explainability surfaced in the interviews, and the use of those was based on the insight and opinion of the AI-team. Sometimes AI was hidden on purpose to provide better UX. This practice supports the ideas of [24] and is in line with the results of [18]. Hence, clearer principles for when and how explainability and transparency should be addressed would be beneficial not only for the user, but also for the development company. Important HCAI related themes, like ethics, responsibility, and sustainability, did not arise in the interviews. That might mean that these themes are not considered at all, or that they might be considered at a later phase of the development. When companies reach higher maturity in AI development, they may have more capacity to look for ways to include human-centeredness in their ways of working.

5.3 Challenges and opportunities for advancing HCAI in practice

Based on the findings and above reflections, we identify the following four main challenges in early-phase HCAI practices.

5.3.1 Practical HCAI challenge 1: Lack of end-user viewpoint in the early design-related activities. Human-centeredness requires continuous involvement of end-users throughout the design and development process. In this sample of companies, end-users have not been actively involved in the early-phase of AI application design. Instead, the clients are used as the source of requirements that affect the functionality choices of the application. Furthermore, the UX work is mostly detached from the early development phases, and the developer and UX teams work separately on their tasks.

5.3.2 Practical HCAI challenge 2: The capabilities of AI are unclear to clients who set the end-user requirements. End-users should be the primary and direct source of their needs and requirements. However, clients are often the primary source of end-user understanding and they may have unrealistic expectations of AI capabilities, also in terms of what it can offer to the end-user. Characteristics related to the operation of AI, such as adaptability and transparency, may be foreign to non-AI experts. The functioning of AI might be difficult to explain and to understand. To address this issue, the development company needs to demonstrate AI's capability even before actual development starts and educate or concretely inform the client about what AI can do.

5.3.3 Practical HCAI challenge 3: Uncertainty and burden of data-driven design and testing. AI is dynamic in its nature and hence it may be hard to predict its outcome. From the HCAI viewpoint this means that all important user cases may be impossible to test – some of the uncertainty necessarily persists. The quality and quantity of data – both for training the AI and for long-term use – is essential for being able to ensure the wanted outcome of an AI application. Practices around data collection can be time consuming, and the continuous learning and adaptive nature of AI requires iterative testing. Without sufficient data, different usage scenarios cannot be tested and the outcome of using AI can remain uncertain, which makes it hard if not impossible to demonstrate the functionality of the AI application.

5.3.4 Practical HCAI challenge 4: Lack of value-based design understanding of AI's impacts on transparency and other ethical issues. Companies developing AI applications have high levels of knowledge and self-confidence regarding the technical skills for developing AI-based solutions. More generally, the characteristics of HCAI are not reflected in the design approaches and related ways of working, but companies’ values are often focused on the technical excellence. Ways for tackling the need of AI transparency (through explainability but hiding complexity), fairness and other HCAI principles are less emphasised, in some cases non-existent.

The challenges identified have similarities and differences from the challenges identified in the literature. The first challenge is well known from the HCD practice as well as from the research on HCD practices in industry [27, 29]. Challenges 2-4 are specific to AI applications and need to be supported by further development of methodologies and guidelines. Academia and industry both have participated in HCAI development by suggesting guidelines, methods, approaches, and frameworks to support HCAI development. However, based on the findings of our interviews, AI developer companies have not used these existing aids. Therefore, we suggest that a HCAI capability model be constructed for AI developer companies, to help them advance human-centeredness of AI applications, and to complement their existing design approaches and methods for AI application design. The capability maturity model (e.g., [26, 37]) are widely used in IT to evaluate the capability or maturity of the company or their processes in certain context or domain. HCAI specific model should include step-wise suggestions for methodological, resource, and organisational aspects of HCAI design in a company and be implementable in design-relevant activities in different phases of the development process.

5.4 Limitations and future work

This study investigated early-phase AI application development and design practices through 12 interviews across 12 companies in Finland. This sample is geographically and culturally limited, so the results should not be generalised to represent software industry as a whole. Still, the findings give a good basis for understanding the potential challenges related to HCAI. Another issue to take into consideration is that the participating companies were mostly SMEs, and they could be said to have relatively low maturity of human-centered design. If more mature – usually larger – companies were studied, the results would probably reveal more advanced HCAI activities. However, investigating companies with lower maturity sheds light on challenges, and this understanding can provide valuable input to methodological requirements.

For future work, several interesting study directions can be outlined. Studying a broader set of companies of different sizes and maturity levels would provide further understanding of more versatile HCAI practices (or lack thereof). Furthermore, additional analyses of exiting company practices, as well as expert workshops can produce input for the construction of a HCAI capability maturity model. Supporting the capability maturity model, investigating suitable design tools and methods to support HCAI design could be beneficial for AI developer companies and designers. Methodological trials with AI-specific adaptations of HCD methods could be conducted. Suggested HCAI practices could be transformed into guidelines and methodological understanding for advancing HCAI.

6 CONCLUSION

We investigated companies’ views and practices regarding the development of AI applications, with a specific focus on human-centered viewpoints of the practices. The development companies in this study do not have established AI-specific processes, but they rely on their software development experience, and on their technical skills with AI. The early-phase decisions address the user needs and are based on the developers’ own rationale and conversations with the client, without including the end-user. The success of an AI project is strongly dependent on the amount and the quality of the data. Testing the AI's functionality in the early phase of the development with quality data is considered crucial, as the uncertainty of AI is one of the biggest challenges in AI application development. Overall, the HCAI viewpoint or practices were not integrated in AI application development. The understanding of the human viewpoint of AI applications is strongly dependent on data availability and the client's preferences. Still, clients are often not aware of AI's capabilities, which creates a potential communication gap in the development. Based on our findings, we identified four main challenges for HCAI practices, related to (i) disintegration of HCAI work from the technical development, (ii) clients’ central role as the source of user requirements, (iii) uncertain nature of AI, and (iv) lack of value-based understanding of AI in companies. The findings of this study can be used in further research and development of HCAI practices. Special attention is needed in designing and testing concrete HCAI methods as ways to improve maturity and capabilities of AI application development in real settings of AI development companies. Eventually, this will enhance the quality of AI in the real world.

ACKNOWLEDGEMENTS

We thank the interviewees for their time. We are also grateful for the funders of KITE project in which this research was conducted: European Regional Development Fund, Business Tampere and University of Tampere.

REFERENCES

Amershi, Saleema, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice. ACM, Montreal, Canada, 291-300
Amershi, Saleema, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh et al. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow, 1-13. DOI:10.1145/3290605.3300233
Arrieta, Alejandro Barredo, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García et al. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities, and challenges toward responsible AI. Information Fusion 58 (2020): 82-115. DOI:10.1016/j.inffus.2019.12.012
Auernhammer, Jan. 2020. Human-centered AI: The role of Human-centered Design Research in the development of AI. In Synergy - DRS International Conference 2020. Held online. DOI:10.21606/drs.2020.282
Balaji, S., and M. Sundararajan Murugaiyan. 2012. Waterfall vs. V-Model vs. Agile: A comparative study on SDLC. International Journal of Information Technology and Business Management 2, 1 (2012), 26-30.
van Berkel, Niels, Tag, Benjamin, Goncalves, Jorge, and Hosio, Simo. 2020. Human-centred artificial intelligence: a contextual morality perspective. Behaviour & Information Technology. 1-17.
Braun, Virginia, and Clarke, Victoria. 2012. Thematic analysis. APA handbook of research methods in psychology, Vol. 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. American Psychological Association, Washington. DOI: 10.1037/13620-000
Bond, Raymond R., Mulvenna, Maurice D., Wan, Hui, Finlay, Dewar D., Wong, Alexander, Ansgar, Koene, Brisk, Rob, Boger, Jennifer and Adel, Tameem. 2019. Human Centered Artificial Intelligence: Weaving UX into Algorithmic Decision Making. In Proceedings of 16th International Conference on Human-Computer Interaction. Bucharest, 2-9.
Bogers, Sander, Joep Frens, Janne Van Kollenburg, Eva Deckers, and Caroline Hummels. 2016. Connected Baby Bottle: A Design Case Study Towards a Framework for Data-Enabled Design. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 301-311. DOI: 10.1145/2901790.2901855
Cantamessa, Marco, Montagna, Francesca, Altavilla, Stefania, and Casagrande-Seretti, Alessandro. 2020. Data-Driven Design: The New Challenges of Digitalization on Product Design and Development. Design Science, 6. DOI: 10.1017/dsj.2020.25
Cerejo, Joana. 2021. The design process of human-centered AI. Retrieved Feb 8, 2022, from https://bootcamp.uxdesign.cc/human-centered-ai-design-process-part-1-8cf7e3ce00
Chandra, Yanto, and Shang, Liang. 2019. Inductive coding. In Qualitative research using R: A systematic approach. Springer, Singapore.
Dellermann, Dominik, Adrian Calma, Nikolaus Lipusch, Thorsten Weber, Sascha Weigel, and Philipp Ebel. 2021. The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems. arXiv preprint arXiv:2105.03354.
Dignum, Virginia. 2019. HumanE AI. Project report. Retrieved Oct 8, 2021, from https://www.humane-ai.eu/wp-content/uploads/2019/11/D13-HumaneAI-framework-report.pdf
Dove, Graham, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX design innovation: Challenges for working with machine learning as a design material. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Neew York, 278-288. DOI: 10.1145/3025453.3025739
Feuerriegel, Stefan, Mateusz Dolata, and Gerhard Schwabe. 2020. Fair AI: Challenges and Opportunities. Business & information systems engineering 62, 4 (2020): 379-384. DOI: 10.1007/s12599-020-00650-3
Eiband, M., Völkel, S. T., Buschek, D., Cook, S., & Hussmann, H. (2019, March). When people and algorithms meet: User-reported problems in intelligent everyday applications. In Proceedings of the 24th international conference on intelligent user interfaces (pp. 96-106).
Ghallab, Malik. 2019. Responsible AI: requirements and challenges. AI Perspectives, 1:1. 1-7. DOI: 10.1186/s42467-019-0003-z
Giacomin, Joseph. 2014. What is human centred design? The Design Journal 17, 4 (2014): 606-623. DOI:10.2752/175630614X14056185480186
Girardin, Fabien, and Neal Lathia. 2017. When user experience designers partner with data scientists. In AAAI Spring Symposium Series on Designing the User Experience of Machine Learning Systems. The IEEE press, Palo Alto.
Guidotti, Riccardo, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys 51, 5 (2018). 1-42.
Hagendorff, Thilo. 2020. The ethics of AI ethics: An evaluation of guidelines. Minds and Machines 30, 1 (2020). 99-120. DOI:10.1007/s11023-020-09517-8
Harper, Richard. 2019. The Role of HCI in the Age of AI. International Journal of Human–Computer Interaction, 35, 15 (2019). 1331-1344. DOI: 10.1080/10447318.2019.1631527
HAX. 2021. Microsoft Human-centered AI eXperience Toolkit. Retrieved Feb 8, 2022, from https://www.microsoft.com/en-us/haxtoolkit/
Herbsleb, James, David Zubrow, Dennis Goldenson, Will Hayes, and Mark Paulk. 1997. Software quality and the capability maturity model. Communications of the ACM 40, 6 (1997). 30-40. DOI: 10.1145/255656.25569
Hesenius, Marc, Nils Schwenzfeier, Ole Meyer, Wilhelm Koop, and Volker Gruhn. 2019. Towards a software engineering process for developing data-driven applications. In Proceedings of 7th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering. ACM, Montreal, 35-41. DOI: 10.1109/RAISE.2019.00014
Hill, Charles, Bellamy, Rachel, Erickson, Thomas, and Burnett, Margaret. 2016. Trials and Tribulations of Developers of Intelligent Systems: A Field Study. In 2016 IEEE Symposium on Visual Languages and Human-Centric Computing.162-170. DOI: 10.1109/VLHCC.2016.7739680
Holmquist, Lars Erik. 2017. Intelligence on tap: artificial intelligence as a new design material. Interactions 24, no. 4 (2017). 28-33. DOI:10.1145/3085571
IBM Design for AI. 2019. Retrieved Oct 8, 2021 from https://www.ibm.com/design/ai/
Jin, Xiaoneng, Evans, Mark, Dong, Hua, and Yao, Anqi. 2021. Design Heuristics for Artificial Intelligence: Inspirational Design Stimuli for Supporting UX Designers in Generating AI-powered Ideas. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1-8. DOI: 10.1145/3411763.3451727
Lu, Jiahao, Ortega, Alejandra. G., Gonçalves, Milene, and Bourgeois, Jacky. 2021. The impact of data on the role of designers and their process. Proceedings of the Design Society, 1. 3021-3030. DOI: 10.1017/pds.2021.563
Maguire, Martin. 2001. Methods to support human-centred design. International journal of human-computer studies 55, 4 (2001). 587-634. DOI:10.1006/ijhc.2001.0503
Oppermann, Leif, Alexander Boden, Britta Hofmann, Wolfgang Prinz, and Stefan Decker. 2019. Beyond HCI and CSCW: Challenges and useful practices towards a human-centred vision of AI and IA. In Proceedings of the Halfway to the Future Symposium. ACM, 1-5. DOI: 10.1145/3363384.3363481
PAIR with Google. People + AI Research. Retrieved Oct 8 from https://pair.withgoogle.com.
Paulk, Mark C., Curtis, Bill, Chrissis, Mary Beth, and Weber, Charles V. 1993. Capability maturity model, version 1.1. IEEE software 10, 4 (1993). 18-27. DOI: 10.1109/52.219617
Passi, Samir and Jackson, Steven J. 2018. Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects. Proceedings of the ACM on Human-Computer Interaction, 2. 1-28. DOI: 10.1145/3274405
Piorkowski, David, Park, Soya, Wang, April Y., Wang, Dakuo, Muller, Michael, and Portnoy, Felix. 2021. How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study. Proceedings of the ACM on Human-Computer Interaction, 5. 1-25. DOI: 10.1145/3449205
Saldaña, Johnny. 2014. Coding and analysis strategies. In The Oxford handbook of qualitative research. DOI: 10.1093/oxfordhb/9780199811755.013.001
Shneiderman, Ben. 2020. Bridging the gap between ethics and practice: Guidelines for reliable, safe, and trustworthy Human-Centered AI systems. ACM Transactions on Interactive Intelligent Systems, 10, 4 (2020). 1-31. DOI: 10.1145/3419764
Shneiderman, Ben. 2020. Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020). 495-504. DOI: 10.1080/10447318.2020.1741118
Silberg, Jake and Manyika, James. 2019. Notes from the AI frontier: Tackling bias in AI (and in humans). McKinsey Global Institute. 1-6.
Steen, Marc. 2011. Tensions in Human-Centred Design. CoDesign 7, 1 (2011). 45-60. DOI: 10.1080/15710882.2011.563314
Subramonyam, Hariharan, Colleen Seifert, and Eytan Adar. 2021. ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In Proceedigns of 26th International Conference on Intelligent User Interfaces. ACM, Virtual, 48–58. DOI: 10.1145/3397481.3450640
Verheij, Bart. 2020. Artificial intelligence as law. Artificial Intelligence Law 28, 2 (2020). 181-206.
Wan, Zhiyuan, Xia, Xin, Lo, David, and Murphy, Gail C. 2019. How Does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering, 47:9. 1857-1871 DOI: 10.1109/TSE.2019.2937083
Wickramasinghe, Chathurika S., Marino, Daniel L., Grandio, Javier, and Manic, Milos. 2020. Trustworthy AI Development Guidelines for Human System Interaction. In International Conference on Human System Interaction. 130-136. DOI: 10.1109/HSI49210.2020.9142644
Windl, Maximiliane, Feger, Sebastian S., Zijlstra, Lara, Schmidt, Albrecht, and Wozniak, Pawel W. 2022. ‘It Is Not Always Discovery Time’: Four Pragmatic Approaches in Designing AI Systems. In CHI Conference on Human Factors in Computing Systems. 1-12. DOI: 10.1145/3491102.3501943
van Wynsberghe, Aimee. 2021. Sustainable AI: AI for sustainability and the sustainability of AI. AI and Ethics, 1:3. 213-218. DOI: 10.1007/s43681-021-00043-6
Zhang, Jie M., Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes, and horizons. IEEE Transactions on Software Engineering. DOI: 10.1109/TSE.2020.3038802
Xu, Wei. 2019. Toward Human-Centered AI: A Perspective from Human-Computer Interaction. Interactions 26, 4 (2019). 42-46. DOI: 10.1145/3328485
Yang, Qian, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu, 1-13. DOI: 10.1145/3313831.3376301

CC-BY license image
This work is licensed under a Creative Commons Attribution International 4.0 License.

NordiCHI '22, October 08–12, 2022, Aarhus, Denmark