Understanding How Visitors Interact with Voice-Based Conversational Systems

The use of new technologies in museums encourages the creation of new genres of experience, which need to be developed with excellent design, marketing campaigns and proper functioning, fundamental characteristics for a customer of goods and services, and who will also look for this in experiences (Pine and Gilmore 1999). However, what does it take to make an experience unforgettable? (Boswijk et al. 2006) describe characteristics based on an extensive literature inspired by the concept of flow (Csikszentmihalyi 1991), a mental state that is reached when the individual is involved in an activity, with absolute focus, making it spontaneous and productive.

Another way to make an experience enjoyable is to use the conversational experience that the museum exhibits promote to infer meaning and reflection on certain perspectives or challenges. Traditionally, conversation promotes a space for learning in museums: visitors engage with museum content and develop conversations from exhibits for learning (Leinhardt 2002, Falk and Dierking 2016). To understand how to introduce and design conversations with chatbots in museums, researchers should know the purposes and intents of the design as well as the purposes and experience of visitors. Content plays a central role in the experience of informal learning by promoting conversations and reflections generated by exhibits. What are the main ways museums can connect with visitors? Several cultural and scientific institutions have adopted technologies to connect beyond labels displayed next to the artworks. These include: chatbots (Boiano 2018), robots (Shiomi 2006), QR codes (Schultz 2013), RFID tags (Hsi 2005), and augmented reality (Wojciechowski 2004). This paper presents a conversational voice-based system and, supporting user studies for understanding how visitors engage with the experience and content by acting in society.

2.1 Artificial Intelligence in Museums

The state of the art of museum experiences points to a long list of studies on the use of technology to captivate the visitor (Falco and Vassos 2017). For Wright (2017, p. 109), “technology is inevitable in a museum, so what is the advantage of museums in resisting them?”. However, the author emphasizes that experiments should not be developed around technology, which changes so quickly that museums lose time and money invested in hardware that can quickly become out of date. For the author, investments in digital experiences need to be in content - the narratives that museums want to tell about their collections, the place, and their stories. Falco and Vassos 2017 present different technological artifacts that would contribute to the visitor’s journey. For example, there are mobile apps, such as audio guides or video guides, interactive with virtual reality (VR) or augmented reality (AR); wearable; sensor technologies. Those might employ voice or movement to trigger specific commands; and natural language processing technologies - NLP. Majd and Safabakhsh (2017) explain that machine learning technologies have improved the experience of users in museums, going beyond its technical function of extracting information of any kind for later analysis of the institution. The same authors explain that machine learning and computer vision have opened up a new way to access information in museums more naturally and with less invasive methods. Vassos (2016) shows in a study the use of machine learning to create dialogues between the visitor and works of art at the Mario Praz Museum, in Rome, Italy. The exhibition also used the Messenger application. The conversational exhibition had the objective of drawing the attention of “digital natives” (Vassos et al. 2016, p. 434). According to our review, there is a small number of conversational applications based on AI technologies in museums, that aim to engage visitors to act in the society.

3.1 The Experience of IRIS+ at Museum of Tomorrow

The Museum of Tomorrow offers a narrative about how we can live and shape our next 50 years on this planet. The museum traces a journey towards possible futures. From the big questions that humanity has always asked, such as: Where do we come from? Who are we? What are we? Where are we going? How do we want to get there?. This space also seeks to promote innovation, spread the word about the advances of science, and publish news about the vital signs of the planet. IRIS+ is the first expansion of the central exhibition held at the Museum of Tomorrow since its opening. The launch of the new experience gives new meaning to the museum’s call: Tomorrow is Today and Today is the Place of Action. The dialogue system (IRIS+) placed at the end of the exhibition trail was developed to questioning visitors who passed through the central exhibition (Cosmos, Earth, Anthropocene, Tomorrows and Us). The IRIS+ inspires the visitor to think about their role in society and truly participate in the search for more awareness, tolerance and a sustainable tomorrow. The initial interaction of the visitor with Iris+ is through a voice dialogue. The conversation system guides this discussion. The dialog begins with a question from IRIS+: Considering what you saw in the Main exhibition, what are you most concerned about?. The visitors can answer, and IRIS+ will provide more questions based on the visitor’s response. Follow an example of visitor interaction with Iris+:

At the end of a conversation, IRIS+ recommends some social initiatives connected to the concerns mentioned by the visitor. IRIS+ has a database of previously registered efforts, and a recommendation component is responsible for identifying up to 3 social initiatives that are consistent with the concerns of the visitor. Subsequently, to seeing recommendations, the visitor can take an optional photo. This photo is projected on a large visualization video wall to highlight the most relevant themes for visitors, and it reveals clusters of people concerned about the same issues. The whole experience lasts 5 to 7 min.

3.2 The Technology Behind IRIS+

The IBM Watson Assistant Service^{Footnote 1} was employed to obtain the IRIS+ response for the text transcription of the visitors’ utterances from a set of pre-defined statements created by the museum curators. The recognition of user intents (phrases of visitors) was trained before the start of the exhibition. The curators (Subject Matter Experts - SMEs) from the museum are responsible for the curatorship activities.

To have most of the conversation by voice, the solution uses Speech to Text and Text to Speech components. The solution also uses a recommendation algorithm to recommends social initiatives to visitors. A high-level description of the IRIS+ architecture is presented in Fig. 2.

The intents, dialogs, and initiatives can be updated by curators over time using an administrative web interface. This allows them to change the solution behavior in real-time.

4.1 Quantitative Survey

Considering that one of the leading objectives of IRIS+ is to inspire people to embrace initiatives related to their concerns for the Planet, we undertook a study six months after the launch of interactive AI, to discover to what degree visitors had actually engaged. The six-month period was established as a means to allow visitors to absorb the experience and reflect on their visit, giving them time to get involved with an initiative. The study took the form of a quantitative survey combined with a set of open-ended questions (Onwuegbuzie and Leech 2005). We conducted the survey with Museum of Tomorrow visitors aged over 15 years old and who had interacted with the IRIS+. Aware of the importance of gathering information on visitors from all over the world, we recruited the participants based on their location precedence to answer the online questionnaire (Evans and Mathur 2005). The survey was answered by 116 participants between July 17, 2018 and August 4, 2018. It was divided into three sets of questions that, together, contained 13 closed-ended questions and 10 open-ended questions. Responses were tabulated and analyzed using Excel and R software.

Sociodemographic and Cultural Profile of Participants.

Regarding the profile of the sample, there was a balance between male (51%) and female (49%) participants. There was a higher number of visitors aged 30 and over (73%) and, 27% of visitors were under 30 years old. There was an expressive participation of visitors from Other States of Brazil beside Rio de Janeiro (53%). Additionally, most of the visitors were within the bracket of up to 2 minimum wages (33%), followed by a group with monthly income ranging from 5 to 10 minimum wages (27%). For most of the participants (81%), it was the first visit to the Museum of Tomorrow.

Results.

Follow our results after analyzing the data. It is important to mention that survey was conducted after six months of the visitors interacted with IRIS+, and our aim was also to identify visitors’ engagement in the social actions recommended by the exhibit.

Perceptions Regarding the Interaction.

Visitors were given the opportunity to assess their experiences through a score ranging from 1 to 5, with the higher score indicating a better evaluation. Most participants (63%) assessed the conversation with IRIS+ positively, allocating the highest possible score, while only 1% scored the experience between 1 and 2. The 4.53 average, along with public opinion, seems to show that the idea of interactive AI was approved. The most frequent comments were related to raising visitors’ awareness about contemporary problems; the acknowledgment that we are all responsible when it comes to environmental issues; and reflections on human-machine interaction. This can be observed in participant 25 and participant 28 quotes:

“This project with IRIS+ is essential in reminding visitors to the Museum of Tomorrow about their responsibility when it comes to environmental issues” (P25)

“Despite feeling “strange/uncomfortable in interacting with artificial intelligence” it was really fantastic! I’ll certainly be back” (P98)

When questioned on possible reflections as a result of the conversation, visitors were emphatic in identifying the striking aspects of IRIS+, whether through comments like “My interaction led me to think that the planet I want for the future is not the one I cultivate in my daily routine” (P14), admitting they should already be doing more for the planet than what they are, or in other comments like “It provoked me unexpectedly, I had never stopped to think about what humans have done and are doing to our planet. It opened my eyes to the new challenges in my life” (P1) by visitors who have begun questioning themselves about global issues based on the interaction.

Engagement.

We asked visitors whether, six months after the launch of IRIS+ if they had become engaged in a social project or changed their individual behaviors based on the exhibit recommendations. Of all the participants, 37% had become engaged in an initiative related to their concerns about the planet. Assessing the most common topics in this engagement, we noted links with the most frequent concerns recorded through the IRIS+ system (Table 1). This data reveals that visitors seem more likely to act on issues part of their daily routine, like environmental conservation and waste management.

Among engaged visitors, we noted a greater probability to get involved in personal initiatives, that is, those which do not require association to any institution. Selective waste collection, saving water and electricity and cutting down on the use of plastic are examples of individual practices adopted by the participants. Of the 37% that took action, 86% opted for this type of engagement described here by participants 14, 50 and 105.

“A personal initiative, about saving water at home.” (P14)

“I’m involved in an initiative to help youths and adolescents find their first job. It’s a way I found to apply my expertise to benefit society.” (P50)

“It’s not an institution. I organized a campaign to recycle waste and cooking oil in the building complex where I live.” (P105)

Among the group that became affiliated with an institution (14% of those engaged) there were people involved with NGOs, collaborating and promoting environmental causes; there were teachers providing lessons on environmental conservation and education; and even some related to reducing the use of fossil fuel powered vehicles.

“Campaigns warning about the use of fossil fuel powered vehicles, leveraging the generation of new energies. I encourage people to walk and pedal.” (P84)

“I came up with material on EcoEducators (for “AppaiEducar” magazine), through which Environmental Engineering students undertake volunteer work at schools to raise awareness about environmental preservation.” (P103)

“I help to promote causes (for Greenpeace).” (P110)

Aimed at understanding the potential for engagement in different groups in the study, cross tables were created between the “engagement” variable and the variables of gender, age bracket, origin, monthly income and the dichotomous question of “Have you previously visited the Museum of Tomorrow?”. Table 2 illustrates the main variable leveraging engagement, namely monthly income. To reach this conclusion, a dummy variable (Caudill 1987) was created, wherein we allocated all the individuals that earned up to (and including) 5 minimum wages per month in a group coded as 0; and those individuals who earned over 5 minimum wages per month in a group coded as 1. The first exploratory analysis of the data shows us that, of the visitors earning up to 5 minimum wages per month, 27% were engaged. Those who earned over 5 minimum wages per month presented an engagement rate of 47%. We chose to use Fisher’s exact test to assess the null hypothesis that answers were evenly distributed in the contingency table.

The p-value of 0.04928 is an indication that the null hypothesis at a 5% significance level was not rejected, allowing us to infer that there are signs that visitors with a monthly income over 5 minimum wages are more likely to be engaged.

Another interesting result was linked to the age bracket. We parameterized age brackets according to dummy variables whereby individuals aged up to and including 30 years old were coded as 0, while those aged over 30 were coded as 1. The first exploratory analysis showed that 29% of the younger visitors (aged up to 30) had become engaged against 39% of the older visitors (over 30 years old). Although Fisher’s test results are inconclusive for this cross reference (p > 0.1), we feel it important to look more closely at this data. The IRIS+ system shows that of the users who initiated a session with the AI, 52% are aged up to 30 years old. On the other hand, of those who completed the experience all the way to the end (when the photo appears on the panel), only 32% are aged up to 30 years old.

This data, along with information from the previous paragraph, led us to reflect on the potential engagement among younger visitors. If they make up the majority of visitors to the Museum, and they are also the majority in initiating a session with IRIS+, why are they the minority in the “conversations completed with IRIS+” indicator and why do they present a lower engagement percentage according to the survey data?

Now, we will look at the subject of “no engagement”. We asked visitors who had not engaged what their leading motives were. A lack of time appears most common, together with the convenience factor of statements like “Involvement with other personal issues, but which do not really justify my lack of involvement” (P100) and “Not entirely convenient, unfortunately” (P2). In other words, these are visitors who are aware of the problems but who, due to other priorities, end up doing nothing. We also discovered that non-engaged visitors living outside the city of Rio de Janeiro were unaware of any initiatives close to home, through statements like “There’s no time! I don’t know of any initiatives in my city” (P34).

This valuable information can help bolster the responsibility of the Museum of Tomorrow in studying and adding new institutions, organizations and foundations from all over Brazil to the system.

4.2 Observation Studies in the Wild

The objective of this study was to understand in the wild the user experience with IRIS+ (Schuman 1987). The field study included observations and brief interviews with visitors. The semi-structured interviews were designed to be short and not to disturb or delay visitors. Twelve visitors described their experience to a researcher. The interviews were audio-recorded and consisted of only one question

Q1: Please tell us how you would describe your experience with this exhibit for a friend that will not be able to visit it.

Four employees also contributed to the study sharing their views. The interviews also served as a clarification of the behaviors observed by the researcher during visitor’s sessions. We also gathered the text interaction logs of 380 visitors, and audio/video recorded a day of visit interactions with IRIS+.

The Data Analysis.

This analysis was a first attempt to understand visitors experience with IRIS+ in situ. We investigated, inspired by other conversation analysis studies (Porcheron 2016), (Moore 2013), Suchman 1987), (Yamazaki 2009) how visitors structure their interaction with Iris+ in a public museum space and which kind of social actions occurred because of this interaction. To investigate the rational social action of visitors, we first explore here onboarding interaction situations that were directly observable and reportable to people present in situ. (Garfinkel 1967). And then, we describe the perceptions of visitors reporting their own experience with IRIS+.

Results.

We highlight the main issues identified in the observation studies grounded by the video and audio recordings gathered during this investigation. Analyzing the sequence details of interactions, we identified visitors’ reactions to an intelligent voice-based exhibition. We selected a couple of interaction fragments to illustrate attitudes and some strategies visitors used to interact with IRIs+.

Visitors’ Interaction Strategies.

Iris+ is localized at the end of the main museum trail. Although most of the visitors interact with it after seeing the main spaces of the museum, some go across the corridor and interact first with Iris+. We notice an evident difference from the ones who interacted with the museum spaces before. Those visitors, we call here experienced visitors, they knew what to ask and answered the questions with more words and property. The others requested ideas of what to ask from the museum attendants or give up more easily in the middle of the interaction. The experienced visitors, in most of the observed cases, knew how to start the interaction using the museum card, it is also used in other museum spaces. (Figure 3 (6)). Due to similar shape displayed on the tablet screen, not experienced visitors more often tap the card on the screen mistakenly. The right place to tap it is on the figure on the wall beside the tablet. This behavior happens even though, there is written information on the screen: “To start, tap on the logo beside the tablet and wear the pair of headphones.” We also observed that both types of visitors often laid the card on the logo beside the tablet through the whole experience. They were afraid the exhibit would stop working if they take the card out. What it was a misconception.

Another curious and misconception behavior was to lean forward and to whisper to IRIS+. It was like they were telling a secret to Iris+. Visitors were aware that Headphones had microphones because attendants advised them to hold the microphones to have a better experience, even so, they engaged in this behavior. In Fig. 3 (7), we can see a visitor pointing to the tablet of his partner to advise to talk near the screen. Visitors are familiar with audio guides in museums, but not used to respond back to devices. The museum attendants helped visitors that did not know how to react to IRIS+ advising them that was a voice-based interaction. We observe visitors’ behavior before asking for this kind of help. In those situations, they tap the screen for more information, they looked at the next visitor interacting with the installation, or they verbalized the need for assistance to the museum attendant. An excerpt of the data illustrates this:

Visitors also enjoyed the exhibit accompanied. We observed the cases when visitors were alone; they frequently repeated the experience inviting acquaintances for the second or third time. Second, third-time visitors usually taught the novices they invited and evoked conversations around the IRIS+. (Figure 3 (9)). Additionally, we noticed that several times when visitors were accompanied, only one was wearing the headphones, which caused some misinterpretation from the person who was only seeing the screen. The person who was observing sometimes tap on the screen, resulting in discomfort for the one wearing the headphones, who had to gesticulate signs (waiting, stop hand signs).

The flow of the interaction was also shaped by the multi-modal interaction proposed by the system. Visitors tapped the card and were requested to use the voice-based mode to dialogue with IRIS following the experience. Visitors have an option to take a picture and share their concerns with other visitors that previously interacted with IRIS+. For having this option available for them, they are asked to fill out a form typing their home city, age, and their email. Only people that have more than 18 years old are allowed to take their pictures to display on the Initiative panel. Visitors have the option to skip the form, and we observed that most of them act like that. They were concerned with the use of their data that was not so clear on the interface. Also, some visitors with their parents observed that they could only show up their concerns visually if they filled out the form, so they lied about their age to participate in this step of the interaction. The change of mode from voice to the text input in this stage was also confusing for old visitors that asked the guide for help in this stage, some of them did not have emails, and for this reason, they could not complete the form. After filling out the form, IRIS+ change the mode back to voice-interaction, so users continue tapping on the screen, even though the mode of input was not the same. When visitors saw the initiatives suggested by IRIS+, they also tapped on the screen to know more about each of them, although the square of the initiatives were not touchable. They also took pictures of the screen (Fig. 1), to record the suggested initiatives. It was not clear the initiatives were sent by e-mail if they filled out the form.

Visitor experience with IRIS+.

Full size image

High-level architecture

Full size image

(1) Visitor tap the card to start the experience. (2) IRIS+ Interface shows a question asked by IRIS+. (3) Visitors interacting with IRIS+. (4) Visitors taking pictures of their displayed pictures. (5) Panel showing visualizations with clusters of causes. (6) Visitor using a card to interact with IRIS+. (6) Visitor showing his partner how to interact with the installation. (7) Visitor lean forward to talk to IRIS+. (8) Social interaction between visitors.

Full size image

Another change in interaction mode also affected the visitors’ experience. When the option to take the picture was available, we observed visitors taking off the headphones to tap the screen to take their pictures, and they did not put them back. Therefore, those users did not hear the final statements of IRIS+. They did not know the voice-based interaction was not over yet.

Visitors Perceptions of Talking with Intelligent Systems.

Visitors also reported how they felt talking to an artificial intelligence system. IRIS+ invited participants to reflect on museum themes and connect those to their own lives. It surprised participants and created a social situation where sharing those concerns with a machine was not natural. A participant accompanied by his partner commented:

More visitors displayed the same behavior, and many More visitors displayed the same reaction, and many verbalized they did not know what to answer to Iris+. In that situation, IRIS+ employed repairs actions (Schegloff 1977) to avoid interaction breakdowns in the dialogue. In line 03 and line 05 of next fragment, Iris+ applies repair mechanisms to bring P8 back to the dialogue subject.

IRIS+ also captured some environment noises, likewise a sound similar to waves from another exhibit and utterances of visitors talking to each other. In those situations, IRIS+ continued the script and recognized the utterance as not related to the questions asked. Visitors hear a repair question when a not recognized utterance is verbalized for the first time (line 3). And for the second time, IRIS+ processes the ((sound of another exhibit)) and asks a new question (line7).

In this case, P11 answered the second question (line 3) and responded to the question in line 7 considering the line 5 information provided by Iris and his previous answer (line 4). In this case, P11 ignored the question at the end of line 5.

Next we show a transcription of a semi-structured interview with three visitors that know each other (P3, P4, P5). P3 grabbed the audio recorder from the researcher (R) and interviewed her companions.

In this fragment, we notice that P4 share her feelings of embarrassment to others by feeling she was talking by herself in a public space (line 03). And P5 demonstrates his anxiety of controlling the interaction (line 11 and 13). We also notice, P3 expectation of why P4 felt stupid (line 4). P4 and P5 also leaned towards the tablet and whispered to the machine, what shows evidence of P4 uncomfortable feelings in public. Likewise, P4 and P6, other visitors we interviewed reported similar feelings.

This analysis shows that integration with visual and verbal elements are essential for onboarding interaction with the intelligent voice-based devices. We also unveiled visitors’ social actions in situ while interacting with IRIS+. Highlighting those behaviors may help designers, developers and museum curators to think carefully on how to tailor conversation technologies to visitors and how to take advantage of those social actions to intensify visitors experience in museums and promotes the engagement in social themes outside the institutions. In the next session we present the main lessons learned and challenges to deploy AI voice-based systems in museums.

We identified the following challenges for AI voice-based installations in public spaces.

• Diverse modes of interaction in the same exhibition might affect the visitors’experience.

• Social interaction is an essential element in informal learning spaces.

• Share personal information and opinions in public spaces might affect participation.

• Repairing the conversation with visitors is essential.

• Design integrated conversational interactions considering previous knowledge of the museum content acquired by visitors.

• Engaging audiences of different ages in the experience.

• Enroll participants of different location and economic backgrounds displaying information that motivate them to act.

The second phase of IRIS+ is in development. The new design of IRIS+ is using more colors and an advanced format to catch the dialog (using more voice mode and less text mode input). The contents are now linked with the Sustainable Development Goals (SDGs) from United Nations, and there is a connection between the initiatives of action presented by IRIS+ connected to the themes from the Main Exhibition of the museum.

The IRIS+ project allows knowing concerns based on different profiles of visitors, being able to recognize the significant worries and desires that are influencing the collective awareness about the future. The storytelling of the experience was developed to inspire the visitors for a sustainable and plural tomorrow, connecting to the SDG issues. Besides that, IRIS+ allows its partners to come with strategic narratives that may push people into acting and to be engaged with their communities. In 2020, the project is going to be enhanced in their second stage of development, putting the visitors in discovery about the concerns linking into the areas of the Museum’s content, making them more deeply knowledgeable about the themes that they want to connect. The research about the audience of IRIS+ was fundamental because it helped to create new parameters and interactions in the experience to attract the attention of the young audience and address some of the challenges described in the previously. The curatorial team has searched initiatives all over the country related to youngsters and social themes as inequality, gender equality, and women empowerment. The IRIS+ appearance was redesigned, and the model presented in its new version allows the visitor to notice his social connection with other visitors, meaning that more people can engage in similar themes. Because of this, they are willing to change reality.

Artificial Intelligence is a data science tool that has positively impacted much of the goods and services we use. Its effectiveness in use in Museums is not yet fully known, but institutions have taken the risk of learning more about this technology. The results should be known later, over time. Far from being thought of as a solution for the development of an unforgettable experience, the application of the use of Artificial Intelligence needs to be understood before being used in the context of museums. It is necessary, above all, to reflect on the content that will cover the technology, more important than presenting it itself - the AI - only as an innovation, as there is a risk of obsolescence of the activity.

In the case of the use of AI in an experiment, another point to be emphasized is the need to identify what enhances the participation and engagement of visitors, checking if the methods adopted and the results meet the expectations created by the developers and the users. We understand that its use in this context requires a good deal of understanding and learning on the part of Museum teams and professionals involved in the implementation of a complex project such as the IRIS+ object and that this is particularly one of the significant challenges in our reality.

This research helped to unveil those issues and redesign the experience being more social and considering the issues that affected the visitors ‘experience with the exhibit. The idea is that Artificial Intelligence applied in Museums will, in the future, be a possible way to search for experiential solutions aimed at engagement. It is understood that the use of Artificial Intelligence in the IRIS+ experience to instigate engagement, according to data from the Museum of Tomorrow, efficiently promoted the activation of the visitor to actions related to social or environmental issues. Moreover, the results showed that this experience allowed visitors to the Museum of Tomorrow to experience a dynamic experience with Artificial Intelligence and made accountable their perceptions of having a conversation with an AI.

We hope museum curators, designers and developers use our study as a motivation to use AI in their projects, and be aware of expectations and actions visitors might have while and after interacting with voice-based AI systems.