Embodied Conversational Agent Appearance for Health Assessment of Older Adults: Explorative Study

Frailty Assessment App

The ECA under study was embedded within a frailty assessment web app developed as part of a larger platform designed to counter frailty by offering older adults training modules in the domains of healthy nutrition and physical and cognitive training to maintain a healthy lifestyle [20]. Initial and continued use of the platform is stimulated by integrating gamification elements. In this study, we focused on the stand-alone frailty assessment app.

The frailty assessment app consisted of an index page (Figure 1) and a dialogue page (Figure 2). On the index page, an agent was displayed next to a blackboard. The blackboard provided a list of available dialogues: introductory small talk, questionnaire assessing aspects of the older adult’s health, and small talk explaining the results of the questionnaires. When a dialogue was finished, the user returned to the index page. Before the questionnaire dialogues were performed, only the introductory small talk was available on the blackboard. In this dialogue, users were introduced to the agent and the goal of the frailty assessment. Afterward, the questionnaire dialogues were unlocked and shown on the blackboard. Three validated questionnaires were implemented to assess the older adult’s frailty status covering multiple health domains. The 36-item Short-Form Health Survey [21] contains 36 multiple-choice questions related to health topics (eg, physical functioning, social functioning). The Alzheimer Disease Detection [22] tests for functional decline in memory using 8 yes or no items. The Mini Nutritional Assessment [23] tests for malnutrition with 6 multiple-choice questions related to nutrition and weight. We translated the three frailty assessment questionnaires into dialogues between the agent and older adults. After questionnaires were completed, the result dialogue was unlocked on the blackboard. In this dialogue, users received the outcomes of the assessment.

Only one dialogue was available at a time. Clicking on the start button of a dialogue opened the dialogue page (Figure 2). A dialogue consisted of multiple dialogue steps. Each dialogue step consisted of a statement by the agent and one or more reply options that could be selected by the user. The statement by the agent was shown in the white box with the orange border and the reply options for the user were listed in the black box. After finishing a dialogue with the agent, the user returned to the index page and available dialogues listed on the blackboard were updated.

The agents used in the frailty assessment app (Figure 3) are Sylvia, a young female peer agent, and Egbert, an older male peer agent. By a peer agent, we mean an agent who is not a medical expert. Agent designs were selected based on findings from a previous study [16], in which the static images of eight agents were evaluated. The agent images differed on three features: age (young or old), gender (male or female), and role (experts had a high level of health expertise, and peers had a low level of health expertise). In an online questionnaire, images of all agents were shown to the participant at once, with participant selecting agent they preferred most (to be their health coach) at first glance. Afterward, participant rated characteristics for each agent. Results showed that a young female agent was preferred most and an older male agent was preferred least in both a general and elderly population (ie, these designs were extremes in terms of user preference). This study builds on the previous study by evaluating users’ impressions of these two agents, both at first glance and after a short interaction with the agents. A blinking eyes animation was implemented for both agents. In addition, when the agent spoke (ie, when a new dialogue step was loaded), a mouth animation of a fixed duration was played.

Study Design

We applied a within-subject design in which we counterbalanced the order in which agents were presented to participants. Half of the participants started the frailty assessment with the young, female peer agent and finished with the older, male peer agent (Figure 4, top). The other half of the participants were first presented with the older male peer agent, followed by the young, female peer agent (Figure 4, bottom). The study was performed in a lab setting, taking place either at a research institute or a local physiotherapy practice. The nature of this general study with healthy volunteers from the general population does not require formal medical ethical approval according to Dutch law. All participants provided their informed consent.

Participants

Participants should be aged 65 years or above and fluent in the Dutch language in order to be included. In addition, they should be cognitively able to work with an ECA as assessed via the Mini-Mental State Examination, scoring at least 23 out of 30 points [24]. We recruited the respondents via a Dutch panel of adults that indicated they were interested in participating in research on eHealth. Participants were also recruited via a local physiotherapy practice.

Measurements

Data Analyses

SPSS Statistics 25 (IBM Corporation) software was used to perform statistical analyses. Since the underlying data were nonparametric, for all relations testing differences between the two agents, a Wilcoxon signed-rank test was conducted. All tests used a 95% confidence interval. All variables were tested for statistically significant differences between the two agents by means of a model consisting of Wilcoxon signed-rank tests for cross-over designs. Effect size was calculated by r=Z/√N, using 0.1, 0.3, and 0.5 as cutoff values for a small, medium, and large effects, respectively.

The audio recordings of the thinking aloud sessions and interviews were transcribed and inductively thematically analyzed. In addition, screen captures of the interaction with the frailty assessment app were aligned with the audio recordings. This way, the screen captures were used to verify the thoughts of the participants on the audio recordings. All themes were coded using ATLAS.ti 8 (ATLAS.ti Scientific Software Development GmbH) based on an empirical method proposed by Pope and Mays [27]. One researcher (StS) created a first coding scheme based on the data and then labeled the transcripts. A second researcher (MB) used the coding scheme to code a subset of the data so that a discussion could be held between the first and second coder for improving the coding scheme. The procedure of creating a first coding scheme, labeling the data by two researchers, and discussing the coding scheme was repeated a second time leading to a final coding scheme. The final coding scheme was used by the first coder to code all data for final analyses. The final coding scheme contained the following codes: agent characteristics, appearance agents, interaction with agents, preference agent, content questionnaires, language usage in dialogues, presentation information, interaction with app, design, navigation, general computer interaction, and intention to use.

Participants

A total of 21 participants began the study (Table 1). One participant was not able to complete the protocol due to a lack of computer experience and was excluded. The average age of participants was 72.2 (SD 3.5) years, and 13 males and 7 females participated. Ten participants started with the young, female agent, and ten participants started with the older, male agent.

Agent Appearance

Usability and Intention to Use Frailty Assessment App

Questionnaire results show that the usability of and intention to use the frailty assessment app were high: the 20 usability ratings displayed a median of 6.1 (interquartile range [IQR] 6.1-7.0) and the 20 intention-to-use ratings displayed a median of 6.0 (IQR 4.0-6.0) on a 7-point Likert scale.

During the thinking aloud session and interviews, participants pointed out usability issues of the frailty assessment app or provided suggestions for improvements to the app. The following themes were identified: content questionnaires (mentioned 107 times), language usage in dialogues (mentioned 41 times), presentation information (mentioned 21 times), interaction with app (mentioned 14 times), design (mentioned 7 times), navigation (mentioned 7 times), and general computer interaction (mentioned 6 times).

Most comments or suggested improvements related to the content of the questionnaires and the language in the app. The majority of the participants reported that the questionnaires did not fit their personal situation and contained a lot of repetition or ambiguity. Participants suggested adapting the questionnaires according to previous answers given. In addition, participants commented on the language used: words being ambiguous, too popular or too old fashioned, unnecessary, patronizing, or not being known by people with a lower education or older adults. Furthermore, participants commented on the length and structure of the sentences and pointed out spelling mistakes. A participant suggested adapting the language in the app to the education of the user. Considerably fewer comments related to the presentation of information, interaction with the app, design or navigation of the app, and general computer interaction. As an example, with respect to navigation, some participants indicated they would like to be able to go back to a previous dialogue step.

With respect to the intention to use, the thinking aloud sessions and interviews showed that a minority of the participants would like to use the app. A participant indicated he would not use the app but would recommend the app to others who might benefit from it. In addition, some participants clearly indicated they would not use the app. The majority of the participants indicated that the agents did not add any benefit to the app, arguing that the app was not personal since answer options were limited and the opportunity to explain them was missing. A participant stated that for the app to be beneficial, it should also provide advice on what actions the user should perform to become more healthy. Another participant explicitly stated that he would use the app when the text was replaced by speech.

Principal Findings

Our results show that the appearance of an agent, in particular age and gender, affects users’ perceptions of agent authority but does not affect users’ perceptions of other agent characteristics, users’ feelings of rapport, or users’ likeliness of following agent advice. Compared with a young female agent, an older male agent is only seen as more authoritative. These results are not in line with our expectation that agents are perceived differently after a short interaction with a user. To the best of our knowledge, there is no existing research comparing users’ impressions of agents at first glance with those after short interactions. But research shows that in human-human interaction, first impressions, formed within milliseconds [28], are difficult to lose. Therefore, we assumed that the differences in perceptions of characteristics of a static image of a young female agent and an older male agent, as found in a previous study [16], would still be present after a short interaction with these agents. An explanation for this inconsistency could be that impressions in human-agent interaction differ from impressions in human-human interaction. Users’ judgments of agents may modify with ongoing interaction, as research shows that agents do have a second chance to make a first impression [13,29]. Therefore, differences in perceptions of both agents may have been present at first glance but disappeared after interaction. Further research is needed to confirm this finding. Future research could study users’ perceptions of agent characteristics with a larger study population. Eventually agents will be used in a long-term setting; therefore, it is interesting to research not only users’ perceptions at first glance and after short-term interaction, but also after long-term interaction.

How do we explain the difference in perceptions of agent authority after a short interaction? Although research on short-term interaction with an agent indicates that an agent’s appearance, including clothing [18], racial concordance with the user [30,31], and similarity with the user [30,32], could affect users’ perceptions of the agent, to the best of our knowledge there is no research on agent authority after short interaction in particular. From a previous study [16], we see that at first glance, static images of male and older agents are indeed seen as more authoritative than female and young agents, respectively. In addition, the study shows that the differences found in authority are often higher compared with differences found for other characteristics tested, which could explain why the difference in authority level is still present after short interaction. However, since we did not control the age and gender of the agents in this study independently, it is difficult to say whether the difference in perception of agent authority is caused by agent age or gender in particular or solely by the combination. Future research could study which factors actually control the difference, researching users’ perceptions of agent authority by independently controlling the age and gender of the agents. In addition, future research could study how an agent’s authority is perceived after long-term interaction.

We expect that the effect of the first impression established by agent age and gender on the impression after short interaction is small compared with the effect of other design features, such as the content and language of the messages, (absence of) agent speech, and the amount of embodiment. Our study shows that the majority of participants perceived the agents not as humans but as machines or cartoons and found interaction with the agents impersonal or artificial. They did not have the feeling of being in a conversation. These perceptions may indicate users had a negative adaptation gap [29], which occurs when a user overestimates the competency of an agent, creating a negative gap between expected and actual competency of the agent and resulting in the user being disappointed. This negative adaptation gap may have been caused by the content and language of agent messages, agents lacking speech, or agents having little embodiment, as supported by remarks made by participants during the thinking aloud sessions and interviews. Therefore, we believe it is important to manage users’ expectations of agent characteristics and functionality up front, ensuring users’ expectations match actual agent capabilities by explaining what the users can expect from the agent. Future research could study how an agent’s content, language, speech, and embodiment affect users’ perceptions of the conversation with the agent (eg, how these factors could make the conversation with an agent more human-like).

Although our study shows agent age and gender have little effect on users’ impressions of the agent after short interaction, we believe that adapting these features to the user is important because they affect users’ impressions of the agent at first glance [17,19,33], and research shows that people with favorable impressions of someone tend to interact more with that person than they do others who gave unfavorable impressions [34]. Selecting an agent with the right age and gender could thus lower the threshold to interact with the agent and use the app.

Second, our results show that usability of the developed frailty assessment app was judged positively overall; issues identified by participants related to the content or language of the questionnaires. We suggest tailoring the content and language toward the personal characteristics of the user, as confirmed by existing research [35], and adapting the content to previous answers given by the user.

Third, not all participants show an intention to use the app. Research indicates that older adults put effort into learning new digital technologies as long as they are believed to be worthy of time and dedication (eg, when technology can be used to keep in touch with others to foster relationships [36]). Similarly, research shows that the elderly value apps that address a social problem [37]. The app used in our study did not address a social problem, which could have resulted in some participants not seeing the added value of the app and not showing an intention to use the app. In addition, intention to use digital technologies in elderly persons is, next to the quality of the technology itself, affected by their personal context (eg, their ability to concentrate) and social context (eg, whether family is around to provide technical support) [37]. Both factors might have affected participant intentions to use the frailty assessment app in our study.

More specifically, the majority of participants do not believe the agent adds value to the frailty assessment app. Therefore, we suggest updating the design of the agent. We believe that the agent should convey additional information to its message in text via its embodiment. Existing research provides evidence for implementation of animations of the agent’s embodiment, showing that animations positively affect users’ impressions of the agent [38-40] and interaction time [13,39]. In addition, the use of speech is recommended because it could increase the sense of personality of an agent [41] and could be used to describe feelings [42]. Low-literate users could benefit from multiple output modalities [43]. Furthermore, participants indicated they would like the app to provide advice on what actions they should perform in order to become more healthy. We see an opportunity for using the agent to provide this advice. As an example, the agent could show videos of exercises to improve physical strength.

Strengths and Limitations

This is the first study that specifically evaluates effects of agent appearance after short interaction with the agent. In addition, this study uses actual health content, which is scarce in research on agent design.

Our study also has some limitations. First, the negative adaptation gap between user expectations of agent capabilities and actual agent capabilities suggests the app used might not have been mature enough. The agent conveyed the majority of the information via text. Participants might have been focused on reading the text and therefore paid little attention to the agent, resulting in participants having difficulties in creating impressions of agent characteristics and establishing rapport. Second, interaction time with the agents might have been too short to create impressions of agent characteristics and establish rapport. Third, although we found a difference in users’ perceptions of authority of the young female and the older male ECA, it is difficult to identify whether this was caused by the ECA’s gender or age, since these factors were not independently controlled in the study.

Toward Digital Frailty Assessment With Embodied Conversational Agents: Recommendations for Future Research

Conclusions

Our study shows that an agent’s appearance, in particular age and gender, only affects users’ perceptions of agent authority after short-term interaction. We conclude that adapting agent age and gender to users’ preferences is important to lower the threshold to interact, whereas the content and language of the agent’s messages and agent speech and embodiment are important factors for users’ impressions of the agent after short interaction.

We believe that ECAs have potential to be used in digital frailty assessment, but future research is needed. Future research could study users’ perceptions of agents after long-term interaction, whether users’ perceptions of agent authority are related to agent age or gender in particular, and how an agent’s content, language, speech, and embodiment affect users’ perceptions of the conversation with the agent.