How Important are Detailed Hand Motions for Communication for a Virtual Character Through the Lens of Charades?

We use virtual characters to communicate in many ways: carefully animated as a core element when telling an engaging story in movies and games, as embodied intelligent agents (controlled by algorithms) such as virtual tutors, and as avatars, controlled by people in real-time, e.g., VTubers or users of social VR applications. In effective interpersonal communication, hand movements play a key role [Goldin-Meadow 1999; Kendon 2004; McNeill 2008]. We use gestures to describe, emphasize, and clarify what we want to convey and even to replace words entirely. When it comes to capturing human motions or animating them, the arm and detailed hand motions are often treated separately for technical reasons [Wheatland et al. 2015]. In VR, some head mounted displays (HMDs) include cameras in the headset that can track the hands. However, hand tracking in real-time still fails frequently due to the hand being outside of the camera’s field of view, occlusions, or motion blur [Ferstl et al. 2021].

Previous research has shown that small changes in finger motions are perceptible and can change the interpretation of a scenario [Jörg et al. 2010] or the perceived personality of a virtual character [Wang et al. 2016]. However, the impact of errors or lack of accurate hand motions in a communicative setting is not known. What are the consequences of inaccurate or missing hand motions on comprehension or on the impression we get of an avatar? What can we do if hand motions can not be captured at all? It is important to understand how inaccuracies and errors in these motions might affect conversations, from changing the whole message to making a conversational partner appear to be unfriendly or untrustworthy.

In this article, we aim at answering these questions by investigating the role of detailed hand motions when conveying content. We base our experiments on charades, a game which requires players to communicate exclusively with gestures and asks viewers to guess specific answers. Gestures are used differently in charades than in typical conversations. They are made to convey content on purpose and include more gestures that act out objects or concepts. The main advantage, however, of using charades in this research is that they provide us with a straightforward, quantitative way to measure comprehension: the number of correct answers. Further advantages supporting this choice are the option to use charades as videos that can be post-processed offline, and the absence of speech that eliminates a confounding factor.

While our primary goal is to investigate how detailed hand motions affect the viewer’s ability to understand the character (comprehension), for a more complete investigation we also examine the effect on the personality of the character, on the social presence of the viewer, and we ask how comfortable viewers are with the avatar in VR. These effects might be influenced by avatar appearance, which is why we incorporate two characters with different levels of realism into our study.

Our study consists of three experiments. To explore different hand motion alterations, we use Amazon Mechanical Turk to gather large numbers of participants for our first two experiments, in which participants watch videos of our stimuli. We then repeat a small subset of the conditions in a third experiment in VR. In our first experiment, the Alteration Experiment, we alter the hand motions of a motion captured character to simulate errors or changes that would typically happen when creating, tracking, or post-processing hand motions (see Figure 1). This experiment asks: How do hand animation and avatar appearance affect the viewer’s ability to understand the character and the viewer’s impressions of the character? Can we create acceptable hand motions without any data? In our second experiment, the Intensity Experiment, we ask: Which intensities of specific alterations, namely Jitter, Popping, and Smooth, are acceptable? What are acceptable thresholds for these errors? We vary their intensity from subtle to extreme to observe their effects. In the third experiment, the VR Experiment, we verify some of our results in a virtual environment and evaluate participants’ comfort level when watching the character in VR.

We find that the absence of finger motion reduces comprehension and social presence. It also negatively influenced the viewer’s perception of a virtual character. However, hand motions with slight inaccuracies and errors, and surprisingly even with larger errors, achieved the same levels of comprehension as accurate hand motions, but they influenced the perception of the character in some cases. A large jitter was seen as most negative, still without affecting comprehension. In the absence of captured hand motions, adding some motions, even random ones, improved the perception of the avatar and the viewer’s social presence compared to no motions at all, but it did not improve comprehension. These results have important implications for the animation of engaging virtual characters, for the creation of effective virtual agents, and for the development of VR technologies.

This work builds up on previous research from several areas, such as motion capture and animation, virtual reality, gestures and communication, and the perception of virtual characters. We provide an overview of previous related research in the following paragraphs.

Capturing and Synthesizing Hand Motions. Capturing detailed hand motions remains an important and difficult problem, which is illustrated by the diversity of suggested approaches. Wheatland et al. [2015] provide an overview of technologies to capture detailed hand motions; Jörg et al. [2020] focus on state of the art virtual reality methods. Approaches include marker-based optical systems, individual and multiple cameras, and sensored gloves [Glauser et al. 2019; Han et al. 2020, 2018; Mueller et al. 2019; Wang et al. 2020]. Further approaches try to synthesize hand motions or to recreate them with incomplete information [Jörg et al. 2012; Schröder et al. 2015; Wheatland et al. 2013; Zhang et al. 2021; Zhao et al. 2013]. While the progress in the past years has been impressive, it is still not possible to capture accurate hand motions with consumer technology [Schneider et al. 2021]. When using HMDs, the limits of the field of view of the camera, visual occlusions, or motion blur create many hand tracking errors [Ferstl et al. 2021]. We investigate the consequences of such errors.

Perception of Hand Motions. Previous work has shown that subtle changes in the timing of hand motions can be noticed and can alter the interpretation of a character’s actions [Jörg et al. 2010]. While very subtle changes can be noticed, they are not noticed all of the time: Hoyet et al. [2012] showed that the inaccuracies resulting from using a reduced marker set with eight markers compared to 20 markers were only detected at a significant rate for one out of nine different actions. Such changes are by far more subtle than errors in consumer capture devices and the ones we use in our experiments. Wang et al. [2016] found that hand poses and motions can convey personality. For example, spreaded fingers are perceived as conveying extraversion and openness whereas a resting hand pose conveys high emotional stability and agreeableness. Smith and Neff [2017] showed that it is possible to influence the perception of the personality of a virtual character (measured with the Big Five personality model based on the traits extraversion, openness, emotional stability, agreeableness, and conscientiousness) by editing the timing and poses of gestures. While most of the changes apply to the arm motions, they showed that extending the fingers increased extraversion and a slight disfluency—which could be compared to a slight jitter or popping—reduced conscientiousness, agreeableness, and emotional stability.

Gestures, Communication, and Charades. Regardless of whether errors are perceived and what personality is conveyed, hand and arm motions are important for communication [Goldin-Meadow 1999; Kendon 2004; McNeill 1992, 2008]. As McNeill describes, gestures orchestrate speech [McNeill 2015]. Gestures can, for example, be used to emphasize a point or to show the size of an object. Gestures can also replace words altogether, for example, to indicate that something is “okay” by touching the index and thumb fingers together to form a circle. Altering such gestures can inadvertently misrepresent what a speaker is trying to say. McNeill [1992] defined several categories of gestures which frequently accompany speech: iconic where a person acts out an action or object as they describe it; metaphoric where a person treats an abstract concept as a physical object, for example, using a giving gesture to indicate generosity; beat where the hand moves with the rhythm of speech; and deictic which refers to pointing gestures. Different types of gestures occur during different types of speech. Iconic gestures typically accompany utterances when describing concrete objects and events, for example, during narration when telling a story, whereas metaphoric gestures can often be observed when referring to the structure of speech [Goldin-Meadow 2003; McNeill 1992]. There are also other types of categorizations, such as the ones by Kendon or Ekman and Friesen [Ekman and Friesen 1969; Kendon 2004]. Independent of the classification used, the proportions of different types of gestures vary immensely depending on the subject and type of a conversation and many other factors such as cultural background or age [Colletta et al. 2015; Goldin-Meadow 2003].

The focus of research involving gestures lies on gestures used in combination with speech. McNeill [1992] also uses the word gesticulation to specifically denominate speech-based gestures. The gestures during charades are categorized as pantomime and emblems on the gesture continuum. In contrast to gesticulation, pantomime is made with a purpose, it is improvised, characterized by the absence of speech, and it does not necessarily have gesture phases [McNeill 2015; Żywiczyński et al. 2018]. Emblems, such as the “thumbs up” gesture, have a specific, socially defined meaning and shape. They are also made on purpose to communicate and do not require speech. We provide further details on our charades and how they differ from other types of communication in Sections 3.1 and 7. As the gestures when performing charades are made on purpose to convey meaning and do not involve speech, they are an ideal test case for this study.

Avatar Appearance. Finally, the appearance of the character can affect how viewers interpret its motions. For example, Hodgins et al. [1998] found that viewers could detect differences in motions more easily with a polygonal character than with a stick figure. Chaminade et al. [2007] also showed that artificial and biological motions are perceived differently depending on the character model. McDonnell et al. [2012] found that realistic avatars were more unpleasant than stylistic avatars when they moved unnaturally and Baylor [2011] showed that the appearance of a virtual tutor influences the motivation of the learner. It has also been shown that the presence and appearance of avatars has an effect on people’s perceptions in virtual reality settings. The use of avatars has been shown to increase social presence in VR [Aseeri and Interrante 2021; Smith and Neff 2018]. Kilteni et al. [2013] demonstrated that people’s drumming skills improved when they were given a dark-skinned self-avatar in VR. Further research has shown that the appearance of virtual hands influences the perceived ownership in VR [Argelaguet et al. 2016; Lin and Jörg 2016]. To investigate the influence of the virtual character’s appearance, we create two virtual characters with different levels of realism for our experiments: a simple mannequin and a more realistic character.

In summary, hand motions are an essential aspect of body language during communication, contributing to comprehension and influencing the perception of the character. Subtle errors in hand motions have been shown to matter in some cases. In this work, we investigate the effect of such errors in hand motions when we communicate. Our results are important to help us understand how we can successfully convey the information and personality we aim at conveying with virtual characters.

Our study consists of three experiments: Alteration, Intensity, and VR (see Table 1). In our experiments, among other questions, participants are asked to guess acted movie titles or words displayed on a virtual character, in a similar way to the game “charades” where players pantomime words or phrases.

For both the Alteration and Intensity experiments, our independent variables are the Motion Condition (Motion Alteration or Motion Intensity), Avatar, and Clip Length. Our dependent variables are the participants’ comprehension, perception of character, and social presence. Both experiments use a between-group design so that no participant would be asked to guess the same movie or word more than once. Therefore, each participant viewed a series of clips with consistent motion condition, avatar, and clip length. For both experiments, participants were recruited online through Amazon Mechanical Turk.

The VR experiment was conducted in-person at Clemson University and consists of three parts: VR Compare, VR Comfort, and VR Rank. VR Compare repeats the Alteration Experiment with a small subset of conditions. VR Comfort and VR Rank evaluate whether the hand motions influence how comfortable participants feel while watching the character: VR Comfort asks participants for comfort ratings and VR Rank asks them to rank motion conditions based on comfort in direct comparison. The independent variable in all three parts is the Motion Condition. The dependent variables of the first part are the same as in the Alteration and Intensity Experiments; in the second and third part they are the participants’ comfort and ranking.

Our hypotheses are as follows:

The Alteration Experiment examines the role of hand animation accuracy and character appearance on participants’ comprehension, perception of the character, and social presence. A between-group design was used, where each participant saw either 15 short clips or six long motions on one avatar (out of two) with one motion alteration (out of eight), leading to a total of 32 different conditions: 8 (Motion Alteration) \(\times\) 2 (Avatar) \(\times\) 2 (Clip Length).

In our first experiment, we found that our Partial Motion conditions did not lead to many differences compared to our Original condition. They did not reduce Motion Comprehension or Perceived Comprehension at all. However, if the intensities of these errors are increased, at some point we expect them to influence comprehension as there is no meaningful data left. For example, smoothing a motion to an extreme point would result in a static, averaged hand pose, which corresponds to our Static condition. Increasing the intensity of Popping to an extreme level would result in one or a few random poses being held for a long time, and exaggerating the Jitter condition would result in an erratic, random-type motion. The levels of errors we added were very reasonable. Therefore, in our second experiment, we test higher levels of errors with the goal of finding thresholds up to which the errors would be acceptable.

In this article, we investigate the effects of errors in or the lack of hand motions on comprehension, perception of a virtual character, perceived social presence, and comfort when watching the character. We summarize our key findings in Table 2 and as follows:

Our experiments confirm the importance of detailed hand motions for communication, social presence, and for accurately conveying personality. Furthermore, we found several surprising results. We expected to see negative effects when showing static hands, but also when seeing hand motions with large errors. However, errors in hand motions did not reduce comprehension, even when the errors were very obvious and larger than what one would encounter in practice as was the case in our Intensity experiment. We assume that the redundancy in motions is large enough, so that viewers are able to extract or infer the information necessary for comprehension or forming impressions of a character even if the data is incomplete or noisy. The thresholds for reducing comprehension were higher than the values we tested. Jitter was perceived negatively at lower intensities in some cases, but still did not reduce comprehension even with a high intensity.

While adding random or passive motions to the hands in the absence of data did not help with comprehension, it did at least improve social presence and how the character was perceived to some degree. Both chosen methods (conditions Passive and Random) were rather simple. It would be interesting to see if other, more complex methods of creating hand motions when no data is available might help with motion comprehension or increase, for example, the perceived naturalness.

Finally, we found that character design is important (not surprisingly), and that it might be even more important when the character is seen for longer times and in virtual reality. This might be due to viewers noticing more details when given more time and when sharing a virtual environment with a character. While in our case the realistic character was perceived more positively than the mannequin when seen for longer times, this result might be different based on the exact design of the character, e.g., if we had used a highly appealing cartoony character our results might be different.

While we were able to answer many questions with our experiments, it also has limitations. In these experiments, the virtual character plays charades. We chose this type of motions as we specifically were looking for tasks with expressive motions where body motions and hand motions might be important and for a quantitative way to measure comprehension. We were furthermore trying to avoid any confounding effects that audio might have. However, gestures are used differently when playing charades where one might use more iconic and metaphoric gestures than during typical conversations where beat gestures are more common and detailed hand motions might therefore be less important. To quantify those differences, the gestures of all of the charades motions used in this study were labeled by two graduate students and classified as iconic, metaphoric, beat, or deictic based on the descriptions by McNeill [1992, 2015]. In the 3 minutes and one second of charades, we detected 32 iconic, 37 metaphoric, 8 deictic, and not a single beat gesture. As a comparison, the same process was applied to two motion databases from Jörg et al. [2012], one called Conversations database that includes 8 minutes and 5 seconds of narrations and one called Debates database from the same actor as the charades and with 9 minutes and 34 seconds of debates. In the narrations, 42 gestures were coded as iconic, 27 as metaphoric, 9 as deictic, and 66 as beat gestures. In the debates, 28 gestures were categorized as iconic, 77 as metaphoric, 29 as deictic, and 74 as beat gestures. As expected, the charades show a higher frequency of iconic and metaphoric gestures and a lower frequency of beat gestures than the conversations or the debates. We also compare this distribution to findings from the literature: McNeill provides statistics of six cartoon narratives by English-language speaking university students. There are 261 iconic, 43 metaphoric, 28 deictic, and 268 beat gestures in an estimated 49 minutes of narration. Such a distribution would be expected in a narrative scenario and is closest to our conversations database.

While results might differ depending on the exact type of communication, accurate hand motions are likely to be less relevant in a conversational scenario with audio and more beat gestures. We conclude that, if errors in hand motions did not reduce motion comprehension when playing charades, they are unlikely to affect comprehension during a typical conversation with audio. Still, gestures are an integral part of conversations and the knowledge that the complete lack of hand motion reduces comprehension in some cases might be enough to attempt to add at least some hand motions all of the time.

A second design choice and limitation was to not animate the face of the realistic avatar to avoid confounding factors with the mannequin. Of course, the presence of detailed facial animations is likely to influence our results. We assume that differences between conditions would be less pronounced as facial animation might convey additional information and distract the viewer from the hands. Changes are likely to depend on the detailedness of the facial animation. In current VR social rooms, facial animation, if present, typically only includes motion of the jaw matching the audio. As that animation is very limited, we assume that our results apply well to current VR scenarios. However, future work will have to show the influence of accurate hand motions when detailed facial motions are present.

Future work would be able to further investigate these effects and answer further questions. Would people in a live scenario adapt and move differently or speak more clearly if errors occur in the motions? Do the results vary depending on the expressiveness of gestures, the personalities of the performers, the information conveyed, and the emotional content of the conversation? For a complete picture, many variables need to be taken into account in future work. The influence of the design of the avatar from stylized floating upper bodies with floating hands to realistic virtual characters should be investigated further. It would also be interesting to find out if hand motions can be learned that actually contribute to comprehension. Our experimental setup could serve as a test bed for such approaches.

Based on our findings, we have several recommendations for developers and animators to consider when creating virtual characters or interactions with avatars in VR. We recommend to capture at least partial hand motion whenever possible even if they contain some errors. Smaller errors and even most of the larger ones we tested did not affect comprehension or social presence or how the character was perceived. The main exception was jitter, which should be avoided or smoothed. However, even highly jittery motion contributes to comprehension. If no hand motions can be acquired, creating some substitute motions is still better than leaving the fingers immobile when it comes to social presence and how the character is perceived.

The authors thank Adam Wentworth for spending many hours on post-processing motion capture files and Christian Sharpe for fitting our character model to our skeleton.