Perceptual Study on Facial Expressions

Perceptual Study on Facial Expressions Eva G. Krumhuber and Lina Skora Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Early Beginnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Dynamic Advantage in Facial Expression Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Temporal Characteristics: Directionality and Speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Effects of Facial Motion on Perception and Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Ratings of Authenticity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Person Judgments and Behavioral Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Facial Mimicry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Neuroscientific Evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Cross-References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Abstract Facial expressions play a paramount role in character animation since they reveal much of a person’s emotions and intentions. Although animation techniques have become more sophisticated over time, there is still need for knowledge in terms of what behavior appears emotionally convincing and believable. The present chapter examines how motion contributes to the perception and interpretation of facial expressions. This includes a description of the early beginnings in research on facial motion and more recent work, pointing toward a dynamic advantage in facial expression recognition. Attention is further drawn to the potential characteristics (i.e., directionality and speed) that facilitate such dynamic advantage. This is followed by a review on how facial motion affects perception and E.G. Krumhuber (*) • L. Skora University College London, London, UK e-mail: e.krumhuber@ucl.ac.uk # Springer International Publishing AG 2016 B. Müller, S.I. Wolf (eds.), Handbook of Human Motion, DOI 10.1007/978-3-319-30808-1_18-1 1 2 E.G. Krumhuber and L. Skora behavior more generally, with the neural systems that underlie the processing of dynamic emotions. The chapter concludes by discussing remaining challenges and future directions for the animation of natural occurring emotional expressions in dynamic faces. Keywords Motion • Dynamic • Facial expression • Emotion • Perception Introduction Among the 12 principles of animation developed in the early 1930s, animators at the Walt Disney Studios considered motion to be fundamental for creating believable characters. They were convinced that the type and speed of an action help define a character’s intentions and personality (Kerlow 2004). Since the early days of character animation, much has changed in animation techniques and styles. From hand-drawn cartoon characters to real-time three-dimensional computer animation, the field has seen a major shift toward near-realistic characters that exhibit humanlike behavior. Whether those are used for entertainment, therapy, or education, the original principle of motion continues to be of interest in research and design. This particularly applies to the topic of facial animation as subtle elements of the character’s thoughts and emotions are conveyed through the face (Kappas et al. 2013). Facial expressions provide clues and insight about what the character thinks and feels. They act as a powerful medium in conveying emotions. Although the tools for facial animation have become more sophisticated over time (i.e., techniques for capturing and synthesizing facial expressions), there is still need for knowledge about how humans respond to emotional displays in moving faces. Only if the character appears emotionally convincing and believable will the user/audience feel comfortable in interaction. The present chapter aims to help with this task by providing an overview of the existing literature on the perception of dynamic facial expressions. Given the predominant focus on static features of the face in past research, we seek to highlight the beneficial role of facial dynamics in the attribution of emotional states. This includes a description of the early beginnings in research on facial motion and more recent work pointing toward a dynamic advantage in facial expression recognition. The next section draws attention to the potential characteristics that facilitate such dynamic advantage. This is followed by a review on how facial motion affects perception and behavior more generally. Neural systems in the processing of dynamic emotions and their implications for action representation are also outlined. The final section concludes the paper by discussing remaining challenges and future directions for the animation of natural occurring emotional expressions in dynamic faces. Perceptual Study on Facial Expressions 3 State of the Art Early Beginnings In everyday settings, human motion and corresponding properties (e.g., shapes, texture) interact to produce a coherent percept. Yet, motion conveys important cues for recognition even in isolation from the supportive information. The human visual system, having evolved in dynamic conditions, is highly attuned to dynamic signals within the environment (Gibson 1966). It can use this information to identify an agent or infer its actions purely by the motion patterns inherent to living organisms, called biological motion (Johansson 1973). Investigations of biological motion of the face suggest that the perception of faces is aided by the presence of nonrigid facial movements, such as stretching, bulging, or flexing of the muscles and the skin. In an early and now seminal point-light paradigm (Bassili 1978), all static features of actors’ faces, such as texture, shape, and configuration, were obscured with the use of black makeup. Subsequently, the darkened faces were covered with approximately 100 luminescent white dots and video recorded in a dark room displaying a range of nonrigid motion, from grimaces to the basic emotional expressions (happiness, sadness, fear, anger, surprise, and disgust). The dark setup resulted in only the bright points being visible to the observer, moving as a result of facial motion. In a recognition experiment, the moving dots were recognized as faces significantly better than when the stimulus was shown as a sequence of static frames or as a static image. Similarly, moving point-light faces enabled above-chance recognition of the six basic emotional expressions in comparison to motionless point-light displays (Bassili 1979; Bruce and Valentine 1988). This suggests that when static information about the face is absent, biological motion alone is distinctive enough to provide important cues for recognition. Dynamic Advantage in Facial Expression Recognition Subsequent research has pointed toward a motion advantage especially when static facial features are compromised. This is of particular relevance for computergenerated, synthetic faces (e.g., online avatars, game characters). In comparison to natural human faces, synthetic faces are still inferior in terms of their realistic representation of the finer-grained features, such as textures, skin stretching, or skin wrinkling. Such impairment in quality of static information can be remedied by motion. Numerous studies have shown that expression recognition in dynamic synthetic faces consistently outperforms recognition in static synthetic faces (Ehrlich et al. 2000; Wallraven et al. 2008; Wehrle et al. 2000; Weyers et al. 2006). This suggests that motion is able to add a relevant layer of information when synthetic features fail to provide sufficient cues for recognition. The effect is found both under uniform viewing quality and when the featural or textural information is degraded (e.g., blurred). 4 E.G. Krumhuber and L. Skora For natural human faces, however, the dynamic advantage is weaker or inexistent when the quality of both static and dynamic displays is comparably good (Fiorentini and Viviani 2011; Kamachi et al. 2001; Kätsyri and Sams 2008). As such, motion is likely to provide additional cues for recognition when key static information is missing (i.e., in degraded and obscured expressions). Its benefits may be redundant when the observer can draw enough information from the static properties of the face. This applies to static stimuli that typically portray expressions at the peak of emotionality. Such stimuli, prominently used in face perception research, are characterized by their stereotypical depiction of a narrow range of basic emotions. They are often also posed upon instructions by the researcher and follow a set of prototypical criteria (e.g., Facial Action Coding System, FACS; Ekman and Friesen 1978). In this light, it is likely that stylized static expressions contain the prototypical markers of specific emotions, thereby facilitating recognition. Yet, everyday emotional expressions are spontaneous and often include non-prototypical emotion blends or patterns. They are normally also of lower intensity, potentially becoming more difficult to identify without supportive cues such as motion. For instance, low-intensity expressions, which tend to be more difficult to identify the less intense they get, are recognized significantly better in a dynamic than static form (Ambadar et al. 2005; Bould and Morris 2008). In this context, motion appears to provide additional perceptual cues, making up for insufficient informative signals. Temporal Characteristics: Directionality and Speed How can we explain the motion advantage in expression recognition? Could it simply derive from an increase in the amount of cues in a dynamic sequence? Early hypotheses point out that a moving sequence contains a greater number of static information from which to infer emotion judgments than a single static portrayal (Ekman and Friesen 1978). Arguably, as a dynamic sequence unfolds, it provides multiple samples of the developing expression compared to a single sample in static displays. To test this assumption, Ambadar et al. (2005) compared emotion recognition performances between dynamic, static, and multi-static expressions. In the multi-static condition, static frames constituting a video were interspersed with visual noise masks disrupting the fluidity of motion. Out of these, dynamic expressions were recognized with a significantly greater accuracy than both multi-static and static portrayals (see also Bould and Morris 2008). This suggests that the intrinsic temporal quality of the unfolding expression is what helps to disambiguate its content rather than a mere increase in static frames. A likely candidate that facilitates the dynamic advantage is the directionality of change in the expression over time. Research shows that humans are highly sensitive to the direction in which the expression unfolds. For example, they are able to accurately detect the directionality in a set of scrambled images and arrange them into a temporally correct sequence (Edwards 1998). Similarly, disrupting the natural temporal direction of the expression results in worse recognition accuracy than when Perceptual Study on Facial Expressions 5 the expressions unfold naturally. In a series of experiments, Cunningham and Wallraven (2009b) demonstrated this by applying various manipulation techniques to the direction of unfolding, such as scrambling the frames in a dynamic sequence or playing them backward. Their results indicate that the identification of emotional expressions suffers considerably when natural motion is interrupted. Recognition performance also appears to be better in sequences in which the temporal unfolding is preserved, thereby allowing the directionality of change to be observed as the expression emerges (Bould et al. 2008, but see Ambadar et al. 2005 for a contrasting result). Yet, it is noteworthy that this effect might not affect all emotions equally. For example, happiness is typically recognized better than other basic emotions regardless of condition. In addition to the movement direction, the velocity of unfolding plays a crucial role in emotion perception. Changes in viewing speed, such as slowing down or speeding up of the dynamic sequences, significantly affect expression recognition accuracy. This effect appears to be different between emotions based on the differences in their intrinsic optimum velocities. For example, sadness is naturally slow; so slowed-down viewing conditions do not impact it negatively as much as they impact recognition accuracy for all other tested emotions (Kamachi et al. 2001). Conversely, surprise is naturally fast, and it could be its natural velocity that distinguishes it from the morphologically similar expression of fear which is slower (Sato and Yoshikawa 2004). Importantly, changing the speed throughout an entire expression results in different effects as changes to the duration of the peak. This suggests that the beneficial effects of natural movements cannot simply be explained by the mere exposure time to the expression (Kamachi et al. 2001; Recio et al. 2013). Overall, altering the speed of expression unfolding appears to influence perception without affecting the direction of change. As such, the intrinsic velocities of particular emotional expressions are likely to provide stronger cues for recognition than the perception of change alone (Bould et al. 2008). Finally, the perception of dynamic faces is also linked to the quality of motion. While expressions in real faces unfold in a biologically natural manner (i.e., nonlinearly), facial animations have been often characterized by linear techniques. Such linearly unfolding facial expressions (e.g., dynamic displays morphed from individual static displays) yield slower and poorer recognition accuracy in comparison to natural, nonlinear unfolding, as well as worse naturalness and genuineness ratings (Cosker et al. 2010; Wallraven et al. 2008). As a result, linear morphs might not constitute a good representation of the real-life quality of facial motion, which is particularly relevant to the construction of realistic synthetic faces. However, recent developments within the field of affective computing identify multiple parameters linked to naturalistic expression unfolding that can improve the quality of motion in computer-generated faces and raise their recognition rates, such as appropriate speeds, action unit (AUs) activations, intensities, asymmetries, and textures (Krumhuber et al. 2012; Recio et al. 2013; Yang et al. 2013). As such, the benefits provided by motion appear to be more than the perception of motion itself. Instead, it is a comprehensive set of information deriving from the temporal characteristics 6 E.G. Krumhuber and L. Skora including the perception of change, intrinsic velocity of an expression, and the quality of motion. Effects of Facial Motion on Perception and Behavior In addition to the supportive role in expression recognition, motion also affects a number of perceptual and behavioral factors. Those include expression judgments such as intensity and authenticity, as well as behavioral responses and even mimicry. Firstly, emotions expressed in a dynamic form are perceived to be more intense than the same emotions in a static form (Biele and Grabowska 2006; Cunningham and Wallraven 2009a). Motion appears to enhance intensity estimates because of the changes in the expression as it develops from neutral to fully emotive. While static portrayals retain the same intensity level throughout the presentation time, dynamic changes highlight the contrast between the neutral and fully emotional expression. As such, the contrast makes the expression seem more intense (Biele and Grabowska 2006). Another explanation for this effect was offered in terms of representational momentum (RP). RP is a visual perception phenomenon in which the observer exaggerates the final position of a gradually moving stimulus. It often involves a forward displacement. For example, when a moving object disappears from the visual field, observers tend to report its final position as displaced further down its trajectory than it objectively was. In a study about dynamic facial expressions and RP, Yoshikawa and Sato (2008) found that participants exaggerated the last – fully emotive – frame of the dynamic sequence and remembered it as more intense that it was in reality. The effect also got more pronounced with increasing velocity of expression unfolding. As such, it seems that the gradual shift from neutral to emotional in dynamic expressions generates a forward displacement, inducing an exaggerated and intensified perception of the final frame in the sequence. Ratings of Authenticity Motion also appears to help observers assess the authenticity of an expression better than static portrayals can. Authenticity refers to more than correct identification of the emotional expression observed. It is a quality telling us whether the emotion is genuinely experienced or not. Smiles have been prominently used to study this dimension. Being universal and widespread in everyday interactions, smiles can indicate a range of feelings, from happiness and amusement to politeness and embarrassment (Ambadar et al. 2009). However, smiles can also be easily used to mask real emotions or to deceive others (e.g., Ekman 1985). As such, they constitute a good stimulus to study the genuineness of the underlying feeling. Traditionally, the so-called Duchenne marker has been considered as an indicator of smile authenticity (Ekman et al. 1990), where its presence signals that a smile is genuine (“felt”) as opposed to false (“unfelt”). The Duchenne marker involves, in addition to the lip corner puller (zygomaticus major muscle), the activation of the Perceptual Study on Facial Expressions 7 orbicularis oculi muscle surrounding the eye. This results in wrinkling on the sides of the eyes, commonly referred to as crow’s feet. While the validity of the Duchenne marker in the perception of static expressions is well documented, motion properties are crucial for assessing smile authenticity in dynamic displays (e.g., Korb et al. 2014; Krumhuber and Manstead 2009). For example, genuine smiles differ in lip corner and eyebrow movements from deliberate, false smiles (Schmidt et al. 2006; Schmidt et al. 2009). More specifically, Frank et al. (1993) highlighted three dynamic markers of genuine smiles: expression duration, synchrony in muscle activation (between zygomaticus major and orbicularis oculi muscles), and smoothness of mouth movements. Overall, genuine smiles last between 500 and 4000 ms, whereas false smiles tend to be shorter or longer (Ekman and Friesen 1982). Furthermore, the smoothness and duration of the expressive components of smiles are meaningful indicators. Bugental (1986) and Weiss et al. (1987) were first to show that the onset and offset in false smiles tend to be faster in comparison to felt smiles (see also Hess and Kleck 1990). To investigate whether these differences affect expression perception, Krumhuber and Kappas (2005) manipulated onset, apex, and offset timings of computergenerated smiles. Their results confirmed the proposition that each dynamic element of a smiling expression has an intrinsic duration range at which it looks genuine. In particular, expressions are perceived as more authentic the longer their onsets and offsets, while a long apex is linked to lower genuineness ratings. Person Judgments and Behavioral Responses Besides their effects on authenticity ratings, dynamic signals influence trait attributions and behavioral responses to the target expressing an emotion. For instance, people displaying dynamic genuine smiles (long onset and offset) are rated as more trustworthy, more attractive, and less dominant than those who show smile expressions without those characteristics (Krumhuber et al. 2007b). In addition, facial movement helps to regulate interpersonal relations by shaping someone’s intention to approach or cooperate with another person. In economic trust games, participants can receive a financial gain if their counterpart cooperates but incur a loss if the counter-player fails to cooperate. As such, their performance depends on accurate assessment of the counterpart’s intentions. Krumhuber and colleagues (Krumhuber et al. 2007a) showed that people are more likely to trust and engage in an interaction with a counterpart who displays a dynamic authentic smile than a dynamic false smile or neutral expression. Participants with genuinely smiling counterparts also ascribe more positive emotions and are more inclined to meet them again. Furthermore, people showing dynamic genuine smiles are evaluated more favorably and considered more suitable candidates in a job interview than those who do not smile or smile falsely (Krumhuber et al. 2009). Notably, this effect applies to real human faces as well as to computer-generated ones. When comparing static and dynamic facial features, it appears that they contribute to different evaluations and social decisions. Static and morphological features, 8 E.G. Krumhuber and L. Skora such as bone structure or width, have been found to affect judgments of ability and competence. In turn, features that are dynamic and malleable, like muscular patterns in emotional expressions, affect judgments of intentions (Hehman et al. 2015). Given that these facial signals are also linked to evaluations of trustworthiness and likeability, they are likely to drive decision-making in social interactions. In line with this argument, participants were shown to choose a financial advisor, a role requiring trust, based on dynamic rather than static facial properties (Hehman et al. 2015). Facial Mimicry Existing evidence suggests that dynamic facial displays elicit involuntary and subtle imitative responses more evidently than do static versions of the same expression (Rymarczyk et al. 2011; Sato et al. 2008; Weyers et al. 2006). Those responses, interpretable as mimicry, are a result of activity in facial muscles corresponding to a given perceived expression (i.e., lowering the eyebrows in anger, pulling the lip corners in happiness). They occur spontaneously and swiftly (about 800–900 ms) after detecting a change in the observed face. While involuntary facial mimicry is a subtle rather than full-blown replication of a witnessed emotion, it is evident enough to be distinguished in terms of its valence (positive or negative quality) by independent observers (Sato and Yoshikawa 2007a). Crucially, the presence of mimicry has a supporting role in emotion perception. For example, being able to mimic helps observers to recognize the emotional valence of expressions (Sato et al. 2013). Happiness and disgust are less well identified when corresponding muscles are engaged by biting on a pen which effectively blocks mimicry in the lower part of the face (Oberman et al. 2007; Ponari et al. 2012). In a similar vein, blocking mimicry in the upper part of the face by drawing together two stickers placed above the eyebrows impairs the recognition of anger. Mimicry also appears useful in detecting changes in expressions. Having to identify the point at which an expression transforms from one emotion into another (e.g., happiness to sadness) proves more difficult when mimicry is blocked by holding a pen sideways between the teeth. For this task, participants who are free to mimic are quicker in spotting changes in the dynamic trajectory of facial expressions (Niedenthal et al. 2001). Furthermore, mimicry aids emotion judgments, particularly in the context of smile authenticity. Dynamic felt smiles are more easily distinguished from dynamic false ones when expressions can be freely mimicked compared to when mimicry is blocked by a mouth-held pen (Maringer et al. 2011; Rychlowska et al. 2014). Overall, those findings suggest that facial mimicry helps to make inferences about dynamic emotional faces such as emotion recognition and trajectory changes or authenticity judgments. As such, it adds to the evidence that facial motion conveys information that is essential to comprehensive expression perception, while also driving behavioral responses. Perceptual Study on Facial Expressions 9 Neuroscientific Evidence Evidence from neuroscience suggests that differences in the processing of dynamic and static facial stimuli begin at a neural level. For example, studies of patients with brain lesions or neurological disorders point toward a dissociation in the neural routes for processing dynamic and static faces. In the most notable cases, patients who are unable to recognize emotions from static displays can easily do so from moving displays (Adolphs et al. 2003; Humphreys et al. 1993). In healthy people, dynamic facial expressions evoke significantly larger and more widespread activation patterns in the brain than static expressions (LaBar et al. 2003; Sato et al. 2004). This enhanced activation is apparent in a range of brain regions, starting with the visual area V5 which subserves motion perception. It has also been observed in the fusiform face area (FFA), a number of frontal and parietal regions, and the superior temporal sulcus (STS), areas implicated in the processing of faces, emotion, and biological motion, respectively (Kessler et al. 2011; Trautmann et al. 2009). The STS has been given particular consideration due to its involvement in interpreting social signals, in addition to biological motion. As such, enhanced activation in the STS in response to dynamic facial stimuli could be related to extracting socially relevant information (i.e., intentions) from the changeable features of the face (Arsalidou et al. 2011; Kilts et al. 2003). Additionally, in an electroencephalograph (EEG) study, attention-related brain activity was found to be greater and longer when participants observed dynamic compared to static stimuli (Recio et al. 2011). This higher activity continued throughout the duration of an expression, contributing to more elaborate processing of dynamic faces. Such enhanced and more widespread brain activation in response to facial motion could be caused by the fact that dynamic expressions are inherently more complex to process. Equally, it could derive from greater evolutionary experience with moving faces and the need to extract social meaning from them for effective communication. In this light, neurological evidence lends support to the behavioral findings. Improved recognition accuracy, sensitivity to the temporal characteristics, and the ability to make inferences about genuineness, trustworthiness, or approachability could be an effect of enhanced processing of dynamic faces. Besides phenomena of neural adaptation, there is work suggesting that brain activity while observing facial movements may encompass regions which are linked to one’s own experience of emotional states, as well as areas reported to contain mirror neurons (Dapretto et al. 2006). Initially observed in macaque monkeys, mirror neurons fire both when performing an action and when watching the action in others (Rizzolatti et al. 1996). Emotion perception may therefore be partially subserved by the mirror neuron system (i.e., premotor and parietal regions, superior temporal sulcus; Iacoboni and Dapretto 2006; Rizzolatti and Craighiero 2004) which activates an internal representation of the observed state almost as if it was felt by oneself. Supportive evidence comes from research showing that facial mimicry in response to observed expressions activates similar patterns in the brain of the perceiver (Lee et al. 2006). Also, observing an emotional experience of someone elicits 10 E.G. Krumhuber and L. Skora corresponding subjective arousal in oneself (Lundqvist and Dimberg 1995) which is found to be stronger for dynamic than static faces (Sato and Yoshikawa 2007b). Importantly, it has been proposed that this mirror neuron system has evolved to produce an implicit internal understanding of others’ mental states and intentions (Dimberg 1988; Gallese 2001). Following from this assumption, mirroring brain activity in response to facial expressions could be the driving force behind higherorder cognitive processes such as empathy or mentalizing (Iacoboni 2009). For example, witnessing a painful expression on someone’s face and feeling pain oneself activate largely overlapping neural pathways which are correlated with regions linked to empathy (Botvinick et al. 2005; Singer et al. 2004). The ability to mimic expressions was also shown to cause greater prosocial behavior, arguably mediated by greater empathy derived from mimicry and shared activations (Stel et al. 2008). Overall, this has been taken to suggest that humans understand, empathize with, and make inferences about mental states of others because the action-perception overlap activates internal experiences of the same state (Schulte-Rüther et al. 2007). Future Directions From the literature reviewed above, there is conclusive evidence suggesting that humans have remarkable abilities to perceive and understand the actions of others. Driven by the universal need for social connection, the efficient detection and interpretation of social signals appears essential for successful interaction. Given the rapid advances in technology, these uniquely adaptive skills are likely to be translated to a new form of social partners in the near future. With the move of computing into the social domain, nonhuman agents are envisaged to become integral parts of our daily lives, from the workplace to social and private applications (Küster et al. 2014). As a result, many interactions will not occur in their traditional form (i.e., human to human) but instead involve computer-generated avatars and social robots. In order to build animated systems that emit appropriate social cues and behavior, it is imperative to understand the factors that influence perception. Facial expressions prove to play a vital part in this process since they reveal much of a character’s emotions and intentions. While animation techniques offer more control than ever over visual elements, subtle imperfections in the timing of facial expressions could evoke negative reactions from the viewer. In 1970, Masahiro Mori described a phenomenon called the “uncanny valley” (UV) in which human-realistic characters are viewed negatively if they are almost but not quite perfectly human. As such, increased human-likeness may result in unease when appearance or behavior fall short of emulating those of real human beings. Classic examples can be found in computer-animated films, such as The Polar Express and Final Fantasy: The Spirits Within, which many viewers find disturbing due to their human-realistic but eerie characters (Geller 2008). According to Mori, this perceived deviation from normal human behavior is further pronounced when movement is added. Particularly, if the appearance is more advanced than the behavior, violated perceptual expectations could make the moving character less Perceptual Study on Facial Expressions 11 acceptable. In line with this argument, Saygin et al. (2012) showed that androids that look human but don’t move in a humanlike (biological) manner elicit a prediction error that leads to stronger brain activity in the perceiver. Furthermore, virtual characters are more likely to be rated as uncanny when their facial expressions lack movement in the forehead and eyelids (Tinwell et al. 2011). Although the exact role of motion in the UV remains an issue of debate (see Kätsyri et al. 2015), there is increasing evidence suggesting that natural human motion positively influences the acceptability of characters, particularly those that would fall appearance-wise into the UV (i.e., zombies; Piwek et al. 2014). As such, high-quality motion has the potential to improve ratings of familiarity and humanlikeness by eliciting higher affinity (McDonnnell et al. 2012, Thompson et al. 2011). In order for natural motion to become the standard in animation, it is essential to rely on behavior representative of the real world. At the moment, databases depicting dynamic emotional expressions are still limited in the range and type of facial movements being captured. The majority of them contain deliberately posed affective displays recorded under highly constrained conditions (for a review see Krumhuber et al. in press). Such acted portrayals may not provide an optimal basis for the modeling of naturally occurring emotions. For progress to occur in the future, efforts that target the dynamic analysis and synthesis of spontaneous behavior will prove fruitful. This also includes the study of how multiple dynamic cues interact to produce a coherent percept. Only once the dynamic nature of facial expressions is fully understood will it be possible to successfully incorporate this knowledge into animation models. The present chapter underscores the importance of this task by showing that perceivers are highly sensitive to the motion dynamics in the perceptual study of facial expressions. Cross-References ▶ Blendshape Facial Animation ▶ Real-Time Full Body (or face) Posing ▶ Video-Based Performance Driven Facial Animation References Adolphs R, Tranel D, Damasio AR (2003) Dissociable neural systems for recognizing emotions. Brain Cogn 52:61–69. doi:10.1016/S0278-2626(03)00009-5 Ambadar Z, Cohn JF, Reed LI (2009) All smiles are not created equal: morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. J Nonverbal Behav 33:17–34. doi:10.1007/s10919-008-0059-5 Ambadar Z, Schooler JW, Cohn JF (2005) Deciphering the enigmatic face. The importance of facial dynamics in interpreting subtle facial expressions. Psychol Sci 16:403–410. doi:10.1111/j.09567976.2005.01548.x Arsalidou M, Morris D, Taylor MJ (2011) Converging evidence for the advantage of dynamic facial expressions. Brain Topogr 24:149–163. doi:10.1007/s10548-011-0171-4 12 E.G. Krumhuber and L. Skora Bassili JN (1978) Facial motion in the perception of faces and of emotional expression. J Exp Psychol Hum Percept Perform 4:373–379. doi:10.1037/0096-1523.4.3.373 Bassili JN (1979) Emotion recognition: the role of facial movement and the relative importance of upper and lower face. J Pers Soc Psychol 37:2049–2058. doi:10.1037//0022-3514.37.11.2049 Biele C, Grabowska A (2006) Sex differences in perception of emotion intensity in dynamic and static facial expressions. Exp Brain Res 171:1–6. doi:10.1007/s00221-005-0254-0 Botvinick M, Jha AP, Bylsma LM, Fabian SA, Solomon PE, Prkachin KM (2005) Viewing facial expressions of pain engages cortical areas involved in the direct experience of pain. Neuroimage 25:312–319. doi:10.1016/j.neuroimage.2004.11.043 Bould E, Morris N (2008) Role of motion signals in recognizing subtle facial expressions of emotion. Br J Psychol 99:167–189. doi:10.1348/000712607X206702 Bould E, Morris N, Wink B (2008) Recognising subtle emotional expressions: the role of facial movements. Cognit Emot 22:1569–1587. doi:10.1080/02699930801921156 Bruce V, Valentine T (1988) When a nod’s as good as a wink: the role of dynamic information in facial recognition. In: Gruneberg MM, Morris PE, Sykes RN (eds) Practical aspects of memory: current research and issues, vol 1. John Wiley and Sons, New York, pp 169–174 Bugental DB (1986) Unmasking the “polite smile”. Situational and personal determinants of managed affect in adult-child interaction. Pers Soc Psychol Bull 12:7–16. doi:10.1177/ 0146167286121001 Cosker D, Krumhuber EG, Hilton A (2010) Perception of linear and nonlinear motion properties using a FACS validated 3D facial model. In: Proceedings of the symposium on applied Perception in graphics and visualization (APGV), Los Angeles Cunningham DW, Wallraven C (2009a) The interaction between motion and form in expression recognition. In: Bodenheimer B, O’Sullivan C (eds) proceedings of the 6th symposium on applied perception in graphics and visualization (APGV2009), New York Cunningham DW, Wallraven C (2009b) Dynamic information for the recognition of conversational expressions. J Vis 9:1–17. doi:10.1167/9.13.7 Dapretto M, Davies MS, Pfeifer JH, Scott AA, Sigman M, Bookheimer SY, Iacoboni M (2006) Understanding emotions in others: mirror neuron dysfunction in children with autism spectrum disorders. Nat Neurosci 9:28–30. doi:10.1038/nn1611 Dimberg U (1988) Facial electromyography and the experience of emotion. J Psychophysiol 2:277–282 Edwards K (1998) The face of time: temporal cues in facial expressions of emotion. Psychol Sci 9:270–276. doi:10.1111/1467-9280.00054 Ehrlich SM, Schiano DJ, Sheridan K (2000) Communicating facial affect: it’s not the realism, it’s the motion. In: Proceedings of ACM CHI 2000 conference on human factors in computing systems, New York Ekman P (1985) Telling lies. Norton, New York Ekman P, Friesen WV (1982) Felt, false, and miserable smiles. J Nonverbal Behav 6:238–252. doi:10.1007/BF00987191 Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto Ekman P, Davidson RJ, Friesen WV (1990) The Duchenne smile: emotional expression and brain physiology: II. J Pers Soc Psychol 58:342–353. doi:10.1037/0022-3514.58.2.342 Fiorentini C, Viviani P (2011) Is there a dynamic advantage for facial expressions. J Vis 11:1–15. doi:10.1167/11.3.17 Frank MG, Ekman P, Friesen WV (1993) Behavioral markers and recognizability of the smile of enjoyment. J Pers Soc Psychol 64:83–93. doi:10.1037/0022-3514.64.1.83 Gallese V (2001) The ‘shared manifold’ hypothesis. From mirror neurons to empathy. J Conscious Stud 8:33–50 Geller T (2008) Overcoming the uncanny valley. IEEE Comput Graph Appl 28:11–17. doi:10.1109/ MCG.2008.79 Perceptual Study on Facial Expressions 13 Gibson JJ (1966) The senses considered as perceptual systems. Houghton Mifflin, Boston doi:10.1080/00043079.1969.10790296 Hehman E, Flake JK, Freeman JB (2015) Static and dynamic facial cues differentially affect the consistency of social evaluations. Pers Soc Psychol Bull 41:1123–1134. doi:10.1177/ 0146167215591495 Hess U, Kleck RE (1990) Differentiating emotion elicited and deliberate emotional facial expressions. Eur J Soc Psychol 20:369–385. doi:10.1002/ejsp.2420200502 Humphreys GW, Donnely N, Riddoch MJ (1993) Expression is computed separately from facial identity, and is computed separately for moving and static faces: neuropsychological evidence. Neuropsychologia 31:173–181. doi:10.1016/0028-3932(93)90045-2 Iacoboni M (2009) Imitation, empathy, and mirror neurons. Annu Rev Psychol 60:653–670. doi:10.1146/annurev.psych.60.110707.163604 Iacoboni M, Dapretto M (2006) The mirror neuron system and the consequences of its dysfunction. Nat Rev Neurosci 7:942–951. doi:10.1038/nrn2024 Johansson G (1973) Visual perception of biological motion and a model for its analysis. Percept Psychophys 14:201–211. doi:10.3758/BF03212378 Kamachi M, Bruce V, Mukaida S, Gyoba J, Yoshikawa S, Akamatsu S (2001) Dynamic properties influence the perception of facial expressions. Perception 30:875–887. doi:10.1068/p3131 Kappas A, Krumhuber EG, Küster D (2013) Facial behavior. In: Hall JA, Knapp ML (eds) Nonverbal communication (Handbooks of Communication Science, HOCS 2). Mouton de Gruyter, Berlin, pp 131–165 Kätsyri J, Sams M (2008) The effect of dynamics on identifying basic emotions from synthetic and natural faces. Int J Hum Comput Stud 66:233–242. doi:10.1016/j.ijhcs.2007.10.001 Kätsyri J, Förger K, Mäkäräinen M, Takala T (2015) A review of empirical evidence on different uncanny valley hypotheses: support for perceptual mismatch as one road to the valley of eeriness. Front Psychol 6:390. doi:10.3389/fpsyg.2015.00390 Kerlow IV (2004) The art of 3D computer animation and effects, 3rd edn. John Wiley and Sons, Hoboken Kessler H, Doyen-Waldecker C, Hofer C, Hoffmann H, Traue HC, Abler B (2011) Neural correlates of the perception of dynamic versus static facial expressions of emotion. GMS Psychosoc Med 8:1–8. doi:10.3205/psm000072 Kilts CD, Egan G, Gideon DA, Ely TD, Hoffmann JM (2003) Dissociable neural pathways are involved in the recognition of emotion in static and dynamic facial expressions. Neuroimage 18:156–168. doi:10.1006/nimg.2002.1323 Korb S, With S, Niedenthal PM, Kaiser S, Grandjean D (2014) The perception and mimicry of facial movements predict judgments of smile authenticity. PLoS One 9:e99194. doi:10.1371/ journal.pone.0099194 Krumhuber EG, Tamarit L, Roesch EB, Scherer KR (2012) FACSGen 2.0 animation software: generating three-dimensional FACS-valid facial expressions for emotion research. Emotion 12:351–363. doi:10.1037/a0026632 Krumhuber EG, Kappas (2005) Moving smiles: the role of dynamic components for the perception of the genuineness of smiles. J Nonverbal Behav 29:3–24. doi:10.1007/s10919-004-0887-x Krumhuber EG, Manstead ASR (2009) Can Duchenne smiles be feigned? New evidence on felt and false smiles. Emotion 9:807–820. doi:10.1037/a0017844 Krumhuber EG, Manstead ASR, Cosker D, Marshall D, Rosin PL (2009) Effects of dynamic attributes of smiles in human and synthetic faces: a simulated job interview setting. J Nonverbal Behav 33:1–15. doi:10.1007/s10919-008-0056-8 Krumhuber EG, Manstead ASR, Cosker D, Marshall D, Rosin PL, Kappas A (2007a) Facial dynamics as indicators of trustworthiness and cooperative behavior. Emotion 7:730–735. doi:10.1037/1528-3542.7.4.730 Krumhuber EG, Manstead ASR, Kappas A (2007b) Temporal aspects of facial displays in person and expression perception: the effects of smile dynamics, head-tilt and gender. J Nonverbal Behav 31:39–56. doi:10.1007/s10919-006-0019-x 14 E.G. Krumhuber and L. Skora Krumhuber EG, Skora P, Küster D, Fou L (in press) A review of dynamic datasets for facial expression research. Emotion Rev Küster D, Krumhuber EG, Kappas A (2014) Nonverbal behavior online: a focus on interactions with and via artificial agents and avatars. In: Kostic A, Chadee D (eds) Social psychology of nonverbal communications. Palgrave MacMillan, New York, pp 272–302 LaBar KS, Crupain MJ, Vovodic JT, McCarthy G (2003) Dynamic perception of facial affect and identity in the human brain. Cereb Cortex 13:1023–1033. doi:10.1093/cercor/13.10.1023 Lee TW, Josephs O, Dolan RJ, Critchley HD (2006) Imitating expressions: emotion specific neural substrates in facial mimicry. Soc Cogn Affect Neurosci 1:122–135. doi:10.1093/scan/nsl012 Lundqvist L, Dimberg U (1995) Facial expressions are contagious. J Psychophysiol 9:203–211 Maringer M, Krumhuber EG, Fischer AH, Niedenthal P (2011) Beyond smile dynamics: mimicry and beliefs in judgments of smiles. Emotion 11:181–187. doi:10.1037/a0022596 McDonnnell R, Breidt M, Buelthoff HH (2012) Render me real? Investigating the effect of render style on the perception of animated virtual humans. ACM Trans Graph 31:1–11. doi:10.1145/ 2185520.2185587 Mori M (1970) Bukimi No Tani. The Uncanny Valley (MacDorman KF and Minato T, Trans). Energy 7:33–35 Niedenthal P, Brauer M, Halberstadt JB, Innes-Ker AH (2001) When did her smile drop? Facial mimicry and the influences of emotional state on the detection of change in emotional expression. Cognit Emot 15:853–864. doi:10.1080/02699930143000194 Oberman LM, Winkielman P, Ramachandran VS (2007) Face to face: blocking facial mimicry can selectively impair recognition of emotional expressions. Soc Neurosci 2:167–178. doi:10.1080/ 17470910701391943 Piwek L, McKay LS, Pollick FE (2014) Empirical evaluation of the uncanny valley hypothesis fails to confirm the predicted effect of motion. Cognition 130:271–277. doi:10.1016/j. cognition.2013.11.001 Ponari M, Conson M, D’Amico NP, Grossi D, Trojano L (2012) Mapping correspondence between facial mimicry and emotion recognition in healthy subjects. Emotion 12:1398–1403. doi:10.1037/a0028588 Recio G, Schacht A, Sommer W (2013) Classification of dynamic facial expressions of emotion presented briefly. Cognit Emot 27:1486–1494. doi:10.1080/02699931.2013.794128 Recio G, Sommer W, Schacht A (2011) Electrophysiological correlates of perceiving and evaluating static and dynamic facial emotional expressions. Brain Res 1376:66–75. doi:10.1016/j. brainres.2010.12.041 Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192. doi:10.1146/annurev.neuro.27.070203.144230 Rizzolatti G, Fadiga L, Gallese V, Fogassi L (1996) Premotor cortex and the recognition of motor actions. Cogn Brain Res 3:131–141. doi:10.1016/0926-6410(95)00038-0 Rychlowska M, Canadas E, Wood A, Krumhuber EG, Niedenthal P (2014) Blocking mimicry makes true and false smiles look the same. PLoS One 9:e90876. doi:10.1371/journal. pone.0090876 Rymarczyk K, Biele C, Grabowska A, Majczynski H (2011) EMG activity in response to static and dynamic facial expressions. Int J Psychophysiol 79:330–333. doi:10.1016/j. ijpsycho.2010.11.001 Sato W, Yoshikawa S (2004) Brief report. The dynamic aspects of emotional facial expressions. Cognit Emot 18:701–710. doi:10.1080/02699930341000176 Sato W, Yoshikawa S (2007a) Spontaneous facial mimicry in response to dynamic facial expressions. Cognition 104:1–18. doi:10.1109/DEVLRN.2005.1490936v Sato W, Yoshikawa S (2007b) Enhanced experience of emotional arousal in response to dynamic facial expressions. J Nonverbal Behav 31:119–135. doi:10.1007/s10919-007-0025-7 Sato W, Fujimura T, Suzuki N (2008) Enhanced facial EMG activity in response to dynamic facial expressions. Int J Psychophysiol 70:70–74. doi:10.1016/j.ijpsycho.2008.06.001 Perceptual Study on Facial Expressions 15 Sato W, Fujimura T, Kochiyama T, Suzuki N (2013) Relationships among facial mimicry, emotional experience, and emotion recognition. PLoS One 8:e57889. doi:10.1371/journal.pone.0057889 Sato W, Kochiyama T, Yoshikawa S, Naito E, Matsumura M (2004) Enhanced neural activity in response to dynamic facial expressions of emotion: an fMRI study. Cogn Brain Res 20:81–91. doi:10.1016/S0926-6410(04)00039-4 Saygin AP, Chaminade T, Ishiguro H, Driver J, Frith C (2012) The thing that should not be: predictive coding and the uncanny valley in perceiving human and humanoid robot actions. Soc Cogn Affect Neurosci 7:413–422. doi:10.1093/scan/nsr025 Schmidt KL, Ambadar Z, Cohn J, Reed LI (2006) Movement differences between deliberate and spontaneous facial expressions: zygomaticus major action in smiling. J Nonverbal Behav 301:37–52. doi:10.1007/s10919-005-0003-x Schmidt KL, Bhattacharya S, Delinger R (2009) Comparison of deliberate and spontaneous facial movement in smiles and eyebrow raises. J Nonverbal Behav 33:35–45. doi:10.1007/s10919008-0058-6 Schulte-Rüther M, Markowitsch HJ, Fink GR, Piefke M (2007) Mirror neuron and theory of mind mechanisms involved in face-to-face interactions: a functional magnetic resonance imaging approach to empathy. J Cogn Neurosci 19:1354–1372. doi:10.1162/jocn.2007.19.8.1354 Singer T, Seymour B, O’Doherty JP, Frith CD (2004) Empathy for pain involves the affective but not the sensory components of pain. Science 303:1157–1162. doi:10.1126/science.1093535 Stel M, Van Baaren RB, Vonk R (2008) Effects of mimicking: acting prosocially by being emotionally moved. Eur J Soc Psychol 38:965–976. doi:10.1002/ejsp.472 Thompson JC, Trafton JG, McKnight P (2011) The perception of humanness from the movements of synthetic agents. Perception 40:695–704. doi:10.1068/p6900 Tinwell A, Grimshaw M, Nabi DA, Williams A (2011) Facial expression of emotion and perception of the Uncanny Valley in virtual characters. Comput Hum Behav 27:741–749. doi:10.1016/j. chb.2010.10.018 Trautmann SA, Fehr T, Hermann M (2009) Emotions in motion: dynamic compared to static facial expressions of disgust and happiness reveal more widespread emotion-specific activations. Brain Res 1284:100–115. doi:10.1016/j.brainres.2009.05.075 Wallraven C, Breidt M, Cunningham DW, Bülthoff H (2008) Evaluating the perceptual realism of animated facial expressions. ACM Trans Appl Percept 4:1–20. doi:10.1145/1278760.1278764 Wehrle T, Kaiser S, Schmidt S, Scherer K (2000) Studying the dynamics of emotional expression using synthetized facial muscle movement. J Pers Soc Psychol 78:105–119. doi:10.1037/00223514.78.1.105 Weiss F, Blum GS, Gleberman L (1987) Anatomically based measurement of facial expressions in simulated versus hypnotically induced affect. Motiv Emot 11:67–81. doi:10.1007/BF00992214 Weyers P, Mühlberger A, Hefele C, Pauli P (2006) Electromyographic responses to static and dynamic avatar emotional facial expressions. Psychophysiology 43:45–453. doi:10.1111/ j.1469-8986.2006.00451.x Yang M, Wang K, Zhang L (2013) Realistic real-time facial expression animation via 3D morphing target. J Softw 8:418–425. doi:10.4304/jsw.8.2.418-425 Yoshikawa S, Sato W (2008) Dynamic facial expressions of emotion induce representational momentum. Cogn Affect Behav Neurosci 8:25–31. doi:10.3758/CABN.8.1.25