Abstract
This paper provides a framework for recording, analyzing and modeling of 3 dimensional emotional movements for embodied game applications. To foster embodied interaction, we need interfaces that can develop a complex, meaningful understanding of intention—both kinesthetic and emotional—as it emerges through natural human movement. The movements are emulated on robots or other devices with sensory-motor features as a part of games that aim improving the social interaction skills of children. The design of an example game platform that is used for training of children with autism is described since the type of the emotional behaviors depends on the embodiment of the robot and the context of the game. The results show that quantitative movement parameters can be matched to emotional state of the embodied agent (human or robot) using the Laban movement analysis. Emotional movements that were emulated on robots using this principle were tested with children in the age group 7–9. The tests show reliable recognition on most of the behaviors.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Teaching of socially relevant behavior to children through games and robotic toys is an area of emerging interest [1–6]. Autonomous robots are of special interest in this area since they provide an opportunity to use the robot as a replacement of a caregiver, as a partner, or a mediator of play [1–9]. A number of purely autonomous robots, both with and without anthropomorphic or zoomorphic shape have been used in games with children [5–13]. Robots in these studies, however, perform simple behavior (as for instance lifting of a hand) that aims to provoke reciprocal human reaction [4, 5, 12] or predict the position or the direction of the movement which is referred to as predicting motor intentions [4, 5, 10, 44], instead of being a tool for more general social learning. Robotic studies on social learning that have the potential to be extended to human–robot interaction scenarios are reported in [10, 14–16, 44].
In this work we aim to go beyond sensory–motor interaction in robotic models of embodied cognition by also taking into account the interplay between constitutive and interactive aspects of autonomy. This implies that sensory–motor interaction has to be enriched with intentional, emotional and reward aspects of interaction. Specifically, we focus on the emotions that are conveyed by movement behavior.
Keltner and Kring [17] point out a highly dependent link between emotion and social meaning. They argue that emotions serve a set of functions that are critical for coordinating social interactions. These functions are:
-
to provide information to the conspecifics about the surrounding environment (e.g., fear may indicate the presence of a predator);
-
to elicit both complementary and similar emotions in others, depending on the context;
-
to be an incentive that promotes social relationships.
This motivates us to pursue a more general social learning framework with robots that include emotional facial expressions, bodily posture and actions of others, and triggers appropriate emotional responses.
Although it is not shown that emotional resonance (response to perceived emotion) and emotion understanding and recognition are related components of the emotional system [7], teaching to just recognize emotions will be beneficial for instance to persons (especially children) with autism which have atypical social and emotional development [18]. Benefits in using robots for behavioral training of autistic children have been investigated in [6, 8, 19, 42].
Teaching emotional and social movement patterns can take many forms. Since we target children, we choose to design game scenarios that will include recognition of emotional movements. Most obvious value of games is enjoyment and sharing of social experiences with others [20]. In addition, play, is widely used as a preferred educational activity for young children [21, 22]. Salen and Zimmerman [23] point out the difference between games and play. They pose that a game is a subset of play, a formalized and focused interaction that occurs when players follow the rules of a game in order to play it. Since the ultimate goal of this research is enhancing the social skills of children with autism who feel comfortable with structure and clearly defined rules, using games to stimulate social interaction between them is a good way to approach this user group.
Physical play [11] is based on motor activities of the players. Using robots for social training raises the interest for nonverbal social communication, expressed via movement and postures. Present day robots express movement as a response to their sensing and decision making, while natural language understanding and synthesis is far beyond the reach of the contemporary science. That is the reason why speaking robots can be used for social tasks only in a tele-operated mode, see for instance [24].
The nonverbal ways for conveying socially relevant information include facial expressions and body movements [25, 26]. Neurological studies suggest that understanding nonverbal communication of facial expressions and body movements involves the mirror neuron system [26, 27]. However, Montgomery and Haxby [27] showed that these two forms of nonverbal social communication have distinct representations within that system, so it is plausible to study them separately. We investigate a specific aspect of social interaction behaviors, namely the perception and expression of emotions and emotional body language, as conveyed by body movements, which is based on our understanding of the common coding principle in the mirror neuron system [14]. However, the construction of a bottom up model of the mirror neuron system on neuronal or functional level is not plausible for this study. Instead, we use Laban movement analysis [28] to embody the common coding principle on behavioral level. Laban movement analysis (LMA) it is a systematic framework for describing all forms of human movement and emphasizes how internal feelings and intentions govern the patterning of movement throughout the whole body. It provides a complex understanding of intention. Similar to common coding/mirroring paradigm LMA is useful to describe the interaction in the physical world, which is caused by physical robots that move or perceive movements of humans and other agents (robots). Therefore using Laban movement analysis for social interaction through physical objects gives many possibilities : to design a robot (intelligent object) that understands the emotional state of a human player and responds in an adequate manner; to design robot behaviors that imitate, enhance or counteract an emotional state of a person; to design socially believable robotic (embodied) characters that provoke social interaction; to create constantly adapting interaction based on movement/emotional understanding of a robot.
In this particular study we develop a framework for expressing and interpreting emotional movements. Based on this framework we recorded and analyzed human movement patterns and designed and implemented emotional movements on robots. User testing of these movements was performed with typical children to validate the plausibility of this design. Also we developed games that include the emotional robot behaviors for training autistic children.
This paper is organized as follows. In Sect. 2 will be discussed how the stages of social development can be advanced by games with robotic toys. Section 3 features the framework for recording and interpreting emotional movements. The common elements of the framework that make it able to analyze and model human movements are outlined. Several applications for modeling and analyzing emotional movements on and by robots or intelligent objects using this framework are described in Sect. 4. Also a platform and a game scenario that involves recognition and control over emotional behaviors is shown in this Section. Section 5 provides a discussion.
2 Stages of social development matched by games with robotic toys
Play behaviour is an expression of physical, social and emotional development of children. There are several stages of play corresponding to the level of social development.
Early researchers categorized the play they observed in young children into age related stages [29]. Later theorists described the stages of play development in specific domains, such as social, motor, and socio-cultural development [30–32].
These stages correspond to the stages of motor, social and the emotional development of the children. Exploratory and sensorimotor types of play are primary during the early infancy period. Symbolic play starts in the first years of child development and it is characterized with reciprocal interaction between the children and the emergence of social games. Wachs [33] distinguishes between object play (exploratory, manipulative, or functional play) as well as social play, related to peer and adult–child interactions. We have argued [34] that interaction with objects is a developmental stage of social behavior, and that shortcomings in the motor level of interaction can result in impaired social behavior [34].This argument is based on our understanding on the role mirror neuron system in social learning and interaction, that has emerged as an additional functionality of the neural structures that have primarily developed for gross and fine motor skills, such as grasping and precision grip. Therefore, difficulties in planning and executing simple discrete movements, for instance, can lead to problems in learning to coordinate diverse muscle groups into a unitary movement pattern. Moreover, when a person is unable to respond to another’s action in a timely fashion they will miss the positive reinforcement associated with interpersonal interaction.
Lack of motor imitation, as for instance observed by children with autism, results in difficulties to understand each other’s behavior. The complementary reciprocal play in early toddlerhood, such as run and chaise [35] may be prerequisite for cooperative social pretend play later on.
Considering each stage of social development, we design games that promote a wider spectrum of social and emotional experiences. We focus on physical play, i.e. play with physical objects that aims to enhance the interaction between the children [1–3]. To trigger more advanced forms of social play, we aim at developing physical objects that have own means to stimulate social interaction. These include: sensors so that the objects can record the changes in the surrounding world, some learning or adaptation mechanism that will facilitate decision making and insure a level of autonomy, and actuators so the objects can express behaviour. A physical object with such features is in a broad sense a robot independently of its shape or means of behaviour. As mentioned in the introduction of this paper, we aim at higher level of autonomy than the largely available robots have at present. The difference by this new level of autonomy is in the robot's ability to interpret human movement behaviour and to behave in such a way that humans understand. Specifically, we aim at designing robotic toys that can themselves express emotional behaviours, as well as understand emotional expressions in other agents. We consider 3 different levels of awareness for robots that can be used in game-planning scenarios:
-
Robots that understand the emotional state of a human player(s) and respond in an adequate manner;
-
Robots that imitate, enhance or counteract an emotional state of a human;
-
Socially believable robotic (embodied) characters that provoke social interaction;
-
Emergence of constantly adapting interaction based on movement/emotional understanding of a robot.
3 Framework for emulating and interpreting emotional movements
This section proposes a framework that will make it possible to express and interpret emotional movement patterns. The framework is based on the understanding that there is a close relation between expressed and perceived emotion which will make it possible for the robot to (1) model or synthesize emotion (e.g. on robot or another object with sensory-motor features) in a way that it can be understood by a human observer and (2) to recognize or interpret human emotional movement.
Evidence from independent studies in neuroscience, social psychology, and behavioral studies strongly suggest that our ability to perceive the actions of others results from the massive experience we have accumulated in planning and executing self-produced actions. Sensory representations used during action perception and motor representations for action planning are coded in the same brain areas. Different connectivity between these areas and additional signals determine whether the movement will be perceived as own or observed. This common coding principle (as known in psychological studies), or mirror neuron principle (as known in neuroscience) is a basis for designing the social interaction systems [14, 15].
A ‘social perception’ system that is directly connected to, and partially overlapping with the mirror neuron system, processes emotionally-rich facial expressions, bodily posture and actions of others, and triggers an appropriate emotional responses. It detects other persons’ assumed social motor intentions, triggers automatic social/emotional response plans, and so facilitates automatic intentional and emotional responses.
If these ideas are used in an interactive robot the same modules will be used for recording and interpreting emotional movements. The common coding principle makes it possible to create an interchangeable interaction through whole body movement between two robots, as shown in [14]. We want to extend this motor interaction to emotional body language that is expressed by movement. Moreover, we would like to make it possible to have mutual interaction and understanding between a robot and a human. This implies that a high level representation of movement primitives is characterized by certain emotional patterns. Such a high level representation which characterizes the human movement in terms of expressed and perceived intentions and emotions is searched in the framework of Laban movement analysis.
In order to identify the contribution of body movements to the recognition of emotion, it is important to have a clear and suitable description of these movements. The Laban Movement Analysis [28] is a well-established, effective method for observing, describing, annotating, and interpreting human movement, however it does not provide a straightforward way to assign quantitative measures to the movement qualities. It provides descriptors for the content of human body movements in terms of the following categories: Body, Space, Effort and Relationship. The Effort category relates to the dynamic and expressive characteristics of the movement. It is comprised of four movement qualities: weight, space (not to be confused with the Space category), time, and flow. Each quality represents a continuum between opposite polarities: weight varies between strength and lightness, space, between direct and indirect, time between sudden and sustained and flow between bound and free. Effort qualities usually appear in combinations called “states” or “drives”. When two motion qualities are combined, it is called inner attitudes or incomplete effort. When three motion qualities are combined, they form externalized drives. Of special interest is the Passion drive, which combines time, weight, and flow. Using these motion determinants that body takes in space gives a way to noticeably differentiate expressive and emotional actions.
For instance, the difference between punching someone in anger and reaching for a glass is slight in terms of body organization—both rely on extension of the arm. However, the attention to the strength of the movement, the control of the movement and the timing of the movement are very different. This example shows how the three qualities, namely weight, flow, and time, respectively, help characterize the emotional load of movements.
According to [36], the Laban Effort parameters can be translated into low-level movement parameters such as curvature, velocity, and acceleration.
After consulting a certified Laban movement analyst, and reviewed the related literature [36–39] we concluded that this relation is bidirectional, i.e. based on the physical characteristics of the movement we can determine the Laban characteristics of the movement, and vice versa. If the factors that are the components of the Passion drive (i.e. time, weight, and flow) are combined, a judgment about the emotional load of the movement is possible to be made. For this purpose, based on the expert’s opinion, our knowledge of modeling movement, and making a number of simple experiments we concluded that the acceleration, the velocity, and the profile (curvature, amplitude, and variability) of the movement can be used in the following way:
-
Time can be characterized by looking at the movement’s acceleration.
-
Weight is characterized by a combination of acceleration and velocity.
-
Flow needs the values of all three components in order to be assessed.
This relation is visualized in the Figs. 1, 2 and 3:
Connecting the Laban factors to emotional states is a two stage process. First, the right correspondence between the Laban factors and emotions has to be established. Camurri et al. [37] and Fagerberg et al. [38], have independently used LMA to classify dance gestures in terms of basic emotions; anger, fear, grief and joy. Their analysis overlaps for 3 of these emotions, namely anger, fear, and joy (happiness). We use the coding that corresponds to the relation between Laban movement qualities and emotions as introduced by Camurri et al.
Second, since we work with partially subjective parameters a classification method that can deal with sufficient degree of uncertainty has to be deployed. For this purpose we used variations of neural classifiers for both analysis and synthesis of emotional movement.
In summary, the framework for analyzing and simulating emotional movements takes the following steps.
-
1.
Description of movement that allows for an easy bidirectional transformation between the dynamic characteristics of a signal (velocity, acceleration and variability) to Laban Effort factors.
-
2.
Learning the dependence between Laban Effort factors and the basic emotions.
-
3.
Prediction of movement as an element of the interaction.
4 Examples: using the framework for expressing and interpreting emotional movements by robots
The framework was tested in several prototypes of behaving objects for social play that correspond to one of the levels of awareness of the robots as listed in Sect. 2. Once more we would like to state that our definition of a robot is a device or everyday object with sensory, motor, and processing capabilities. Movement is the most recognizable expression of behavior, but other controlled behavioral expression, such as speech, light changes are also possible and considered as valid motor expression.
4.1 Simple behavioral response based on interpreting the emotional state of human players
Analyzing the emotional state of the human player is a very complex task. The complexity also depends on the sensors, which can provide simple data in the range of binary values (i.e. on and off) to camera images that require complex signal processing. We restrict the complexity of the analysis by assuming that either the robot or the player will be static at every moment, so when the robot observes moving body parts it does not need to recalculate them with respect to its own movement.
In order to abstract information about the movement qualities by a static object with sensory-motor features, we constructed a behaving walk-in closet equipped with 3 types of sensors and a learning algorithm that decides the closet behavior. We placed sensors in the surrounding floor area, in the shelves and in the vertical space in the walk-in closet. Human players were asked to act out several scenarios. From the analysis of the sensor data it became apparent that the walking movement and the hand movement are most informative about the emotional state of the human subjects. These two movements relate to two kind of movement analysis: micro analysis of the movements in the kinesphere of the user (that relates to a body part in our case arms) and macro analysis to describe the larger (whole body) movements in the space. To capture macro movement we placed 12 sensor mats (595 mm × 170 mm) in the floor which are able to capture the walking movement in one dimension. In the shelves we placed 8 passive infrared sensors to measure micro movement activity. The sensors were connected through three Arduino microcontrollers to a PC. A Neural gas learning algorithm was implemented on Max/MSP and connected to the actuators that controlled the intensity, the speed, and the order of lighting of the lamps.
The analysis of the data showed that the movement qualities space and time were sufficient for the learning algorithm embedded in the closet to distinguish 4 different states of the players. For every combination one scenario was created which would be tested in an experiment. All the scenarios had as main context a selection of clothes for different occasions. The goal of the scenario was to evoke a certain state on the user, such as the feeling of hurry, relaxation, or enjoyment.
The closet is currently not prepared to distinguish between the different players but its behavior can be changed by increasing the activity in the object handling or in whole body movements of the game participants. So far the collective play behavior is not tested systematically. Our focus has been on the mood analysis and on the simulation of submissive and dominant behaviors of the closet.
4.2 Social mediation of play based on recognition and expression of emotions by movement
For the purpose of mediation of play between children, a robot that can engage in reciprocal social interaction through movement was used. In particular, we design behaviors to support collective games for the humanoid robot NAO [40]. The underlying idea is that the robot can recognize and imitate human movements and give feedback on the behavior of the human. The game scenarios that can be constructed on the basis of this interactive behaviors aim to help children with autism learn to recognize emotional movements from a robot partner and to produce similar movements.
For this type of interaction behavior a more precise method for analyzing the human movement is needed. The NAO robot has 25 degrees of freedom, 5 in each leg and arm, and 1 in each hand. Further it has 2 degrees of freedom in its head and one in the pelvis. The platform contains 2 color cameras with a maximum resolution of 640 × 480 pixels at a speed of 30 frames per second. The platform contains an embedded AMD Geode 500 MHz processor and is shipped with an embedded Linux distribution. A software library called NaoQi is used to control the robot. This API provides an easy to use C++ interface to the robot’s sensors and actuators. Due to this library it is relatively easy to control the robots actuators and make use of advanced routines that let the robot move and talk using text to speech conversion.
Challenging, however, remains the recognition of the emotional behaviors, which we resolved within the framework outlined in Sect. 3. A performer and certified Laban movement analyst was asked to enact waving behaviors with different emotional coloring, namely angry, happy, sad and polite. The waving was chosen for several reasons. First it is a behavior that is used exclusively in a social context. Second, it can relatively easily be tracked by a single camera, and third, several emotional states can naturally be expressed through waving. In fact our goal is not to confine to the type of the movement, but the extraction of a dynamic primitives that are typical for a certain emotion.
Figure 5 depicts pairs of 4 emotional waving patterns recorded in an experimental scenario as shown in Fig. 4, using color images of 640 × 480 pixels at a speed of 29 frames per second. The image processing consists of a combination of skin color and motion detection with the aim of tracking a single body part per person that can be associated with the emotional waving movement. The black rectangular regions capture the center and the boundaries of the skin color areas, green areas capture moving objects and the blue areas give a combination of a moving skin colored area. Due to parallel processing on GPU’s, multiple moving hands can be followed, which makes this method suitable for collective games.
Fifteen recordings of 20 seconds have been made with waving patterns that enact happiness, anger, sadness, or politeness (Fig. 5). At each row two examples of the same enacted emotion are plotted. Note that the trajectories of the movements have been different by enacting the same emotional state. For instance for angry emotional state the performer acted movement resembling pushing away a fly and a movement resembling an angry mother calling for her child to come. The first movement starts close to the body and moves laterally away. The second starts frontally, away from the body and finishes frontally, close to the body. The plots, however, show the acceleration profile of the movement which is similar for each emotion, independently of the trajectory of the enacted movement. From Fig. 6 the following can be observed:
-
1.
Happy waving provides a regular waving pattern with a relatively high frequency.
-
2.
Angry waving demonstrates bursts with tremendous acceleration
-
3.
Sad waving demonstrates a profile of low acceleration; its frequency is relatively low and appears to have a lower frequency compared to the other three emotions.
-
4.
Polite waving is a regular pattern with a relatively high frequency that is obtained by using minimal energy.
To analyze whether the emotional patterns can be properly classified by a robot, an average acceleration-frequency matrix was plotted. Figure 6 shows that four distinctive clusters are formed.
4.3 Expressing emotional behaviors in social game with robots
The Laban movement guidelines as suggested by Camurri et al. [37] were used to design emotional behaviors on a mobile robot. Note that for this experiment we did not use directly the human movement primitives extracted from the signals that were analyzed in Sect. 4.2. This choice was made since the recordings that are illustrated in Sect. 4.2 are not made with a representative group of participants. To test solely the perceived emotion from the movement, the experiments with the e-puck [41] robot that does neither have anthropomorphic nor zoomorphic shape were performed. A control user group of 42 typically developing children were tested to observe the emotional behaviors enacted by the robots. The outcome of the tests showed a good recognition of several basic emotions (Table 1).
The chart shows the recognition rate in percents of the designed emotional behaviors for each emotion. It is important to mention that the children were not provided with a list of possible emotions. They were asked how they think that the robot is feeling at the moment.
The low recognition rate on anger and happiness emotions was the reason to construct the confusion matrix out from the user test outcomes. From this matrix become apparent that the two emotional movement behaviors, anger and happiness were mistaken for each other. These two emotions are characterized with high intensity but have different valence (i.e. happiness is a strong positive emotion, while anger is a strong negative emotion). Moreover, the Laban descriptions of these emotions, as suggested by [37] have many common elements, as evident from Table 2.
In contrast, the acceleration plots as made in our experiments with robot vision and described in Sect. 4.2 show very different acceleration profiles. The recorded waving pattern that represents Happiness provides a regular waving pattern with a relatively high frequency (Fig. 5a, b), while the Anger waving demonstrates bursts with tremendous acceleration, and pauses in between (Fig. 5c, d).
To analyze the robot behaviors we attached the Wiimote to the robot when it performed the emotional movements. The plots did not show the typical acceleration profile as by the emotional movements enacted by a human demonstrator. The recording of the acceleration profiles from human motion patterns are a substantial step for redesigning the robot emotional behavior.
The so designed emotional behaviors were included in a game for promoting associative play [1] by children with autism. To design the game the following shortcomings of the children were targeted: inability to share and socially interact, inability to understand expression of emotions and link them to context, preference to learn by examples and logic rather than by trial and error.
To account for these problems, a combined approach of a game that will require negotiations and working towards a common goal, together with recognition of emotional states was made. The game uses a storyline that describes various situations involving different emotions. When the children recognize the emotion described in the story they have to command a robot to either perform or contradict this emotion. The robot is commanded by the collective physical behavior of the children. At least two children have to step on one site of a large disk to make the disk tilt (Fig. 7). The disk can be tilted in several directions denoted with colored LED-lighted arrows. Every tilt direction will provoke movement behavior of the robot that expresses particular emotion. The tilted disk will trigger a movement of a robot that corresponds to the corresponding emotion. The LED arrows have the color that resembles the emotion in a similar way as the traffic light metaphor that was used in schools for autistic children to illustrate children’s emotions. The bottom line of the game is that the children have to identify the correct emotion based on the story and movement. The teacher can use different storylines referring to an emotion, that will consequently be acted by the movement of the robot. With this multivalent approach we aimed at an integrated understanding of the acted emotion.
For the purpose of the game a huge round disc that could accommodate several children was made (Fig. 7). The disc can control the movement of a robot by being tilted in a certain direction. The robot in the game situations is drawn as a small object near the platform. It resembles the shape and the size of the e-puck robot [41] that was used for this experiment. The e-puck is a two-wheel mobile robot that was originally developed at Swiss Federal Institute of Technology (EPFL). The robot is equipped with a dsPIC processor and can be controlled by blue tooth through the computer, or a very simple program can be uploaded on the dsPIC processor. We used computer-mediated control.
The children, standing on top of the platform, had to negotiate their positions since a single child could not tilt the disc. Once multiple children moved to the right position and the disc tilted at that direction, the corresponding behavior of the robot was triggered. To change the robot emotional behavior, the children had to agree on their next position and move together. When conflicting views occur, it was an opportunity for the children to learn to negotiate and get aware that they need others’ help.
5 Discussion
We proposed a framework for expressing and interpreting emotional movements that is based on actual recording, analyzing and emulating of emotional movements on robots. It uses the same way for representing the emotional behavior in the agent which expresses and the agent which observes (or interacts). In this way the proposed framework enables with the same modeling, sensing, and analysis tools to design games and scenarios with robots and other intelligent objects that have different degrees of awareness. By using this framework, different levels of skills of the robots, namely their abilities to analyze the emotional state of the human player, to express own emotional social behavior, or take part in reciprocal social interaction can be achieved with no need for re-design.
Laban movement analysis was used to provide the possibility for a unified way of expressing and interpreting emotional movements. A distinctive feature of scenarios that use Laban movement analysis is that they allow for realistic interpretation of the human (emotional) state. Differently from the existing studies that employ the Laban movement analysis we choose for analysis of the qualities of the actual movements with elaborate image processing techniques. The analysis of the human movements with a camera, showed different acceleration profiles of the movement as the one that were suggested by other studies on emotional movements that use the Laban framework, for instance by [37]. This fact suggests that using signal analysis can improve the designed emotional and social behaviors by capturing features of the movement that are omitted by observations. Indeed, when the robot behaviors for the described game were designed purely on the recommendations of [37], two emotions were largely confused by the participants in the user test. After the acceleration patterns of the so created emotional movements were recorded and compared with the emotional movement by the human observer, there was an obvious difference in the acceleration profiles of the misinterpreted signals. Moreover, our approach is straightforward for the design of interactive behaviors, since it gives a clear description of the behavior.
Last, but not least, the parallel processing framework that we previously suggested in [43] and used in the experiments shown in Sect. 4.2, allows for parallel tracking and interacting with multiple human or robot agents, which makes our framework especially suitable for games with multiple human participants, and therefore for design for social and physical play. This is a subject for our future research.
References
Barakova EI (2008) Emotion recognition in robots in a social game for autistic children. In: Sturm J, Bekker MM (eds) Proceedings of the 1st workshop on design for social interaction through physical play; Eindhoven, The Netherlands, pp 21–26
Barakova EI, Feijs LMG, Gillesen JCC (2009) Social training of autistic children with interactive intelligent agents. J Integr Neurosci 8(1):23–34
Barakova EI, van Wanrooij G, van Limpt R, Menting M (2007) Using an emergent system concept in designing interactive games for autistic children. 6th international conference on interaction design and children (IDC07), pp 73–77, Aalborg Denmark, June 2007. ACM 978-1-59593-747-6
Billard BR, Dautenhahn K, Nadel J (2007) Building Robota, a mini- humanoid robot for the rehabilitation of children with Autism. RESNA Assistive Technol J 19(1):37–49
Nadel J (2004) Early imitation and a sense of agency. Proceedings of the 4th international workshop on epigenetic robots
Robins B, Dautenhahn K, Dickerson P, Stribling P (2004) Robot mediated joint attention in children with autism. Interact Stud 5:161–198
Nadel J, Simon M, Canet P, Soussignan R, Blancard P, Canamero L, Gaussier P (2006) Human responses to an expressive robot. Proceedings of the sixth international workshop on epigenetic robotics. Lund University Cognitive Studies, vol 128, pp 79–86
Robins B, Dautenhahn K, Dickerson P (2005) Robots as embodied beings—interactionally sensitive body movements in interactions among autistic children and a robot’. ROMAN 2005. IEEE international workshop on robot and human interactive communication, pp 54–59
Werry I, Dautenhahn K, Ogden B, Harwin W (2001) Can social interaction skills be taught by a social agent: the role of a robotic mediator in autism therapy. In: Beynon M, Nehaniv CL, Dautenhahn K (eds) Cognitive technology: instruments of mind: 4th international conference, CI 2001, Warwick, UK, August 6–9
Barakova EI, Vanderelst D (2009) From spreading of behavior to dyadic interaction—a robot learns what to imitate. Int J Intell Syst (in press)
Bekker MM, Sturm J, Barakova EI (2008) Designing for social interaction through physical play. In: Sturm JA, Bekker MM, Barakova EI (eds) Proceedings of the 1st workshop on design for social interaction through physical play; Eindhoven, the Netherlands; October 22, 2008, Technische Universiteit Eindhoven, Eindhoven, pp 7–10
Breazeal C, Buchsbaum D, Gray J, Gatenby D, Blumberg B (2006) Learning from and about others: towards using imitation to bootstrap the social understanding of others by robots. Artif Life 11(1/2):3–62
Lourens T, Barakova EI (2009) My sparring partner is a humanoid robot. IWINAC (2) 2009:344–352, LNCS
Barakova EI, Lourens T (2009) Mirror neuron framework yields representations for robot interaction. Neurocomputing 72(4–6):895–900
Ezequiel A (2000) Di Paolo, behavioral coordination, structural congruence and entrainment in a simulation of acoustically coupled agents. Adapt Behav 8:27–48
Vanderelst D, Ahn R, Barakova E (2008) Simulated trust: towards robust social learning. In: Bullock S, Noble J, Watson R, Bedau MA (eds) Artificial life XI: proceedings of the eleventh international conference on the simulation and synthesis of living systems. MIT Press, Cambridge, pp 632–639
Keltner D, Kring AM (1998) Emotion, social function, and psychopathology. Rev Gen Psychol 2:320–342
Diagnostic criteria for autistic disorder, Indiana Resource Center for Autism, Indiana Institute on Disability and Community, Indiana University. Retrieved 27 Feb 2007
Pierno AC, Morena M, Dean L, Umberto C (2008) Robotic movement elicits visuomotor priming in children with autism, Neuropsychologia 46(2):448–454
Melamed L Games for growth. In: Celebration of play, vol 180, Croom Helm Ltd, London, p 162,
Alessandrini N (1949) A play—a child’s world. Am J Occup Ther 3:9–12
Parham LD, Fazio LS (2008) Play in occupational therapy for children, 2nd edn. Mosby Elsevier, St. Louis
Salen K, Zimmermann E (2004) Rules of play: game design fundamentals. MIT Press, Massachusetts
Markopoulos P, de Ruyter B, Privender S, van Breemen A (2005) Case study: bringing social intelligence into home dialogue systems. Interactions 12(4):37–44
Breazeal C, Velasquez J (1998) Toward teaching a robot ‘infant’ using emotive communication acts. In: Proceedings of the SAB’98, pp 25–40
Parr LA, Waller BM, Fugate J (2005) Emotional communication in primates: implications for neurobiology. Curr Opin Neurobiol 15:716–720
Montgomery KJ, Haxby JV, Mirror neuron system differentially activated by facial expressions and social hand gestures: a functional magnetic resonance imaging study. J Cogn Neurosci 20(10):1–12
Laban R, Lawrence FC (1947) Effort. Macdonald & Evans Ltd, London
Parten M (1932) Social participation among preschool children. J Abnorm Soc Psychol 27:242–269
Erikson EH (1972/1976) Play and actuality. In: Bruner JS, Jolly A, Silva K (eds) Play: its role in development and evolution. Harmondsworth, Penguin, pp 688–703
Piaget J (1976) The grasp of consciousness: action and concept in the young child. Harvard University Press, Cambridge
Vygotsky LS (1966) Play and its role in the mental development of the child. Voprosy Psikhologii 12:62–67
Wachs TD (1993) Multidimensional correlates of individual variability in play and exploration. New Dir Child Adolesc Dev 59:43–53
Barakova EI, Chonnaparamutt W (2009) Timing sensory integration for robot simulation of autistic behavior. IEEE Robotics Autom Magazine 16(3):51–58
Howes C (1987) Howes, social competency with peers: contributions from child care. Early Child Res Quart 2:155–167
Joerg Rett J, Dias J (2007) Computational Laban movement analysis using probability calculus
Camurri A, Lagerlöf I, Volpe G (2003) Recognizing emotions from dance movement: comparison of spector recognition and automated techniques. Int J Hum Comput Stud 59(1/2):213–225
Fagerberg P, Ståhl A, Höök K (2003) Designing gestures for affective input: an analysis of shape, effort and valence. In: Proceedings of mobile ubiquitous and multimedia, MUM
Amaya K, Bruderlin A, Calvert T (1996) Emotion from motion. Graphics Interface ‘96, pp 222–229
Aldebran NAO robot. http://www.aldebaran-robotics.com/eng/
E-puck education robot. http://www.e-puck.org
Boucenna S, Gaussier P, Andry P (2008) What should be taught first: the emotional expression or the face? In: 8th international conference on epigenetic robotics, EPIROB 2008, Brighton, UK
Lourens T (2004) TiViPE -Tino’s visual programming environment. In: The 28th annual international computer software and applications conference, IEEE COMPSAC 2004, pp 10–15
Vanderelst D, Barakova EI (2008) Autonomous parsing of behavior in a multi-agent setting. Intelligent systems 2008: IS ‘08; 4th international IEEE conference, 6–8 September, 2008, Varna, Bulgaria. vol 2, pp 10-7–10-13)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License ( https://creativecommons.org/licenses/by-nc/2.0 ), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Barakova, E.I., Lourens, T. Expressing and interpreting emotional movements in social games with robots. Pers Ubiquit Comput 14, 457–467 (2010). https://doi.org/10.1007/s00779-009-0263-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00779-009-0263-2