Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Generating Listening Behaviour

  • Chapter
  • First Online:
Emotion-Oriented Systems

Part of the book series: Cognitive Technologies ((COGTECH))

Abstract

In face-to-face conversations listeners provide feedback and comments at the same time as speakers are uttering their words and sentence. This ‘talk’ in the backchannel provides speakers with information about reception and acceptance – or lack thereof – of their speech. Listeners, through short verbalisations and non-verbal signals, show how they are engaged in the dialogue. The lack of incremental, real-time processing has hampered the creation of conversational agents that can respond to the human interlocutor in real time as the speech is being produced. The need for such feedback in conversational agents is, however, undeniable for reasons of naturalism or believability, to increase the efficiency of communication and to show engagement and building of rapport. In this chapter, the joint activity of speakers and listeners that constitutes a conversation is more closely examined and the work that is devoted to the construction of agents that are able to show that they are listening is reviewed. Two issues are dealt with in more detail. The first is the search for appropriate responses for an agent to display. The second is the study of how listening responses may increase rapport between agents and their human partners in conversation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Goffman (1976) provides a very insightful analysis of this process of replies and responses.

  2. 2.

    Similar experiments were reported on in Heylen et al., (2007b) and Heylen, (2007).

  3. 3.

    See also the chapter by Marinetti et al. ‘Emotions in Social Interactions: Unfolding Emotional Experience’ in Handbook Area at the beginning of this volume.

  4. 4.

    It should be noted that interactions with virtual characters can vary depending on if subjects believe the character is an avatar (controlled by a human) or an agent (controlled by software). In the results we report here, subject were led to believe they were interacting with an avatar to assess the impact of the quality of feedback while holding other factors constant.

  5. 5.

    It should be noted that listening agents that produced more head nods were also rated as more insincere, arguing for some caution when generating listening feedback.

References

  • Allwood J (1993) Feedback in second language acquisition. In: Perdue C. (edi) Adult language acquisition. Cross linguistic perspectives. Cambridge University Press, Cambridge, NY, pp 196–232

    Google Scholar 

  • Allwood J, Cerrato L (2003) A study of gestural feedback expressions. In: Paggio P, Jokinen K, Jonsson A. (eds) First nordic symposium on multimodal communication, Copenhagen, 23–24 September, pp 7–22

    Google Scholar 

  • Allwood J, Nivre J, Ahlsén E (1992) On the semantics and pragmatics of linguistic feedback. Semantics 9(1):1–26

    Google Scholar 

  • Ambady N, Bernieri FJ, Richeson JA (2000) Toward a histology of social behavior = Judgment accuracy from thin slices of the behavioral stream. Academic, San Diego, CA, pp 201–271

    Google Scholar 

  • Austin JA (1962) How to do things with words. Oxford University Press, London

    Google Scholar 

  • Bailenson JN, Yee N (2005) Digital chameleons = automatic assimilation of nonverbal gestures in immersive virtual environments. Psychol Sci 16:814–819

    Article  Google Scholar 

  • Bakhtin M (1999) The problem of speech genres. In: Jaworski A, Coupland N. (eds) The discourse reader. Routledge

    Google Scholar 

  • Bavelas JB, Chovil N (1997) Faces in dialogue. In: Russell J, Fernandez-Dols JM, (ed) The psychology of facial expressions. Cambridge University Press, Cambridge, pp 334–346

    Google Scholar 

  • Bavelas JB, Coates L, Johnson T (2000) Listeners as co-narrators. J Pers Soc Psychol 79(6):941–952

    Article  Google Scholar 

  • Bernieri FJ, Rosenthal R (1991) Interpersonal coordination, behavior matching and interactional synchrony. In: Feldman RS, Rimé B, (ed) Fundamentals of nonverbal behavior. Cambridge University Press, Cambridge

    Google Scholar 

  • Bevacqua E, Heylen D, Pelachaud C, Tellier M (2007) Facial feedback signals for ECAs. In: Proceedings of AISB’07: artificial and ambient intelligence, Newcastle University, Newcastle upon Tyne, UK, April 2007

    Google Scholar 

  • Bouhuys AL, van den Hoofdakker RH (1991) The interrelatedness of observed behavior of depressed patients and of a psychiatrist. An ethological study on mutual influence. J Affect Disorders 23:63–74

    Article  Google Scholar 

  • Brand M (1999) Voice puppetry. In: ACM SIGGRAPH. ACM Press/Addison-Wesley, New York, NY

    Google Scholar 

  • Breazeal C, Aryananda L (2002) Recognition of affective communicative intent in robot-directed speech. Autono Robots 12:83–104

    Article  MATH  Google Scholar 

  • Burgoon J, Hale J (1987) Validation and measurement of the fundamental themes of relational communication. Commun Monogr 54:19–41

    Article  Google Scholar 

  • Burns M (1984) Rapport and relationships. The basis of child care. J Child Care 2:47–57

    Google Scholar 

  • Capella JN (1990) On defining conversational coordination and rapport. Psychol Inquiry 1(4):303–305

    Article  Google Scholar 

  • Cassell J, Thórisson KR (1999) The power of a nod and a glance, envelope vs. emotional feedback in animated conversational agents. Int J Appl Artif Intell 13(4–5):519–538

    Article  Google Scholar 

  • Cassell J, Bickmore T, Billinghurst M, Campbell L, Chang K, Vilhjálmsson H, Yan H (1999) Embodiment in conversational interfaces. Rea. In: Conference on human factors in computing systems, Pittsburgh, PA, pp 520–527

    Google Scholar 

  • Castelfranchi C (2000) Affective appraisal versus cognitive evaluation in social emotions and interactions. In: Paiva A, (ed) Affective interactions. Towards a new generation of computer interfaces. Springer, Berlin

    Google Scholar 

  • Cauldwell R (2000) Where did the anger go? The role of context in interpreting emotion in speech. In: Proceedings of the ISCA workshop on speech and emotion, Northern Ireland, pp 127–131

    Google Scholar 

  • Chartrand TL, Bargh JA (1999) The chameleon effect, the perception-behavior link and social interaction. J Personal Soc Psychol 76(6):893–910

    Article  Google Scholar 

  • Chovil N (1991) Social determinants of facial displays. J Nonverbal Behav 15:141–154

    Article  Google Scholar 

  • Clark H, Krych MA (2004) Speaking while monitoring addressees for understanding. J Mem Lang 50:62–81 Clark.Krych.04.pdf

    Article  Google Scholar 

  • Clark H, Schaefer E (1991) Contributing to discourse. Cogn Sci 13:259–294

    Google Scholar 

  • Clark HH (1996) Using language. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Cogger JW (1982) Are you a skilled interviewer? Personnel J 61:840–843

    Google Scholar 

  • Condon WS, Ogston WD (1966) Sound film analysis of normal and pathological behavior patterns. J Nerv Dis 143(4):338–347

    Article  Google Scholar 

  • Drolet AL, Morris MW (2000) Rapport in conflict resolution = accounting for how face-to-face contact fosters mutual cooperation in mixed-motive conflicts. Exp Soc Psych 36:26–50

    Article  Google Scholar 

  • Duncan S, Fiske DW (1977) Face-to-face Interaction. Erlbaum, Hillsdale NJ

    Google Scholar 

  • Duncan SD, Niederehe G (1974) On signalling that its your turn to speak. J Exp Soc Psychol 10:234–47

    Article  Google Scholar 

  • Ekman P (1977) Biological and cultural contributions to body and facial movement. In: Blacking J, (ed) The anthropology of the body. Academic, London, pp 39–84

    Google Scholar 

  • Ekman P (1979) About brows: emotional and conversational signals. In: Cranach von M, Foppa K, Lepenies W, Ploog D, (ed) Human ethology. Cambridge University Press/Editions de la Maison des Sciences de l’Homme, Cambridge, pp 169–202

    Google Scholar 

  • Fuchs D (1987) Examiner familiarity effects on test performance, implications for training and practice. Top Early Child Spec Educ 7:90–104

    Article  Google Scholar 

  • Goffman E (1976) Replies and responses. Lang Soc 5(3):2257–313

    Article  Google Scholar 

  • Goodwin C (1984) Notes on story structure and the organization of participation. In: Atkinson MJ, Heritage J, (ed) Structures of social action. studies in conversation analysis. Cambridge University Press, Cambridge, pp 225–246

    Google Scholar 

  • Grahe JE, Bernieri FJ (1999) The importance of nonverbal cues in judging rapport. J Nonverbal Behav 23:253–269

    Article  Google Scholar 

  • Gratch J, Okhmatovskaia A, Duncan S (2006a) Virtual humans for the study of rapport in cross cultural settings. In: 25th army science conference, Orlando, FL, 27–30 November 2006

    Google Scholar 

  • Gratch J, Okhmatovskaia A, Lamothe F, Marsella S, Morales M, van der Werf R, Morency LP (2006b) Virtual rapport. In: 6th international conference on intelligent virtual agents, Springer, Berlin Marina del Rey, CA

    Google Scholar 

  • Gratch J, Wang N, Gerten J, Fast E (2007a) Creating rapport with virtual agents. In: 7th international conference on intelligent virtual agents, Paris, France, 2007a

    Google Scholar 

  • Gratch J, Wang N, Okhmatovskaia A, Lamothe F, Morales M, van der Werf R, Morency L-P (2007b) Can virtual humans be more engaging than real ones? In 12th international conference on human-computer interaction, Beijing, China

    Google Scholar 

  • Grice HP (1975a) Logic and conversation. In: Cole P, Morgan JL, (ed) Syntax and semantics: Vol 3: speech acts. Academic, San Diego, CA, pp 41–58

    Google Scholar 

  • Grice HP (1975b) Meaning. Phil Rev 66(3):377–388

    Article  Google Scholar 

  • Gumperz J (1982) Discourse strategies. Cambridge University Press, Cambridge, England

    Book  Google Scholar 

  • Hadar U, Steiner TJ, Rose CF (1985) Head movement during listening turns in conversation. J Nonverbal Behav 9(4):214–228

    Article  Google Scholar 

  • Hall J, Bernieri F. (ed) Interpersonal sensitivity. LEA, Mahwah, NJ

    Google Scholar 

  • Heylen D (2007) Multimodal backchannel generation for conversational agents. In: Proceedings of the workshop on multimodal output generation (MOG 2007), University of Twente, 2007. CTIT Series, p 8192, 25–26 January 2007

    Google Scholar 

  • Heylen D, Bevacqua E, Tellier M, Pelachaud C (2007a) Searching for prototypical facial feedback signals. In: Proceedings of the 7th international conference on intelligent virtual agents, Paris, France, 17–19 September, pp 147–153

    Google Scholar 

  • Heylen D, Nijholt A, Poel M (2007b) Generating nonverbal signals for a sensitive artificial listener. In: Esposito A, Faunder-Zanny M, Keller E, Marinaro M. (ed) Verbal and nonverbal communication behaviours, Lecture notes in computer science. Springer, Berlin, pp 264–274

    Chapter  Google Scholar 

  • Jefferson G (1984) Notes on a systematic deployment of the acknowledgement tokens ‘yeah’ and ‘mm hm’. Papers in Linguistics, 17:197–206

    Google Scholar 

  • Jonsdottir GR, Gratch J, Fast E, Thórisson KR (2007) Fluid semantic back-channel feedback in dialogue: challenges and progress. In Proceedings of the 7th international conference on intelligent virtual agents, Paris, France, 17–19 September, pp 154–160

    Google Scholar 

  • Kallmann M, Marsella S (2005) Hierarchical motion controllers for real-time autonomous virtual humans. In: Intelligent virtual agents, Kos, Greece, 2005. Springer, Berlin

    Google Scholar 

  • Kendon A (1970) Movement coordination in social interaction: some examples described. Acta Psychol 32:100–125

    Google Scholar 

  • Kendon A (1967) Some functions of gaze direction in social interaction. Acta Psychol 26:22–63

    Article  Google Scholar 

  • Kraut RE, Lewis SH, Swezey LW (1982) Listener responsiveness and the coordination of conversation. J Pers Soc Psychol 43(4):718–731

    Article  Google Scholar 

  • LaFrance M (1979) Nonverbal synchrony and rapport: analysis by the cross-lag panel technique. Soc Psychol Quart 42(1):66–70

    Article  Google Scholar 

  • Lafrance M, Ickes W (1981) Posture mirroring and interactional involvement: sex and sex typing effects. J nonverb behav 5:139–154

    Article  Google Scholar 

  • Lakin JL, Jefferis VA, Cheng CM, Chartrand TL (2003) Chameleon effect as social glue, evidence for the evolutionary significance of nonconscious mimicry. J Nonverb Behav 27(3):145–162

    Article  Google Scholar 

  • McClave EZ (2000) Linguistic functions of head movements in the context of speech. Journal of Pragmatics, 32:855–878

    Article  Google Scholar 

  • Morency L-P, Sidner C, Lee C, Darrell T (2005) Contextual recognition of head gestures. In: 7th international conference on multimodal interactions, Torento, Italy, 4–6 October 2005, pp 18–24

    Google Scholar 

  • Oppenheim AV, Schafer RW (2004) From frequency to quefrency. A history of the cepstrum. IEEE Signal Process Mag September:95–106

    Google Scholar 

  • Pelachaud C, Bilvi M (2003) Computational model of believable conversational agents. In: Huget M-P (ed) Communication in multiagent systems, vol 2650 of Lecture notes in computer science. Springer, Berlin, pp 300–317

    Google Scholar 

  • Pertaub D-P, Slater M, Barker C (2001) An experiment on public speaking anxiety in response to three different types of virtual audience. Presence Teleoperators and Virtual Environ 11(1):68–78

    Article  Google Scholar 

  • Poggi I (2007) Minds, hands, face and body. Weidler Buchverlag, Berlin

    Google Scholar 

  • Poggi I, Germani M (2003) Emotions at work. In: International conference on human aspects of advanced manufacturing: agility and hybrid automation, Rome, Italy, 27–30 May 2003, pp 461–468

    Google Scholar 

  • Rosenfeld HM (1987) Conversational control functions of nonverbal behavior. In: Siegman AW, Feldstein S (eds) Nonverbal behavior and communication. Lawrence Erlbaum Associates, Hillsdale NY, pp 563–601

    Google Scholar 

  • Rosenfeld HM, Hancks M (1980) The nonverbal context of verbal listener responses. In: Key MR (ed) The relationship of verbal and nonverbal communication. Mouton, The Hague, pp 193–206

    Google Scholar 

  • Sacks H, Schegloff EA, Jefferson G (1974) A simplest systematics for the organization of turn-taking for conversation. Language 50:696–735

    Article  Google Scholar 

  • Schegloff EA (1982) Discourse as interactional achievement: some uses of “uh huh” and other things that come between sentences. In: Tannen D. (ed) Analyzing discourse, text, and talk. Georgetown University Press, Washington, DC, pp 71–93

    Google Scholar 

  • Schegloff EA, Sacks H (1973) Opening up closings. Semiotica 8:289–327

    Article  Google Scholar 

  • Scherer K (1994) Affect bursts. In: van Goozen SHM, van de Poll NE, Sergeant JA. (ed) Emotions: Essays on emotion theory. Lawrence Erlbaum, Hillsdale, NJ, pp 161–193

    Google Scholar 

  • Schröder M (2003) Experimental study of affect bursts. Speech Commun Special Issue Speech a Emot 40(1–2):99–116

    MATH  Google Scholar 

  • Schröder M, Heylen D, Poggi I (2006) Perception of non-verbal emotional listener feedback. In: Proceedings of speech prosody 2006, Dresden, Germany, 2–5 May 2006

    Google Scholar 

  • Searle JR (1969) Speech acts: an essay in the philosophy of language. Cambridge University Press, Cambridge

    Google Scholar 

  • Sidner CL, Lee C (2007) Attentional gestures in dialogues between people and robots. In: Nishida T (ed) Conversational informatics. Wiley, New York, NY

    Google Scholar 

  • Sonnby-Borgstrom M, Jonsson P, Svensson O (2003) Emotional empathy as related to mimicry reactions at different levels of information processing. J Nonverb Behav 27(1):3–23

    Article  Google Scholar 

  • Tatar D (1997) Social and personal consequences of a preoccupied listener. PhD thesis, Department of Psychology, Stanford University, Stanford, CA

    Google Scholar 

  • Thórisson KR (1996) Communicative Humanoids: a computational model of psycho-social dialogue skills. PhD thesis, Massachusetts Institute of Technology

    Google Scholar 

  • Tickle-Degnen L, Rosenthal R (1990) The nature of rapport and its nonverbal correlates. Psychol Inquiry 1(4):285–293

    Article  Google Scholar 

  • Tosa N (1993) Neurobaby. In: SIGGRAPH-93 visual proceedings, tomorrow’s realities, ACM, pp 212–213

    Google Scholar 

  • Trouvain J (2004) Non-verbal vocalisations – the case of laughter. Paper presented at Evolution of Language: Fifth International Conference, 2004. Max Planck Institute for Evolutionary Anthropology, Leipzig, Germany, 31 March – 3 April 2004

    Google Scholar 

  • Trouvain J, Schröder M (2004) How (not) to add laughter to synthetic speech. In: Proceedings of Workshop on affective dialogue systems, Kloster Irsee, Germany, 14–16 June 2004, pp 229–232

    Google Scholar 

  • Tsui P, Schultz GL (1985) Failure of rapport. Why psychotherapeutic engagement fails in the treatment of Asian clients. Am J Orthopsychiatry, 55:561–569

    Article  Google Scholar 

  • Vilhjalmsson H, Cantelmo N, Cassell J, Chafai NE, Kipp M, Kopp S, Mancini M, Marsella S, Marshall AN, Pelachaud C, Ruttkay ZS, Thorisson KR, van Welbergen H, van der Werf R (2007) The behavior markup language, recent developments and challenges. In: International conference on intelligent virtual agents, Paris, France, 2007. Springer, Berlin

    Google Scholar 

  • Ward N, Tsukahara W (2000) Prosodic features which cue back-channel responses in English and Japanese. J Pragmatics 23:1177–1207

    Article  Google Scholar 

  • Yngve VH (1970) On getting a word in edgewise. In: Papers from the sixth regional meeting of the Chicago Linguistic Society. Chicago Linguistic Society, Chicago, IL, pp 567–577

    Google Scholar 

Download references

Acknowledgments

Many of the sections of this chapter are based on previous work of the authors that was done in collaboration with other colleagues. Therefore they wish to acknowledge the big help they got from Sue Duncan, Marion Tellier and Rieks op den Akker.

Author information

Authors and Affiliations

  1. Enschede Faculty of Electrical Engineering Mathematics and Computer Science, University of Twente, Enschede, The Netherlands

    Dirk Heylen

  2. University of Paris 8, now at CNRS, Telecom-ParisTech, Paris, France

    Elisabetta Bevacqua

  3. CNRS-LTCI, TELECOM ParisTech, Paris, France

    Catherine Pelachaud

  4. University of Rome 3, Rome, Italy

    Isabella Poggi

  5. University of Southern California, Marina del Rey, CA, USA

    Jonathan Gratch

  6. Deutsches Forschungsinstitut für Künstliche Intelligenz, Saarbrücken, Germany

    Marc Schröder

Authors
  1. Dirk Heylen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elisabetta Bevacqua
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Catherine Pelachaud
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Isabella Poggi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jonathan Gratch
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marc Schröder
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dirk Heylen .

Editor information

Editors and Affiliations

  1. , Department of Psychology, Queen's University Belfast, Belfast, BT7 1NN, United Kingdom

    Roddy Cowie

  2. TELECOM ParisTech, CNRS - LTCI, rue Dareau 37, Paris, 75014, France

    Catherine Pelachaud

  3. for Artificial Intelligence (OFAI), Austrian Research Institute, Freyung 6/6, Vienna, 1010, Austria

    Paolo Petta

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Heylen, D., Bevacqua, E., Pelachaud, C., Poggi, I., Gratch, J., Schröder, M. (2011). Generating Listening Behaviour. In: Cowie, R., Pelachaud, C., Petta, P. (eds) Emotion-Oriented Systems. Cognitive Technologies. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15184-2_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-15184-2_17

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-15183-5

  • Online ISBN: 978-3-642-15184-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation