research-article

Estimating communication skills using dialogue acts and nonverbal features in multiple discussion datasets

Authors: Shogo Okada, Yoshihiko Ohtake,

Yukiko I. Nakano,

Hung-Hsuan Huang,

Yutaka Takase, Katsumi NittaAuthors Info & Claims

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Pages 169 - 176

https://doi.org/10.1145/2993148.2993154

Published: 31 October 2016 Publication History

Abstract

This paper focuses on the computational analysis of the individual communication skills of participants in a group. The computational analysis was conducted using three novel aspects to tackle the problem. First, we extracted features from dialogue (dialog) act labels capturing how each participant communicates with the others. Second, the communication skills of each participant were assessed by 21 external raters with experience in human resource management to obtain reliable skill scores for each of the participants. Third, we used the MATRICS corpus, which includes three types of group discussion datasets to analyze the influence of situational variability regarding to the discussion types. We developed a regression model to infer the score for communication skill using multimodal features including linguistic and nonverbal features: prosodic, speaking turn, and head activity. The experimental results show that the multimodal fusing model with feature selection achieved the best accuracy, 0.74 in R² of the communication skill. A feature analysis of the models revealed the task-dependent and task-independent features to contribute to the prediction performance.

References

[1]

O. Aran and D. Gatica-Perez. One of a kind: Inferring personality impressions in meetings. In Proc. of ACM ICMI, pages 11–18, 2013.

Digital Library

[2]

L. F. Bachman. Fundamental considerations in language testing. Oxford University Press, 1990.

[3]

L. Batrinca, N. Mana, B. Lepri, N. Sebe, and F. Pianesi. Multimodal personality recognition in collaborative goal-oriented tasks. IEEE Trans. on Multimedia, 18(4):659–673, 2016.

[4]

J.-I. Biel, L. Teijeiro-Mosquera, and D. Gatica-Perez. Facetube: predicting personality from facial expressions of emotion in online conversational video. In Proc. of ACM ICMI, pages 53–56, 2012.

Digital Library

[5]

P. Boersma and D. Weenink. Praat: doing phonetics by computer. {Computer program},Version 5.3.51, 2013.

[6]

J. A. Bonito. The effect of contributing substantively on perceptions of participation. Small Group Research, 31(5):528–553, 2000.

[7]

J. Carletta, S. Garrod, and H. Fraser-Krauss. Placement of authority and communication patterns in workplace groups the consequences for innovation. Small Group Research, 29(5):531–559, 1998.

[8]

L. Chen and M. P. Harper. Multimodal floor control shift detection. In proc. of ACM ICMI-MLMI, pages 15–22, 2009.

Digital Library

[9]

M. G. Core and J. Allen. Coding dialogs with the damsl annotation scheme. In AAAI fall symposium on communicative action in humans and machines, pages 28–35, 1997.

[10]

J. E. Driskell, B. Olmstead, and E. Salas. Task cues, dominance cues, and influence in task groups. Journal of Applied Psychology, 78(1):51, 1993.

[11]

D. Gatica-Perez. Automatic nonverbal analysis of social interaction in small groups: A review. Image Vision Computing, 27(12):1775–1787, 2009.

Digital Library

[12]

J. O. Greene and B. R. Burleson. Handbook of communication and social interaction skills. Psychology Press, 2003.

[13]

D. Hymes. On Communicative Competence. Harmondsworth: penguin, 1972.

[14]

D. B. Jayagopi, D. Sanchez-Cortes, K. Otsuka, J. Yamato, and D. Gatica-Perez. Linking speaking and looking behavior patterns with group composition, perception, and performance. In Proc. of ACM ICMI, pages 433–440, 2012.

Digital Library

[15]

T. Kudo, K. Yamamoto, and Y. Matsumoto. Applying conditional random fields to japanese morphological analysis. In EMNLP, volume 4, pages 230–237, 2004.

[16]

C. Lai, J. Carletta, and S. Renals. Modelling participant affect in meetings with turn-taking features. In Proc. of WASSS 2013, 2013.

[17]

L. Nguyen, D. Frauendorfer, M. Mast, and D. Gatica-Perez. Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior. IEEE Trans. on Multimedia, 16(4):1018–1031, 2014.

Digital Library

[18]

F. Nihei, Y. I. Nakano, Y. Hayashi, H.-H. Hung, and S. Okada. Predicting influential statements in group discussions using speech and head motion information. In Proc. of ACM ICMI, pages 136–143, 2014.

Digital Library

[19]

S. Okada, O. Aran, and D. Gatica-Perez. Personality trait classification via co-occurrent multiparty multimodal event discovery. In Proc. of ACM ICMI, pages 15–22, 2015.

Digital Library

[20]

S. Park, H. S. Shim, M. Chatterjee, K. Sagae, and L.-P. Morency. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In Proc. of ACM ICMI, pages 50–57, 2014.

Digital Library

[21]

F. Pianesi, N. Mana, A. Cappelletti, B. Lepri, and M. Zancanaro. Multimodal recognition of personality traits in social interactions. In Proc. of ACM ICMI, pages 53–60, 2008.

Digital Library

[22]

V. Ramanarayanan, C. W. Leong, L. Chen, G. Feng, and D. Suendermann-Oeft. Evaluating speech, face, emotion and body movement time-series features for automated multimodal presentation scoring. In Proc. of ACM ICMI, pages 23–30, 2015.

Digital Library

[23]

G. Rickheit and H. Strohner. Handbook of communication competence, volume 1. Walter de Gruyter, 2008.

[24]

R. E. Riggio, H. R. Riggio, C. Salinas, and E. J. Cole. The role of social and emotional communication skills in leader emergence and effectiveness. Group Dynamics: Theory, Research, and Practice, 7(2):83, 2003.

[25]

R. B. Ruback, J. M. Dabbs, and C. H. Hopper. The process of brainstorming: An analysis with individual and group vocal parameters. Journal of Personality and Social Psychology, 47(3):558, 1984.

[26]

D. Sanchez-Cortes, O. Aran, M. S. Mast, and D. Gatica-Perez. A nonverbal behavior approach to identify emergent leaders in small groups. IEEE Trans. on Multimedia, 14(3):816–832, 2012.

Digital Library

[27]

S. Scherer, N. Weibel, L.-P. Morency, and S. Oviatt. Multimodal prediction of expertise and leadership in learning groups. In Proc. of the International Workshop on Multimodal Learning Analytics, pages 1–8, 2012.

Digital Library

[28]

E. Shriberg, R. Dhillon, S. Bhagat, J. Ang, and H. Carvey. The icsi meeting recorder dialog act (mrda) corpus. In Proc. of SIGDIAL, pages 97–100, 2004.

[29]

S. Siegel and N. Castellan. Nonparametric statistics for the behavioral sciences. McGraw–Hill, Inc., second edition, 1988.

[30]

A. Stolcke, N. Coccaro, R. Bates, P. Taylor, C. Van Ess-Dykema, K. Ries, E. Shriberg, D. Jurafsky, R. Martin, and M. Meteer. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373, 2000.

Digital Library

[31]

G. Tur and R. De Mori. Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons, 2011.

[32]

F. Valente, S. Kim, and P. Motlicek. Annotation and recognition of personality traits in spoken conversations from the ami meetings corpus. In Proc. of INTERSPEECH, pages 1183–1186, 2012.

[33]

T. Wörtwein, M. Chollet, B. Schauerte, L.-P. Morency, R. Stiefelhagen, and S. Scherer. Multimodal public speaking performance assessment. In Proc. of ACM ICMI, pages 43–50, 2015.

Digital Library

Cited By

Amat APlunk AAdiani DWilkes DSarkar N(2024)Prediction Models of Collaborative Behaviors in Dyadic Interactions: An Application for Inclusive Teamwork Training in Virtual EnvironmentsSignals10.3390/signals50200195:2(382-401)Online publication date: 3-Jun-2024
https://doi.org/10.3390/signals5020019
Isshiki YHuang H(2024)Investigation on the Use of Mora in Assessment of L2 Speakers’ Japanese Language ProficiencySocial Computing and Social Media10.1007/978-3-031-61305-0_5(67-83)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-61305-0_5
Kinoshita SOnishi TAzuma NIshii RFukayama ANakamura TMiyata ALugrin BLatoschik Mvon Mammen SKopp SPécune FPelachaud C(2023)A Study of Prediction of Listener's Comprehension Based on Multimodal InformationProceedings of the 23rd ACM International Conference on Intelligent Virtual Agents10.1145/3570945.3607304(1-4)Online publication date: 19-Sep-2023
https://dl.acm.org/doi/10.1145/3570945.3607304
Show More Cited By

Index Terms

Estimating communication skills using dialogue acts and nonverbal features in multiple discussion datasets
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics
  2. Machine learning
    1. Machine learning algorithms
      1. Feature selection
2. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Social engineering (social sciences)

Recommendations

Exploiting emotions to disambiguate dialogue acts
IUI '04: Proceedings of the 9th international conference on Intelligent user interfaces

This paper describes an attempt to reveal the user's intention from dialogue acts, thereby improving the effectiveness of natural interfaces to pedagogical agents. It focuses on cases where the intention is unclear from the dialogue context or utterance ...
Dialog acts in greeting and leavetaking in social talk
ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents

Conversation proceeds through dialogue moves or acts, and dialog act annotation can aid the design of artificial dialog. While many dialogs are task-based or instrumental, with clear goals, as in the case of a service encounter or business meeting, ...
Multifunctionality in dialogue

This paper studies the multifunctionality of dialogue utterances, i.e. the phenomenon that utterances in dialogue often have more than one communicative function. It is argued that this phenomenon can be explained by analyzing the participation in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

October 2016

605 pages

ISBN:9781450345569

DOI:10.1145/2993148

General Chairs:
Yukiko I. Nakano
Seikei University, Japan
,
Elisabeth André
Augsburg University, Germany
,
Toyoaki Nishida
Kyoto University, Japan
,
Program Chairs:
Louis-Philippe Morency
Carnegie Mellon University, USA
,
Carlos Busso
University of Texas at Dallas, USA
,
Catherine Pelachaud
ISIR, France / University of Paris6, France

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '16

Sponsor:

SIGCHI

ICMI '16: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 12 - 16, 2016

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
525
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)7

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Amat APlunk AAdiani DWilkes DSarkar N(2024)Prediction Models of Collaborative Behaviors in Dyadic Interactions: An Application for Inclusive Teamwork Training in Virtual EnvironmentsSignals10.3390/signals50200195:2(382-401)Online publication date: 3-Jun-2024
https://doi.org/10.3390/signals5020019
Isshiki YHuang H(2024)Investigation on the Use of Mora in Assessment of L2 Speakers’ Japanese Language ProficiencySocial Computing and Social Media10.1007/978-3-031-61305-0_5(67-83)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-61305-0_5
Kinoshita SOnishi TAzuma NIshii RFukayama ANakamura TMiyata ALugrin BLatoschik Mvon Mammen SKopp SPécune FPelachaud C(2023)A Study of Prediction of Listener's Comprehension Based on Multimodal InformationProceedings of the 23rd ACM International Conference on Intelligent Virtual Agents10.1145/3570945.3607304(1-4)Online publication date: 19-Sep-2023
https://dl.acm.org/doi/10.1145/3570945.3607304
Lin YLiu YLee C(2023)An Interaction-process-guided Framework for Small-group Performance PredictionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/355876819:2(1-25)Online publication date: 6-Feb-2023
https://dl.acm.org/doi/10.1145/3558768
Bai CBolonkin MRegunath VSubrahmanian V(2023) DIPS: A Dyadic Impression Prediction System for Group Interaction VideosACM Transactions on Multimedia Computing, Communications, and Applications10.1145/353286519:1s(1-24)Online publication date: 23-Jan-2023
https://dl.acm.org/doi/10.1145/3532865
Tun SOkada SHuang HLeong C(2023)Multimodal Transfer Learning for Oral Presentation AssessmentIEEE Access10.1109/ACCESS.2023.329583211(84013-84026)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3295832
Mawalim COkada SNakano YUnoki M(2023)Personality trait estimation in group discussions using multimodal analysis and speaker embeddingJournal on Multimodal User Interfaces10.1007/s12193-023-00401-017:2(47-63)Online publication date: 8-Feb-2023
https://doi.org/10.1007/s12193-023-00401-0
Masuda THirose HTsuchiya T(2023)Structuring of Discourse and Annotation Method for Contribution Assessment in Collaborative DiscussionsFuture Data and Security Engineering. Big Data, Security and Privacy, Smart City and Industry 4.0 Applications10.1007/978-981-99-8296-7_6(76-91)Online publication date: 17-Nov-2023
https://doi.org/10.1007/978-981-99-8296-7_6
Plunk AAmat AWilkes DSarkar N(2023)Automated Behavior Labeling During Team-Based Activities Involving Neurodiverse and Neurotypical Partners Using Multimodal DataPattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges10.1007/978-3-031-37660-3_14(193-209)Online publication date: 30-Jul-2023
https://doi.org/10.1007/978-3-031-37660-3_14
Huang H(2023)Analysis on the Language Use of L2 Japanese Speakers Regarding to Their Proficiency in Group Discussion ConversationsSocial Computing and Social Media10.1007/978-3-031-35915-6_5(55-67)Online publication date: 9-Jul-2023
https://doi.org/10.1007/978-3-031-35915-6_5
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents