research-article

Computational Analysis of Persuasiveness in Social Multimedia: A Novel Dataset and Multimodal Prediction Approach

Authors:

Moitreya Chatterjee,

Louis-Philippe MorencyAuthors Info & Claims

ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

Pages 50 - 57

https://doi.org/10.1145/2663204.2663260

Published: 12 November 2014 Publication History

Abstract

Our lives are heavily influenced by persuasive communication, and it is essential in almost any types of social interactions from business negotiation to conversation with our friends and family. With the rapid growth of social multimedia websites, it is becoming ever more important and useful to understand persuasiveness in the context of social multimedia content online. In this paper, we introduce our newly created multimedia corpus of 1,000 movie review videos obtained from a social multimedia website called ExpoTV.com, which will be made freely available to the research community. Our research results presented here revolve around the following 3 main research hypotheses. Firstly, we show that computational descriptors derived from verbal and nonverbal behavior can be predictive of persuasiveness. We further show that combining descriptors from multiple communication modalities (audio, text and visual) improve the prediction performance compared to using those from single modality alone. Secondly, we investigate if having prior knowledge of a speaker expressing a positive or negative opinion helps better predict the speaker's persuasiveness. Lastly, we show that it is possible to make comparable prediction of persuasiveness by only looking at thin slices (shorter time windows) of a speaker's behavior.

References

[1]

Ambady, N. and Rosenthal, R. 1992. Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin 111, 2 (Mar 1992), 256--274.

[2]

Biel, J., Teijeiro-Mosquera, L., and Gatica-Perez, D. 2012. FaceTube: Predicting personality from facial expression of emotion in online Table 3. Top computational descriptors for positive and negative reviews which were automatically selected using Information Gain. conversational video. In Proc. of the 14th ACM Int'l Conf. on Multimodal Interaction. ICMI '12, 53--56.

Digital Library

[3]

Burgoon, J., Birk, T., and Pfau, M. 1990. Nonverbal behaviors, persuasion, and credibility. Human Communication Research 17, 1 (Sept. 1990), 140--169.

[4]

Busso, C., Lee, S. and Narayanan, S. 2009. Analysis of emotionally salient aspects of fundamental frequency for emotion detection. IEEE Transactions on Audio, Speech, and Language Processing, 17, 4 (May 2009), 582--596.

Digital Library

[5]

Chaiken, S. and Eagly, A. 1983. Communication modality as a determinant of message persuasiveness and message comprehensibility. Journal of Personality and Social Psychology 34, 4 (Oct. 1976), 605--614.

[6]

Chaiken, S., Liberman, A., and Eagly, A. Heuristic and systematic information processing within and beyond the persuasion context. In J. Uleman and J. Bargh (Eds.), Unintended Thought: Limits of Awareness, Intention, and Control (pp. 212--252). New York: Guildford Press.

[7]

Chang, C.-C. and Lin, C.-J. 2011. LIBSVM: A library for support vector machine. ACM Transactions on Intelligent Systems and Technology 2, 3 (Apr. 2011), 27:1--27:25.

Digital Library

[8]

Crano, W. and Prislin, R. 2006. Attitudes and persuasion. Annual Review of Psychology 57 (Jan. 2006), 345--374.

[9]

Curhan, J. and Pentland, A. 2007. Thin slices of negotiation: Predicting outcomes from conversational dynamics within the first 5 minutes. Journal of Applied Psychology 92, 3 (May 2007), 802--811.

[10]

Degottex, G., Kane, J., Drugman, T., Raitio, T., and Scherer, S. 2014. COVAREP - A collaborative voice analysis repository for speech technologies. In Proc. of the 39th Int'l Conf. on Acoustics, Speech, and Signal Processing. ICASSP '14.

[11]

Ekman, P. and Rosenberg, E. 1997. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press, New York, NY.

[12]

FaceFX. http://www.facefx.com/

[13]

FACET. http://www.emotient.com/cert

[14]

Gobl, C. and Chasaide, A. 2003. The role of voice quality in communication emotion, mood and attitude. Speech Communication 40, 1--2 (Apr. 2003), 189--212.

Digital Library

[15]

Hosman, L. 2002. Language and persuasion. In J. Dillard and M. Pfau (Eds.), The Persuasion Handbook: Developments in Theory and Practice (pp. 371--390). New York: Sage.

[16]

Kane, J., Scherer, S., Aylett, M., Morency, L.-P., and Gobl, C. 2013. Speaker and language independent voice quality classification applied to unlabeled corpora of expressive speech. In Proc. of the 38th Int'l Conf. on Acoustics, Speech, and Signal Processing. ICASSP '13, 7982--7986.

[17]

Kane, J., Scherer, S., Morency, L.-P., and Gobl, C. 2013. A comparative study of glottal open quotient estimation techniques. In Proc. of the 14th Annual Conf. of the Int'l Speech Communication Association. Interspeech '13, 1658--1662.

[18]

Krippendorff, K. 2004. Content Analysis: An Introduction to Its Methodology. Sage, Beverly Hills, CA.

[19]

LaCrosse, M. 1975. Nonverbal behavior and perceived counselor attractiveness and persuasiveness. Journal of Counseling Psychology 22, 6 (Nov. 1975), 563--566.

[20]

Mahl, G. 1956. Disturbances and silences in the patient's speech in psychotherapy. The Journal of Abnormal and Social Psychology 53, 1 (Jul. 1956), 1--15.

[21]

Mason, W. and Suri, S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (Mar. 2012), 1--23.

[22]

Mehrabian, A. 1971. Silent messages. Wadsworth Publishing Company, Inc., Belmont, CA.

[23]

Mehrabian, A. and Williams, M. 1969. Nonverbal concomitants of perceived and intended persuasiveness. Journal of Personality and Social Psychology 13, 1 (Sept. 1969), 37--58.

[24]

Meyers-Levy, J. and Malaviya, P. 1999. Consumers' processing of persuasive advertisements: An integrative framework of persuasion theories. Journal of marketing 63, 4 (Oct. 1999), 45--60.

[25]

Miller, N., Maruyama, G., Beaber, R., and Valone, K. 1976. Speed of speech and persuasion. Journal of Personality and Social Psychology 34, 4 (Oct. 1976), 615--624.

[26]

Morency, L.-P., Whitehill, J., and Movellan, J. 2008. Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation. In Proc. of the 8th Int'l Conf. on Automatic Face and Gesture Recognition. FG '08, 1--8.

[27]

OKAO Vision. http://www.omron.com/r_d/coretech/vision/okao.html

[28]

O'Keefe. 2002. Persuasion: Theory and Research. Sage, Thousand Oaks, CA.

[29]

O'Keefe, D. and Jensen, J. 2007. The relative persuasiveness of gainframed loss-framed messages for encouraging disease prevention behaviors: A meta-analytic review. Journal of Health Communication 12, 7 (Oct. 2007), 623--644.

[30]

Pearce, W. and Brommel, B. 1972. Vocalic communication in persuasion. Quarterly Journal of Speech 58, 3 (1972), 298--306.

[31]

Perloff, R. 2010. The Dynamics of Persuasion: Communication and Attitudes in the Twenty-First Century. Routledge, New York, NY.

[32]

Petty, R. and Cacioppo, J. 1986. Communication and Persuasion: Central and Peripheral Routes to Attitude Change. New York: SpringerVerlag.

[33]

Pittam, J. 1990. The relationship between perceived persuasiveness of nasality and source characteristics for Australian and American listeners. The Journal of Social Psychology 130, 1 (1990), 81--87.

[34]

Pornpitakpan, C. 2004. The persuasiveness of source credibility: A critical review of five decades? evidence. Journal of Applied Social Psychology 34, 2 (Feb. 2004), 243--281.

[35]

Rammstedt, B. and John, O. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41, 1 (Feb. 2007), 203--212.

[36]

Rosenfeld, H. 1966. Approval-seeking and approval-inducing functions of verbal and nonverbal responses in the dyad. Journal of Personality and Social Psychology 4, 6 (Dec. 1966), 597--605.

[37]

Scherer, S., Kane, J., Gobl, C., and Schwenker, F. Investigating fuzzyinput fuzzy-output support vector machines for robust voice quality classification. Computer Speech and Language 27, 1 (Jan. 2013), 263--287.

Digital Library

[38]

Scherer, S., London, H., and Wolf, J. 1973. The voice of confidence: Paralinguistic cues and audience evaluation. Journal of Research in Personality 7, 1 (June 1973), 31--44.

[39]

Schuller, B., Steidl, S., Batliner, A., Schiel, F., and Krajewski, J. 2011. The Interspeech 2011 speaker state challenge. In Proc. of the 12th Annual Conf. of the Int'l Speech Communication Association. Interspeech '11, 3201--3204.

[40]

Stern, S., Mullennix, J., and Wilson, S. 2002. Effects of perceived disability on persuasiveness of computer-synthesized speech. Journal of Applied Psychology 87, 2 (Apr. 2002), 411--417.

[41]

Voss, J. 2005. The science of persuasion: An exploration of advocacy and the science behind the art of persuasion in the courtroom, Law and Psychology Review 29 (2005), 301--327.

[42]

Worchel, S., Andreoli, V., and Eason, J. 1975. Is the medium the message? A study of the effects of media, communicator, and message characteristics on attitude change. Journal of Applied Social Psychology 5, 2 (June 1975), 157--172.

[43]

Yang, Y. and Pedersen, J. 1997. A comparative study on feature selection in text categorization. In Proc. of the 14th Int'l Conf. on Machine Learning. ICML '97, 412--420.

Digital Library

[44]

Young, J., Martell, C., Anand, P., Ortiz, P., and Gilbert, H. 2011. A microtext corpus for persuasion detection in dialog. In Proc. of the AAAI11 Workshop on Analyzing Microtext, 80--85.

Cited By

Wang ZXu GZhou XKim JZhu HDeng L(2024)Deep Tensor Evidence Fusion Network for Sentiment ClassificationIEEE Transactions on Computational Social Systems10.1109/TCSS.2022.319799411:4(4605-4613)Online publication date: Aug-2024
https://doi.org/10.1109/TCSS.2022.3197994
Liu WGong YZhang GLu JZhou YLiao J(2024)GLMDriveNet: Global–local Multimodal Fusion Driving Behavior Classification NetworkEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107575129(107575)Online publication date: Mar-2024
https://doi.org/10.1016/j.engappai.2023.107575
Li YDu MSong RWang XSun MWang Y(2024)Mitigating Social Biases of Pre-trained Language Models via Contrastive Self-Debiasing with Double Data AugmentationArtificial Intelligence10.1016/j.artint.2024.104143(104143)Online publication date: Apr-2024
https://doi.org/10.1016/j.artint.2024.104143
Show More Cited By

Index Terms

Computational Analysis of Persuasiveness in Social Multimedia: A Novel Dataset and Multimodal Prediction Approach
1. Applied computing
  1. Law, social and behavioral sciences
2. Information systems
  1. World Wide Web
    1. Web applications
      1. Internet communications tools

Recommendations

Deep multimodal fusion for persuasiveness prediction
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Persuasiveness is a high-level personality trait that quantifies the influence a speaker has on the beliefs, attitudes, intentions, motivations, and behavior of the audience. With social multimedia becoming an important channel in propagating ideas and ...
Multimodal Analysis and Prediction of Persuasiveness in Online Social Multimedia
Regular Articles and Special Issue on Highlights of ICMI 2014 (Part 2 of 2)

Our lives are heavily influenced by persuasive communication, and it is essential in almost any type of social interaction from business negotiation to conversation with our friends and family. With the rapid growth of social multimedia websites, it is ...
Predicting Persuasiveness of Participants in Multiparty Conversations
IUI '22 Companion: Companion Proceedings of the 27th International Conference on Intelligent User Interfaces

Persuasiveness is an important capability in communication skills. This study aims to estimate the persuasiveness of participants in group discussions. First, human annotators rated the level of persuasiveness of each of four participants in group ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

November 2014

558 pages

ISBN:9781450328852

DOI:10.1145/2663204

General Chairs:
Albert Ali Salah
Boğaziçi University, Turkey
,
Jeffrey Cohn
University of Pittsburgh, USA
,
Björn Schuller
University of Passau, Germany and Imperial College London, UK
,
Program Chairs:
Oya Aran
Idiap Research Institute, Switzerland
,
Louis-Philippe Morency
University of Southern California, USA
,
Philip R. Cohen
Adapx, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Division of Information and Intelligent Systems

Conference

ICMI '14

Sponsor:

SIGCHI

ICMI '14: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 12 - 16, 2014

Istanbul, Turkey

Acceptance Rates

ICMI '14 Paper Acceptance Rate 51 of 127 submissions, 40%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
1,250
Total Downloads

Downloads (Last 12 months)137
Downloads (Last 6 weeks)21

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZXu GZhou XKim JZhu HDeng L(2024)Deep Tensor Evidence Fusion Network for Sentiment ClassificationIEEE Transactions on Computational Social Systems10.1109/TCSS.2022.319799411:4(4605-4613)Online publication date: Aug-2024
https://doi.org/10.1109/TCSS.2022.3197994
Liu WGong YZhang GLu JZhou YLiao J(2024)GLMDriveNet: Global–local Multimodal Fusion Driving Behavior Classification NetworkEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107575129(107575)Online publication date: Mar-2024
https://doi.org/10.1016/j.engappai.2023.107575
Li YDu MSong RWang XSun MWang Y(2024)Mitigating Social Biases of Pre-trained Language Models via Contrastive Self-Debiasing with Double Data AugmentationArtificial Intelligence10.1016/j.artint.2024.104143(104143)Online publication date: Apr-2024
https://doi.org/10.1016/j.artint.2024.104143
Thomas CUdhayanan PYadav APurvaj SJayagopi D(2024)Multimodal Unsupervised Domain Adaptation for Predicting Speaker Characteristics from VideoSN Computer Science10.1007/s42979-024-02723-65:5Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1007/s42979-024-02723-6
Biancardi BChollet MClavel C(2024)Introducing the 3MT_French dataset to investigate the timing of public speaking judgementsLanguage Resources and Evaluation10.1007/s10579-023-09709-5Online publication date: 23-Mar-2024
https://doi.org/10.1007/s10579-023-09709-5
Lorini EOchs MSabouret N(2024)Cognitive Planning for Persuasive Multimodal InteractionEngineering Multi-Agent Systems10.1007/978-3-031-71152-7_6(93-104)Online publication date: 16-Sep-2024
https://doi.org/10.1007/978-3-031-71152-7_6
Ramaswamy MPalaniswamy S(2024)Multimodal emotion recognition: A comprehensive review, trends, and challengesWIREs Data Mining and Knowledge Discovery10.1002/widm.156314:6Online publication date: 8-Oct-2024
https://doi.org/10.1002/widm.1563
Luo XJia NOuyang EFang Z(2024)Introducing machine‐learning‐based data fusion methods for analyzing multimodal data: An application of measuring trustworthiness of microenterprisesStrategic Management Journal10.1002/smj.359745:8(1597-1629)Online publication date: 5-Mar-2024
https://doi.org/10.1002/smj.3597
Xian TDu PLiao C(2023)Theory and Data-Driven Competence Evaluation with Multimodal Machine Learning—A Chinese Competence Evaluation Multimodal DatasetApplied Sciences10.3390/app1313776113:13(7761)Online publication date: 30-Jun-2023
https://doi.org/10.3390/app13137761
Ito ANakano YNihei FSakato TIshii RFukayama ANakamura T(2023)Estimating and Visualizing Persuasiveness of Participants in Group DiscussionsJournal of Information Processing10.2197/ipsjjip.31.3431(34-44)Online publication date: 2023
https://doi.org/10.2197/ipsjjip.31.34
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents