research-article

Public Access

Multimodal Analysis and Prediction of Persuasiveness in Online Social Multimedia

Authors:

Moitreya Chatterjee,

Louis-Philippe MorencyAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 6, Issue 3

Article No.: 25, Pages 1 - 25

https://doi.org/10.1145/2897739

Published: 17 October 2016 Publication History

Abstract

Our lives are heavily influenced by persuasive communication, and it is essential in almost any type of social interaction from business negotiation to conversation with our friends and family. With the rapid growth of social multimedia websites, it is becoming ever more important and useful to understand persuasiveness in the context of social multimedia content online. In this article, we introduce a newly created multimedia corpus of 1,000 movie review videos with subjective annotations of persuasiveness and related high-level characteristics or attributes (e.g., confidence). This dataset will be made freely available to the research community. We designed our experiments around the following five main research hypotheses. First, we study if computational descriptors derived from verbal and nonverbal behavior can be predictive of persuasiveness. We further explore combining descriptors from multiple communication modalities (acoustic, verbal, para-verbal, and visual) for predicting persuasiveness and compare with using a single modality alone. Second, we investigate how certain high-level attributes, such as credibility or expertise, are related to persuasiveness and how the information can be used in modeling and predicting persuasiveness. Third, we investigate differences when speakers are expressing a positive or negative opinion and if the opinion polarity has any influence in the persuasiveness prediction. Fourth, we further study if gender has any influence in the prediction performance. Last, we test if it is possible to make comparable predictions of persuasiveness by only looking at thin slices (i.e., shorter time windows) of a speaker's behavior.

References

[1]

Nalini Ambady and Robert Rosenthal. 1992. Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psych. Bull. 111, 2 (Mar. 1992), 256--274.

[2]

Joan-Isaac Biel, Lucía Teijeiro-Mosquera, and Daniel Gatica-Perez. 2012. FaceTube: Predicting personality from facial expression of emotion in online conversational video. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI’12). ACM Press, New York, 53--56. 10.1145/2388676.2388689

Digital Library

[3]

Judee Burgoon, Thomas Birk, and Michael Pfau. 1990. Nonverbal behaviors, persuasion, and credibility. Hum. Commun. Res. 17, 1 (Sept. 1990), 140--169.

[4]

Carlos Busso, Sungbok Lee, and Shrikanth Narayanan. 2009. Analysis of emotionally salient aspects of fundamental frequency for emotion detection. IEEE Trans. Audio, Speech Lang. Process. 17, 4 (May 2009), 582--596. 10.1109/TASL.2008.2009578

Digital Library

[5]

John Campbell. 1998. Participation in videoconferenced meetings: User disposition and meeting context. Inf. Manage. 34, 6 (Dec. 2006), 329--338.

Digital Library

[6]

Linda Carli, Suzanne LaFleur, and Christopher Loeber. 1995. Nonverbal behavior, gender, and influence. J. Personal. Social Psych. 68, 6 (Jun. 1995), 1030--1041.

[7]

Shelly Chaiken. 1979. Communicator physical attractiveness and persuasion. J. Personal. Social Psych. 37, 8 (Aug. 1979), 1387--1397.

[8]

Shelly Chaiken and Alice Eagly. 1976. Communication modality as a determinant of message persuasiveness and message comprehensibility. Journal of Personality and Social Psychology 34, 4 (Oct. 1976), 605--614.

[9]

Shelly Chaiken, Akiva Liberman, and Alice Eagly. 1989. Heuristic and systematic information processing within and beyond the persuasion context. In Unintended Thought: Limits of Awareness, Intention, and Control, James Uleman and John Bargh (Eds.). Guildford Press, New York, 212--252.

[10]

Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machine. ACM Trans. Intell Syst Tech. 2, 3 (Apr. 2011), 27, 1--27, 25.

Digital Library

[11]

Moitreya Chatterjee, Sunghyun Park, Han Suk Shim, Kenji Sagae, and Louis-Philippe Morency. 2014. Verbal behaviors and persuasiveness in online multimedia content. In Proceedings of the 2^nd Social NLP Workshop (SocialNLP’14), 50--58.

[12]

Milton Chen. 2003. Conveying Conversational Cues through Video. Ph.D. Dissertation. Stanford University, Palo Alto, CA.

Digital Library

[13]

William Crano and Radmila Prislin. 2006. Attitudes and persuasion. Ann. Rev. Psych. 57 (Jan. 2006), 345--374.

[14]

Jared Curhan and Alex Pentland. 2007. Thin slices of negotiation: Predicting outcomes from conversational dynamics within the first 5 minutes. J. Appl. Psych. 92, 3 (May 2007), 802--811.

[15]

Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. COVAREP - A collaborative voice analysis repository for speech technologies. In Proceedings of the 39^th International Conference on Acoustics, Speech, and Signal Processing (ICASSP’14). IEEE, 960--964.

[16]

Paul Ekman. 1997. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, New York, NY.

[17]

FaceFX. http://www.facefx.com/.

[18]

Kurt Frey and Alice Eagly. 1993. Vividness can undermine the persuasiveness of messages. J. Personal. Social Psych. 65, 1 (July 1993), 32--44.

[19]

Chris Fullwood. 2007. The effect of mediation on impression formation: A comparison of face-to-face and video-mediated conditions. Appl. Ergon. 38, 3 (May 2007), 267--273.

[20]

Christer Gobl and Ailbhe Chasaide. 2003. The role of voice quality in communication emotion, mood and attitude. Speech Commun. 40, 1--2 (Apr. 2003), 189--212.

Digital Library

[21]

Lawrence Hosman. 2002. Language and persuasion. In The Persuasion Handbook: Developments in Theory and Practice, James Dillard and Michael Pfau (Eds.). New York: Sage, 371--390.

[22]

Matthew Inglis and Juan Mejia-Ramos. 2009. The effect of authority on the persuasiveness of mathematical arguments. Cognit. Instruct. 27, 1 (2009), 25--50.

[23]

John Kane, Stefan Scherer, Matthew Aylett, Louis-Philippe Morency, and Christer Gobl. 2013a. Speaker and language independent voice quality classification applied to unlabeled corpora of expressive speech. In Proceedings of the 38th International Conference on Acoustics, Speech, and Signal Processing (ICASSP’13), IEEE, 7982--7986. 10.1109/ICASSP.2013.6639219

[24]

John Kane, Christer Gobl, Stefan Scherer, and Louis-Philippe Morency. 2013b. A comparative study of glottal open quotient estimation techniques. In Proceedings of the 14th Annual Conference of the International Speech Communication Association (Interspeech’13). 1658--1662.

[25]

Klaus Krippendorff. 2012. Content Analysis: An Introduction to Its Methodology (3^rded.). Sage, Beverly Hills, CA.

[26]

Michael LaCrosse. 1975. Nonverbal behavior and perceived counselor attractiveness and persuasiveness. Journal of Counseling Psychology 22, 6 (Nov. 1975), 563--566.

[27]

Shihong Lao and Masato Kawade. 2005. Vision-based face understanding technologies and their applications. In Advances in Biometric Person Authentication, Stan Li, Jianhuang Lai, Tieniu Tan, Guocan Feng, and Yunhong Wang (Eds.). Lecture Notes in Computer Science, Vol. 3338, Springer Berlin Heidelberg, 339--348.

Digital Library

[28]

Gwen Littlewort, Jacob Whitehill, Tingfan Wu, Ian Fasel, Mark Frank, Javier Movellan, and Marian Barlett. 2011. The computer expression recognition toolbox (CERT). In Proceedings of the 9^th IEEE International Conference on Automatic Face and Gesture Recognition (FG’11). IEEE, 298--305.

[29]

James Maddux and Ronald Rogers. 1980. Effects of source expertness, physical attractiveness, and supporting arguments on persuasion: A case of brains over beauty. J. Personal. Social Psych. 39, 2 (Aug. 1980), 235--244.

[30]

George Mahl. 1956. Disturbances and silences in the patient's speech in psychotherapy. J. Abnor. Social Psych. 53, 1 (July1956), 1--15.

[31]

Catha Maslow, Kathryn Yoselson, and Harvey London. 2011. Persuasiveness of confidence expressed via language and body language. Brit. J. Social Clin. Psych. 10, 3 (Sept. 1971), 234--240.

[32]

Winter Mason and Siddharth Suri. 2011. Conducting behavioral research on Amazon's mechanical turk. Behav. Res. Meth. 44, 1 (Mar. 2012), 1--23.

[33]

Albert Mehrabian. 1971. Silent messages. Wadsworth Publishing Company, Inc., Belmont, CA.

[34]

Albert Mehrabian and Martin Williams. 1969. Nonverbal concomitants of perceived and intended persuasiveness. J. Personality Social Psych. 13, 1 (Sept. 1969), 37--58.

[35]

Joan Meyers-Levy and Prashant Malaviya. 1999. Consumers’ processing of persuasive advertisements: An integrative framework of persuasion theories. Journal of Marketing 63, Special Issue (1999), 45--60.

[36]

Norman Miller, Geoffrey Maruyama, Rex Beaber, and Keith Valone. 1976. Speed of speech and persuasion. J. Personal. Social Psych. 34, 4 (Oct. 1976), 615--624.

[37]

Louis-Philippe Morency, Jacob Whitehill, and Javier Movellan. 2008. Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation. In Proceedings of the 8^th IEEE International Conference on Automatic Face and Gesture Recognition (FG’08). IEEE, 1--8.

[38]

Daniel O’Keefe. 2002. Persuasion: Theory and Research. Sage, Thousand Oaks, CA.

[39]

Daniel O’Keefe and Jakob Jensen. 2007. The relative persuasiveness of gain-framed loss-framed messages for encouraging disease prevention behaviors: A meta-analytic review. J. Health Commun.: Int. Perspect. 12, 7 (Oct. 2007), 623--644.

[40]

Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. 2014. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In Proceedings of the 16^th ACM International Conference on Multimodal Interaction (ICMI’14). ACM, New York, 50--57.

Digital Library

[41]

W. Barnett Pearce and Bernard Brommel. 1972. Vocalic communication in persuasion. Quart. J. Speech 58, 3 (1972), 298--306.

[42]

Richard Perloff. 2010. The Dynamics of Persuasion: Communication and Attitudes in the Twenty-First Century. Routledge, New York, NY.

[43]

Richard Petty and John Cacioppo. 1986. Communication and Persuasion: Central and Peripheral Routes to Attitude Change.: Springer-Verlag, New York.

[44]

Jeffery Pittam. 1990. The relationship between perceived persuasiveness of nasality and source characteristics for Australian and American listeners. J. Social Psych. 130, 1 (1990), 81--87.

[45]

Chanthika Pornpitakpan. 2004. The persuasiveness of source credibility: A critical review of five decades’ evidence. J. Appl. Social Psych. 34, 2 (Feb. 2004), 243--281.

[46]

Beatrice Rammstedt and Oliver John. 2007. Measuring personality in one minute or less: A 10-item short version of the big five inventory in English and German. J. Res. Personal. 41, 1 (Feb. 2007), 203--212.

[47]

Howard Rosenfeld. 1966. Approval-seeking and approval-inducing functions of verbal and nonverbal responses in the dyad. J. Personal. Social Psych. 4, 6 (Dec. 1966), 597--605.

[48]

Roni Rosenfeld. 2000. Two decades of statistical language modeling: Where do we go from here? Proc. IEEE 88, 8 (Aug. 2000), 1270--1278.

[49]

Stefan Scherer, John Kane, Christer Gobl, and Fiedhelm Schwenker. 2013. Investigating fuzzy-input fuzzy-output support vector machines for robust voice quality classification. Comput. Speech Lang. 27, 1 (Jan. 2013), 263--287.

Digital Library

[50]

Björn Schuller, Stefan Steidl, Anton Batliner, Florian Schiel, and Jarek Krajewski. 2011. The Interspeech 2011 speaker state challenge. In Proceedings of the 12^th Annual Conference of the International Speech Communication Association (Interspeech’11). 3201--3204.

[51]

Han Suk Shim, Sunghyun Park, Moitreya Chatterjee, Stefan Scherer, Kenji Sagae, and Louis-Philippe Morency. 2015. Acoustic and para-verbal indicators of persuasiveness in social multimedia. In Proceedings of the 40^th International Conference on Acoustics, Speech, and Signal Processing (ICASSP’15).

[52]

Steven Stern, John Mullennix, and Stephen Wilson. 2002. Effects of perceived disability on persuasiveness of computer-synthesized speech. J. Appl. Psych. 87, 2 (Apr. 2002), 411--417.

[53]

John Storck and Lee Sproull. 1995. Through a glass darkly: What do people learn in videoconferences? Hum. Commun. Res. 22, 2 (Dec. 1995), 197--219.

[54]

Jansen Voss. 2005. The science of persuasion: An exploration of advocacy and the science behind the art of persuasion in the courtroom. Law and Psychology Review 29 (2005), 301--327.

[55]

Ederyn Williams. 1977. Experimental comparisons of face-to-face and mediated communication: A review. Psychological Bulletin 84, 5 (Sept. 1977), 963--976.

[56]

Stephen Worchel, Virginia Andreoli, and Joe Eason. 1975. Is the medium the message? A study of the effects of media, communicator, and message characteristics on attitude change. J. Appl. Social Psych. 5, 2 (Jun. 1975), 157--172.

[57]

Yiming Yang and Jan Pedersen. 1997. A comparative study on feature selection in text categorization. In Proceedings of the 14th International Conference on Machine Learning (ICML’97). 412--420.

Digital Library

[58]

Joel Young, Craig Martell, Pranav Anand, Pedro Ortiz, and Henry Gilbert. 2011. A microtext corpus for persuasion detection in dialog. In Proceedings of the AAAI-11 Workshop on Analyzing Microtext. 80--85.

Digital Library

Cited By

Zhang TJin XMa XPeng XZhao YLiu JYu X(2025)Can question-texts improve the recognition of handwritten mathematical expressions in respondents’ solutions?Knowledge-Based Systems10.1016/j.knosys.2024.112731307(112731)Online publication date: Jan-2025
https://doi.org/10.1016/j.knosys.2024.112731
Peng HGu XLi JWang ZXu H(2024)Text-Centric Multimodal Contrastive Learning for Sentiment AnalysisElectronics10.3390/electronics1306114913:6(1149)Online publication date: 21-Mar-2024
https://doi.org/10.3390/electronics13061149
Zhao BCheng TZhang YCheng YFeng RZhang XCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)CT2C-QA: Multimodal Question Answering over Chinese Text, Table and ChartProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681053(3897-3906)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681053
Show More Cited By

Index Terms

Multimodal Analysis and Prediction of Persuasiveness in Online Social Multimedia

Recommendations

Deep multimodal fusion for persuasiveness prediction
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Persuasiveness is a high-level personality trait that quantifies the influence a speaker has on the beliefs, attitudes, intentions, motivations, and behavior of the audience. With social multimedia becoming an important channel in propagating ideas and ...
Computational Analysis of Persuasiveness in Social Multimedia: A Novel Dataset and Multimodal Prediction Approach
ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

Our lives are heavily influenced by persuasive communication, and it is essential in almost any types of social interactions from business negotiation to conversation with our friends and family. With the rapid growth of social multimedia websites, it ...
Persuasiveness in social multimedia: the role of communication modality and the challenge of crowdsourcing annotations
ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

With an exponential growth in social multimedia contents online, there is an increasing importance of understanding why and how some contents are perceived as persuasive while others are ignored. This paper outlines my research goals in understanding ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 6, Issue 3

Regular Articles and Special Issue on Highlights of ICMI 2014 (Part 2 of 2)

October 2016

136 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/2997043

Editor:
Michelle Zhou
Juji, Inc., USA

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2016

Accepted: 01 August 2016

Revised: 01 August 2016

Received: 01 June 2015

Published in TIIS Volume 6, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Science Foundation
U.S. Army

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
908
Total Downloads

Downloads (Last 12 months)287
Downloads (Last 6 weeks)49

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang TJin XMa XPeng XZhao YLiu JYu X(2025)Can question-texts improve the recognition of handwritten mathematical expressions in respondents’ solutions?Knowledge-Based Systems10.1016/j.knosys.2024.112731307(112731)Online publication date: Jan-2025
https://doi.org/10.1016/j.knosys.2024.112731
Peng HGu XLi JWang ZXu H(2024)Text-Centric Multimodal Contrastive Learning for Sentiment AnalysisElectronics10.3390/electronics1306114913:6(1149)Online publication date: 21-Mar-2024
https://doi.org/10.3390/electronics13061149
Zhao BCheng TZhang YCheng YFeng RZhang XCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)CT2C-QA: Multimodal Question Answering over Chinese Text, Table and ChartProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681053(3897-3906)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681053
Wang XWang MCui HZhang Y(2024)A dual-channel multimodal sentiment analysis framework based on three-way decisionEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109174137(109174)Online publication date: Nov-2024
https://doi.org/10.1016/j.engappai.2024.109174
Liu ZYang TChen WChen JLi QZhang J(2024)Sentiment analysis of social media comments based on multimodal attention fusion networkApplied Soft Computing10.1016/j.asoc.2024.112011164(112011)Online publication date: Oct-2024
https://doi.org/10.1016/j.asoc.2024.112011
Pandey AVishwakarma D(2024)Progress, achievements, and challenges in multimodal sentiment analysis using deep learningApplied Soft Computing10.1016/j.asoc.2023.111206152:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.asoc.2023.111206
Thomas CUdhayanan PYadav APurvaj SJayagopi D(2024)Multimodal Unsupervised Domain Adaptation for Predicting Speaker Characteristics from VideoSN Computer Science10.1007/s42979-024-02723-65:5Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1007/s42979-024-02723-6
Lv CZhang XZhang H(2024)Visual audio and textual triplet fusion network for multi-modal sentiment analysisSignal, Image and Video Processing10.1007/s11760-024-03563-w18:12(9505-9513)Online publication date: 19-Sep-2024
https://doi.org/10.1007/s11760-024-03563-w
Vinitha VJayanthi RThirukumaran SVedaiyan RRaja G(2023)An Enhanced Approach for Multi-Modal Sentimental Analysis in Natural Language ProcessingRecent Developments in Machine and Human Intelligence10.4018/978-1-6684-9189-8.ch006(73-89)Online publication date: 30-Jun-2023
https://doi.org/10.4018/978-1-6684-9189-8.ch006
孙睿(2023)A Review of Multimodal Sentiment Analysis for Measuring Visual InformationStatistics and Application10.12677/SA.2023.12101512:01(128-138)Online publication date: 2023
https://doi.org/10.12677/SA.2023.121015
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents