research-article

Modeling and Computational Characterization of Twitter Customer Service Conversations

Authors:

Mansurul Bhuiyan,

Pritam Gundecha,

Rama AkkirajuAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 9, Issue 2-3

Article No.: 18, Pages 1 - 28

https://doi.org/10.1145/3213014

Published: 18 March 2019 Publication History

Abstract

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understanding trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained “dialogue acts” frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real time, and showcase this using our “PredDial” portal. We characterize differences between customer and agent behavior in Twitter customer service conversations and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes and present actionable rules based on our findings. We explore the correlations between different dialogue acts and the outcome of the conversations in detail using an actionable-rule discovery task by leveraging a state-of-the-art sequential rule mining algorithm while modeling a set of conversations as a set of sequences. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.

References

[1]

James Allen, George Ferguson, and Amanda Stent. 2001. An architecture for more realistic conversational systems. In Proceedings of the 6th International Conference on Intelligent User Interfaces. ACM, 1--8.

Digital Library

[2]

Yasemin Altun, Ioannis Tsochantaridis, and Thomas Hofmann. 2003. Hidden Markov support vector machines. In International Conference on Machine Learning (ICML’03).

Digital Library

[3]

John L. Austin. 1962. How to Do Things with Words. Harvard University Press, Cambridge, MA.

[4]

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media.

Digital Library

[5]

Kristy Elizabeth Boyer, Eun Young Ha, Robert Phillips, Michael D. Wallis, Mladen A. Vouk, and James C. Lester. 2010. Dialogue act modeling in a complex task-oriented domain. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL’10). Association for Computational Linguistics, 297--305.

Digital Library

[6]

Harry Bunt, Jan Alexandersson, Jean Carletta, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David Traum. 2010. Towards an ISO standard for dialogue act annotation. In 7th Conference on International Language Resources and Evaluation (LREC’10), 2548--2555.

[7]

Mark G. Core and James Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In AAAI Fall Symposium on Communicative Action in Humans and Machines, Vol. 56.

[8]

Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378--382.

[9]

Philippe Fournier-Viger, Antonio Gomariz, Manuel Campos, and Rincy Thomas. 2014. Fast vertical mining of sequential patterns using co-occurrence information. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. 40--52.

[10]

Milica Gasic, Catherine Breslin, Matthew Henderson, Dongho Kim, Martin Szummer, Blaise Thomson, Pirros Tsiakoulis, and Steve Young. 2013. POMDP-based dialogue manager adaptation to extended domains. In Proceedings of the SIGDIAL 2013 Conference. Association for Computational Linguistics, 214--222.

[11]

Jonathan Herzig, Guy Feigenblat, Michal Shmueli-Scheuer, David Konopnicki, and Anat Rafaeli. 2016. Predicting customer satisfaction in customer support conversations in social media using affective features. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization (UMAP’16). ACM, New York, 115--119. Retrieved from

Digital Library

[12]

Jonathan Herzig, Guy Feigenblat, Michal Shmueli-Scheuer, David Konopnicki, Anat Rafaeli, Daniel Altman, and David Spivak. 2016. Classifying emotions in customer support dialogues in social media. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, 64--73.

[13]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computing 9, 8 (Nov. 1997), 1735--1780.

Digital Library

[14]

Courtenay Honeycutt and Susan C. Herring. 2009. Beyond microblogging: Conversation and collaboration via Twitter. In Proceedings of the 42nd Hawai’i International Conference on System Sciences (HICSS-42’09).IEEE Computer Society, 1--10.

Digital Library

[15]

Edward Ivanovic. 2005. Dialogue act tagging for instant messaging chat sessions. Proceedings of the Association for Computational Linguistics Student Research Workshop (ACL’05), 79--84.

Digital Library

[16]

Edward Ivanovic. 2006. Using dialogue acts to suggest responses in support services via instant messaging. Australasian Language Technology Workshop. 159--160.

[17]

Edward Ivanovic. 2008. Automatic instant messaging dialogue using statistical models and dialogue acts. Master’s Research Thesis.

[18]

Thorsten Joachims, Thomas Finley, and Chun-Nam John Yu. 2009. Cutting-plane training of structural SVMs. Machine Learning 77, 1 (Oct. 2009), 27--59. Retrieved from

Digital Library

[19]

Shafiq R. Joty and Enamul Hoque. 2016. Speech act modeling of written asynchronous conversations with task-specific embeddings and conditional structured models. In Association for Computer Linguistics (ACL'16). 1746--1756.

[20]

Dan Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard-DAMSL Labeling Project Coder’s Manual, Draft 13. Technical Report 97-02, University of Colorado, Institute of Cognitive Science, Boulder, CO.

[21]

Su Nam Kim, Lawrence Cavedon, and Timothy Baldwin. 2010. Classifying dialogue acts in one-on-one live chats. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP’10). Association for Computational Linguistics, 862--871.

Digital Library

[22]

Su Nam Kim, Lawrence Cavedon, and Timothy Baldwin. 2012. Classifying dialogue acts in multi-party live chats. In 26th Pacific Asia Conference on Language, Information and Computation, 463--472.

[23]

Tina Klüwer, Hans Uszkoreit, and Feiyu Xu. 2010. Using syntactic and semantic based relations for dialogue act recognition. In Proceedings of the 26th International Conference on Computational Linguistics (COLING'10), 570--578.

Digital Library

[24]

John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning (ICML’01). Morgan Kaufmann Publishers, San Francisco, CA, 282--289.

Digital Library

[25]

J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159--174.

[26]

Ramesh Manuvinakurike, Maike Paetzel, Cheng Qu, David Schlangen, and David DeVault. 2016. Toward incremental dialogue act segmentation in fast-paced interactive dialogue systems. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, 252--262.

[27]

Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text (CAAGET’10). Association for Computational Linguistics, 26--34.

Digital Library

[28]

Ralph A. Morelli, Joseph D. Bronzino, and J. W. Goethe. 1991. A computational speech-act model of human-computer conversations. In Proceedings of the 1991 IEEE 17th Annual Northeast Bioengineering Conference, 1991. 263--264.

[29]

Shereen Oraby, Pritam Gundecha, Jalal Mahmud, Mansurul Bhuiyan, and Rama Akkiraju. 2017. “How may I help you?”: Modeling Twitter customer service conversations using fine-grained dialogue acts. In Proceedings of the 22nd International Conference on Intelligent User Interfaces (IUI’17). ACM, New York, 343--355. Retrieved from

Digital Library

[30]

Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[31]

Martin L. Puterman. 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley 8 Sons, Inc., New York.

[32]

Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of Twitter conversations. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 172--180.

Digital Library

[33]

Harvey Sacks. 1992. In Harvey Sacks: Lectures on Conversation, Gail Jefferson (Ed.). Blackwell Publishing.

[34]

Amanda Schiffrin. 2005. Modelling speech acts in conversational discourse (PhD thesis). University of Leeds, Department of Computing, May 2005.

[35]

John R. Searle. 1975. A taxonomy of illocutionary acts. In Language Mind and Knowledge, Vol. 7. University of Minnesota Press, 344--369.

[36]

Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, C. V. Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics 26, 3 (2000), 339--373. arXiv:cs/0006023

Digital Library

[37]

Ming Sun, Yun-Nung Chen, and Alexander I. Rudnicky. 2016. An intelligent assistant for high-level task understanding. In Proceedings of the 21st International Conference on Intelligent User Interfaces. ACM, 169--174.

Digital Library

[38]

Gokhan Tur, Umit Guz, and Dilek Hakkani-Tür. 2006. Model adaptation for dialog act tagging. In Proceedings of the 2006 IEEE ACL Spoken Language Technology Workshop (SLT’06). 94--97.

[39]

Soroush Vosoughi and Deb Roy. 2013. A semi-automatic method for efficient detection of stories on social media. International Association for the Advancement of Artificial Intelligence Conference (AAAI) on Web and Social Media (ICWSM'16). 707--710.

[40]

Soroush Vosoughi and Deb Roy. 2016. Tweet acts: A speech act classifier for twitter. International Association for the Advancement of Artificial Intelligence Conference (AAAI) on Web and Social Media (ICWSM'16). 711--714.

[41]

Mohammed J. Zaki and Wagner Meira Jr. 2014. Data Mining and Analysis: Fundamental Concepts and Algorithms. Cambridge University Press, New York.

Digital Library

[42]

Elina Zarisheva and Tatjana Scheffler. 2015. Dialog act annotation for Twitter conversations. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, 114--123.

[43]

Renxian Zhang, Dehong Gao, and Wenjie Li. 2011. What are Tweeters doing: Recognizing speech acts in Twitter. Analyzing Microtext WS-11-05 (2011), 86--91.

Digital Library

Cited By

Li CZhang Y(2023)Construction of Intelligent Customer Service System on E-Commerce Platform and Its Impact on User Experience2023 International Conference on Network, Multimedia and Information Technology (NMITCON)10.1109/NMITCON58196.2023.10276323(1-7)Online publication date: 1-Sep-2023
https://doi.org/10.1109/NMITCON58196.2023.10276323
Cheng XQiao LYang BLi Z(2023)An investigation on the influencing factors of elderly people's intention to use financial AI customer serviceInternet Research10.1108/INTR-06-2022-040234:3(690-717)Online publication date: 16-Mar-2023
https://doi.org/10.1108/INTR-06-2022-0402
Hellman JChen JUddin MCheng JGuo JBaldassarre MPrechelt LRodríguez-Pérez G(2022)Characterizing user behaviors in open-source software user forumsProceedings of the 15th International Conference on Cooperative and Human Aspects of Software Engineering10.1145/3528579.3529178(46-55)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3528579.3529178
Show More Cited By

Index Terms

Modeling and Computational Characterization of Twitter Customer Service Conversations
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification

Recommendations

"How May I Help You?": Modeling Twitter Customer ServiceConversations Using Fine-Grained Dialogue Acts
IUI '17: Proceedings of the 22nd International Conference on Intelligent User Interfaces

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understand trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop ...
Customer Opinion Summarization Based on Twitter Conversations
WIMS '16: Proceedings of the 6th International Conference on Web Intelligence, Mining and Semantics

As Twitter gains popularity, millions of messages are generated every day via this platform allowing people to communicate with each other through daily chatter and public conversations. This social content is usually considered more subjective than ...
Modeling Human Intelligence in Customer-Agent Conversation Using Fine-Grained Dialogue Acts
Natural Language Processing and Chinese Computing
Abstract
Smart service chatbot, aiming to provide efficient, reliable and natural customer service, has grown rapidly in recent years. The understanding of human-agent conversation, especially modeling the conversational behavior, is essential to enhance ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 9, Issue 2-3

Special Issue on Highlights of ACM IUI 2017

September 2019

324 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/3320251

Editor:
Michelle X. Zhou
Juji, Inc., USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2019

Accepted: 01 March 2019

Revised: 01 November 2017

Received: 01 June 2017

Published in TIIS Volume 9, Issue 2-3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
336
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)1

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li CZhang Y(2023)Construction of Intelligent Customer Service System on E-Commerce Platform and Its Impact on User Experience2023 International Conference on Network, Multimedia and Information Technology (NMITCON)10.1109/NMITCON58196.2023.10276323(1-7)Online publication date: 1-Sep-2023
https://doi.org/10.1109/NMITCON58196.2023.10276323
Cheng XQiao LYang BLi Z(2023)An investigation on the influencing factors of elderly people's intention to use financial AI customer serviceInternet Research10.1108/INTR-06-2022-040234:3(690-717)Online publication date: 16-Mar-2023
https://doi.org/10.1108/INTR-06-2022-0402
Hellman JChen JUddin MCheng JGuo JBaldassarre MPrechelt LRodríguez-Pérez G(2022)Characterizing user behaviors in open-source software user forumsProceedings of the 15th International Conference on Cooperative and Human Aspects of Software Engineering10.1145/3528579.3529178(46-55)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3528579.3529178
Wei YLu WCheng QJiang TLiu S(2022)How humans obtain information from AIInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10283859:2Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1016/j.ipm.2021.102838

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents