research-article

Stance and Sentiment in Tweets

Authors:

Saif M. Mohammad,

Parinaz Sobhani,

Svetlana KiritchenkoAuthors Info & Claims

ACM Transactions on Internet Technology (TOIT), Volume 17, Issue 3

Article No.: 26, Pages 1 - 23

https://doi.org/10.1145/3003433

Published: 12 June 2017 Publication History

Abstract

We can often detect from a person’s utterances whether he or she is in favor of or against a given target entity—one’s stance toward the target. However, a person may express the same stance toward a target by using negative or positive language. Here for the first time we present a dataset of tweet–target pairs annotated for both stance and sentiment. The targets may or may not be referred to in the tweets, and they may or may not be the target of opinion in the tweets. Partitions of this dataset were used as training and test sets in a SemEval-2016 shared task competition. We propose a simple stance detection system that outperforms submissions from all 19 teams that participated in the shared task. Additionally, access to both stance and sentiment annotations allows us to explore several research questions. We show that although knowing the sentiment expressed by a tweet is beneficial for stance classification, it alone is not sufficient. Finally, we use additional unlabeled data through distant supervision techniques and word embeddings to further improve stance classification.

References

[1]

Pranav Anand, Marilyn Walker, Rob Abbott, Jean E. Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool! Classifying stance in online debate. In Proceedings of the Workshop on Computational Approaches to Subjectivity and Sentiment Analysis. 1--9.

Digital Library

[2]

Ion Androutsopoulos and Prodromos Malakasiotis. 2010. A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research 38, 1, 135--187.

[3]

Yoshua Bengio, Rejean Ducharme, and Pascal Vincent. 2001. A neural probabilistic language model. In Advances in Neural Information Processing Systems. 1--7.

Digital Library

[4]

Luisa Bentivogli, Peter Clark, Ido Dagan, Hoa Dang, and Danilo Giampiccolo. 2011. The seventh PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference.

[5]

Adam Bermingham and Alan Smeaton. 2011. On using Twitter to monitor political sentiment and predict election results. In Proceedings of the Workshop on Sentiment Analysis Where AI Meets Psychology. 2--10.

[6]

Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the International Conference on Machine Learning. 160--167.

Digital Library

[7]

Michael Conover, Bruno Gonçalves, Jacob Ratkiewicz, Alessandro Flammini, and Filippo Menczer. 2011a. Predicting the political alignment of Twitter users. In Proceedings of the IEEE International Conference on Privacy, Security, Risk, and Trust. 192--199.

[8]

Michael Conover, Jacob Ratkiewicz, Matthew R. Francisco, Bruno Gonçalves, Filippo Menczer, and Alessandro Flammini. 2011b. Political polarization on Twitter. In Proceedings of the International AAAI Conference on Weblogs and Social Media. 89--96.

[9]

Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In Proceedings of the Conference on Intelligent Systems for Molecular Biology. 77--86.

Digital Library

[10]

Ido Dagan and Oren Glickman. 2004. Probabilistic textual entailment: Generic applied modeling of language variability. In Proceedings of the PASCAL Workshop on Text Understanding and Mining. 26--29.

[11]

Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Morgan 8 Claypool.

[12]

Lingjia Deng, Janyce Wiebe, and Yoonjung Choi. 2014. Joint inference and disambiguation of implicit sentiments via implicature constraints. In Proceedings of the International Conference on Computational Linguistics. 79--88.

[13]

Myroslava O. Dzikovska, Rodney D. Nielsen, and Claudia Leacock. 2016. The joint student response analysis and recognizing textual entailment challenge: Making sense of student responses in educational applications. Language Resources and Evaluation 50, 1, 67--93.

Digital Library

[14]

Adam Faulkner. 2014. Automated classification of stance in student essays: An approach using stance target information and the Wikipedia link-based measure. In Proceedings of the Flairs Conference.

[15]

Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.

Digital Library

[16]

Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter Sentiment Classification Using Distant Supervision. Technical Report. Stanford University, Stanford, CA.

[17]

Jennifer Golbeck and Derek Hansen. 2011. Computing political preference among Twitter followers. In Proceedings of the Conference on Human Factors in Computing Systems. 1105--1108.

Digital Library

[18]

Kazi Saidul Hasan and Vincent Ng. 2013. Stance classification of ideological debates: Data, models, features, and constraints. In Proceedings of the International Joint Conference on Natural Language Processing. 1348--1356.

[19]

Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 168--177.

Digital Library

[20]

Stephanie D. Husby and Denilson Barbosa. 2012. Topic classification of blog posts using distant supervision. In Proceedings of the Workshop on Semantic Analysis in Social Media. 28--36.

Digital Library

[21]

Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif M. Mohammad. 2014b. NRC-Canada-2014: Detecting aspects and sentiment in customer reviews. In Proceedings of the International Workshop on Semantic Evaluation.

[22]

Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014a. Sentiment analysis of short informal texts. 50, 723--762.

Digital Library

[23]

Florian Kunneman, Christine Liebrecht, and Antal van den Bosch. 2014. The (un)predictability of emotional hashtags in Twitter. In Proceedings of the Workshop on Language Analysis for Social Media. 26--34.

[24]

Vasileios Lampos, Daniel Preotiuc-Pietro, and Trevor Cohn. 2013. A user-centric model of voting intention from social media. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 993--1003.

[25]

Bing Liu. 2015. Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press.

[26]

Bing Liu and Lei Zhang. 2012. A survey of opinion mining and sentiment analysis. In Mining Text Data, C. C. Aggarwal and C.-X. Zhai (Eds.). Springer, 415--463.

[27]

Walid Magdy, Hassan Sajjad, Tarek El-Ganainy, and Fabrizio Sebastiani. 2015. Bridging social media via distant supervision. Social Network Analysis and Mining 5, 1, 1--12.

[28]

Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 Task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the International Workshop on Semantic Evaluation.

[29]

Diana Maynard and Adam Funk. 2011. Automatic detection of political opinions in tweets. In Proceedings of the ESWC Workshop on the Semantic Web. 88--99.

Digital Library

[30]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. 3111--3119.

Digital Library

[31]

Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. 1003--1011.

Digital Library

[32]

Andriy Mnih and Geoffrey E. Hinton. 2009. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems. 1081--1088.

Digital Library

[33]

Saif M. Mohammad. 2012. #Emotional tweets. In Proceedings of the Joint Conference on Lexical and Computational Semantics. 246--255.

Digital Library

[34]

Saif M. Mohammad. 2016a. A practical guide to sentiment annotation: Challenges and solutions. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis.

[35]

Saif M. Mohammad. 2016b. Sentiment analysis: Detecting valence, emotions, and other affectual states from text. In Emotion Measurement, H. L. Meiselman (Ed.). Elsevier, 201--237.

[36]

Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 Task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation.

[37]

Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the International Workshop on Semantic Evaluation.

[38]

Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. 26--34.

Digital Library

[39]

Saif M. Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, emotion, purpose, and style in electoral tweets. Information Processing and Management 51, 480--499.

Digital Library

[40]

Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2, 1--2, 1--135.

Digital Library

[41]

Ankur P. Parikh, Hoifung Poon, and Kristina Toutanova. 2015. Grounded semantic parsing for complex knowledge extraction. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics.

[42]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, et al. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825--2830.

Digital Library

[43]

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1532--1543.

[44]

Maria Pontiki, Dimitrios Galanis, Haris Papageogiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 Task 12: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation.

[45]

Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation.

[46]

Kevin Reschke, Martin Jankowiak, Mihai Surdeanu, Christopher D. Manning, and Daniel Jurafsky. 2014. Event extraction using distant supervision. In Proceedings of the International Conference on Language Resources and Evaluation. 4527--4531.

[47]

Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1524--1534.

Digital Library

[48]

Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif M. Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval-2015 Task 10: Sentiment analysis in Twitter. In Proceedings of the International Workshop on Semantic Evaluations.

[49]

Parinaz Sobhani, Diana Inkpen, and Stan Matwin. 2015. From argumentation mining to stance classification. In Proceedings of the Workshop on Argumentation Mining. 67--77.

[50]

Parinaz Sobhani, Saif M. Mohammad, and Svetlana Kiritchenko. 2016. Detecting stance in tweets and analyzing its interaction with sentiment. In Proceedings of the Joint Conference on Lexical and Computational Semantics (^*Sem).

[51]

Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. 116--124.

Digital Library

[52]

Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for Twitter sentiment classification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 1555--1565.

[53]

Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from congressional floor-debate transcripts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 327--335.

Digital Library

[54]

Andranik Tumasjan, Timm O. Sprenger, Philipp G. Sandner, and Isabell M. Welpe. 2010. Election forecasts with Twitter: How 140 characters reflect the political landscape. Social Science Computer Review 29, 4, 402--418.

Digital Library

[55]

Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 384--394.

Digital Library

[56]

Peter D. Turney. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the Association for Computational Linguistics. 417--424.

Digital Library

[57]

Marilyn A. Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012a. Stance classification using dialogic properties of persuasion. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 592--596.

Digital Library

[58]

Marilyn A. Walker, Jean E. Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012b. A corpus for research on deliberation and debate. In Proceedings of the International Conference on Language Resources and Evaluation. 812--817.

[59]

Theresa Wilson, Zornitsa Kozareva, Preslav Nakov, Sara Rosenthal, Veselin Stoyanov, and Alan Ritter. 2013. SemEval-2013 Task 2: Sentiment analysis in Twitter. In Proceedings of the International Workshop on Semantic Evaluation.

[60]

Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. 347--354.

Digital Library

[61]

Guido Zarrella and Amy Marsh. 2016. MITRE at SemEval-2016 Task 6: Transfer learning for stance detection. In Proceedings of the International Workshop on Semantic Evaluation.

Cited By

Semeraro AVilella SImprota RDe Duro EMohammad SRuffo GStella M(2025)EmoAtlas: An emotional network analyzer of texts that merges psychological lexicons, artificial intelligence, and network scienceBehavior Research Methods10.3758/s13428-024-02553-757:2Online publication date: 27-Jan-2025
https://doi.org/10.3758/s13428-024-02553-7
Paramonov IPoletaev A(2025)Annotation of Text Corpora by Sentiment and Irony in a Project of Citizen ScienceAutomatic Control and Computer Sciences10.3103/S014641162470026358:7(797-807)Online publication date: 12-Feb-2025
https://doi.org/10.3103/S0146411624700263
Loschke CBraungardt SRieger J(2025)What motivates and demotivates energy savings in times of crisis? – An argument mining analysis using X/Twitter dataEnergy Efficiency10.1007/s12053-024-10283-018:1Online publication date: 2-Jan-2025
https://doi.org/10.1007/s12053-024-10283-0
Show More Cited By

Index Terms

Stance and Sentiment in Tweets
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Sentiment analysis
  2. Information systems applications
    1. Data mining
    2. Decision support systems
      1. Data analytics

Recommendations

Challenges for Sentence Level Opinion Detection in Blogs
ICIS '09: Proceedings of the 2009 Eigth IEEE/ACIS International Conference on Computer and Information Science

The Opinion Detection from blogs has always been a challenge for researchers. However with the introduction of Blog track in TREC 2006, a considerable improvement has been seen in this field at document level. But now it is the time when researchers are ...
Followee recommendation based on topic extraction and sentiment analysis from tweets
iiWAS '15: Proceedings of the 17th International Conference on Information Integration and Web-based Applications & Services

Twitter has become a popular social media service, accumulating and distributing vast amounts of information for its numerous users. One feature of Twitter is that it enables a user to follow other users, who can obtain the information her/his followees ...
Aggregation Methods for Proximity-Based Opinion Retrieval

The enormous amount of user-generated data available on the Web provides a great opportunity to understand, analyze, and exploit people’s opinions on different topics. Traditional Information Retrieval methods consider the relevance of documents to a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Internet Technology

ACM Transactions on Internet Technology Volume 17, Issue 3

Special Issue on Argumentation in Social Media and Regular Papers

August 2017

201 pages

ISSN:1533-5399

EISSN:1557-6051

DOI:10.1145/3106680

Editor:
Munindar P. Singh
Department of Computer Science, North Carolina State University

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2017

Accepted: 01 September 2016

Revised: 01 August 2016

Received: 01 January 2016

Published in TOIT Volume 17, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Natural Sciences and Engineering Research Council of Canada under the CREATE program

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

233
Total Citations
View Citations
2,032
Total Downloads

Downloads (Last 12 months)160
Downloads (Last 6 weeks)8

Reflects downloads up to 14 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Semeraro AVilella SImprota RDe Duro EMohammad SRuffo GStella M(2025)EmoAtlas: An emotional network analyzer of texts that merges psychological lexicons, artificial intelligence, and network scienceBehavior Research Methods10.3758/s13428-024-02553-757:2Online publication date: 27-Jan-2025
https://doi.org/10.3758/s13428-024-02553-7
Paramonov IPoletaev A(2025)Annotation of Text Corpora by Sentiment and Irony in a Project of Citizen ScienceAutomatic Control and Computer Sciences10.3103/S014641162470026358:7(797-807)Online publication date: 12-Feb-2025
https://doi.org/10.3103/S0146411624700263
Loschke CBraungardt SRieger J(2025)What motivates and demotivates energy savings in times of crisis? – An argument mining analysis using X/Twitter dataEnergy Efficiency10.1007/s12053-024-10283-018:1Online publication date: 2-Jan-2025
https://doi.org/10.1007/s12053-024-10283-0
Zhang HNguyen QBhattacharya PGao WWong LLoh BSimons JAn J(2025)Enhancing Stance Classification on Social Media Using Quantified Moral FoundationsSocial Networks Analysis and Mining10.1007/978-3-031-78541-2_19(305-319)Online publication date: 24-Jan-2025
https://doi.org/10.1007/978-3-031-78541-2_19
Zareie ABontcheva KScarton C(2025)A Lightweight Approach for User and Keyword Classification in Controversial TopicsSocial Networks Analysis and Mining10.1007/978-3-031-78538-2_21(243-253)Online publication date: 25-Jan-2025
https://doi.org/10.1007/978-3-031-78538-2_21
Dias dos Santos PHideko Goya D(2024)Detecção de Posicionamento e Rotulação Automática de Usuários do Twitter: o caso da CPI da Covid-19iSys - Brazilian Journal of Information Systems10.5753/isys.2023.300816:1Online publication date: 14-Jan-2024
https://doi.org/10.5753/isys.2023.3008
Kar PWang Z(2024)Identifying COVID-19 Fake News on Social Networks Using Deep Learning: You will not know what happens next!WSEAS TRANSACTIONS ON BUSINESS AND ECONOMICS10.37394/23207.2024.21.16021(1973-1987)Online publication date: 10-Oct-2024
https://doi.org/10.37394/23207.2024.21.160
Mathebula MModupe AMarivate V(2024)Fine-Tuning Retrieval-Augmented Generation with an Auto-Regressive Language Model for Sentiment Analysis in Financial ReviewsApplied Sciences10.3390/app14231078214:23(10782)Online publication date: 21-Nov-2024
https://doi.org/10.3390/app142310782
Liyanage CGokani RMago V(2024)GPT-4 as an X data annotator: Unraveling its performance on a stance classification taskPLOS ONE10.1371/journal.pone.030774119:8(e0307741)Online publication date: 15-Aug-2024
https://doi.org/10.1371/journal.pone.0307741
Mets MKarjus AIbrus ISchich M(2024)Automated stance detection in complex topics and small languages: The challenging case of immigration in polarizing news mediaPLOS ONE10.1371/journal.pone.030238019:4(e0302380)Online publication date: 26-Apr-2024
https://doi.org/10.1371/journal.pone.0302380
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents