research-article

Joint modeling of causal phrases-sentiments-aspects using Hierarchical Pitman Yor Process

Authors:

V.R.P.S. Sastry Yadavilli,

Karthick Seshadri,

Nagesh Bhattu S.Authors Info & Claims

Volume 61, Issue 4

https://doi.org/10.1016/j.ipm.2024.103753

Published: 18 July 2024 Publication History

Abstract

Traditional sentiment-aware topic models assume that topic or sentiment transition occurs from either a sentence to the next sentence or from a word to the next word. Such models cannot capture a topic or sentiment transition at phrase boundaries. Further, most of the models adopt a sentiment lexicon to initialize sentiment priors and this approach induces coverage problems. To overcome the above-cited limitations, we have proposed a topic model that extracts aspects, sentiments, and causal phrases simultaneously by leveraging Hierarchical Pitman Yor Process (HPYP) that is modified using a sentiment component, a word-tagger to guide the causal phrase generation and a sentiment prior initialized through a sequential model to address coverage problems. We have evaluated our model on six datasets and found that the proposed model outperforms the baselines in terms of perplexity by 14%, topical coherence by 20%, topic diversity by 5%, sentiment classification task’s accuracy by 4% and, precision, recall and F1 score by 2%. Ablation studies assert that sequence model based sentiment prior initialization results in increasing the accuracy of sentiment classification by 2%.

Highlights

•

A first-of-its kind model for joint mining of topics, causal phrases, and sentiments.

•

A first-of-its-kind adaptation of HPYP for phrase generation using a word tagger.

•

VAE-LSTM induced prior yielded an improvement of 2% over static sentiment prior.

•

An increase of 14% w.r.t. perplexity and 20% w.r.t. topical coherence were observed.

•

An increase of 5% w.r.t. topical diversity and 4% w.r.t. accuracy were achieved.

References

[1]

Akhtar M.S., Garg T., Ekbal A., Multi-task learning for aspect term extraction and aspect sentiment classification, Neurocomputing 398 (2020) 247–256.

[2]

Başarslan M.S., Kayaalp F., MBi-GRUMCONV: A novel multi bi-GRU and multi CNN-based deep learning model for social media sentiment analysis, Journal of Cloud Computing 12 (1) (2023) 5.

[3]

Blei D.M., Ng A.Y., Jordan M.I., Latent dirichlet allocation, Journal of Machine Learning Research 3 (Jan) (2003) 993–1022.

[4]

Bouma G., Normalized (pointwise) mutual information in collocation extraction, Proceedings of GSCL 30 (2009) 31–40.

[5]

Cai Y., Ke W., Cui E., Yu F., A deep recommendation model of cross-grained sentiments of user reviews and ratings, Information Processing & Management 59 (2) (2022).

Digital Library

[6]

Churchill R., Singh L., Using topic-noise models to generate domain-specific topics across data sources, Knowledge and Information Systems 65 (5) (2023) 2159–2186.

[7]

De Diego I.M., Redondo A.R., Fernández R.R., Navarro J., Moguerza J.M., General performance score for classification problems, Applied Intelligence 52 (10) (2022) 12049–12063.

[8]

Demšar J., Statistical comparisons of classifiers over multiple data sets, The Journal of Machine Learning Research 7 (2006) 1–30.

Digital Library

[9]

Dieng A.B., Ruiz F.J., Blei D.M., Topic modeling in embedding spaces, Transactions of the Association for Computational Linguistics 8 (2020) 439–453.

[10]

Geeganage D.T.K., Xu Y., Li Y., Semantic-based topic representation using frequent semantic patterns, Knowledge-Based Systems 216 (2021).

[11]

Geeganage D.K., Xu Y., Li Y., A semantics-enhanced topic modelling technique: Semantic-LDA, ACM Transactions on Knowledge Discovery from Data (2024).

[12]

Griffiths T.L., Steyvers M., Finding scientific topics, Proceedings of the National Academy of Sciences 101 (suppl_1) (2004) 5228–5235.

[13]

Hammi S., Hammami S.M., Belguith L.H., Advancing aspect-based sentiment analysis with a novel architecture combining deep learning models CNN and bi-RNN with the machine learning model SVM, Social Network Analysis and Mining 13 (1) (2023) 117.

[14]

Huang J., Peng M., Li P., Hu Z., Xu C., Improving biterm topic model with word embeddings, World Wide Web 23 (6) (2020) 3099–3124.

[15]

Huang H., Tang Y.-K., Shi X., Mao X.-L., Dependency-aware neural topic model, Information Processing & Management 61 (1) (2024).

[16]

Jiang, Q., Chen, L., Xu, R., Ao, X., & Yang, M. (2019). A challenge dataset and effective models for aspect-based sentiment analysis. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (pp. 6280–6285).

[17]

Kaibi I., Satori H., et al., A comparative evaluation of word embeddings techniques for twitter sentiment analysis, in: 2019 international conference on wireless technologies, embedded and intelligent systems, IEEE, 2019, pp. 1–4.

[18]

Kiritchenko, S., Mohammad, S., & Salameh, M. (2016). Semeval-2016 task 7: Determining sentiment intensity of english and arabic phrases. In Proceedings of the 10th international workshop on semantic evaluation (pp. 42–51).

[19]

Li F., Huang M., Zhu X., Sentiment analysis with global topics and local dependency, Proceedings of the AAAI conference on artificial intelligence, vol. 24, 2010, pp. 1371–1376.

[20]

Li Q., Peng H., Li J., Xia C., Yang R., Sun L., et al., A survey on text classification: From traditional to deep learning, ACM Transactions on Intelligent Systems and Technology 13 (2) (2022) 1–41.

[21]

Lin S., Clark R., Birke R., Schönborn S., Trigoni N., Roberts S., Anomaly detection for time series using vae-lstm hybrid model, in: ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing, IEEE, 2020, pp. 4322–4326.

[22]

Lin C., He Y., Everson R., Ruger S., Weakly supervised joint sentiment-topic detection from text, IEEE Transactions on Knowledge and Data engineering 24 (6) (2011) 1134–1145.

Digital Library

[23]

Lin H., Zuo Y., Liu G., Li H., Wu J., Wu Z., A pseudo-document-based topical N-grams model for short texts, World Wide Web 23 (2020) 3001–3023.

[24]

Lindsey, R., Headden, W., & Stipicevic, M. (2012). A phrase-discovering topic model using hierarchical pitman-yor processes. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning (pp. 214–222).

[25]

Lu Q., Sun X., Long Y., Gao Z., Feng J., Sun T., Sentiment analysis: Comprehensive reviews, recent advances, and open challenges, IEEE Transactions on Neural Networks and Learning Systems (2023).

[26]

Luo Y., Tang R.L., Understanding hidden dimensions in textual reviews on airbnb: An application of modified latent aspect rating analysis (LARA), International Journal of Hospitality Management 80 (2019) 144–154.

[27]

Mcauliffe J., Blei D., Supervised topic models, Advances in neural information processing systems, vol. 20, 2007, pp. 121–128.

[28]

Mei, Q., Ling, X., Wondra, M., Su, H., & Zhai, C. (2007). Topic sentiment mixture: modeling facets and opinions in weblogs. In Proceedings of the 16th international conference on world wide web (pp. 171–180).

[29]

Meng, Y., Huang, J., Wang, G., Wang, Z., Zhang, C., Zhang, Y., et al. (2020). Discriminative topic mining via category-name guided text embedding. In Proceedings of the web conference 2020 (pp. 2121–2132).

[30]

Meng, Y., Zhang, Y., Huang, J., Zhang, Y., & Han, J. (2022). Topic discovery via latent space clustering of pretrained language model representations. In Proceedings of the ACM web conference 2022 (pp. 3143–3152).

[31]

Meng, Y., Zhang, Y., Huang, J., Zhang, Y., Zhang, C., & Han, J. (2020). Hierarchical topic mining via joint spherical tree and text embedding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 1908–1917).

[32]

Mimno, D., Wallach, H., Talley, E., Leenders, M., & McCallum, A. (2011). Optimizing semantic coherence in topic models. In Proceedings of the 2011 conference on empirical methods in natural language processing (pp. 262–272).

[33]

Moghaddam, S., & Ester, M. (2011). ILDA: Interdependent LDA model for learning latent aspects and their ratings from online product reviews. In Proceedings of the 34th international ACM SIGIR conference on research and development in information retrieval (pp. 665–674).

[34]

Nan F., Ding R., Nallapati R., Xiang B., Topic modeling with wasserstein autoencoders, in: Korhonen A., Traum D., Màrquez L. (Eds.), Proceedings of the 57th annual meeting of the association for computational linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 6345–6381,. URL https://aclanthology.org/P19-1640.

[35]

Nielsen, F. A. r. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In Proceedings of the ESWC2011 workshop on ’making sense of microposts: big things come in small packages (pp. 93–98).

[36]

Osmani A., Mohasefi J.B., Opinion mining using enriched joint sentiment-topic model, International Journal of Information Technology and Decision Making 22 (01) (2023) 313–375.

[37]

Osmani A., Mohasefi J.B., Gharehchopogh F.S., Enriched latent dirichlet allocation for sentiment analysis, Expert Systems 37 (4) (2020).

[38]

Ozyurt B., Akcayol M.A., A new topic modeling based approach for aspect extraction in aspect based sentiment analysis: SS-LDA, Expert Systems with Applications 168 (2021).

[39]

Pimpalkar A., et al., MBiLSTMGloVe: Embedding GloVe knowledge into the corpus using multi-layer BiLSTM deep learning model for social media sentiment analysis, Expert Systems with Applications 203 (2022).

[40]

Poria S., Hazarika D., Majumder N., Mihalcea R., Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research, IEEE Transactions on Affective Computing (2020).

[41]

Ramadhan W., Novianty S.A., Setianingsih S.C., Sentiment analysis using multinomial logistic regression, in: 2017 international conference on control, electronics, renewable energy and communications, IEEE, 2017, pp. 46–49.

[42]

Ruck D.W., Rogers S.K., Kabrisky M., Feature selection using a multilayer perceptron, Journal of Neural Network Computing 2 (2) (1990) 40–48.

[43]

Rupapara V., Rustam F., Amaar A., Washington P.B., Lee E., Ashraf I., Deepfake tweets classification using stacked Bi-LSTM and words embedding, PeerJ Computer Science 7 (2021).

[44]

Seilsepour A., Ravanmehr R., Nassiri R., Topic sentiment analysis based on deep neural network using document embedding technique, The Journal of Supercomputing (2023) 1–39.

[45]

Shen D., Qin C., Wang C., Dong Z., Zhu H., Xiong H., Topic modeling revisited: A document graph-based neural network perspective, Advances in neural information processing systems, vol. 34, 2021, pp. 14681–14693.

[46]

Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., et al. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing (pp. 1631–1642).

[47]

Sokolova M., Lapalme G., A systematic analysis of performance measures for classification tasks, Information Processing & Management 45 (4) (2009) 427–437.

Digital Library

[48]

Terragni S., Fersini E., Messina E., Constrained relational topic models, Information Sciences 512 (2020) 581–594.

Digital Library

[49]

Wan C., Peng Y., Xiao K., Liu X., Jiang T., Liu D., An association-constrained LDA model for joint extraction of product aspects and opinions, Information Sciences 519 (2020) 243–259.

[50]

Wang, Q., Wen, Z., Zhao, Q., Yang, M., & Xu, R. (2021). Progressive self-training with discriminator for aspect term extraction. In Proceedings of the 2021 conference on empirical methods in natural language processing (pp. 257–268).

[51]

Wang D., Xu Y., Li M., Duan Z., Wang C., Chen B., et al., Knowledge-aware Bayesian deep topic model, Advances in Neural Information Processing Systems 35 (2022) 14331–14344.

[52]

Wang R., Zhou D., He Y., Atm: Adversarial-neural topic model, Information Processing & Management 56 (6) (2019).

Digital Library

[53]

Wilson T., Wiebe J., Hoffmann P., Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis, Computational Linguistics 35 (3) (2009) 399–433.

[54]

Wu C., Kanoulas E., de Rijke M., Learning entity-centric document representations using an entity facet topic model, Information Processing & Management 57 (3) (2020).

[55]

Xiong S., Wang K., Ji D., Wang B., A short text sentiment-topic model for product reviews, Neurocomputing 297 (2018) 94–102.

[56]

Yadavilli V.S., Seshadri K., Explainable sentiment analysis for product reviews using causal graph embeddings, Sādhanā 47 (4) (2022) 241.

[57]

Yin Y., Jin Z., Document sentiment classification based on the word embedding, in: 2015 4th international conference on mechatronics, materials, chemistry and computer engineering, Atlantis Press, 2015, pp. 456–461.

[58]

Yu L.-C., Wang J., Lai K.R., Zhang X., Pipelined neural networks for phrase-level sentiment intensity prediction, IEEE Transactions on Affective Computing 11 (3) (2018) 447–458.

[59]

Yuan L., Wang J., Yu L.-C., Zhang X., Encoding syntactic information into transformers for aspect-based sentiment triplet extraction, IEEE Transactions on Affective Computing (2023).

[60]

Zhang Y., Wan C., Xiao K., Wan Q., Liu D., Liu X., rHDP: An aspect sharing-enhanced hierarchical topic model for multi-domain corpus, ACM Transactions on Information Systems 42 (3) (2023) 1–31.

[61]

Zhou X., Bu J., Zhou S., Yu Z., Zhao J., Yan X., Improving topic disentanglement via contrastive learning, Information Processing & Management 60 (2) (2023).

[62]

Zhou J., Huang J.X., Chen Q., Hu Q.V., Wang T., He L., Deep learning for aspect-level sentiment classification: survey, vision, and challenges, IEEE Access 7 (2019) 78454–78483.

[63]

Zhou T., Law K., Creighton D., A weakly-supervised graph-based joint sentiment topic model for multi-topic sentiment analysis, Information Sciences 609 (2022) 1030–1051.

[64]

Zhou K., Long F., Sentiment analysis of text based on CNN and bi-directional LSTM model, in: 2018 24th international conference on automation and computing, IEEE, 2018, pp. 1–5.

[65]

Zhu B., Cai Y., Ren H., Graph neural topic model with commonsense knowledge, Information Processing & Management 60 (2) (2023).

Recommendations

Topic models with power-law using Pitman-Yor process
KDD '10: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining

One important approach for knowledge discovery and data mining is to estimate unobserved variables because latent variables can indicate hidden specific properties of observed data. The latent factor model assumes that each item in a record has a latent ...
A joint model for topic-sentiment modeling from text
SAC '15: Proceedings of the 30th Annual ACM Symposium on Applied Computing

Traditional topic models, like LDA and PLSA, have been efficiently extended to capture further aspects of text in addition to the latent topics (e.g., time evolution, sentiment etc.). In this paper, we discuss the issue of joint topic-sentiment ...
Jointly Discovering Fine-grained and Coarse-grained Sentiments via Topic Modeling
MM '14: Proceedings of the 22nd ACM international conference on Multimedia

The ever-increasing user-generated contents in social media and other web services make it highly desirable to discover opinions of users on all kinds of topics. Motivated by the assumption that individual word and paragraph in documents will deliver ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Information Processing and Management: an International Journal

Information Processing and Management: an International Journal Volume 61, Issue 4

Jul 2024

1167 pages

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 18 July 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents