Nothing Special   »   [go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

Sentiment and hashtag-aware attentive deep neural network for multimodal post popularity prediction

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Social media users articulate their opinions on a broad spectrum of subjects and share their experiences through posts comprising multiple modes of expression, leading to a notable surge in such multimodal content on social media platforms. Nonetheless, accurately forecasting the popularity of these posts presents a considerable challenge. Prevailing methodologies primarily center on the content itself, thereby overlooking the wealth of information encapsulated within alternative modalities such as visual demographics, sentiments conveyed through hashtags and adequately modeling the intricate relationships among hashtags, texts, and accompanying images. This oversight limits the ability to capture emotional connection and audience relevance, significantly influencing post popularity. To address these limitations, we propose a seNtiment and hAshtag-aware attentive deep neuRal netwoRk for multimodAl posT pOpularity pRediction, herein referred to as NARRATOR that extracts visual demographics from faces appearing in images and discerns sentiment from hashtag usage, providing a more comprehensive understanding of factors influencing post popularity. Moreover, we introduce a hashtag-guided attention mechanism that leverages hashtags as navigational cues, guiding the model’s focus toward the most pertinent features of textual and visual modalities, thus aligning with target audience interests and broader social media context. Experimental results demonstrate that NARRATOR outperforms existing methods by a significant margin on two real-world datasets. Furthermore, ablation studies underscore the efficacy of integrating visual demographics, sentiment analysis of hashtags, and hashtag-guided attention mechanisms in enhancing the performance of post popularity prediction, thereby facilitating increased audience relevance, emotional engagement, and aesthetic appeal.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Data will be made available on request.

Notes

  1. https://www.flickr.com/.

  2. https://www.facebook.com/.

  3. https://www.instagram.com/.

  4. https://photutorial.com/flickr-statistics/.

  5. https://www.omnicoreagency.com/instagram-statistics/.

  6. https://stanfordnlp.github.io/CoreNLP/.

References

  1. Shen F, Xia C, Skoric M (2020) Examining the roles of social media and alternative media in social movement participation: a study of Hong Kong’s umbrella movement. Telemat Inform 47:101303. https://doi.org/10.1016/j.tele.2019.101303

    Article  Google Scholar 

  2. Anderson M, Brook A (2021) Social Media Use in 2021 | Pew Research Center

  3. Cao Q, Shen H, Gao J, Wei B, Cheng X (2020) Popularity prediction on social platforms with coupled graph neural networks. In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp 70–78. https://doi.org/10.1145/3336191.3371834

  4. Wu B, Mei T, Cheng W-H, Zhang Y (2016) Unfolding temporal dynamics: Predicting social media popularity using multi-scale temporal decomposition. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 30. https://doi.org/10.1609/aaai.v30i1.9970

  5. Gonçalves MA, Almeida JM, Santos LG, Laender AH, Almeida V (2010) On popularity in the blogosphere. IEEE Internet Comput 14(3):42–49. https://doi.org/10.1109/MIC.2010.73

    Article  MATH  Google Scholar 

  6. Majid A, Chen L, Chen G, Mirza HT, Hussain I, Woodward J (2013) A context-aware personalized travel recommendation system based on geotagged social media data mining. Int J Geogr Inform Sci 27(4):662–684. https://doi.org/10.1080/13658816.2012.696649

    Article  Google Scholar 

  7. Li C, Lu Y, Mei Q, Wang D, Pandey S (2015) Click-through prediction for advertising in twitter timeline. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1959–1968. https://doi.org/10.1145/2783258.2788582

  8. Aven BL, Burgess DA, Haynes JF, Merino JR, Moore PC (2014) Using product and social network data to improve online advertising. Google Patents. US Patent 8,843,406

  9. Roy SD, Mei T, Zeng W, Li S (2013) Towards cross-domain learning for social video popularity prediction. IEEE Trans Multimed 15(6):1255–1267. https://doi.org/10.1109/TMM.2013.2265079

    Article  MATH  Google Scholar 

  10. Gan C, Sun C, Duan L, Gong B (2016) Webly-supervised video recognition by mutually voting for relevant web images and web video frames. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pp 849–866. https://doi.org/10.1007/978-3-319-46487-9_52. Springer

  11. Kim W, Won JH, Park S, Kang J (2015) Demand forecasting models for medicines through wireless sensor networks data and topic trend analysis. Int J Distrib Sens Netw 11(9):907169. https://doi.org/10.1155/2015/907169

    Article  MATH  Google Scholar 

  12. Wang J, Xu B, Zu Y (2021) Deep learning for aspect-based sentiment analysis. In: 2021 International Conference on Machine Learning and Intelligent Systems Engineering (MLISE), pp 267–271. https://doi.org/10.1109/MLISE54096.2021.00056. IEEE

  13. Saura JR (2021) Using data sciences in digital marketing: framework, methods, and performance metrics. J Innov Knowl 6(2):92–102. https://doi.org/10.1016/j.jik.2020.08.001

    Article  MathSciNet  MATH  Google Scholar 

  14. Saura JR, Ribeiro-Soriano D, Palacios-Marqués D (2021) From user-generated data to data-driven innovation: a research agenda to understand user privacy in digital markets. Int J Inform Manag 60:102331. https://doi.org/10.1016/j.ijinfomgt.2021.102331

    Article  MATH  Google Scholar 

  15. Ribeiro-Navarrete S, Saura JR, Palacios-Marqués D (2021) Towards a new era of mass data collection: assessing pandemic surveillance technologies to preserve user privacy. Technol Forecas Soc Change 167:120681. https://doi.org/10.1016/j.techfore.2021.120681

    Article  MATH  Google Scholar 

  16. Xu K, Lin Z, Zhao J, Shi P, Deng W, Wang H (2020) Multimodal deep learning for social media popularity prediction with attention mechanism. In: MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia. https://doi.org/10.1145/3394171.3416274

  17. Lin HH, Lin JD, Ople JJM, Chen JC, Hua KL (2022) Social media popularity prediction based on multi-modal self-attention mechanisms. IEEE Access. https://doi.org/10.1109/ACCESS.2021.3136552

    Article  MATH  Google Scholar 

  18. Nguyen M-T, Le DH, Nakajima T, Yoshimi M, Thoai N (2019) Attention-based neural network: a novel approach for predicting the popularity of online content. In: 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp 329–336. https://doi.org/10.1109/HPCC/SmartCity/DSS.2019.00058. IEEE

  19. Liao D, Xu J, Li G, Huang W, Liu W, Li J (2019) Popularity prediction on online articles with deep fusion of temporal process and content features. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 33, pp 200–207. https://doi.org/10.1609/aaai.v33i01.3301200

  20. Chen J, Liang D, Zhu Z, Zhou X, Ye Z, Mo X (2019) Social media popularity prediction based on visual-textual features with xgboost. In: Proceedings of the 27th ACM International Conference on Multimedia, pp 2692–2696. https://doi.org/10.1145/3343031.335607

  21. Zhang Z, Chen T, Zhou Z, Li J, Luo J (2018) How to become instagram famous: Post popularity prediction with dual-attention. In: 2018 IEEE International Conference on Big Data (big Data), pp 2383–2392. https://doi.org/10.1109/BigData.2018.8622461. IEEE

  22. Zhang W, Wang W, Wang J, Zha H (2018) User-guided hierarchical attention network for multi-modal social image popularity prediction. In: The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018. https://doi.org/10.1145/3178876.3186026

  23. Wang J, Yang S, Zhao H, Yang Y (2023) Social media popularity prediction with multimodal hierarchical fusion model. Comput Speech Lang 80:101490. https://doi.org/10.1016/j.csl.2023.101490

    Article  MATH  Google Scholar 

  24. Caleffi P-M (2015) The ‘hashtag’: a new word or a new rule? SKASE J Theor Linguist https://doi.org/10.24093/awej/call6.6

  25. Zhang S, Yao Y, Xu F, Tong H, Yan X, Lu J (2019) Hashtag recommendation for photo sharing services. In: Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v33i01.33015805

  26. Abousaleh FS, Cheng WH, Yu NH, Tsao Y (2021) Multimodal deep learning framework for image popularity prediction on social media. IEEE Trans Cognit Dev Syst. https://doi.org/10.1109/TCDS.2020.3036690

    Article  MATH  Google Scholar 

  27. Bakhshi S, Shamma DA, Gilbert E (2014) Faces engage us: Photos with faces attract more likes and comments on instagram. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/2556288.2557403

  28. Gelli F, Uricchio T, Bertini M, Bimbo AD, Chang SF (2015) Image popularity prediction in social media using sentiment and context features. In: Proceedings of the 23rd ACM International Conference on Multimedia. https://doi.org/10.1145/2733373.2806361

  29. Li J, Gao Y, Gao X, Shi Y, Chen G (2019) Senti2pop: Sentiment-aware topic popularity prediction on social media. In: Proceedings - IEEE International Conference on Data Mining, ICDM, vol 2019-Nov. https://doi.org/10.1109/ICDM.2019.00143

  30. Mannepalli K, Singh SP, Kolli CS, Raj S, Bojja GR, Rajakumar B, Binu D (2023) Popularity prediction model with context, time and user sentiment information: an optimization assisted deep learning technique. Internat J Uncertain Fuzziness Knowl-Based Syst 31(02):283–302. https://doi.org/10.1142/S0218488523500150

    Article  Google Scholar 

  31. Yang C, Wang X, Jiang B (2020) Sentiment enhanced multi-modal hashtag recommendation for micro-videos. IEEE Access 8:78252–78264. https://doi.org/10.1109/ACCESS.2020.2989473

    Article  Google Scholar 

  32. Liao YY (2022) Leveraging hashtag networks for multimodal popularity prediction of Instagram posts. In: Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Odijk J, Piperidis S (eds) Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp 7191–7198. European Language Resources Association, Marseille, France. https://aclanthology.org/2022.lrec-1.779

  33. Arazzi M, Cotogni M, Nocera A, Virgili L (2023) Predicting tweet engagement with graph neural networks. In: Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, pp 172–180. https://doi.org/10.1145/3591106.3592294

  34. Purba KR, Asirvatham D, Murugesan RK (2021) Instagram post popularity trend analysis and prediction using hashtag, image assessment, and user history features. Int Arab J Inform Technol. https://doi.org/10.34028/iajit/18/1/10

  35. Kumar N, Yadandla A, Suryamukhi K, Ranabothu N, Boya S, Singh M (2017) Arousal prediction of news articles in social media, vol 10682 LNAI. https://doi.org/10.1007/978-3-319-71928-3_30

  36. Lin Z, Huang F, Li Y, Yang Z, Liu W (2019) A layer-wise deep stacking model for social image popularity prediction. World Wide Web. https://doi.org/10.1007/s11280-018-0590-1

  37. Cao Q, Shen H, Gao J, Wei B, Cheng X (2020). Popularity prediction on social platforms with coupled graph neural networks. https://doi.org/10.1145/3336191.3371834

  38. Mannepalli K, Singh SP, Kolli CS, Raj S, Bojja GR, Rajakumar BR, Binu D (2023) Popularity prediction model with context, time and user sentiment information: an optimization assisted deep learning technique. Int J Uncertain Fuzziness Knowl-Based Syst. https://doi.org/10.1142/S0218488523500150

    Article  Google Scholar 

  39. Tan Y, Liu F, Li B, Zhang Z, Zhang B (2022) An efficient multi-view multimodal data processing framework for social media popularity prediction. In: MM 2022 - Proceedings of the 30th ACM International Conference on Multimedia, pp 7200–7204. https://doi.org/10.1145/3503161.3551607

  40. Zappavigna M (2015) Searchable talk: the linguistic functions of hashtags. Soc Semiot 25(3):274–291. https://doi.org/10.1080/10350330.2014.996948

    Article  MATH  Google Scholar 

  41. Liu J, He Z, Huang Y (2018) Hashtag2vec: learning hashtag representation with relational hierarchical embedding model. In: IJCAI, vol 2018-Jul. https://doi.org/10.24963/ijcai.2018/480

  42. Chakrabarti P, Malvi E, Bansal S, Kumar N (2023) Hashtag recommendation for enhancing the popularity of social media posts. Soc Netw Anal Min. https://doi.org/10.1007/s13278-023-01024-9

    Article  MATH  Google Scholar 

  43. Bansal S, Gowda K, Kumar N (2023) A hybrid deep neural network for multimodal personalized hashtag recommendation. IEEE Trans Computat Soc Syst. https://doi.org/10.1109/TCSS.2022.3184307

    Article  MATH  Google Scholar 

  44. Wang J, Yang S, Zhao H, Yang Y (2023) Social media popularity prediction with multimodal hierarchical fusion model. Comput Speech Lang. https://doi.org/10.1016/j.csl.2023.101490

    Article  MATH  Google Scholar 

  45. Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota. https://doi.org/10.18653/v1/N19-1423

  46. Mikolov T, Sutskever I, Chen K, Corrado G, Dean J (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inform Process Syst

  47. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings

  48. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2010) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/cvpr.2009.5206848

  49. Serengil SI, Ozpinar A (2021) Hyperextended lightface: a facial attribute analysis framework. In: 2021 International Conference on Engineering and Emerging Technologies (ICEET). https://doi.org/10.1109/ICEET53442.2021.9659697

  50. Grootendorst M (2022) Bertopic: neural topic modeling with a class-based tf-idf procedure. https://arxiv.org/abs/2203.05794

  51. McInnes L, Healy J, Astels S (2017) hdbscan: hierarchical density based clustering. The J Open Source Softw. https://doi.org/10.21105/joss.00205

  52. McInnes L, Healy J, Saul N, Großberger L (2018) Umap: uniform manifold approximation and projection. J Open Source Softw. https://doi.org/10.21105/joss.00861

  53. Hamilton WL, Ying R, Leskovec J (2017) Inductive representation learning on large graphs. In: Neural Information Processing Systems, vol. 2017-Dec

  54. Manning CD, Surdeanu M, Bauer J, Finkel J, Bethard SJ, McClosky D (2014) The stanford corenlp natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, vol 2014-Jun. https://doi.org/10.3115/v1/p14-5010

  55. Hutto CJ, Gilbert E (2014) Vader: a parsimonious rule-based model for sentiment analysis of social media text. In: Proceedings of the International AAAI Conference on Web and Social Media. https://doi.org/10.1609/icwsm.v8i1.14550

  56. Aloufi S, Zhu S, Saddik AE (2017) On the prediction of flickr image popularity by analyzing heterogeneous social sensory data. Sensors (Switzerland). https://doi.org/10.3390/s17030631

    Article  MATH  Google Scholar 

  57. Jollife IT, Cadima J (2016) Principal component analysis: a review and recent developments. https://doi.org/10.1098/rsta.2015.0202

  58. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15

  59. Wu B, Liu B, Cheng WH, Zeng Z, Liu P, Luo J (2019) Smp challenge: an overview of social media prediction challenge 2019. In: MM 2019 - Proceedings of the 27th ACM International Conference on Multimedia. https://doi.org/10.1145/3343031.3356084

  60. Wu B, Cheng WH, Zhang Y, Huang Q, Li J, Mei T (2017) Sequential prediction of social media popularity with deep temporal context networks. In: IJCAI International Joint Conference on Artificial Intelligence, vol 0 https://doi.org/10.24963/ijcai.2017/427

  61. Ding K, Wang R, Wang S (2019) Social media popularity prediction: a multiple feature fusion approach with deep neural networks. In: Proceedings of the 27th ACM International Conference on Multimedia. https://doi.org/10.1145/3343031.3356062

  62. Gradient boost tree network based on extensive feature analysis for popularity prediction of social posts. In: Proceedings of the 31st ACM International Conference on Multimedia, pp 9451–9455 (2023). https://doi.org/10.1145/3581783.3612843

  63. Mao S, Xi W, Yu L, Lü G, Xing X, Zhou X, Wan W (2023) Enhanced catboost with stacking features for social media prediction. In: Proceedings of the 31st ACM International Conference on Multimedia, pp 9430–9435. https://doi.org/10.1145/3581783.3612839

  64. Spearman C (1904) The proof and measurement of association between two things. The Am J Psychol. https://doi.org/10.2307/1412159

    Article  MATH  Google Scholar 

Download references

Acknowledgements

We are thankful to the Prime Minister Research Fellowship (PMRF) scheme, an initiative of the Government of India for providing the Ph.D. fellowship to Shubhi Bansal (Grant ID: 2101704). We are also thankful for the Young Faculty Research Catalyzing Grant (YFRCG) scheme, an initiative by IIT Indore, for providing research grant to Dr. Nagendra Kumar (Project ID: IITI/YFRCG/2023-24/03).

Funding

There was no funding for this project.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology Indore, Khandwa Road, Simrol, Indore, Madhya Pradesh, 453552, India

    Shubhi Bansal, Mohit Kumar, Chandravardhan Singh Raghaw & Nagendra Kumar

Authors
  1. Shubhi Bansal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohit Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chandravardhan Singh Raghaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nagendra Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shubhi Bansal.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bansal, S., Kumar, M., Raghaw, C.S. et al. Sentiment and hashtag-aware attentive deep neural network for multimodal post popularity prediction. Neural Comput & Applic 37, 2799–2824 (2025). https://doi.org/10.1007/s00521-024-10755-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-024-10755-5

Keywords

Search

Navigation