Abstract
VIsual STorytelling (VIST) is a task that transforms a sequence of images into narrative text stories. A narrative story requires an understanding of the contexts and relationships among images. Our study introduces a story generation process that emphasizes creating a coherent narrative by constructing both image and narrative contexts to control the coherence. First, the image contexts are generated from the content of individual images, using image features and scene graphs that detail the elements of the images. Second, the narrative context is generated by focusing on the overall image sequence. Ensuring that each caption fits within the overall story maintaining continuity and coherence. We also introduce a narrative concept summary, which is external knowledge represented as a knowledge graph. This summary encapsulates the narrative concept of an image sequence to enhance the understanding of its overall content. Following this, both image and narrative contexts are used to generate a coherent and engaging narrative. This framework is based on Long Short-Term Memory (LSTM) with an attention mechanism. We evaluate the proposed method using the VIST dataset, and the results highlight the importance of understanding the context of an image sequence in generating coherent and engaging stories. The study demonstrates the significance of incorporating narrative context into the generation process to ensure the coherence of the generated narrative.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
https://www.flickr.com/ (Accessed Oct. 21, 2024).
References
Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of 43rd Annual Meeting of the Association for Computational Linguistics, pp. 65–72 (2005)
Chen, H., Huang, Y., Takamura, H., Nakayama, H.: Commonsense knowledge aware concept selection for diverse and informative visual storytelling. In: Proceedings of 35th AAAI Conference on Artificial Intelligence, pp. 999–1008 (2021)
Chen, W., Li, X., Su, J., Zhu, G., Li, Y., Ji, Y., Liu, C.: TARN-VIST: Topic aware reinforcement network for VIsual STorytelling. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, pp. 15617–15628 (2024)
Cong, Y., Yang, M.Y., Rosenhahn, B.: RelTR: relation TRansformer for scene graph generation. IEEE Trans. Pattern Anal. Mach. Intell. 45(9), 11169–11183 (2023)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Dunne, C., Shneiderman, B.: Motif Simplification: improving network visualization readability with fan, connector, and clique glyphs. In: Proceedings of 31st Annual SIGCHI Conference on Human Factors in Computing Systems, pp. 3247–3256 (2013)
Girshick, R.: Fast R-CNN. In: Proceedings of 15th IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)
Graves, A., Schmidhuber, J.: Bidirectional LSTM networks for improved phoneme classification and recognition. In: Proceedings of 15th International Conference on Artificial Neural Networks, pp. 799–804 (2005)
Han, X., Yang, J., Hu, H., Zhang, L., Gao, J., Zhang, P.: Image Scene Graph Generation (SGG) benchmark. Comput. Res. Reposit. arXiv Preprint, arXiv:2107.12604 (Jul 2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hsu, C.C., et al.: Knowledge-enriched visual storytelling. In: Proceedings of 34th AAAI Conference on Artificial Intelligence, pp. 7952–7960 (2020)
Hsu, C.Y., Chu, Y.W., Huang, T.H., Ku, L.W.: Plot and rework: modeling storylines for visual storytelling. In: Proceedings of 2021 Findings Association for Computational Linguistics, 11th International Joint Conference on Natural Language Processing, pp. 4443–4453 (2021)
Huang, T.K., et al.: Visual storytelling. In: Proceedings of 15th North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1233–1239 (2016)
Kim, T., Heo, M., Son, S., Park, K., Zhang, B.: GLAC Net: GLocal attention cascading networks for multi-image cued story generation. In: Proceedings of 17th North American Chapter of the Association for Computational Linguistics (Workshop), pp. 1–6 (2018)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of 3rd International Conference on Learning Representations, pp. 1–13 (2014)
Krishna, R., et al.: Visual Genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 123(1), 32–73 (2017)
Kuznetsova, A., et al.: The Open Images dataset V4: unified image classification, object detection, and visual relationship detection at scale. Int. J. Comput. Vis. 128(7), 1956–1981 (2020)
Li, T., Wang, H., He, B., Chen, C.W.: Knowledge-enriched attention network with group-wise semantic for visual storytelling. IEEE Trans. Pattern Anal. Mach. Intell. 45(7), 8634–8645 (2022)
Liu, H., et al.: AOG-LSTM: an adaptive attention neural network for visual storytelling. Neurocomputing 552(126486), 1–13 (2023)
McCarthy, P.M., Jarvis, S.: MTLD, VOCD-D, and HD-D: a validation study of sophisticated approaches to lexical diversity assessment. Behav. Res. Methods 42(2), 381–392 (2010)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceeding of 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002)
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1532–1543 (2014)
Sellam, T., Das, D., Parikh, A.P.: BLEURT: Learning robust metrics for text generation. In: Proceedings of 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881–7892 (2020)
Speer, R., Chin, J., Havasi, C.: ConceptNet 5.5: an open multilingual graph of general knowledge. In: Proceedings of 31st AAAI Conference on Artificial Intelligence, pp. 4444–4451 (2017)
Tang, K., Niu, Y., Huang, J., Shi, J., Zhang, H.: Unbiased scene graph generation from biased training. In: Proceedings of 2020 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3716–3725 (2020)
Wang, B., Ma, L., Zhang, W., Jiang, W., Zhang, F.: Hierarchical photo-scene encoder for album storytelling. In: Proceedings of 33rd AAAI Conference on Artificial Intelligence, pp. 8909–8916 (2019)
Wang, E., Han, C., Poon, J.: SCO-VIST: social interaction COmmonsense knowledge-based VIsual STorytelling. In: Proceedings of 18th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1602–1616 (2024)
Wang, E., Han, S.C., Poon, J.: RoViST: learning robust metrics for visual STorytelling. In: Proceedings of 13th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2691–2702 (2022)
Wang, R., Wei, Z., Li, P., Zhang, Q., Huang, X.: Storytelling from an image stream using scene graphs. In: Proc. 34th AAAI Conference on Artificial Intelligence & 32nd Innovative Applications of Artificial Intelligence Conference, pp. 9185–9192 (2020)
Wang, X., Chen, W., Wang, Y., Wang, W.Y.: No metrics are perfect: adversarial reward learning for visual storytelling. In: Proceedings of 56th Annual Meeting of the Association for Computational Linguistics, pp. 899–909 (2018)
Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)
Xu, C., Yang, M., Li, C., Shen, Y., Ao, X., Xu, R.: Imagine, reason and write: visual storytelling with graph knowledge and relational reasoning. In: Proceedings of 35th AAAI Conference on Artificial Intelligence, pp. 3022–3029 (2021)
Yu, L., Bansal, M., Berg, T.L.: Hierarchically-attentive RNN for album summarization and storytelling. In: Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing, pp. 966–971 (2017)
Zellers, R., Yatskar, M., Thomson, S., Choi, Y.: Neural Motifs: scene graph parsing with global context. In: Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831–5840 (2018)
Zhang, M., Cui, Z., Neumann, M., Chen, Y.: An end-to-end deep learning architecture for graph classification. In: Proceedings of 32nd AAAI Conference on Artificial Intelligence, pp. 4438–4445 (2018)
Acknowledgments
This work was supported in part by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan, through Grants-in-Aid for Scientific Research JP21H03519, JP23K16945, and JP24H00733.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Phueaksri, I., Kastner, M.A., Kawanishi, Y., Komamizu, T., Ide, I. (2025). Towards Visual Storytelling by Understanding Narrative Context Through Scene-Graphs. In: Ide, I., et al. MultiMedia Modeling. MMM 2025. Lecture Notes in Computer Science, vol 15523. Springer, Singapore. https://doi.org/10.1007/978-981-96-2071-5_17
Download citation
DOI: https://doi.org/10.1007/978-981-96-2071-5_17
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-96-2070-8
Online ISBN: 978-981-96-2071-5
eBook Packages: Computer ScienceComputer Science (R0)