Abstract
Research on textual style transfer has observed that the concept of style can vary across domains. This research examines the encoding of style across the sentiment and formality domains and observes that formality appears to be more globally encoded, and sentiment more locally encoded. The work also shows how the encoding of a style can inform the appropriate choice of method to compute content preservation during textual style transfer.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The code is released: https://github.com/somayeJ/Transformer-based-style-transfer.
- 2.
Other hyperparameters adapted from (http://github.com/fastnlp/style-transformer).
- 3.
- 4.
Training of our T-based model on Yelp took around 36 h using single Quadro RTX 8000 ss GPU, compared to 75 h while applying the training regime from [4].
References
Bahdanau, D., Cho, K.H., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, ICLR 2015 (2015)
Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. In: Proceedings of the 28th Neural Information Processing Systems (NIPS), Workshop on Deep Learning (2014)
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., Baroni, M.: What you can cram into a single vector: probing sentence embeddings for linguistic properties. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), vol. 1: Long Papers, pp. 2126–2136 (2018)
Dai, N., Liang, J., Qiu, X., Huang, X.: Style transformer: unpaired text style transfer without disentangled latent representation. CoRR abs/1905.05621 (2019). http://arxiv.org/abs/1905.05621
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Elazar, Y., Goldberg, Y.: Adversarial removal of demographic attributes from text data. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 11–21. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1002. https://aclanthology.org/D18-1002
Fu, Z., Tan, X., Peng, N., Zhao, D., Yan, R.: Style transfer in text: exploration and evaluation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., Xing, E.P.: Controllable text generation. CoRR abs/1703.00955 (2017). http://arxiv.org/abs/1703.00955
Jafaritazehjani, S., Lecorvé, G., Lolive, D., Kelleher, J.: Style versus content: a distinction without a (learnable) difference? In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 2169–2180. International Committee on Computational Linguistics, Barcelona, Spain (2020). https://doi.org/10.18653/v1/2020.coling-main.197, https://aclanthology.org/2020.coling-main.197
Jafaritazehjani, S., Lecorvé, G., Lolive, D., Kelleher, J.D.: Style as sentiment versus style as formality: the same or different? In: ICANN (2021)
Jin, D., Jin, Z., Hu, Z., Vechtomova, O., Mihalcea, R.: Deep learning for text style transfer: a survey. Comput. Linguist. 48(1), 155–205 (2022). https://doi.org/10.1162/coli_a_00426
John, V., Mou, L., Bahuleyan, H., Vechtomova, O.: Disentangled representation learning for non-parallel text style transfer. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 424–434 (2019)
Kelleher, J.D.: Deep Learning. MIT Press, Cambridge (2019)
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746–1751 (2014)
Lample, G., Subramanian, S., Smith, E., Denoyer, L., Ranzato, M., Boureau, Y.L.: Multiple-attribute text rewriting. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=H1g2NhC5KQ
Leeftink, W., Spanakis, G.: Towards controlled transformation of sentiment in sentences. In: Proceedings of the 11th International Conference on Agents and Artificial Intelligence, vol. 2: ICAART, pp. 809–816. SCITEPRESS (2019)
Li, J., Jia, R., He, H., Liang, P.: Delete, retrieve, generate: a simple approach to sentiment and style transfer. In: Proceedings of the 16th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), vol. 1 (Long Papers), pp. 1865–1874 (2018)
Ma, S., Sun, X.: A semantic relevance based neural network for text summarization and text simplification. Comput. Linguist. 1(1) (2017)
Nedumpozhimana, V., Kelleher, J.: Finding BERT’s idiomatic key. In: Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pp. 57–62. Association for Computational Linguistics (2021). https://doi.org/10.18653/v1/2021.mwe-1.7, https://aclanthology.org/2021.mwe-1.7
Nedumpozhimana, V., Klubička, F., Kelleher, J.D.: Shapley idioms: analysing BERT sentence embeddings for general idiom token identification. Front. Artif. Intell. 5, 813967 (2022). https://doi.org/10.3389/frai.2022.813967, https://www.frontiersin.org/article/10.3389/frai.2022.813967
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)
Prabhumoye, S., Tsvetkov, Y., Salakhutdinov, R., Black, A.W.: Style transfer through back-translation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pp. 866–876. Association for Computational Linguistics (2018). http://aclweb.org/anthology/P18-1080
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
Rao, S., Tetreault, J.R.: Dear sir or madam, may i introduce the GYAFC dataset: corpus, benchmarks and metrics for formality style transfer. In: NAACL-HLT (2018)
Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (2019). https://arxiv.org/abs/1908.10084
Romanov, A., Rumshisky, A., Rogers, A., Donahue, D.: Adversarial decomposition of text representation. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), vol. 1 (Long and Short Papers), pp. 815–825 (2019)
Shen, T., Lei, T., Barzilay, R., Jaakkola, T.: Style transfer from non-parallel text by cross-alignment. In: Guyon, I., et al. (eds.) Proceedings of the Conference in Neural Information Processing Systems, vol. 30 (NIPS), pp. 6830–6841. Curran Associates, Inc. (2017). http://papers.nips.cc/paper/7259-style-transfer-from-non-parallel-text-by-cross-alignment.pdf
Singh, A., Palod, R.: Sentiment transfer using seq2seq adversarial autoencoders. CoRR abs/1804.04003 (2018). http://arxiv.org/abs/1804.04003
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Proceedings of the Conference in Neural Information Processing Systems (NIPS), pp. 3104–3112 (2014)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Zhao, J., Kim, Y., Zhang, K., Rush, A., LeCun, Y.: Adversarially regularized autoencoders. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 5902–5911. PMLR, Stockholm (2018). http://proceedings.mlr.press/v80/zhao18b.html
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Jafaritazehjani, S., Lecorvé, G., Lolive, D., Kelleher, J.D. (2023). Local or Global: The Variation in the Encoding of Style Across Sentiment and Formality. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds) Artificial Neural Networks and Machine Learning – ICANN 2023. ICANN 2023. Lecture Notes in Computer Science, vol 14263. Springer, Cham. https://doi.org/10.1007/978-3-031-44204-9_41
Download citation
DOI: https://doi.org/10.1007/978-3-031-44204-9_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44203-2
Online ISBN: 978-3-031-44204-9
eBook Packages: Computer ScienceComputer Science (R0)