Abstract
The goal of the JOKER track series is to bring together linguists, translators, and computer scientists to foster progress on the automatic interpretation, generation, and translation of wordplay. Being clearly important for various applications, these tasks are still extremely challenging despite significant recent progress in AI in information retrieval and natural language processing. Building on the lessons learned from last year’s edition, JOKER-2023 held three shared tasks aligned with human approaches to the translation of wordplay, or more specifically of puns in English, French, and Spanish: detection, location and interpretation, and finally translation. In this paper, we define these three tasks and describe our approaches to corpus creation and evaluation. We then present an overview of the participating systems, including the summaries of their approaches and a comparison of their performance. As in JOKER-2022, this year’s track also solicited contributions making further use of our data (an “unshared task”), which we also report on.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, M. (eds.): Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings, CEUR-WS.org (2023)
Anjum, A., Lieberum, N.: Exploring humor in natural language processing: a comprehensive review of JOKER tasks at CLEF symposium 2023. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Bell, N.D.: Learning about and through humor in the second language classroom. Lang. Teach. Res. 13(3), 241–258 (2009). https://doi.org/10.1177/1362168809104697
Brunelière, O., Germann, C., Salina, K.: CLEF 2023 JOKER Task 2: using Chat GPT for pun location and interpretation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Delabastita, D.: There’s a Double Tongue: an Investigation into the Translation of Shakespeare’s Wordplay, with Special Reference to Hamlet. Rodopi, Amsterdam (1993)
Delabastita, D.: Wordplay as a translation problem: a linguistic perspective. In: Ein internationales Handbuch zur Übersetzungsforschung, vol. 1, pp. 600–606. De Gruyter Mouton, June 2008. https://doi.org/10.1515/9783110137088.1.6.600
Dsilva, R.R.: AKRaNLU @ CLEF JOKER 2023: using sentence embeddings and multilingual models to detect and interpret wordplay. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Dubreuil, Q.: UBO team @ CLEF JOKER 2023 track for task 1, 2 and 3 - applying AI models in regards to pun translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Ermakova, L., Gwenn-Bosser, A., Jatowt, A., Miller, T.: The JOKER corpus: English-French parallel data for multilingual wordplay recognition. In: SIGIR ’23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY (2023). https://doi.org/10.1145/3539618.3591885
Ermakova, L., Miller, T., Bosser, A.G., Preciado, V.M.P., Sidorov, G., Jatowt, A.: Science for fun: the CLEF 2023 JOKER track on automatic wordplay analysis. In: Kamps, J., et al. (eds.) Advances in Information Retrieval. ECIR 2023. LNCS, vol. 13982, pp. 546–556. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28241-6_63
Ermakova, L., et al.: Overview of JOKER@CLEF 2022: automatic wordplay and humour translation workshop. In: Barron-Cedeno, A., et al. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2022. LNCS, vol. 13390, pp. 447–469. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-13643-6_27
Ermakova, L., et al.: Overview of the CLEF 2022 JOKER task 3: pun translation from English into French. In: Faggioli, G., Ferro, N., Hanbury, A., Potthast, M. (eds.) Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, 5th to 8th September 2022. CEUR Workshop Proceedings, vol. 3180, pp. 1681–1700, August 2022
Glemarec, L., Bosser, A., Boccou, J., Ermakova, L.: Humorous wordplay generation in French. In: Faggioli, G., Ferro, N., Hanbury, A., Potthast, M. (eds.) Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, 5th to 8th September 2022. CEUR Workshop Proceedings, vol. 3180, pp. 1793–1806. CEUR-WS.org (2022)
Glemarec, L., Charles, F.: BU-Pier team @ CLEF JOKER 2023 open task: slip of the tongue generation to improve social interaction with virtual agents. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Jentzsch, S., Kersting, K.: ChatGPT is fun, but it is not funny! Humor is still challenging large language models (2023)
Kolb, W., Miller, T.: Human-computer interaction in pun translation. In: Hadley, J.L., Taivalkoski-Shilov, K., Teixeira, C.S.C., Toral, A. (eds.) Using Technologies for Creative-Text Translation, pp. 66–88. Routledge (2022). https://doi.org/10.4324/9781003094159-4
Komorowska, J., Čatipović, I., Vujica, D.: CLEF2023’ JOKER working notes. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Lieber, O., Sharir, O., Lentz, B., Shoham, Y.: Jurassic-1: technical details and evaluation. White paper, AI21 Labs, August 2021. https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)
Miller, T., Hempelmann, C.F., Gurevych, I.: SemEval-2017 Task 7: detection and interpretation of English puns. In: Proceedings of the 11th International Workshop on Semantic Evaluation, pp. 58–68, August 2017. https://doi.org/10.18653/v1/S17-2005
Ohnesorge, F., Gutiérrez, M.Á., Plichta, J.: CLEF 2023 JOKER tasks 2 and 3: using NLP models for pun location, interpretation and translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Palmer, M., Ng, H.T., Dang, H.T.: Evaluation of WSD systems. In: Agirre, E., Edmonds, P. (eds.) Word Sense Disambiguation. Text, Speech and Language Technology, vol. 33, pp. 75–106. Springer, Dordrecht (2007). https://doi.org/10.1007/978-1-4020-4809-8_4
Popova, O., Dadić, P.: Does AI have a sense of humor? CLEF 2023 JOKER tasks 1, 2 and 3: using BLOOM, GPT, SimpleT5, and more for pun detection, location, interpretation and translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Preciado, V.M.P., Preciado, C.P., Sidorov, G.: NLPalma @ CLEF 2023 JOKER: a BLOOMZ and BERT approach for wordplay detection and translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Prnjak, A., Davari, D.R., Schmitt, K.: CLEF 2023 JOKER task 1, 2, 3: pun detection, pun interpretation, and pun translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Reicho, S., Jatowt, A.: Innsbruck @ CLEF JOKER 2023 track’s task 1: data augmentation techniques for humor recognition in text. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Thomas-Young, T.: Why sentiment analysis is a joke with JOKER data? word-level and interpretation analysis (CLEF 2023 JOKER Task 2). In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)
Acknowledgements
This project has received a government grant managed by the National Research Agency under the program “Investissements d’avenir” integrated into France 2030, with the Reference ANR-19-GURE-0001. JOKER is supported by La Maison des sciences de l’homme en Bretagne. We thank Carolina Palma Preciado, Leopoldo Jesús Gutierrez Galeano, Khatima El Krirh, Nathalie Narváez Bruneau, and Rachel Kinlay for their help and support in the first Spanish pun translation contest. We also thank Quentin Dubreuil, Keith Salina, Constance Germann, Océane Brunelière, Aurianne Damoy, Angelique Robert, and all other colleagues and students who participated in data construction, the translation contests, and the CLEF JOKER track.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ermakova, L., Miller, T., Bosser, AG., Palma Preciado, V.M., Sidorov, G., Jatowt, A. (2023). Overview of JOKER – CLEF-2023 Track on Automatic Wordplay Analysis. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2023. Lecture Notes in Computer Science, vol 14163. Springer, Cham. https://doi.org/10.1007/978-3-031-42448-9_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-42448-9_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42447-2
Online ISBN: 978-3-031-42448-9
eBook Packages: Computer ScienceComputer Science (R0)