Nothing Special   »   [go: up one dir, main page]
Skip to main content
SpringerLink
Log in

Overview of the CLEF 2024 JOKER Track

Automatic Humour Analysis

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14959))

  • 35 Accesses

Abstract

The JOKER Lab series at the Conference and Labs of the Evaluation Forum (CLEF) was established in 2022 to promote collaborative, interdisciplinary research on the automated processing of wordplay and verbal humour. This paper provides an overview of the setup and results of the Lab’s 2024 edition. We describe the data and evaluation metrics used for the Lab’s three shared tasks (on humour-aware information retrieval, humour classification according to genre and technique, and translation of puns from English to French) and introduce and compare the systems that participated in each task, with particular attention to their approaches and performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.joker-project.com/.

  2. 2.

    https://pypi.org/project/wikipedia/.

  3. 3.

    https://github.com/mjpost/sacrebleu/.

  4. 4.

    https://pypi.org/project/bert-score/.

References

  1. Abu Farha, I., Oprea, S.V., Wilson, S., Magdy, W.: SemEval-2022 task 6: iSarcasmEval, intended sarcasm detection in English and Arabic. In: Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 802–814. Association for Computational Linguistics, July 2022. https://doi.org/10.18653/v1/2022.semeval-1.111

  2. Baguian, H., Ashley, H.N.: JOKER Track @ CLEF 2024: the Jokesters’ approaches for retrieving, classifying, and translating wordplay. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1811–1817. CEUR-WS.org (2024)

    Google Scholar 

  3. Balaji, S.S., et al.: Vayam Solve Kurmaha @ CLEF 2024: task 2: humor classification according to genre and technique using BERT embeddings and transformers. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1895–1901. CEUR-WS.org (2024)

    Google Scholar 

  4. Bogireddy, N.R., Suresh, S., Rai, S.: I’m out of breath from laughing! I think? A dataset of COVID-19 humor and its toxic variants. In: Companion Proceedings of the ACM Web Conference 2023, pp. 1004–1013. Association for Computing Machinery, New York (2023). https://doi.org/10.1145/3543873.3587591

  5. Schuurman, E., Cazemier, M., Buijs, L., Kamps, J.: University of Amsterdam at the CLEF 2024 joker track. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1909–1922. CEUR-WS.org (2024)

    Google Scholar 

  6. Dhanani, F., Abbas, R.: Translating English puns to French. In: Faggioli, G., Ferro, N., Galuscakova, P., García Seco de Herrera, A. (eds.) Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  7. Dsilva, R.R., Bhardwaj, N.: PunDerstand @ CLEF JOKER 2024: Who’s Laughing Now? Humor Classification by Genre & Technique In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1826–1833. CEUR-WS.org (2024)

    Google Scholar 

  8. Elagina, R., Vučić, P.: Working Notes of CLEF 2024: Effective Humor Analysis and Translation. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1834–1840. CEUR-WS.org (2024)

    Google Scholar 

  9. Epron, P., Guibon, G., Couceiro, M.: ORPAILLEUR & SyNaLP at CLEF 2024 Task 2: Good Old Cross Validation for Large Language Models Yields the Best Humorous Detection. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1841–1856 CEUR-WS.org (2024)

    Google Scholar 

  10. Ermakova, L., Bosser, A.G., Jatowt, A., Miller, T.: The JOKER Corpus: English–French parallel data for multilingual wordplay recognition. In: SIGIR 2023: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2796–2806. Association for Computing Machinery, New York (2023). https://doi.org/10.1145/3539618.3591885

  11. Ermakova, L., Miller, T., Bosser, A.G., Palma Preciado, V.M., Sidorov, G., Jatowt, A.: Overview of JOKER 2023 automatic wordplay analysis task 1 – pun detection. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, M. (eds.) Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings, vol. 3497, pp. 1785–1803, October 2023

    Google Scholar 

  12. Ermakova, L., Miller, T., Bosser, A.G., Palma Preciado, V.M., Sidorov, G., Jatowt, A.: Overview of JOKER 2023 automatic wordplay analysis task 2 – pun location and interpretation. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, M. (eds.) Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings, vol. 3497, pp. 1804–1817, October 2023

    Google Scholar 

  13. Ermakova, L., Miller, T., Bosser, A.G., Palma Preciado, V.M., Sidorov, G., Jatowt, A.: Overview of JOKER 2023 automatic wordplay analysis task 3 – pun translation. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, M. (eds.) Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings, vol. 3497, pp. 1818–1827, October 2023

    Google Scholar 

  14. Ermakova, L., Miller, T., Bosser, A.G., Palma Preciado, V.M., Sidorov, G., Jatowt, A.: Overview of JOKER - CLEF-2023 track on automatic wordplay analysis. In: Arampatzis, A., et al. (eds.) CLEF 2023. LNCS, vol. 14163, pp. 397–415. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-42448-9_26

    Chapter  Google Scholar 

  15. Ermakova, L., Miller, T., Bosser, A.G., Palma Preciado, V.M., Sidorov, G., Jatowt, A.: Science for fun: the CLEF 2023 JOKER track on automatic wordplay analysis. In: Kamps, J., et al. (eds.) ECIR 2023. LNCS, vol. 13982, pp. 546–556. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28241-6_63

    Chapter  Google Scholar 

  16. Ermakova, L., et al.: CLEF workshop JOKER: automatic wordplay and humour translation. In: Hagen, M., et al. (eds.) ECIR 2022. LNCS, vol. 13186, pp. 355–363. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99739-7_45

    Chapter  Google Scholar 

  17. Ermakova, L., et al.: Overview of JOKER@CLEF 2022: automatic wordplay and humour translation workshop. In: Barrón-Cedeño, A., et al. (eds.) CLEF 2022. LNCS, vol. 13390, pp. 447–469. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-13643-6_27

    Chapter  Google Scholar 

  18. Ermakova, L., et al.: Overview of the CLEF 2024 JOKER task 1: humour-aware information retrieval. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1775–1785. CEUR-WS.org (2024)

    Google Scholar 

  19. Ermakova, L., et al.: Overview of the CLEF 2024 JOKER task 3: translate puns from English to French. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1800–1810. CEUR-WS.org (2024)

    Google Scholar 

  20. Frenda, S., et al.: EPIC: multi-perspective annotation of a corpus of irony. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 13844–13857. Association for Computational Linguistics, July 2023. https://doi.org/10.18653/v1/2023.acl-long.774

  21. Gepalova, A., Chifu, A.G., Fournier, S.: CLEF 2024 JOKER task 1: exploring pun detection using the T5 transformer model. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1857–1861. CEUR-WS.org (2024)

    Google Scholar 

  22. Kováčiková, J.V., Šuppa, M.: Thinking, fast and slow: from the speed of FastText to the depth of retrieval augmented large language models for humour classification. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1862–1867. CEUR-WS.org (2024)

    Google Scholar 

  23. Macdonald, C., Tonellotto, N.: Declarative experimentation in information retrieval using PyTerrier. In: Proceedings of ICTIR 2020 (2020)

    Google Scholar 

  24. Mann, R., Mikulandric, T.: CLEF 2024 JOKER tasks 1–3: humour identification and classification. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1868–1875. CEUR-WS.org (2024)

    Google Scholar 

  25. Narayanan, S., et al.: CLEF 2024 JOKER task 3 : using RoBERTa and BERT-uncased for humour classification according to genre and technique. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1876–1879. CEUR-WS.org (2024)

    Google Scholar 

  26. Palma Preciado, V.M., Palma Preciado, C., Sidorov, G.: NLPalma Joker 2024: yet, no humor with humorousness – task 2 humour classification according to genre and technique. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1880–1888. CEUR-WS.org (2024)

    Google Scholar 

  27. Palma-Preciado, V.M., et al.: Overview of the CLEF 2024 JOKER task 2: humour classification according to genre and technique. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1786–1799. CEUR-WS.org (2024)

    Google Scholar 

  28. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002). https://doi.org/10.3115/1073083.1073135. https://www.aclweb.org/anthology/P02-1040

  29. Post, M.: A call for clarity in reporting BLEU scores. In: Proceedings of the Third Conference on Machine Translation: Research Papers, Belgium, Brussels, pp. 186–191. Association for Computational Linguistics, October 2018. https://www.aclweb.org/anthology/W18-6319

  30. Saipranav, M., Sridharan, J., et al.: CLEF 2024 JOKER task 2: using BERT and random forest classifier for humor classification according to genre and technique. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1902–1908. CEUR-WS.org (2024)

    Google Scholar 

  31. Shapira, N., Kalinsky, O., Libov, A., Shani, C., Tolmach, S.: Evaluating humorous response generation to playful shopping requests. In: Kamps, J., et al. (eds.) ECIR 2023. LNCS, vol. 13981, pp. 617–626. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28238-6_53

    Chapter  Google Scholar 

  32. Subramanian, R., Sivaraman, V., et al.: Team humour insights at JOKER 2024 task 2: humour classification according to genre and technique. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1923–1932. CEUR-WS.org (2024)

    Google Scholar 

  33. Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

  34. Van Gysel, C., de Rijke, M.: Pytrec_eval: an extremely fast python interface to trec_eval. In: SIGIR. ACM (2018)

    Google Scholar 

  35. Varadi, D.P., Bartulović, A.: University of Split and University of Malta (Team AB &DPV) at the CLEF 2024 JOKER Track: From ’LOL’ to ’MDR’ Using Artificial Intelligence Models to Retrieve and Translate Puns. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1818–1825. CEUR-WS.org (2024)

    Google Scholar 

  36. Wu, S.H., Huang, Y.F., Lau, T.Y.: Humour classification by fine-tuning LLMs: CYUT at CLEF 2024 JOKER Lab subtask humour classification according to genre and technique. In: Faggioli, G., et al. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024). CEUR Workshop Proceedings, pp. 1933–1947. CEUR-WS.org (2024)

    Google Scholar 

  37. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTscore: evaluating text generation with BERT. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=SkeHuCVFDr

Download references

Acknowledgments

This project has received a government grant managed by the National Research Agency under the program “Investissements d’avenir” integrated into France 2030, with the Reference ANR-19-GURE-0001. This track would not have been possible without the great support of numerous individuals. We want to thank in particular the colleagues and the students who participated in data construction and evaluation, in particular the students of the Université de Bretagne Occidentale. Please visit the JOKER website for more details on the track (https://joker-project.com/).

Author information

Authors and Affiliations

  1. Université de Bretagne Occidentale, HCTI, Brest, France

    Liana Ermakova & Victor Manuel Palma Preciado

  2. École Nationale d’Ingénieurs de Brest, Lab-STICC CNRS UMR 6285, Plouzané, France

    Anne-Gwenn Bosser

  3. Department of Computer Science, University of Manitoba, Winnipeg, Canada

    Tristan Miller

  4. Austrian Research Institute for Artificial Intelligence (OFAI), Vienna, Austria

    Tristan Miller

  5. Centro de Investigación en Computación (CIC), Instituto Politécnico Nacional (IPN), Mexico City, Mexico

    Victor Manuel Palma Preciado & Grigori Sidorov

  6. University of Innsbruck, Innsbruck, Austria

    Adam Jatowt

Authors
  1. Liana Ermakova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anne-Gwenn Bosser
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tristan Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Victor Manuel Palma Preciado
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Grigori Sidorov
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Adam Jatowt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liana Ermakova .

Editor information

Editors and Affiliations

  1. Université Grenoble Alpes, CNRS, Grenoble, France

    Lorraine Goeuriot

  2. Université Grenoble Alpes, CNRS, Grenoble, France

    Philippe Mulhem

  3. Université Grenoble Alpes, CNRS, Grenoble, France

    Georges Quénot

  4. Université Grenoble Alpes, CNRS, Grenoble, France

    Didier Schwab

  5. University of Padova, Padua, Italy

    Giorgio Maria Di Nunzio

  6. Sorbonne University, Paris, France

    Laure Soulier

  7. University of Stavanger, Stavanger, Norway

    Petra Galuščáková

  8. University of Essex, Colchester, UK

    Alba García Seco de Herrera

  9. University of Padova, Padua, Italy

    Guglielmo Faggioli

  10. University of Padova, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ermakova, L., Bosser, AG., Miller, T., Palma Preciado, V.M., Sidorov, G., Jatowt, A. (2024). Overview of the CLEF 2024 JOKER Track. In: Goeuriot, L., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2024. Lecture Notes in Computer Science, vol 14959. Springer, Cham. https://doi.org/10.1007/978-3-031-71908-0_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-71908-0_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-71907-3

  • Online ISBN: 978-3-031-71908-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation