Overview of EXIST 2024 — Learning with Disagreement for Sexism Identification and Characterization in Tweets and Memes
Authors: 
Laura Plaza, Jorge Carrillo-de-Albornoz, Víctor Ruiz, Alba Maeso, Berta Chulvi, Paolo Rosso, Enrique Amigó, Julio Gonzalo, Roser Morante, 
Damiano SpinaAuthors Info & Claims
Pages 93 - 117
Abstract
In recent years, the rapid increase in the dissemination of offensive and discriminatory material aimed at women through social media platforms has emerged as a significant concern. This trend has had adverse effects on women’s well-being and their ability to freely express themselves. The EXIST campaign has been promoting research in online sexism detection and categorization in English and Spanish since 2021. The fourth edition of EXIST, hosted at the CLEF 2024 conference, consists of three groups of tasks, which are a continuation of EXIST 2023: sexism identification, source intention identification, and sexism categorization. However, while EXIST 2023 focused on processing tweets, the novelty of this edition is that the three tasks are also applied to memes, resulting in a total of six tasks. The “learning with disagreement” paradigm is adopted to address disagreements in the labelling process and promote the development of equitable systems that are able to learn from different perspectives on the sexism phenomena. The 2024 edition of EXIST has exceeded the success of previous editions, with the participation of 57 teams submitting 412 runs. This lab overview describes the tasks, dataset, evaluation methodology, participant approaches and results.
References
[1]
Amigó, E., Delgado, A.: Evaluating extreme hierarchical multi-label classification. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, vol. Volume 1: Long Papers, pp. 5809–5819. ACL, Dublin, Ireland (2022)
[2]
Aru, G., Emmolo, N., Piras, A., Marzeddu, S., Raffi, J., Passaro, L.C.: RoBEXedda: enhancing sexism detection in tweets for the EXIST 2024 challenge. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[3]
Azadi, A., Ansari, B., Zamani, S.: Bilingual sexism classification: fine-tuned XLM-RoBERTa and GPT-3.5 few-shot learning. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[4]
Barua, D.D., et al.: Penta ML at EXIST 2024: tagging sexism in online multimodal content with attention-enhanced modal context. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[5]
Billig M Humour and hatred: the racist jokes of the Ku Klux Klan Discourse Soc. 2014 12 3 267-289
[6]
Carrillo-Casado, Á., Román-Pásaro, J., Mata-Vázquez, J., Pachón-Álvarez, V.: I2C-UHU at EXIST 2024: transformer-based detection of sexism and source intention in memes using a learning with disagreement approach. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[7]
Chakravarthi, B.R., et al.: Overview of shared task on multitask meme classification - unraveling misogynistic and trolls in online memes. In: Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion, pp. 139–144 (2024)
[8]
Chulvi, B., Fontanella, L., Labadie-Tamayo, R., Rosso, P.: Social or individual disagreement? Perspectivism in the annotation of sexist jokes. In: Proceedings of the NLPerspectives 2023: 2nd Workshop on Perspectivist Approaches to Disagreement in NLP, co-locotaed with ECAI-2023 (2023)
[9]
Fan, S., Frick, R.A., Steinebach, M.: FraunhoferSIT@EXIST2024: leveraging stacking ensemble learning for sexism detection. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[10]
Fang, Y.Z., Lee, L.H., Huang, J.D.: NYCU-NLP at EXIST 2024 – leveraging transformers with diverse annotations for sexism identification in social networks. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[11]
Fersini, E., et al.: SemEval-2022 Task 5: multimedia automatic misogyny identification. In: Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 533–549 (2022)
[12]
Gasparini F, Rizzi G, Saibene A, and Fersini E Benchmark dataset of memes with text transcriptions for automatic detection of multi-modal misogynistic content Data Brief 2022 44
[13]
Guerrero-García, M., Cerrejón-Naranjo, M., Mata-Vázquez, J., Pachón-Álvarez, V.: I2C-UHU at EXIST2024: learning from divergence and perspectivism for sexism identification and source intent classification. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[14]
Hodson G, Rush J, and MacInnis CC A joke is just a joke (except when it isn’t): cavalier humor beliefs facilitate the expression of group dominance motives J. Pers. Soc. Psychol. 2010 99 4 660-682
[15]
Jimenez-Martinez, M.P., Raygoza-Romero, J.M., Sánchez-Torres, C.E., Lopez-Nava, I.H., Montes-y Gómez, M.: Enhancing sexism detection in tweets with annotator-integrated ensemble methods and multimodal embeddings for memes. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[16]
Keinan, R.: Sexism identification in social networks using TF-IDF embeddings, preproccessing, feature selection, word/Char N-grams and various machine learning models in Spanish and English. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[17]
Khan, S., Pergola, G., Jhumka, A.: Multilingual sexism identification via fusion of large language models. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[18]
Kirk, H.R., Yin, W., Vidgen, B., Röttger, P.: SemEval-2023 task 10: explainable detection of online sexism. In: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval) (2023)
[19]
Labadie-Tamayo, R., Chulvi, B., Rosso, P.: Everybody hurts, sometimes. Overview of HUrtful HUmour at IberLEF 2023: detection of humour spreading prejudice in Twitter. In: Procesamiento del Lenguaje Natural (SEPLN), pp. 383–395, No. 71 (2023)
[20]
Ma, J., Li, R.: RoJiNG-CL at EXIST 2024: sexism identification in memes by integrating prompting and fine-tuning. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[21]
Maqbool, F., Fersini, E.: A contrastive learning based approach to detect sexism in memes. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[22]
Maqbool, N.: Sexism identification in social networks: advances in automated detection – a report on the exist task at CLEF. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[23]
Martinez, E., Cuadrado, J., Martinez-Santos, J.C., Puertas, E.: VerbaNex AI at CLEF EXIST 2024: detection of online sexism using transformer models and profiling techniques. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[24]
Mendiburo-Seguel, A., Ford, T.E.: The effect of disparagement humor on the acceptability of prejudice. Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues, pp. No Pagination Specified–No Pagination Specified (2019)
[25]
Menárguez Box, A., Torres Bertomeu, D.: DiTana-PV at sEXism identification in social networks (EXIST) tasks 4 and 6: the effect of translation in sexism identification. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[26]
Naebzadeh, A., Nobakhtian, M., Eetemadi, S.: NICA at EXIST CLEF tasks 2024. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[27]
Obrador Reina, M., García Cucó, A.: LightGMB for sexism identification in memes. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[28]
Pan, R., García Díaz, J.A., Bernal Beltrán, T., Valencia-Garcia, R.: UMUTeam at EXIST 2024: multi-modal identification and categorization of sexism by feature integration. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[29]
Pasha, U.W.: Multilingual sexism detection in memes, a CLIP-enhanced machine learning approach. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[30]
Petrescu, A., Truică, C.O., Apostol, E.S.: Language-based mixture of transformers for EXIST2024. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[31]
Plaza, L., et al.: Overview of EXIST 2023 – learning with disagreement for sexism identification and characterization (Extended Overview). In: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum (2023)
[32]
Plaza, L., et al.: Overview of EXIST 2024 – learning with disagreement for sexism identification and characterization in social networks and memes (Extended Overview). In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[33]
Plaza, L., et al.: Overview of EXIST 2023 – learning with disagreement for sexism identification and characterization (Extended Overview). In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, M. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), vol. 497, pp. 813–854. CEUR Working Notes (2023)
[34]
Quan, L.M., Thin, D.V.: Sexism identification in social networks with generation-based approach. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[35]
Rizzi, G., Gimeno-Gómez, D., Fersini, E., Martínez-Hinarejos, C.D.: PINK at EXIST2024: a cross-lingual and multi-modal transformer approach for sexism detection in memes. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[36]
Rodríguez-Sánchez F et al. Overview of EXIST 2021: sexism identification in social networks Procesamiento del Lenguaje Natural 2021 67 195-207
[37]
Rodríguez-Sánchez F et al. Overview of EXIST 2022: sexism identification in social networks Procesamiento del Lenguaje Natural 2022 69 229-240
[38]
Ruiz, V., Carrillo-de-Albornoz, J., Plaza, L.: Concatenated transformer models based on levels of agreements for sexism detection. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[39]
Shah, A., Gokhale, A.: Team Aditya at EXIST 2024 – detecting sexism in multilingual tweets using contrastive learning approach. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[40]
Shanbhag, A., Jadhav, S., Date, A., Joshi, S., Sonawane, S.: The wisdom of weighing: stacking ensembles for a more balanced sexism detector. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[41]
Shifat, F.T., et al.: Penta-NLP at EXIST 2024 Task 1–3: sexism identification, source intention, sexism categorization in tweets. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[42]
Shimi, G., Mahibha, J., Thenmozhi, D.: Automatic classification of gender stereotypes in social media post. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[43]
Siino, M., Tinnirello, I.: Prompt engineering for identifying sexism using GPT mistral 7B. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[44]
Smith, T., Nie, R., Trippas, J., Spina, D.: RMIT-IR at EXIST lab at CLEF 2024. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[45]
Murari Sreekumar, S.K., Thenmozhi, D., Gopalakrishnan, S., Swaminathan, K.: Sexism identification in tweets using traditional machine learning approaches. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[46]
Tavarez-Rodríguez, J., Sánchez-Vega, F., Rosales-Pérez, A., López-Monroy, A.P.: Better together: LLM and neural classification transformers to detect sexism. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[47]
Uma, A., et al.: SemEval-2021 task 12: learning with disagreements. In: Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pp. 338–347. Association for Computational Linguistics, Online, August 2021
[48]
Usmani, M., Siddiqui, R., Rizwan, S., Khan, F., Alvi, F., Samad, A.: Sexism identification in tweets using BERT and XLM – Roberta. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[49]
Vetagiri, A., Mogha, P., Pakray, P.: Cracking down on digital misogyny with MULTILATE a MULTImodaL hATE detection system. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
[50]
Villarreal-Haro, K., Sánchez-Vega, F., Rosales-Pérez, A., López-Monroy, A.P.: Stacked reflective reasoning in large neural language models. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)
Index Terms
- Overview of EXIST 2024 — Learning with Disagreement for Sexism Identification and Characterization in Tweets and Memes
Index terms have been assigned to the content through auto-classification.
Recommendations
EXIST 2024: sEXism Identification in Social neTworks and Memes
Advances in Information RetrievalAbstractThe paper describes the EXIST 2024 lab on Sexism identification in social networks, that is expected to take place at the CLEF 2024 conference and represents the fourth edition of the EXIST challenge. The lab comprises five tasks in two languages, ...
Overview of EXIST 2023 – Learning with Disagreement for Sexism Identification and Characterization
Experimental IR Meets Multilinguality, Multimodality, and InteractionAbstractIn recent years, the rapid increase in the dissemination of offensive and discriminatory material aimed at women through social media platforms has emerged as a significant concern. This trend has had adverse effects on women’s well-being and ...
Racial microaggressions and perceptions of Internet memes
Although more blatant forms of discrimination have declined, racial prejudice continues to manifest itself in subtle ways. For example, People of Color experience racial microaggressions (i.e., subtle slights or 'put downs') in their face-to-face ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Sep 2024
355 pages
ISBN:978-3-031-71907-3
DOI:10.1007/978-3-031-71908-0
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 19 September 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 13 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in