Report on the FIRE 2020 evaluation initiative
Article No.: 3, Pages 1 - 11
Abstract
This report gives an overview on the Forum for Information Retrieval Evaluation (FIRE) initiative for South-Asian languages1. The FIRE conference was conducted online in December 2020. The event combined a conference including keynotes, peer reviewed paper session with an Evaluation Forum. This report will present an overview of the conference and provide insights into the evaluation tracks. Current domains include legal information access, mixed script information retrieval, semantic analysis and social media posts classification. The tasks are discussed and connections to other evaluation initiatives are shown.
References
[1]
Moumita Basu, Kripabandhu Ghosh, and Saptarshi Ghosh. Information retrieval from Microblogs during disasters: In the light of IRMiDis task. SN Computer Science, 1(1):61, 2020.
[2]
Janek Bevendorff, Berta Chulvi, Gretel Liz De la Peña Sarracén, Mike Kestemont, Enrique Manjavacas, Ilia Markov, Maximilian Mayerl, Martin Potthast, Francisco Manuel Rangel Pardo, Paolo Rosso, Efstathios Stamatatos, Benno Stein, Matti Wiegmann, Magdalena Wolska, and Eva Zangerle. Overview of PAN 2021: Authorship verification, profiling hate speech spreaders on twitter, and style change detection - extended abstract. In 43rd European Conference on IR Research, ECIR, Virtual Event, March 28 - April 1, volume 12657 of LNCS, pages 567--573. Springer, 2021.
[3]
Paheli Bhattacharya, Kripabandhu Ghosh, Saptarshi Ghosh, Arindam Pal, Parth Mehta, Arnab Bhattacharya, and Prasenjit Majumder. Overview of the FIRE 2019 AILA track: Artificial intelligence for legal assistance. In Working Notes of FIRE - Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, volume 2517, pages 1--12. CEUR-WS.org, 2019a. URL http://ceur-ws.org/Vol-2517/T1-1.pdf.
[4]
Paheli Bhattacharya, Shounak Paul, Kripabandhu Ghosh, Saptarshi Ghosh, and Adam Wyner. Identification of rhetorical roles of sentences in Indian legal judgments. In Proc. International Conference on Legal Knowledge and Information Systems (JURIX), 2019b.
[5]
Paheli Bhattacharya, Parth Mehta, Kripabandhu Ghosh, Saptarshi Ghosh, Arindam Pal, Arnab Bhattacharya, and Prasenjit Majumder. Overview of the FIRE 2020 AILA track: Artificial intelligence for legal assistance. In Working Notes of FIRE - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, volume 2826, pages 1--11. CEUR-WS.org, 2020. URL http://ceur-ws.org/Vol-2826/T1-1.pdf.
[6]
Bharathi Raja Chakravarthi, Anand Kumar M, John P. McCrae, B. Premjith, K. P. Soman, and Thomas Mandl. Overview of the track on HASOC-offensive language identification-DravidianCodeMix. In Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, volume 2826, pages 112--120. CEUR-WS.org, 2020a. URL http://ceur-ws.org/Vol-2826/T2-2.pdf.
[7]
Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Shardul Suryawanshi, Navya Jose, Elizabeth Sherly, and John P. McCrae. Overview of the track on sentiment analysis for Dravidian languages in Code-Mixed text. In Forum for Information Retrieval Evaluation, FIRE 2020, page 21--24, New York, NY, USA, 2020b. ACM. ISBN 9781450389785.
[8]
Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Shardul Suryawanshi, Navya Jose, Elizabeth Sherly, and John P. McCrae. Overview of the track on sentiment analysis for Dravidian languages in Code-Mixed text. In Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, volume 2826, pages 480--489. CEUR-WS.org, 2020c. URL http://ceur-ws.org/Vol-2826/T4-1.pdf.
[9]
Bharathi Raja Chakravarthi, Ruba Priyadharshini, Navya Jose, Anand Kumar M, Thomas Mandl, Prasanna Kumar Kumaresan, Rahul Ponnusamy, Hariharan R L, John P. McCrae, and Elizabeth Sherly. Findings of the shared task on offensive language identification in Tamil, Malayalam, and Kannada. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, pages 133--145, Kyiv, April 2021. ACL. URL https://www.aclweb.org/anthology/2021.dravidianlangtech-1.17.
[10]
Gordon V. Cormack, Maura R. Grossman, Bruce Hedin, and Douglas W. Oard. Overview of the TREC 2010 legal track. In Proceedings of The Nineteenth Text REtrieval Conference, TREC 2010, Gaithersburg, Maryland, USA, November 16-19, volume 500--294 of NIST Special Publication. National Institute of Standards and Technology (NIST), 2010. URL https://trec.nist.gov/pubs/trec19/papers/LEGAL10.OVERVIEW.pdf.
[11]
Suchana Datta, Debasis Ganguly, Dwaipayan Roy, Derek Greene, Charles Jochim, and Francesca Bonin. Overview of the causality-driven adhoc information retrieval (CAIR) task at FIRE-2020. In Forum for Information Retrieval Evaluation, pages 14--17, 2020.
[12]
Bhargav Dave, Surupendu Gangopadhyay, Prasenjit Majumder, Pushpak Bhattacharya, Sudeshna Sarkar, and Sobha Lalitha Devi. Overview of the FIRE 2020 EDNIL track: Event detection from news in indian languages. In Working Notes of FIRE - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, volume 2826, pages 806--816. CEUR-WS.org, 2020. URL http://ceur-ws.org/Vol-2826/T10-1.pdf.
[13]
Sobha Lalitha Devi. Anaphora resolution from social media text in indian languages (SocAnaRes-IL)- overview. In FIRE 2020: Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, pages 9--13. ACM, 2020.
[14]
Ali Fadel, Husam Musleh, Ibraheem Tuffaha, Mahmoud Al-Ayyoub, Yaser Jararweh, Elhadj Benkhelifa, and Paolo Rosso. Overview of the PAN@FIRE 2020 task on the authorship identification of SOurce COde. In Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, Dec. 16-20, volume 2826, pages 649--676. CEUR-WS.org, 2020. URL http://ceur-ws.org/Vol-2826/T5-1.pdf.
[15]
Enrique Flores, Paolo Rosso, Lidia Moreno, and Esaú Villatoro-Tello. On the detection of source code re-use. In Proceedings of the Forum for Information Retrieval Evaluation, FIRE 2014, Bangalore, India, December 5-7, 2014, pages 21--30. ACM, 2014.
[16]
Enrique Flores, Paolo Rosso, Esaú Villatoro-Tello, Lidia Moreno, Rosa Alcover, and Vicente Chirivella. PAN@FIRE: Overview of CL-SOCO track on the detection of cross-language source code re-use. In Post Proceedings of the Workshops at the 7th Forum for Information Retrieval Evaluation, Gandhinagar, India, December 4-6, volume 1587, pages 1--5. CEUR-WS.org, 2015. URL http://ceur-ws.org/Vol-1587/T1-1.pdf.
[17]
Thomas J Froehlich. A not-so-brief account of current information ethics: The ethics of ignorance, missing information, misinformation, disinformation and other forms of deception or incompetence. BiD, (39), 2017.
[18]
Debasis Ganguly, Dipasree Pal, Manisha Verma, and Procheta Sen. Overview of RCD-2020, the FIRE-2020 track on retrieval from conversational dialogues. In Forum for Information Retrieval Evaluation, pages 33--36, 2020.
[19]
Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 845--854, Minneapolis, Minnesota, USA, June 2019. ACL. URL https://www.aclweb.org/anthology/S19-2147.
[20]
Prasenjit Ma jumder, Mandar Mitra, Surupendu Gangopadhyay, and Parth Mehta, editors. FIRE 2020: Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, 2020. ACM. ISBN 978-1-4503-8978-5.
[21]
Thomas Mandl, Sandip Modha, Gautam Kishore Shahi, Amit Kumar Jaiswal, Durgesh Nandini, Daksh Patel, Prasenjit Majumder, and Johannes Schäfer. Overview of the HASOC track at FIRE 2020: Hate speech and offensive content identification in Indo-European Languages. In Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, volume 2826, pages 87--111. CEUR-WS.org, 2020. URL http://ceur-ws.org/Vol-2826/T2-1.pdf.
[22]
Parth Mehta, Thomas Mandl, Prasenjit Majumder, and Mandar Mitra, editors. Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, volume 2826 of CEUR Workshop Proceedings, 2021. CEUR-WS.org. URL http://ceur-ws.org/Vol-2826.
[23]
Sandip Modha, Thomas Mandl, Prasenjit Majumder, and Daksh Patel. Overview of the HASOC track at FIRE 2019: Hate speech and offensive content identification in Indo-European Languages. In Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, volume 2517, pages 167--190. CEUR-WS.org, 2019. URL http://ceur-ws.org/Vol-2517/T3-1.pdf.
[24]
Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeño, Rubén Míguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, Alex Nikolov, Gautam Kishore Shahi, Julia Maria Struß, and Thomas Mandl. The CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In 43rd European Conference on IR Research, ECIR Virtual Event, March 28 - April 1, pages 639--649. Springer, 2021.
[25]
Francisco M. Rangel Pardo, Fabio A. González, Felipe Restrepo-Calle, Manuel Montes-y-Gómez, and Paolo Rosso. PAN@FIRE: Overview of the PR-SOCO track on personality recognition in SOurce COde. In Text Processing - FIRE 2016 International Workshop, Kolkata, India, December 7-10, 2016, volume 10478 of LNCS, pages 1--19. Springer, 2016.
[26]
Florina Piroi, Mihai Lupu, and Allan Hanbury. Overview of CLEF-IP 2013 lab - information retrieval in the patent domain. In Information Access Evaluation. Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF, Valencia, Spain, September 23-26. Proceedings, volume 8138 of LNCS, pages 232--249. Springer, 2013.
[27]
Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation, pages 1--47, 2020.
[28]
Juliano Rabelo, Mi-Young Kim, Randy Goebel, Masaharu Yoshioka, Yoshinobu Kano, and Ken Satoh. A summary of the COLIEE 2019 competition. In New Frontiers in Artificial Intelligence - JSAI-isAI International Workshops, JURISIN, AI-Biz, LENLS, Kansei-AI, Yokohama, Japan, November 10-12, volume 12331 of LNCS, pages 34--49. Springer, 2019.
[29]
Gautam Kishore Shahi, Anne Dirkson, and Tim A. Majchrzak. An exploratory study of COVID-19 misinformation on twitter. Online Soc. Networks Media, 22:100104, 2021.
[30]
Manjira Sinha, Tirthankar Dasgupta, and Lipika Dey. CEREX@ FIRE-2020: Overview of the shared task on cause-effect relation extraction. In Forum for Information Retrieval Evaluation, pages 18--20, 2020.
[31]
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. TREC-COVID: constructing a pandemic information retrieval test collection. ACM SIGIR Forum, 54(1):1--12, 2021.
[32]
Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çagri Çöltekin. Semeval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). CoRR, abs/2006.07235, 2020. URL https://arxiv.org/abs/2006.07235.
[33]
Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, and Inho Kang. Overview of the NTCIR-15 dialogue evaluation (DialEval-1) task. In Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, 2020. URL http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/01-NTCIR15-OV-DIALEVAL-ZengZ.pdf.
- Report on the FIRE 2020 evaluation initiative
Recommendations
The FIRE 2008 Evaluation Exercise
The aim of the Forum for Information Retrieval Evaluation (FIRE) is to create an evaluation framework in the spirit of TREC (Text REtrieval Conference), CLEF (Cross-Language Evaluation Forum), and NTCIR (NII Test Collection for IR Systems), for Indian ...
Report on CLEF 2020
This is a report on the tenth edition of the Conference and Labs of the Evaluation Forum (CLEF 2020), (virtually) held from September 22--25, 2020, in Thessaloniki, Greece.
CLEF was a four day event combining a Conference and an Evaluation Forum. The ...
Overview of the CLEF eHealth Evaluation Lab 2020
Experimental IR Meets Multilinguality, Multimodality, and InteractionAbstractIn this paper, we provide an overview of the eight annual edition of the Conference and Labs of the Evaluation Forum (CLEF) eHealth evaluation lab. The Conference and Labs of the Evaluation Forum (CLEF) eHealth 2020 continues our development of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2021 Copyright is held by the owner/author(s).
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 16 July 2021
Published in SIGIR Volume 55, Issue 1
Check for updates
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 86Total Downloads
- Downloads (Last 12 months)12
- Downloads (Last 6 weeks)4
Reflects downloads up to 22 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in