SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024)
Authors: 
Philipp Schaer, 
Christin Katharina Kreutz, 
Krisztian Balog, Timo Breuer, Norbert FuhrAuthors Info & Claims
Pages 3058 - 3061
Abstract
Simulations in various forms have been used to evaluate information access systems, like search engines, recommender systems, or conversational agents. In the form of the Cranfield paradigm, a simulation setup is well-known in the IR community, but user simulations have recently gained interest. While user simulations help to reduce the complexity of evaluation experiments and help with reproducibility, they can also contribute to a better understanding of users. Building on recent developments in methods and toolkits, the Sim4IA workshop aims to bring together researchers and practitioners to form an interactive and engaging forum for discussions on the future perspectives of the field. An additional aim is to plan an upcoming TREC/CLEF campaign.
References
[1]
Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffery Dalton, and Leif Azzopardi. 2024. TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview. arxiv: 2401.01330 [cs.IR]
[2]
Leif Azzopardi, Kalervo J"a rvelin, Jaap Kamps, and Mark D. Smucker. 2010. Report on the SIGIR 2010 workshop on the simulation of interaction. SIGIR Forum, Vol. 44, 2 (2010), 35--47. https://doi.org/10.1145/1924475.1924484
[3]
Krisztian Balog, David Maxwell, Paul Thomas, and Shuo Zhang. 2021. Report on the 1st simulation for information retrieval workshop (Sim4IR 2021) at SIGIR 2021. SIGIR Forum, Vol. 55, 2 (2021), 10:1--10:16. https://doi.org/10.1145/3527546.3527559
[4]
Krisztian Balog and ChengXiang Zhai. 2023 a. Tutorial on User Simulation for Evaluating Information Access Systems. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21--25, 2023, Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhang, and Rodrygo L. T. Santos (Eds.). ACM, 5200--5203. https://doi.org/10.1145/3583780.3615296
[5]
Krisztian Balog and ChengXiang Zhai. 2023 b. User Simulation for Evaluating Information Access Systems. arxiv: 2306.08550 [cs.HC]
[6]
Timo Breuer, Norbert Fuhr, and Philipp Schaer. 2022. Validating Simulations of User Query Variants. In Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10--14, 2022, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 13185), Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer, 80--94. https://doi.org/10.1007/978--3-030--99736--6_6
[7]
Bjö rn Engelmann, Timo Breuer, and Philipp Schaer. 2023. Simulating Users in Interactive Web Table Retrieval. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21--25, 2023, Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhang, and Rodrygo L. T. Santos (Eds.). ACM, 3875--3879. https://doi.org/10.1145/3583780.3615187
[8]
Lukas Gienapp, Harrisen Scells, Niklas Deckers, Janek Bevendorff, Shuai Wang, Johannes Kiesel, Shahbaz Syed, Maik Fröbe, Guido Zuccon, Benno Stein, Matthias Hagen, and Martin Potthast. 2023. Evaluating Generative Ad Hoc Information Retrieval. arxiv: 2311.04694 [cs.IR]
[9]
Julio Gonzalo, V'i ctor Peinado, Paul D. Clough, and Jussi Karlgren. 2009. Overview of iCLEF 2009: Exploring Search Behaviour in a Multilingual Folksonomy Environment. In Multilingual Information Access Evaluation II. Multimedia Experiments - 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 6242), Carol Peters, Barbara Caputo, Julio Gonzalo, Gareth J. F. Jones, Jayashree Kalpathy-Cramer, Henning Mü ller, and Theodora Tsikrika (Eds.). Springer, 13--20. https://doi.org/10.1007/978--3--642--15751--6_2
[10]
Frank Hopfgartner, Krisztian Balog, Andreas Lommatzsch, Liadh Kelly, Benjamin Kille, Anne Schuth, and Martha Larson. 2019. Continuous Evaluation of Large-Scale Information Access Systems: A Case for Living Labs. In Information Retrieval Evaluation in a Changing World, Nicola Ferro and Carol Peters (Eds.). Vol. 41. Springer International Publishing, Cham, 511--543. https://doi.org/10.1007/978--3-030--22948--1_21 Series Title: The Information Retrieval Series.
[11]
Zhiyuan Hu, Yue Feng, Anh Tuan Luu, Bryan Hooi, and Aldo Lipani. 2023. Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulators to Enhance Dialogue System. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM '23). ACM. https://doi.org/10.1145/3583780.3615220
[12]
Peter Ingwersen and Kalervo Järvelin. 2005. The turn - integration of information seeking and retrieval in context. Springer, Dordrecht.
[13]
Rolf Jagerman, Krisztian Balog, Philipp Schaer, Johann Schaible, Narges Tavakolpoursaleh, and Maarten de Rijke. 2017. Overview of TREC OpenSearch 2017. In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017, Gaithersburg, Maryland, USA, November 15--17, 2017 (NIST Special Publication, Vol. 500--324), Ellen M. Voorhees and Angela Ellis (Eds.). National Institute of Standards and Technology (NIST). https://trec.nist.gov/pubs/trec26/papers/Overview-O.pdf
[14]
Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval Systems with Users. Foundations and Trends in Information Retrieval, Vol. 3, 1-2 (2009), 1--224. https://doi.org/10.1561/1500000012
[15]
Benjamin Kille, Andreas Lommatzsch, Gebrekirstos G. Gebremeskel, Frank Hopfgartner, Martha A. Larson, Jonas Seiler, Davide Malagoli, Andrá s Seré ny, Torben Brodt, and Arjen P. de Vries. 2016. Overview of NewsREEL'16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms. In Experimental IR Meets Multilinguality, Multimodality, and Interaction - 7th International Conference of the CLEF Association, CLEF 2016, É vora, Portugal, September 5--8, 2016, Proceedings (Lecture Notes in Computer Science, Vol. 9822), Norbert Fuhr, Paulo Quaresma, Teresa Goncc alves, Birger Larsen, Krisztian Balog, Craig Macdonald, Linda Cappellato, and Nicola Ferro (Eds.). Springer, 311--331. https://doi.org/10.1007/978--3--319--44564--9_27
[16]
Binsheng Liu, Xiaolu Lu, and J. Shane Culpepper. 2021. Strong Natural Language Query Generation. Inf. Retr., Vol. 24, 4--5 (oct 2021), 322--346. https://doi.org/10.1007/s10791-021-09395--3
[17]
Bulou Liu, Yueyue Wu, Fan Zhang, Yiqun Liu, Zhihong Wang, Chenliang Li, Min Zhang, and Shaoping Ma. 2022. Query Generation and Buffer Mechanism: Towards a better conversational agent for legal case retrieval. Information Processing & Management, Vol. 59, 5 (2022), 103051. https://doi.org/10.1016/j.ipm.2022.103051
[18]
Jiqun Liu and Ran Yu. 2021. State-Aware Meta-Evaluation of Evaluation Metrics in Interactive Information Retrieval. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (Virtual Event, Queensland, Australia) (CIKM '21). Association for Computing Machinery, New York, NY, USA, 3258--3262. https://doi.org/10.1145/3459637.3482190
[19]
Gustavo Penha, Arthur Câ mara, and Claudia Hauff. 2022. Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators. In Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10--14, 2022, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 13185), Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer, 397--412. https://doi.org/10.1007/978--3-030--99736--6_27
[20]
Philipp Schaer, Timo Breuer, Leyla Jael Castro, Benjamin Wolff, Johann Schaible, and Narges Tavakolpoursaleh. 2021. Overview of LiLAS 2021 - Living Labs for Academic Search (Extended Overview). In Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021 (CEUR Workshop Proceedings, Vol. 2936), Guglielmo Faggioli, Nicola Ferro, Alexis Joly, Maria Maistro, and Florina Piroi (Eds.). CEUR-WS.org, 1668--1699. https://ceur-ws.org/Vol-2936/paper-143.pdf
[21]
Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023. Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?arxiv: 2302.03495 [cs.IR]
[22]
Saber Zerhoudi, Sebastian Günther, Kim Plassmeier, Timo Borst, Christin Seifert, Matthias Hagen, and Michael Granitzer. 2022. The SimIIR 2.0 Framework: User Types, Markov Model-Based Interaction Simulation, and Advanced Query Generation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (Atlanta, GA, USA) (CIKM '22). Association for Computing Machinery, New York, NY, USA, 4661--4666. https://doi.org/10.1145/3511808.3557711
Index Terms
- SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024)
Recommendations
Second Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2019)
CHIIR '19: Proceedings of the 2019 Conference on Human Information Interaction and RetrievalThe second WEPIR 2019 workshop brings together researchers with different backgrounds interested in continuing to explore and advance the evaluation of personalisation in information retrieval. The workshop builds on the first WEPIR workshop held at ...
Bibliometric-Enhanced Information Retrieval: 14th International BIR Workshop (BIR 2024)
Advances in Information RetrievalAbstractThe series takes place at ECIR 2024 as a full-day workshop. BIR addresses research topics related to academic search and recommendation, at the intersection of ...
SIGIR 2024 Workshop on eCommerce (ECOM24)
SIGIR '24: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information RetrievalECOM24 brings together researchers and practitioners from academia and industry to identify and discuss core research problems in eCommerce search and recommendation. The workshop aims to foster collaboration, to attract research funding, and to ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2024
3164 pages
ISBN:9798400704314
DOI:10.1145/3626772
- General Chairs:
- Grace Hui Yang,
- Hongning Wang,
- Sam Han,
- Program Chairs:
- Claudia Hauff,
- Guido Zuccon,
- Yi Zhang
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 July 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
Conference
SIGIR 2024
Sponsor:
SIGIR 2024: The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
July 14 - 18, 2024
Washington DC, USA
Acceptance Rates
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 63Total Downloads
- Downloads (Last 12 months)63
- Downloads (Last 6 weeks)9
Reflects downloads up to 21 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in