research-article

Report on the 1st simulation for information retrieval workshop (Sim4IR 2021) at SIGIR 2021

Authors:

Krisztian Balog,

Shuo ZhangAuthors Info & Claims

ACM SIGIR Forum, Volume 55, Issue 2

Article No.: 10, Pages 1 - 16

https://doi.org/10.1145/3527546.3527559

Published: 17 March 2022 Publication History

Abstract

Simulation is used as a low-cost and repeatable means of experimentation. As Information Retrieval (IR) researchers, we are no strangers to the idea of using simulation within our own field---such as the traditional means of IR system evaluation as manifested through the Cranfield paradigm. While simulation has been used in other areas of IR research (such as the study of user behaviours), we argue that the potential for using simulation has been recognised by relatively few IR researchers so far.

To this end, the Sim4IR workshop was held online on July 15^th, 2021 in conjunction with ACM SIGIR 2021. Building on past efforts, the goal of the workshop was to create a forum for researchers and practitioners to promote methodology and development of more widespread use of simulation for IR evaluation. Around 80 participants took part over two sessions. A total of two keynotes, three original paper presentations, and eight 'encore talks' were presented. The main conclusions from the resultant discussion were that simulation has the potential to offer solutions to the limitations of existing evaluation methodologies, but there is more research needed toward developing realistic user simulators; and the development and sharing of simulators, in the form of toolkits and online services, is critical for successful uptake.

Date: 15 July, 2021.

Website: https://sim4ir.org.

References

[1]

James Allan, Bruce Croft, Alistair Moffat, and Mark Sanderson. Frontiers, challenges, and opportunities for Information Retrieval: Report from SWIRL 2012, the second strategic workshop on Information Retrieval in Lorne. In ACM SIGIR Forum, volume 46, pages 2--32, 2012.

Digital Library

[2]

Leif Azzopardi. The economics in interactive information retrieval. In Proceedings of the 34^th ACM SIGIR, pages 15--24, 2011.

Digital Library

[3]

Leif Azzopardi, Kalervo Järvelin, Jaap Kamps, and Mark D Smucker. Report on the SIGIR 2010 workshop on the simulation of interaction. In ACM SIGIR Forum, volume 44, pages 35--47, 2011.

[4]

Krisztian Balog, Xianjie Chen, Xu Chen David Maxwell, Paul Thomas, Shuo Zhang, Yi Zhang, and Yongfeng Zhang, editors. CSR-Sim4IR 2021: Joint Proceedings of the Causality in Search and Recommendation (CSR) and Simulation of Information Retrieval Evaluation (Sim4IR) Workshops, volume 2911 of CEUR Workshop Proceedings, 2021. CEUR-WS.org. URL http://ceur-ws.org/Vol-2911.

[5]

Feza Baskaya, Heikki Keskustalo, and Kalervo Järvelin. Time drives interaction: Simulating sessions in diverse searching environments. In Proceedings of the 35^th ACM SIGIR, page 105--114, 2012.

Digital Library

[6]

Feza Baskaya, Heikki Keskustalo, and Kalervo Järvelin. Modeling behavioral factors in interactive information retrieval. In Proceedings of the 22^nd ACM CIKM, pages 2297--2302, 2013.

[7]

Dimitrios Bountouridis, Jaron Harambam, Mykola Makhortykh, Mónica Marrero, Nava Tintarev, and Claudia Hauff. Siren: A simulation framework for understanding the effects of recommender systems in online news environments. In Proceedings of FAT*, page 150--159, 2019.

Digital Library

[8]

Fei Cai and Maarten de Rijke. A survey of query auto completion in information retrieval. Foundations and Trends in Information Retrieval, 10(4):273--363, 2016.

Digital Library

[9]

Ben Carterette, Ashraf Bah, and Mustafa Zengin. Dynamic test collections for retrieval evaluation. In Proceedings of the 1^st ACM ICTIR, pages 91--100, 2015.

Digital Library

[10]

Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. Click Models for Web Search. Morgan & Claypool, 2015.

[11]

Cyril W. Cleverdon, Jack Mills, and Michael Keen. Factors Determining the Performance of Indexing Systems, volume 1:2 of Cranfield Research Projects. 1966.

[12]

Michael D Cooper. A simulation model of an information retrieval system. Information Storage and Retrieval, 9(1):13--32, 1973.

[13]

J Shane Culpepper, Fernando Diaz, and Mark D Smucker. Report from the Third Strategic Workshop on Information Retrieval (SWIRL). 52(1):34--90, 2018.

[14]

Pierre Erbacher, Laure Soulier, and Ludovic Denoyer. State of the art of user simulation approaches for conversational information retrieval. In Balog et al. [2021], pages 32--37. URL http://ceur-ws.org/Vol-2911/paper5.pdf.

[15]

Paul A. Fishwick. Computer simulation: The art and science of digital world construction. Technical report, University of Florida, 1995.

[16]

Jianfeng Gao, Michel Galley, and Lihong Li. Neural Approaches to Conversational AI. Foundations and Trends in Information Retrieval, 13(2--3):127--298, 2019.

[17]

Sebastian Günther and Matthias Hagen. Assessing query suggestions for search session simulation. In Balog et al. [2021], pages 38--45. URL http://ceur-ws.org/Vol-2911/paper6.pdf.

[18]

Donna Harman. Overview of the first trec conference. In Proceedings of the 16^th ACM SIGIR, pages 36--47, 1993.

Digital Library

[19]

David Hawking, Bodo Billerbeck, Paul Thomas, and Nick Craswell. Simulating Information Retrieval Test Collections. Morgan and Claypool, 2020.

[20]

Katja Hofmann, Lihong Li, and Filip Radlinski. Online evaluation for information retrieval. Foundations and Trends in Information Retrieval, 10(1):1--117, 2016.

Digital Library

[21]

Frank Hopfgartner, Allan Hanbury, Henning Müller, Ivan Eggel, Krisztian Balog, Torben Brodt, Gordon V. Cormack, Jimmy Lin, Jayashree Kalpathy-Cramer, Noriko Kando, Makoto P. Kato, Anastasia Krithara, Tim Gollub, Martin Potthast, Evelyne Viegas, and Simon Mercer. Evaluation-as-a-service for the computational sciences: Overview and outlook. Journal of Data and Information Quality, 10(4):15:1--15:32, October 2018.

Digital Library

[22]

Jin Huang, Harrie Oosterhuis, Maarten de Rijke, and Herke van Hoof. Keeping dataset biases out of the simulation: A debiased simulator for reinforcement learning based recommender systems. In Proceedings of the 14^th ACM RecSys, page 190--199, 2020.

Digital Library

[23]

Eugene Ie, Chih-Wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. Recsim: A configurable simulation platform for recommender systems. Computing Research Repository, abs/1909.04847, 2019.

[24]

Rolf Jagerman, Krisztian Balog, and Maarten De Rijke. Opensearch: Lessons learned from an online evaluation campaign. Journal of Data and Information Quality, 10(3):13:1--13:15, September 2018.

Digital Library

[25]

Marc I Kellner, Raymond J Madachy, and David M Raffo. Software process simulation modeling: Why? What? How? Journal of Systems and Software, 46(2--3):91--105, 1999.

[26]

Diane Kelly. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3:1--224, 1 2009.

Digital Library

[27]

R. Kohavi, D. Tang, and Y. Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020.

[28]

Sahiti Labhishetty and Chengxiang Zhai. An exploration of tester-based evaluation of user simulators for comparing interactive retrieval systems. In Proceedings of the 44^th ACM SIGIR, pages 1598--1602, 2021.

Digital Library

[29]

Sahiti Labhishetty, Chengxiang Zhai, Suhas Ranganath, and Pradeep Ranganathan. A cognitive user model for e-commerce search. In Proceedings of the Data Science for Retail and E-Commerce Workshop, 2020.

[30]

Aldo Lipani, Ben Carterette, and Emine Yilmaz. How am I Doing?: Evaluating conversational search systems offline. ACM Transactions on Intelligent Systems and Technology, 2021.

Digital Library

[31]

Jiyun Luo, Sicong Zhang, and Hui Yang. Win-win search: Dual-agent stochastic game in session search. In Proceedings of the 37^th ACM SIGIR, pages 587--596, 2014.

[32]

Jiyun Luo, Sicong Zhang, Xuchu Dong, and Hui Yang. Designing states, actions, and rewards for using pomdp in session search. In Advances in Information Retrieval, pages 526--537, 2015.

[33]

David Maxwell. Modelling Search and Stopping in Interactive Information Retrieval. PhD thesis, University of Glasgow, Scotland, April 2019. URL https://theses.gla.ac.uk/41132/.

Digital Library

[34]

David Maxwell and Leif Azzopardi. Agents, simulated users and humans: An analysis of performance and behaviour. In Proceedings of the 25^th ACM CIKM, pages 731--740, 2016a.

Digital Library

[35]

David Maxwell and Leif Azzopardi. Simulating interactive information retrieval: SimIIR: A framework for the simulation of interaction. In Proceedings of the 39^th ACM SIGIR, pages 1141--1144, 2016b.

Digital Library

[36]

David Maxwell and Leif Azzopardi. Simulating interactive information retrieval: SimIIR: A framework for the simulation of interaction. In Proceedings of the 39^th ACM SIGIR, pages 1141--1144, 2016c.

Digital Library

[37]

David Maxwell and Leif Azzopardi. Information scent, searching and stopping: Modelling SERP level stopping behaviour. In Advances in Information Retrieval, pages 210--222, 2018.

[38]

Teemu Pääkkönen, Kalervo Järvelin, Jaana Kekäläinen, Heikki Keskustalo, Feza Baskaya, David Maxwell, and Leif Azzopardi. Exploring behavioral dimensions in session effectiveness. In Proceedings of the 6^th CLEF, pages 178--189, 2015.

[39]

Nirmal Roy, Arthur Câmara, David Maxwell, and Claudia Hauff. Incorporating widget positioning in interaction models of search behaviour. In Proceedings of the 7^th ACM ICTIR, pages 53--62, 2021.

Digital Library

[40]

Alexandre Salle, Shervin Malmasi, Oleg Rokhlenko, and Eugene Agichtein. Studying the effectiveness of conversational search refinement through user simulation. In Advances in Information Retrieval, pages 587--602, 2021.

Digital Library

[41]

Mark Sanderson. Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4(4):247--375, 2010.

[42]

Manel Slokom. Comparing recommender systems using synthetic data. In Proceedings of the 12^th ACM RecSys, page 548--552, 2018.

Digital Library

[43]

Mark D. Smucker. An analysis of user strategies for examining and processing ranked lists of documents. In Proceedings of the 5^th HCIR, 2011.

[44]

Mark D. Smucker, Gabriella Kazai, and Matthew Lease. Overview of the TREC 2012 crowdsourcing track. In Proceedings of the 21^st TREC, 2012.

[45]

Jean Tague, Michael Nelson, and Harry Wu. Problems in the simulation of bibliographic retrieval systems. In Proceedings of the 3^rd ACM SIGIR, pages 236--255, 1980.

[46]

Suzan Verberne, Maya Sappelli, Kalervo Järvelin, and Wessel Kraaij. User simulations for interactive search: Evaluating personalized query suggestion. In Advances in Information Retrieval, volume 9022, pages 678--690. 2015.

[47]

Ellen M. Voorhees. The evolution of Cranfield. In Nicola Ferro and Carol Peters, editors, Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF, volume 41 of The Information Retrieval Series, pages 45--69. Springer, 2019.

[48]

Junqi Zhang, Jiaxin Mao, Yiqun Liu, Ruizhe Zhang, Min Zhang, Shaoping Ma, Jun Xu, and Qi Tian. Context-aware ranking by constructing a virtual environment for reinforcement learning. In Proceedings of the 28^th ACM CIKM, page 1603--1612, 2019.

Digital Library

[49]

Shuo Zhang and Krisztian Balog. EntiTables: Smart assistance for entity-focused tables. In Proceedings of the 40^th ACM SIGIR, pages 255--264, 2017.

[50]

Shuo Zhang and Krisztian Balog. Evaluating conversational recommender systems via user simulation. In Proceedings of the 26^th ACM SIGKDD, KDD '20, pages 1512--1520, 2020.

Digital Library

[51]

Yinan Zhang, Xueqing Liu, and ChengXiang Zhai. Information retrieval evaluation as search simulation: A general formal framework for ir evaluation. In Proceedings of ICTIR'17, pages 193--200, 2017.

Digital Library

[52]

Yurou Zhao, Jiaxin Mao, and Qingyao Ai. ULTRE framework: a framework for unbiased learning to rank evaluation based on simulation of user behavior. In Balog et al. [2021], pages 46--51. URL http://ceur-ws.org/Vol-2911/paper7.pdf.

Cited By

Bernard NBalog KOosterhuis HBast HXiong C(2024)Towards a Formal Characterization of User Simulation Objectives in Conversational Information AccessProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672529(185-193)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672529
Schaer PKreutz CBalog KBreuer TFuhr NHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024)Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657991(3058-3061)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657991
Breuer TFuhr NSchaer P(2024)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/362364016:1(1-33)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3623640
Show More Cited By

Report on the 1st simulation for information retrieval workshop (Sim4IR 2021) at SIGIR 2021
1. Information systems
  1. Information retrieval

Recommendations

Sim4IR: The SIGIR 2021 Workshop on Simulation for Information Retrieval Evaluation
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

The use of simulation techniques is not foreign to information retrieval. In the past, simulation has been employed, for example, for constructing test collections and for model performance prediction and analysis in a broad array of information access ...
Report on the SIGIR 2016 Workshop on Neural Information Retrieval (Neu-IR)

The SIGIR 2016 workshop on Neural Information Retrieval (Neu-IR) took place on 21 July, 2016 in Pisa. The goal of the Neu-IR (pronounced "New IR") workshop was to serve as a forum for academic and industrial researchers, working at the intersection of ...
ICTIR '21: Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGIR Forum

ACM SIGIR Forum Volume 55, Issue 2

December 2021

247 pages

ISSN:0163-5840

DOI:10.1145/3527546

Issue’s Table of Contents

Copyright © 2022 Copyright is held by the owner/author(s).

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 March 2022

Published in SIGIR Volume 55, Issue 2

Check for updates

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
134
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bernard NBalog KOosterhuis HBast HXiong C(2024)Towards a Formal Characterization of User Simulation Objectives in Conversational Information AccessProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672529(185-193)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672529
Schaer PKreutz CBalog KBreuer TFuhr NHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024)Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657991(3058-3061)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657991
Breuer TFuhr NSchaer P(2024)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/362364016:1(1-33)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3623640
Ekstrand MCarterette BDiaz F(2024)Distributionally-Informed Recommender System EvaluationACM Transactions on Recommender Systems10.1145/36134552:1(1-27)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3613455
Friese J(2024)Reproduction and Simulation of Interactive Retrieval ExperimentsAdvances in Information Retrieval10.1007/978-3-031-56069-9_40(328-330)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56069-9_40
Engelmann BBreuer TSchaer PFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Simulating Users in Interactive Web Table RetrievalProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615187(3875-3879)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615187
Afzali JDrzewiecki ABalog KZhang SChua TLauw HSi LTerzi ETsaparas P(2023)UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender SystemsProceedings of the Sixteenth ACM International Conference on Web Search and Data Mining10.1145/3539597.3573029(1160-1163)Online publication date: 27-Feb-2023
https://dl.acm.org/doi/10.1145/3539597.3573029
Pan XXue Y(2023)Advancements of Artificial Intelligence Techniques in the Realm About Library and Information Subject—A Case Survey of Latent Dirichlet Allocation MethodIEEE Access10.1109/ACCESS.2023.333461911(132627-132640)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3334619
Wang SZuccon GAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Is Non-IID Data a Threat in Federated Online Learning to Rank?Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531709(2801-2813)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531709
Stavinova EGurov ALysenko AChunaev P(2022)Performance Ranking of Recommender Systems on Simulated DataProcedia Computer Science10.1016/j.procs.2022.10.216212:C(142-151)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1016/j.procs.2022.10.216

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents