column

The Dagstuhl Perspectives Workshop on Performance Modeling and Prediction

ACM SIGIR Forum, Volume 52, Issue 1

Pages 91 - 101

https://doi.org/10.1145/3274784.3274789

Published: 31 August 2018 Publication History

Abstract

This paper reports the findings of the Dagstuhl Perspectives Workshop 17442 on performance modeling and prediction in the domains of Information Retrieval, Natural language Processing and Recommender Systems. We present a framework for further research, which identifies five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of prediction models describing the relationship between assumptions, features and resulting performance.

References

[1]

David Banks, Paul Over, and Nien-Fan Zhang. Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval, 1(1-2):7-34, May 1999.

Digital Library

[2]

Ben Carterette. The best published result is random: Sequential testing and its effect on reported effectiveness. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, pages 747-750, New York, NY, USA, 2015. ACM.

Digital Library

[3]

O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected Reciprocal Rank for Graded Relevance. In D. W.-L. Cheung, I.-Y. Song, W. W. Chu, X. Hu, and J. J. Lin, editors, Proc. 18th International Conference on Information and Knowledge Management (CIKM 2009), pages 621-630. ACM Press, New York, USA, 2009.

Digital Library

[4]

M. Ferrante, N. Ferro, and M. Maistro. Injecting User Models and Time into Precision via Markov Chains. In S. Geva, A. Trotman, P. Bruza, C. L. A. Clarke, and K. Järvelin, editors, Proc. 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2014), pages 597-606. ACM Press, New York, USA, 2014.

Digital Library

[5]

M. Ferrante, N. Ferro, and S. Pontarollo. Are IR Evaluation Measures on an Interval Scale? In J. Kamps, E. Kanoulas, M. de Rijke, H. Fang, and E. Yilmaz, editors, Proc. 3rd ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2017), pages 67-74. ACM Press, New York, USA, 2017.

Digital Library

[6]

N. Ferro. Reproducibility Challenges in Information Retrieval Evaluation. ACM Journal of Data and Information Quality (JDIQ), 8(2):8:1-8:4, February 2017.

Digital Library

[7]

N. Ferro, N. Fuhr, K. Järvelin, N. Kando, M. Lippold, and J. Zobel. Increasing Reproducibility in IR: Findings from the Dagstuhl Seminar on "Reproducibility of Data-Oriented Experiments in e-Science"s. SIGIR Forum, 50(1):68-82, June 2016.

Digital Library

[8]

N. Ferro and M. Sanderson. Sub-corpora Impact on System Effectiveness. In N. Kando, T. Sakai, H. Joho, H. Li, A. P. de Vries, and R. W. White, editors, Proc. 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), pages 901-904. ACM Press, New York, USA, 2017.

Digital Library

[9]

N. Ferro and G. Silvello. A General Linear Mixed Models Approach to Study System Component Effects. In R. Perego, F. Sebastiani, J. Aslam, I. Ruthven, and J. Zobel, editors, Proc. 39th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016), pages 25-34. ACM Press, New York, USA, 2016.

Digital Library

[10]

N. Ferro and G. Silvello. Toward an Anatomy of IR System Component Performances. Journal of the American Society for Information Science and Technology (JASIST), 69(2):187-200, February 2018.

[11]

Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindn, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, and Justin Zobel. Building a predictive science for performance of information retrieval, natural language processing, and recommender systems applications (dagstuhl perspectives workshop 17442). Dagstuhl Manifestos, 8, 2018.

[12]

N. Fuhr. Some Common Mistakes In IR Evaluation, And How They Can Be Avoided. SIGIR Forum, 51(3):32-41, December 2017.

Digital Library

[13]

P. J. Huber and E. M. Ronchetti. Robust Statistics. John Wiley & Sons, USA, 2nd edition, 2009.

[14]

T. Kariya and B. K. Sinha. Robustness of Statistical Tests. Academic Press, USA, 1989.

[15]

A. Moffat and J. Zobel. Rank-biased Precision for Measurement of Retrieval Effectiveness. ACM Transactions on Information Systems (TOIS), 27(1):2:1-2:27, 2008.

Digital Library

[16]

S. E. Robertson and E. Kanoulas. On Per-topic Variance in IR Evaluation. In W. Hersh, J. Callan, Y. Maarek, and M. Sanderson, editors, Proc. 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2012), pages 891-900. ACM Press, New York, USA, 2012.

Digital Library

[17]

J. M. Tague-Sutcliffe and J. Blustein. A Statistical Analysis of the TREC-3 Data. In D. K. Harman, editor, The Third Text REtrieval Conference (TREC-3), pages 385-398. National Institute of Standards and Technology (NIST), Special Publication 500-225, Washington, USA, 1994.

[18]

Julin Urbano. Test collection reliability: a study of bias and robustness to statistical assumptions via stochastic simulation. Information Retrieval Journal, 19(3):313-350, December 2015.

Digital Library

[19]

E. M. Voorhees, D. Samarov, and I. Soboroff. Using Replicates in Information Retrieval Evaluation. ACM Transactions on Information Systems (TOIS), 36(2):12:1-12:21, September 2017.

Digital Library

Cited By

Fuhr N(2021)Proof by experimentation?ACM SIGIR Forum10.1145/3483382.348338554:2(1-4)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3483382.3483385
Valcarce DBellogín AParapar JCastells P(2020)Assessing ranking metrics in top-N recommendationInformation Retrieval10.1007/s10791-020-09377-x23:4(411-448)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s10791-020-09377-x
Ferrante MFerro NLosiouk E(2020)How do interval scales help us with better understanding IR evaluation measures?Information Retrieval10.1007/s10791-019-09362-z23:3(289-317)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1007/s10791-019-09362-z
Show More Cited By

Index Terms

The Dagstuhl Perspectives Workshop on Performance Modeling and Prediction
1. Information systems
  1. Information retrieval
  2. Information storage systems

Recommendations

Current perspectives on the software engineering process
Abstract
This volume comprises a selected set of high‐quality and extended articles of the 26th Systems, Software and Services Process Improvement (EuroSPI) Conference, held during September 18–20, 2019 in Edinburgh, UK. Conferences were held in Dublin (...
La programmation a memoire adaptative et les algorithmes pseudo-gloutons: nouvelles perspectives pour les meta-heuristiques
Transitional Flow Prediction in High Reynolds Flows Over Hydrofoils

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGIR Forum

ACM SIGIR Forum Volume 52, Issue 1

June 2018

167 pages

ISSN:0163-5840

DOI:10.1145/3274784

Editors:
Claudia Hauff
Delft University of Technology. The Netherlands
,
Craig Macdonald
University of Glasgow, Glasgow, UK

Issue’s Table of Contents

Copyright © 2018 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 August 2018

Published in SIGIR Volume 52, Issue 1

Check for updates

Qualifiers

Column

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
124
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)3

Reflects downloads up to 23 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fuhr N(2021)Proof by experimentation?ACM SIGIR Forum10.1145/3483382.348338554:2(1-4)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3483382.3483385
Valcarce DBellogín AParapar JCastells P(2020)Assessing ranking metrics in top-N recommendationInformation Retrieval10.1007/s10791-020-09377-x23:4(411-448)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s10791-020-09377-x
Ferrante MFerro NLosiouk E(2020)How do interval scales help us with better understanding IR evaluation measures?Information Retrieval10.1007/s10791-019-09362-z23:3(289-317)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1007/s10791-019-09362-z
Ferro NKim YSanderson M(2019)Using Collection Shards to Study Retrieval Performance Effect SizesACM Transactions on Information Systems10.1145/331036437:3(1-40)Online publication date: 19-Mar-2019
https://dl.acm.org/doi/10.1145/3310364
Soboroff IFerro NFuhr N(2019)Report on GLARE 2018ACM SIGIR Forum10.1145/3308774.330879652:2(132-137)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.1145/3308774.3308796
Ferro NFuhr NMaistro MSakai TSoboroff I(2019)Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility RealmExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-030-28577-7_24(287-300)Online publication date: 9-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-28577-7_24
Fuhr N(2019)Reproducibility and Validity in CLEFInformation Retrieval Evaluation in a Changing World10.1007/978-3-030-22948-1_23(555-564)Online publication date: 14-Aug-2019
https://doi.org/10.1007/978-3-030-22948-1_23
Aker AFuhr N(2018)The Information Retrieval Group at the University of Duisburg-EssenDatenbank-Spektrum10.1007/s13222-018-0290-018:2(113-119)Online publication date: 3-Jul-2018
https://doi.org/10.1007/s13222-018-0290-0

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents