demonstration

Infrastructure support for evaluation as a service

Authors:

Jimmy Lin,

Miles EfronAuthors Info & Claims

WWW '14 Companion: Proceedings of the 23rd International Conference on World Wide Web

Pages 79 - 82

https://doi.org/10.1145/2567948.2577014

Published: 07 April 2014 Publication History

Get Access

Abstract

How do we conduct large-scale community-wide evaluations for information retrieval if we are unable to distribute the document collection? This was the challenge we faced in organizing a task on searching tweets at the Text Retrieval Conference (TREC), since Twitter's terms of service forbid redistribution of tweets. Our solution, which we call "evaluation as a service", was to provide an API through which the collection can be accessed for completing the evaluation task. This paper describes the infrastructure underlying the service and its deployment at TREC 2013. We discuss the merits of the approach and potential applicability to other evaluation scenarios.

References

[1]

C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. SIGIR, 2004.

Digital Library

Google Scholar

[2]

G. Cormack and T. Lynam. TREC 2005 spam track overview. TREC, 2005.

Google Scholar

[3]

D. Harman. Information Retrieval Evaluation. Morgan & Claypool Publishers, 2011.

Crossref

Google Scholar

[4]

V. Lavrenko and W. B. Croft. Relevance-based language models. SIGIR, 2001.

Digital Library

Google Scholar

[5]

I. Ounis, C. Macdonald, J. Lin, and I. Soboroff. Overview of the TREC-2011mboxMicroblog Track. TREC, 2011.

Google Scholar

[6]

M. Sanderson and J. Zobel. Information retrieval system evaluation:\ Effort, sensitivity, and reliability. SIGIR, 2005.

Digital Library

Google Scholar

[7]

I. Soboroff, I. Ounis, C. Macdonald, and J. Lin. Overview of the TREC-2012 Microblog Track. TREC, 2012.

Google Scholar

[8]

E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. SIGIR, 1998.

Digital Library

Google Scholar

[9]

E. M. Voorhees. The philosophy of information retrieval evaluation. CLEF, 2002.

Crossref

Google Scholar

Cited By

View all

Amigó ECarrillo-de-Albornoz JAlmagro-Cádiz MGonzalo JRodríguez-Vidal JVerdejo FKando NSakai TJoho HLi Hde Vries AWhite R(2017)EvALLProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3077136.3084145(1301-1304)Online publication date: 7-Aug-2017
https://dl.acm.org/doi/10.1145/3077136.3084145
Yang PFang HCarterette BFang HLalmas MNie J(2016)A Reproducibility Study of Information Retrieval ModelsProceedings of the 2016 ACM International Conference on the Theory of Information Retrieval10.1145/2970398.2970415(77-86)Online publication date: 12-Sep-2016
https://dl.acm.org/doi/10.1145/2970398.2970415
Rao JLin JEfron M(2015)Reproducible Experiments on Lexical and Temporal Feedback for Tweet SearchAdvances in Information Retrieval10.1007/978-3-319-16354-3_82(755-767)Online publication date: 2015
https://doi.org/10.1007/978-3-319-16354-3_82

Index Terms

Infrastructure support for evaluation as a service
1. Information systems
  1. Information retrieval

Recommendations

Short Query Expansion for Microblog Retrieval

Twitter has emerged as the most popular among microblogging service providers. The content provided in Twitter is large, diverse, and huge in quantity. Given the increasing amount of information available through such microblogging sites, it would be ...
Uprising microblogs: a bayesian network retrieval model for tweet search
SAC '12: Proceedings of the 27th Annual ACM Symposium on Applied Computing

We investigate in this paper the problem of accessing to real-time information and we propose a Bayesian network retrieval model for tweet search. The proposed model interprets tweet relevance as a conditional probability and estimates it using different ...
QuickView: advanced search of tweets
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

Tweets have become a comprehensive repository for real-time information. However, it is often hard for users to quickly get information they are interested in from tweets, owing to the sheer volume of tweets as well as their noisy and informal nature. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

WWW '14 Companion: Proceedings of the 23rd International Conference on World Wide Web

April 2014

1396 pages

ISBN:9781450327459

DOI:10.1145/2567948

General Chair:
Chin-Wan Chung
Korea Advanced Institute of Science and Technology, Korea
,
Program Chairs:
Andrei Broder
Google Inc., USA
,
Kyuseok Shim
Seoul National University, Korea
,
Torsten Suel
New York University, USA

In-Cooperation

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 April 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Demonstration

Funding Sources

Division of Information and Intelligent Systems

Conference

WWW '14

Sponsor:

IW3C2

WWW '14: 23rd International World Wide Web Conference

April 7 - 11, 2014

Seoul, Korea

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
99
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Amigó ECarrillo-de-Albornoz JAlmagro-Cádiz MGonzalo JRodríguez-Vidal JVerdejo FKando NSakai TJoho HLi Hde Vries AWhite R(2017)EvALLProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3077136.3084145(1301-1304)Online publication date: 7-Aug-2017
https://dl.acm.org/doi/10.1145/3077136.3084145
Yang PFang HCarterette BFang HLalmas MNie J(2016)A Reproducibility Study of Information Retrieval ModelsProceedings of the 2016 ACM International Conference on the Theory of Information Retrieval10.1145/2970398.2970415(77-86)Online publication date: 12-Sep-2016
https://dl.acm.org/doi/10.1145/2970398.2970415
Rao JLin JEfron M(2015)Reproducible Experiments on Lexical and Temporal Feedback for Tweet SearchAdvances in Information Retrieval10.1007/978-3-319-16354-3_82(755-767)Online publication date: 2015
https://doi.org/10.1007/978-3-319-16354-3_82

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Short Query Expansion for Microblog Retrieval

Uprising microblogs: a bayesian network retrieval model for tweet search

QuickView: advanced search of tweets

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations