Building Test Collections for Evaluating Temporal IR
Pages 677 - 680
Abstract
Research on temporal aspects of information retrieval has recently gained considerable interest within the Information Retrieval (IR) community. This paper describes our efforts for building test collections for the purpose of fostering temporal IR research. In particular, we overview the test collections created at the two recent editions of Temporal Information Access (Temporalia) task organized at NTCIR-11 and NTCIR-12, report on selected results and discuss several observations we made during the task design and implementation. Finally, we outline further directions for constructing test collections suitable for temporal IR.
References
[1]
R. Campos, G. Dias, A.M. Jorge, and A. Jatowt. Survey of Temporal Information Retrieval and Related Applications. In: ACM Comp. Surv., 15:1--15:41, 2014.
[2]
C. Clarke, M. Kolla, G. Cormack, O. Vechtomova, A. Ashkan, S. Buttcher, and I. MacKinnon. Novelty and Diversity in Information Retrieval Evaluation. In: Proc. of 31st SIGIR, 659--666, 2008.
[3]
A. Jatowt, C.M. Au Yeung and K. Tanaka. Estimating Document Focus Time. In: CIKM'13, 2273--2278.
[4]
H. Joho, A. Jatowt, and R. Blanco. Temporal Information Searching Behaviour and Strategies. In: Inf. Proc. & Manag., 51(6):834--850 (2015).
[5]
H. Joho, A. Jatowt, R. Blanco, H. Naka and S. Yamamoto. Overview of NTCIR-11 Temporal Information Access (Temporalia) Task. In: Proc. of NTCIR-11, 2014.
[6]
H. Joho, A. Jatowt, R. Blanco, H. Yu and S. Yamamoto. Overview of NTCIR-12 Temporal Information Access (Temporalia-2) Task. In: Proc. of NTCIR-12, to appear, 2016.
[7]
N. Kanhabua, R. Blanco and K. Nørv¨ag. Temporal Information Retrieval. In: Foundations and Trends in Information Retrieval, 9(2):91--208, 2014.
[8]
Y. Liu, R. Song, M. Zhang, Z. Dou, T. Yamamoto, M. P. Kato, H. Ohshima, K. Zhou. Overview of the NTCIR-11 IMine Task. In: Proc. of NTCIR-11, 2014.
[9]
M. Matthews, P. Tolchinsky, R. Blanco, J. Atserias, P. Mika, and H. Zaragoza. Searching through Time in the New York Times. In: Proc. of HCIR 2010, 41--44, 2010.
[10]
S. Nunes, C. Ribeiro, and G. David. Use of Temporal Expressions in Web Search. In: Proc. of ECIR 2008, 580--584, 2008.
[11]
O. Popescu, and C. Strapparava. SemEval2015, Task 7: Diachronic Text Evaluation. In: Proc. of the International Workshop on Semantic Evaluation., 2015.
[12]
T. Sakai, Z. Dou, T. Yamamoto, Y. Liu, M. Zhang, M. P. Kato, R. Song, M. Iwata Summary of the NTCIR-10 INTENT-2 Task: Subtopic Mining and Search Result Diversification. In: Proc. of IGIR 2013, 761--764, 2013.
Index Terms
- Building Test Collections for Evaluating Temporal IR
Recommendations
Integrating standard test collections in interactive IR instruction
TLIR'07: Proceedings of the First international conference on Teaching and Learning of Information RetrievalInformation retrieval experiments usually measure the average effectiveness of the IR methods developed. Direct use of an operational retrieval system has the shortcoming that the user is not given feedback about the effectiveness of queries. The Query ...
Test theory for evaluating reliability of IR test collections
Classical test theory offers theoretically derived reliability measures such as Cronbach's alpha, which can be applied to measure the reliability of a set of Information Retrieval test results. The theory also supports item analysis, which identifies ...
Test theory for assessing IR test collections
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrievalHow good is an IR test collection? A series of papers in recent years has addressed the question by empirically enumerating the consistency of performance comparisons using alternate subsets of the collection. In this paper we propose using Test Theory, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2016
1296 pages
ISBN:9781450340694
DOI:10.1145/2911451
- General Chairs:
- Raffaele Perego,
- Fabrizio Sebastiani,
- Program Chairs:
- Javed Aslam,
- Ian Ruthven,
- Justin Zobel
Copyright © 2016 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 July 2016
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
- MEXT Grant-in-Aid for Young Scientists B
- JST Sakigake
Conference
SIGIR '16
Sponsor:
SIGIR '16: The 39th International ACM SIGIR conference on research and development in Information Retrieval
July 17 - 21, 2016
Pisa, Italy
Acceptance Rates
SIGIR '16 Paper Acceptance Rate 62 of 341 submissions, 18%;
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 217Total Downloads
- Downloads (Last 12 months)6
- Downloads (Last 6 weeks)1
Reflects downloads up to 21 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in