Article

IDF revisited: a simple new derivation within the Robertson-Spärck Jones probabilistic model

Author:

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 751 - 752

https://doi.org/10.1145/1277741.1277891

Published: 23 July 2007 Publication History

Get Access

Abstract

There have been a number of prior attempts to theoretically justify the effectiveness of the inverse document frequency (IDF). Those that take as their starting point Robertson and Sparck Jones's probabilistic model are based on strong or complex assumptions. We show that a more intuitively plausible assumption suffices. Moreover, the new assumption, while conceptually very simple, provides a solution to an estimation problem that had been deemed intractable by Robertson and Walker (1997).

References

[1]

K. W. Church and W. A. Gale. Inverse document frequency (IDF): A measure of deviations from Poisson. In Proceedings of the Third Workshop on Very Large Corpora (WVLC), pages 121--130, 1995.

Google Scholar

[2]

W. B. Croft and D. J. Harper. Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 35(4):285--295, 1979. Reprinted in Karen Spärck Jones and Peter Willett, eds., Readings in Information Retrieval, Morgan Kaufmann, pp. 339--344, 1997.

Crossref

Google Scholar

[3]

A. P. de Vries and T. Roelleke. Relevance information: A loss of entropy but a gain for idf? In Proceedings of SIGIR, pages 282--289, 2005.

Digital Library

Google Scholar

[4]

H. Fang, T. Tao, and C. Zhai. A formal study of information retrieval heuristics. In Proceedings of SIGIR, pages 49--56, 2004.

Digital Library

Google Scholar

[5]

W. R. Greiff. A theory of term weighting based on exploratory data analysis. In Proceedings of SIGIR, pages 11--19, New York, NY, USA, 1998.

Digital Library

Google Scholar

[6]

D. Harman. The history of IDF and its in uences on IR and other fields. In Charting a New Course: Natural Language Processing and Information Retrieval: Essays in Honour of Karen Spärck Jones, pages 69--79. Springer, 2005.

Google Scholar

[7]

C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval, chapter 11 (Probabilistic information retrieval). Cambridge University Press, 2007. Draft of April 28.

Google Scholar

[8]

K. Papineni. Why inverse document frequency? In Proceedings of the NAACL, pages 1--8, 1995.

Digital Library

Google Scholar

[9]

S. E. Robertson. Understanding inverse document frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5):503--520, 2004.

Crossref

Google Scholar

[10]

S. E. Robertson and K. Spärck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27(3):129--146, 1976.

Crossref

Google Scholar

[11]

S. E. Robertson and S. Walker. On relevance weights with little relevance information. In Proceedings of SIGIR, pages 16--24, 1997.

Digital Library

Google Scholar

[12]

K. Spärck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11--21, 1972.

Crossref

Google Scholar

[13]

I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, second edition, 1999.

Digital Library

Google Scholar

[14]

S. K. M. Wong and Y. Y. Yao. A note on inverse document frequency weighting scheme {sic}. Technical Report TR-89-990, Cornell University, Ithaca, NY, USA, 1989.

Digital Library

Google Scholar

Cited By

View all

Huang YHsu CChune YLiao IWang HLin YHwu WLee NLai F(2022)Diagnosis of a Single-Nucleotide Variant in Whole-Exome Sequencing Data for Patients With Inherited Diseases: Machine Learning Study Using Artificial Intelligence Variant PrioritizationJMIR Bioinformatics and Biotechnology10.2196/377013:1(e37701)Online publication date: 15-Sep-2022
https://doi.org/10.2196/37701
Jimenez SCucerzan SGonzalez FGelbukh ADueñas G(2018)BM25-CTF: Improving TF and IDF factors in BM25 by using collection term frequenciesJournal of Intelligent & Fuzzy Systems10.3233/JIFS-16947534:5(2887-2899)Online publication date: 24-May-2018
https://doi.org/10.3233/JIFS-169475
Crimp RTrotman A(2017)Automatic Term Reweighting for Query ExpansionProceedings of the 22nd Australasian Document Computing Symposium10.1145/3166072.3166074(1-4)Online publication date: 7-Dec-2017
https://dl.acm.org/doi/10.1145/3166072.3166074
Show More Cited By

Index Terms

IDF revisited: a simple new derivation within the Robertson-Spärck Jones probabilistic model
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

N-gram IDF: A Global Term Weighting Scheme Based on Information Distance
WWW '15: Proceedings of the 24th International Conference on World Wide Web

This paper first reveals the relationship between Inverse Document Frequency (IDF), a global term weighting scheme, and information distance, a universal metric defined by Kolmogorov complexity. We concretely give a theoretical explanation that the IDF ...
Information Retrieval by Modified Term Weighting Method Using Random Walk Model with Query Term Position Ranking
ICSPS '09: Proceedings of the 2009 International Conference on Signal Processing Systems

Term weighting is a core idea behind any information retrieval technique which has crucial importance in document ranking. In graph based ranking algorithm, terms within a document are represented as a graph of that document. Term weights for ...
Exploring the stability of IDF term weighting
AIRS'08: Proceedings of the 4th Asia information retrieval conference on Information retrieval technology

TF IDF has been widely used as a term weighting schemes in today's information retrieval systems. However, computation time and cost have become major concerns for its application. This study investigated the similarities and differences between IDF ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

July 2007

946 pages

ISBN:9781595935977

DOI:10.1145/1277741

General Chairs:
Wessel Kraaij
TNO, The Netherlands
,
Arjen P. de Vries
CWI, The Netherlands
,
Program Chairs:
Charles L. A. Clarke
University of Waterloo, Canada
,
Norbert Fuhr
University of Duisburg-Essen, Germany
,
Noriko Kando
National Institute of Informatics, Japan

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR07

Sponsor:

SIGIR07: The 30th Annual International SIGIR Conference

July 23 - 27, 2007

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
414
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Huang YHsu CChune YLiao IWang HLin YHwu WLee NLai F(2022)Diagnosis of a Single-Nucleotide Variant in Whole-Exome Sequencing Data for Patients With Inherited Diseases: Machine Learning Study Using Artificial Intelligence Variant PrioritizationJMIR Bioinformatics and Biotechnology10.2196/377013:1(e37701)Online publication date: 15-Sep-2022
https://doi.org/10.2196/37701
Jimenez SCucerzan SGonzalez FGelbukh ADueñas G(2018)BM25-CTF: Improving TF and IDF factors in BM25 by using collection term frequenciesJournal of Intelligent & Fuzzy Systems10.3233/JIFS-16947534:5(2887-2899)Online publication date: 24-May-2018
https://doi.org/10.3233/JIFS-169475
Crimp RTrotman A(2017)Automatic Term Reweighting for Query ExpansionProceedings of the 22nd Australasian Document Computing Symposium10.1145/3166072.3166074(1-4)Online publication date: 7-Dec-2017
https://dl.acm.org/doi/10.1145/3166072.3166074
Lu XMoffat ACulpepper JCarterette BFang HLalmas MNie J(2016)Efficient and Effective Higher Order Proximity ModelingProceedings of the 2016 ACM International Conference on the Theory of Information Retrieval10.1145/2970398.2970404(21-30)Online publication date: 12-Sep-2016
https://dl.acm.org/doi/10.1145/2970398.2970404
Trotman APuurula ABurgess B(2014)Improvements to BM25 and Language Models ExaminedProceedings of the 19th Australasian Document Computing Symposium10.1145/2682862.2682863(58-65)Online publication date: 26-Nov-2014
https://dl.acm.org/doi/10.1145/2682862.2682863
Puurula A(2012)Combining Modifications to Multinomial Naive Bayes for Text ClassificationInformation Retrieval Technology10.1007/978-3-642-35341-3_10(114-125)Online publication date: 2012
https://doi.org/10.1007/978-3-642-35341-3_10
Metzler DShanahan JAmer-Yahia SManolescu IZhang YEvans DKolcz AChoi KChowdury A(2008)Generalized inverse document frequencyProceedings of the 17th ACM conference on Information and knowledge management10.1145/1458082.1458137(399-408)Online publication date: 26-Oct-2008
https://dl.acm.org/doi/10.1145/1458082.1458137
Wu HLuk RWong KKwok K(2008)Interpreting TF-IDF term weights as making relevance decisionsACM Transactions on Information Systems10.1145/1361684.136168626:3(1-37)Online publication date: 20-Jun-2008
https://dl.acm.org/doi/10.1145/1361684.1361686

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

N-gram IDF: A Global Term Weighting Scheme Based on Information Distance

Information Retrieval by Modified Term Weighting Method Using Random Walk Model with Query Term Position Ranking

Exploring the stability of IDF term weighting