Article

The maximum entropy method for analyzing retrieval measures

Authors:

Javed A. Aslam,

Virgiliu PavluAuthors Info & Claims

SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 27 - 34

https://doi.org/10.1145/1076034.1076042

Published: 15 August 2005 Publication History

Abstract

We present a model, based on the maximum entropy method, for analyzing various measures of retrieval performance such as average precision, R-precision, and precision-at-cutoffs. Our methodology treats the value of such a measure as a constraint on the distribution of relevant documents in an unknown list, and the maximum entropy distribution can be determined subject to these constraints. For good measures of overall performance (such as average precision), the resulting maximum entropy distributions are highly correlated with actual distributions of relevant documents in lists as demonstrated through TREC data; for poor measures of overall performance, the correlation is weaker. As such, the maximum entropy method can be used to quantify the overall quality of a retrieval measure. Furthermore, for good measures of overall performance (such as average precision), we show that the corresponding maximum entropy distributions can be used to accurately infer precision-recall curves and the values of other measures of performance, and we demonstrate that the quality of these inferences far exceeds that predicted by simple retrieval measure correlation, as demonstrated through TREC data.

References

[1]

A. L. Berger, V. D. Pietra, and S. D. Pietra. A maximum entropy approach to natural language processing. Comput. Linguist., 22:39--71, 1996.

Digital Library

[2]

C. Buckley and E. Voorhees. Evaluating evaluation measure stability. In SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 33--40. ACM Press, 2000.

Digital Library

[3]

W. S. Cooper. On selecting a measure of retrieval effectiveness. part i. In Readings in information retrieval, pages 191--204. Morgan Kaufmann Publishers Inc., 1997.

Digital Library

[4]

T. M. Cover and J. Thomas. Elements of Information Theory. John Wiley & sons, 1991.

Digital Library

[5]

B. Dervin and M. S. Nilan. Information needs and use. In Annual Review of Information Science and Technology, volume~21, pages 3--33, 1986.

[6]

W. R. Greiff and J. Ponte. The maximum entropy approach and probabilistic ir models. ACM Trans. Inf. Syst., 18(3):246--287, 2000.

Digital Library

[7]

E. Jaynes. On the rationale of maximum entropy methods. In Proc.IEEE, volume 70, pages 939--952, 1982.

[8]

E. T. Jaynes. Information theory and statistical mechanics: Part i. Physical Review 106, pages 620--630, 1957a.

[9]

E. T. Jaynes. Information theory and statistical mechanics: Part ii. Physical Review 108, page 171, 1957b.

[10]

Y. Kagolovsky and J. R. Moehr. Current status of the evaluation of information retrieval. J. Med. Syst., 27(5):409--424, 2003.

Digital Library

[11]

P. B. Kantor and J. Lee. The maximum entropy principle in information retrieval. In SIGIR '86: Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval, pages 269--274. ACM Press, 1986.

Digital Library

[12]

D. D. Lewis. Evaluating and optimizing autonomous text classification systems. In SIGIR '95: Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 246--254. ACM Press, 1995.

Digital Library

[13]

R. M. Losee. When information retrieval measures agree about the relative quality of document rankings. J. Am. Soc. Inf. Sci., 51(9):834--840, 2000.

Digital Library

[14]

K. Nigam, J. Lafferty, and A. McCallum. Using maximum entropy for text classification. In IJCAI-99 Workshop on Machine Learning for Information Filtering, pages 61--67, 1999.

[15]

D. Pavlov, A. Popescul, D. M. Pennock, and L. H. Ungar. Mixtures of conditional maximum entropy models. In T. Fawcett and N. Mishra, editors, ICML, pages 584--591. AAAI Press, 2003.

[16]

S. J. Phillips, M. Dudik, and R. E. Schapire. A maximum entropy approach to species distribution modeling. In ICML '04: Twenty-first international conference on Machine learning, New York, NY, USA, 2004. ACM Press.

Digital Library

[17]

V. Raghavan, P. Bollmann, and G. S. Jung. A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inf. Syst., 7(3):205--229, 1989.

Digital Library

[18]

A. Ratnaparkhi and M. P. Marcus. Maximum entropy models for natural language ambiguity resolution, 1998.

[19]

T. Saracevic. Evaluation of evaluation in information retrieval. In SIGIR '95: Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 138--146. ACM Press, 1995.

Digital Library

[20]

C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, pages 379--423 & 623--656, 1948.

[21]

N. Wu. The Maximum Entropy Method. Springer, New York, 1997.

Cited By

Dong HLi LTian DSun YZhao Y(2024)Dynamic link prediction by learning the representation of node-pair via graph neural networksExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122685241:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122685
Khramtsova EZhuang SBaktashmotlagh MWang XZuccon G(2023)Selecting which Dense Retriever to use for Zero-Shot SearchProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625330(223-233)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3625330
Feng DKarmaker S(2023)Joint upper & expected value normalization for evaluation of retrieval systemsInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10340460:4Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.ipm.2023.103404
Show More Cited By

Index Terms

The maximum entropy method for analyzing retrieval measures
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results

Recommendations

Estimating average precision with incomplete and imperfect judgments
CIKM '06: Proceedings of the 15th ACM international conference on Information and knowledge management

We consider the problem of evaluating retrieval systems using incomplete judgment information. Buckley and Voorhees recently demonstrated that retrieval systems can be efficiently and effectively evaluated using incomplete judgments via the bpref ...
A geometric interpretation of r-precision and its correlation with average precision
SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

We consider two of the most commonly cited measures of retrieval performance: average precision and R-precision. It is well known that average precision and R-precision are highly correlated and similarly robust measures of performance, though the ...
Maximum Entropy Principle with General Deviation Measures

An approach to the Shannon and Rényi entropy maximization problems with constraints on the mean and law-invariant deviation measure for a random variable has been developed. The approach is based on the representation of law-invariant deviation measures ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

August 2005

708 pages

ISBN:1595930345

DOI:10.1145/1076034

General Chairs:
Ricardo Baeza-Yates
University of Chile, Chile
,
Nivio Ziviani
Federal University of Minas Gerais, Brazil
,
Program Chairs:
Gary Marchionini
University of North Carolina, USA
,
Alistair Moffat
University of Melbourne, Australia
,
John Tait
University of Sunderland, UK

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 August 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR05

Sponsor:

SIGIR

SIGIR05: The 28th ACM/SIGIR International Symposium on Information Retrieval 2005

August 15 - 19, 2005

Salvador, Brazil

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
1,441
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)4

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dong HLi LTian DSun YZhao Y(2024)Dynamic link prediction by learning the representation of node-pair via graph neural networksExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122685241:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122685
Khramtsova EZhuang SBaktashmotlagh MWang XZuccon G(2023)Selecting which Dense Retriever to use for Zero-Shot SearchProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625330(223-233)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3625330
Feng DKarmaker S(2023)Joint upper & expected value normalization for evaluation of retrieval systemsInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10340460:4Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.ipm.2023.103404
Ferro NKim YSanderson M(2019)Using Collection Shards to Study Retrieval Performance Effect SizesACM Transactions on Information Systems10.1145/331036437:3(1-40)Online publication date: 19-Mar-2019
https://dl.acm.org/doi/10.1145/3310364
Cakir FHe KXia XKulis BSclaroff S(2019)Deep Metric Learning to Rank2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR.2019.00196(1861-1870)Online publication date: Jun-2019
https://doi.org/10.1109/CVPR.2019.00196
Gupta SKutlu MKhetan VLease M(2019)Correlation, Prediction and Ranking of Evaluation Metrics in Information RetrievalAdvances in Information Retrieval10.1007/978-3-030-15712-8_41(636-651)Online publication date: 7-Apr-2019
https://doi.org/10.1007/978-3-030-15712-8_41
Ke WKamps JKanoulas Ede Rijke MFang HYilmaz E(2017)Text Retrieval based on Least Information MeasurementProceedings of the ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3121050.3121075(125-132)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1145/3121050.3121075
Ferro N(2017)What Does Affect the Correlation Among Evaluation Measures?ACM Transactions on Information Systems10.1145/310637136:2(1-40)Online publication date: 29-Aug-2017
https://dl.acm.org/doi/10.1145/3106371
Bao LLo DXia XLi S(2017)Automated Android application permission recommendationScience China Information Sciences10.1007/s11432-016-9072-360:9Online publication date: 28-Jul-2017
https://doi.org/10.1007/s11432-016-9072-3
Bao LLo DXia XLi S(2016)What Permissions Should This Android App Request?2016 International Conference on Software Analysis, Testing and Evolution (SATE)10.1109/SATE.2016.13(36-41)Online publication date: Nov-2016
https://doi.org/10.1109/SATE.2016.13
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents