Article

The document as an ergodic markov chain

Authors:

SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 496 - 497

https://doi.org/10.1145/1008992.1009088

Published: 25 July 2004 Publication History

Get Access

Abstract

In recent years, statistical language models are being proposed as alternative to the vector space model. Viewing documents as language samples introduces the issue of defining a joint probability distribution over the terms.The present paper models a document as the result of a Markov process. It argues that this process is ergodic, which is theoretically plausible, and easy to verify in practice.The theoretical result is that the joint distribution can be easily obtained. This can also be applied for search resolutions other than the document level. We verified this in an experiment on query expansion demonstrating both the validity and the practicability of the method. This holds a promise for general language models.

References

[1]

A cautionary note: The Markov chains used by others, e.g. Lafferty and Zhai {3} represent concepts different than ours. C. Burgess, K. Livesay, and K. Lund. Explorations in context space: Words, sentences, discourse. Discourse Processes, 25:211--257, 1998.

Google Scholar

[2]

E. Hoenkamp. Unitary operators on the document space. Journal of the American Society for Information Science and Technology, 54(4):314--320, 2003.

Digital Library

Google Scholar

[3]

J. Lafferty and C. Zhai. Document language models, query models, and risk minimization for IR. In Proceedings of SIGIR-2001, pages 111--119, 2001.

Digital Library

Google Scholar

[4]

S. E. Robertson, S. Walker, M. Hancock-Beaulieu, A. Gull, and M. Lau. Okapi at TREC. In Text REtrieval Conference, pages 21--30, 1992.

Google Scholar

[5]

R. M. Shiffrin and M. Steyvers. The effectiveness of retrieval from memory. In M. Oaksford and N. Chater, editors, Rational models of cognition, pages 73-9-5. Oxford University Press, 1998.

Google Scholar

Cited By

View all

Kontopoulou EPredari MGallopoulos EHargood CMillard D(2013)Onomatology and content analysis of ergodic literatureProceedings of the 3rd Narrative and Hypertext Workshop10.1145/2462216.2462221(1-5)Online publication date: 1-May-2013
https://dl.acm.org/doi/10.1145/2462216.2462221
Ma CLiu YWang HTeng DDai G(2012)Sketch-Based Annotation and Visualization in Video AuthoringIEEE Transactions on Multimedia10.1109/TMM.2012.219038914:4(1153-1165)Online publication date: 1-Aug-2012
https://dl.acm.org/doi/10.1109/TMM.2012.2190389
Hoenkamp E(2012)Taming the Terabytes: A Human‐Centered Approach to Surviving the Information DelugeInformation Overload10.1002/9781118360491.ch8(145-174)Online publication date: 15-Oct-2012
https://doi.org/10.1002/9781118360491.ch8
Show More Cited By

Index Terms

The document as an ergodic markov chain
1. Information systems
  1. Information retrieval

Recommendations

Document Expansion Using External Collections
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Document expansion has been shown to improve the effectiveness of information retrieval systems by augmenting documents' term probability estimates with those of similar documents, producing higher quality document representations. We propose a method ...
A Pólya Urn Document Language Model for Improved Information Retrieval

The multinomial language model has been one of the most effective models of retrieval for more than a decade. However, the multinomial distribution does not model one important linguistic phenomenon relating to term dependency—that is, the tendency of a ...
Linear dependence of stationary distributions in ergodic Markov decision processes

In ergodic MDPs we consider stationary distributions of policies that coincide in all but n states, in which one of two possible actions is chosen. We give conditions and formulas for linear dependence of the stationary distributions of n+2 such ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

July 2004

624 pages

ISBN:1581138814

DOI:10.1145/1008992

General Chair:
Mark Sanderson
University of Sheffield (UK)
,
Program Chairs:
Kalervo Järvelin
University of Tampere (Finland)
,
James Allan
University of Massachusetts (USA)
,
Peter Bruza
Distributed Systems Technology Centre (Australia)

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR04

Sponsor:

SIGIR04: The 27th ACM/SIGIR International Symposium on Information Retrieval 2004

July 25 - 29, 2004

Sheffield, United Kingdom

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
485
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kontopoulou EPredari MGallopoulos EHargood CMillard D(2013)Onomatology and content analysis of ergodic literatureProceedings of the 3rd Narrative and Hypertext Workshop10.1145/2462216.2462221(1-5)Online publication date: 1-May-2013
https://dl.acm.org/doi/10.1145/2462216.2462221
Ma CLiu YWang HTeng DDai G(2012)Sketch-Based Annotation and Visualization in Video AuthoringIEEE Transactions on Multimedia10.1109/TMM.2012.219038914:4(1153-1165)Online publication date: 1-Aug-2012
https://dl.acm.org/doi/10.1109/TMM.2012.2190389
Hoenkamp E(2012)Taming the Terabytes: A Human‐Centered Approach to Surviving the Information DelugeInformation Overload10.1002/9781118360491.ch8(145-174)Online publication date: 15-Oct-2012
https://doi.org/10.1002/9781118360491.ch8
Hoenkamp E(2011)Trading spacesProceedings of the Third international conference on Advances in information retrieval theory10.5555/2040317.2040326(40-51)Online publication date: 12-Sep-2011
https://dl.acm.org/doi/10.5555/2040317.2040326
Hoenkamp E(2011)Trading Spaces: On the Lore and Limitations of Latent Semantic AnalysisAdvances in Information Retrieval Theory10.1007/978-3-642-23318-0_6(40-51)Online publication date: 2011
https://doi.org/10.1007/978-3-642-23318-0_6
Hoenkamp Evan Dijk S(2006)A fingerprinting technique for evaluating semantics based indexingProceedings of the 28th European conference on Advances in Information Retrieval10.1007/11735106_35(397-406)Online publication date: 10-Apr-2006
https://dl.acm.org/doi/10.1007/11735106_35

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Document Expansion Using External Collections

A Pólya Urn Document Language Model for Improved Information Retrieval

Linear dependence of stationary distributions in ergodic Markov decision processes