Article

Building a distributed full-text index for the Web

Authors:

Sergey Melnik,

Sriram Raghavan,

Beverly Yang,

Hector Garcia-MolinaAuthors Info & Claims

WWW '01: Proceedings of the 10th international conference on World Wide Web

Pages 396 - 406

https://doi.org/10.1145/371920.372095

Published: 01 April 2001 Publication History

Get Access

References

[1]

E. W. Brown, J. P. Callan, W. B. Croft, and J. E. B. Moss. Supporting full-text information retrieval with a persistent object store. In 4th EDBT Conf., Mar 1994.

Digital Library

Google Scholar

[2]

S. Chakrabarti and S. Muthukrishnan. Resource scheduling for parallel database and scientific applications. In 8th ACM Symp. on Parallel Alg. and Architectures, pages 329-335, June 1996.

Digital Library

Google Scholar

[3]

J. Cho and H. Garcia-Molina. The evolution of the web and implications for an incremental crawler. 26th Intl. Conf. on Very Large Databases, Sep 2000.

Digital Library

Google Scholar

[4]

C. Faloutsos and S. Christodoulakis. Signature files: An access method for documents and its analytical performance evaluation. ACM Transactions on Office Information Systems, 2(4):267-288, October 1984.

Digital Library

Google Scholar

[5]

H. Garcia-Molina, J. Ullman, and J. Widom. Database System Implementation. Prentice-Hall, 2000.

Digital Library

Google Scholar

[6]

D. A. Gorssman and J. R. Driscoll. Structuring text within a relation system. In 3rd Intl. Conf. on Database and Expert System Applications, Sep 1992.

Crossref

Google Scholar

[7]

D. Hawking and N. Craswell. Overview of TREC-7 very large collection track. In Proc. of the Seventh Text Retrieval Conf., pages 91-104, Nov 1998.

Google Scholar

[8]

J. Hirai, S. Raghavan, H. Garcia-Molina, and A. Paepcke. WebBase: A repository of web pages. In Proc. of the 9th Intl. WWW Conf., May 2000.

Digital Library

Google Scholar

[9]

B.-S. Jeong and E. Omiecinski. Inverted file partitioning schemes in multiple disk systems. IEEE Trans. on Parallel and Distributed Systems, 6(2):142-153, Feb 1995.

Digital Library

Google Scholar

[10]

S. Lawrence and C. L. Giles. Accessibility of information on the web. Nature, 400:107-109, 1999.

Crossref

Google Scholar

[11]

U. Manber and G. Myers. Suffix arrays: A new method for on-line string searches. In Proc. of the 1st ACM- SIAM Symp. on Discrete Algorithms, 1990.

Digital Library

Google Scholar

[12]

P. Martin, I. A. MacLeod, and B. Nordin. A design of a distributed full text retrieval system. In Proc. ACM Conf. on R&D in Information Retrieval, Sep 1986.

Digital Library

Google Scholar

[13]

S. Melnik et al. Building a distributed full-text index for the web. Technical report, Stanford Digital Library Project, July 2000. Available at wwwdiglib.stanford.edu/cgi-bin/get/SIDL-WP-2000-0140.

Google Scholar

[14]

Mike Burrows. Personal Communication.

Google Scholar

[15]

A. Moffat and J. Zobel. Self-indexing inverted files for fast text retrieval. ACM Transactions on Information Systems, 14(4):349-379, October 1996.

Digital Library

Google Scholar

[16]

A. NgocVo and A. Moffat. Compressed inverted files with reduced decoding overheads. In Proc. of 21st Intl. Conf. on R&D in Information Retrieval, Aug 1998.

Digital Library

Google Scholar

[17]

M. Olson, K. Bostic, and M. Seltzer. Berkeley DB. In 1999 Summer Usenix Technical Conf., Jun 1999.

Digital Library

Google Scholar

[18]

B. Ribeiro-Neto and R. Barbosa. Query performance for tightly coupled distributed digital libraries. In 3rd ACM Conf. on Digital Libraries, June 1998.

Digital Library

Google Scholar

[19]

B. Ribeiro-Neto, E. S. Moura, M. S. Neubert, and N. Ziviani. Efficient distributed algorithms to build inverted files. In 22nd ACM Conf. on R&D in Information Retrieval, Aug 1999.

Digital Library

Google Scholar

[20]

G. Salton. Information Retrieval: Data Structures and Algorithms. Addison-Wesley, Massachussetts, 1989.

Google Scholar

[21]

A. Tomasic and H. Garcia-Molina. Query processing and inverted indices in shared-nothing document information retrieval systems. VLDB Journal, 2(3):243-275, 1993.

Digital Library

Google Scholar

[22]

A. Tomasic, H. Garcia-Molina, and K. Shoens. Incremental update of inverted list for text document retrieval. In 1994 ACM SIGMOD, May 1994.

Digital Library

Google Scholar

[23]

C. L. Viles and J. C. French. Dissemination of collection wide information in a distributed information retrieval system. In Proc. 18th Intl. ACM Conf. on R&D in Information Retrieval, July 1995.

Digital Library

Google Scholar

[24]

Inktomi WebMap. www.inktomi.com/webmap/.

Google Scholar

[25]

I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufman Publ., San Francisco, 1999.

Digital Library

Google Scholar

[26]

J. Zobel, A. Moffat, and R. Sacks-Davis. An efficient indexing technique for full-text database systems. In 18th VLDB Conf., pages 352-362, Aug 1992.

Digital Library

Google Scholar

Cited By

View all

Wu CZhu QZhang YXie XQin HZhou YZhang PYang W(2018)Movement-Oriented Objectified Organization and Retrieval Approach for Heterogeneous GeoVideo DataISPRS International Journal of Geo-Information10.3390/ijgi70702557:7(255)Online publication date: 28-Jun-2018
https://doi.org/10.3390/ijgi7070255
Goodwin BHopcroft MLuu DClemmer ACurmei MElnikety SHe YKando NSakai TJoho HLi Hde Vries AWhite R(2017)BitFunnelProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3077136.3080789(605-614)Online publication date: 7-Aug-2017
https://dl.acm.org/doi/10.1145/3077136.3080789
Jiang KYang Y(2016)Efficient dynamic pruning on largest scores first (LSF) retrievalFrontiers of Information Technology & Electronic Engineering10.1631/FITEE.150019017:1(1-14)Online publication date: 9-Jan-2016
https://doi.org/10.1631/FITEE.1500190
Show More Cited By

Index Terms

Building a distributed full-text index for the Web
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Web-based interaction
2. Information systems

Recommendations

Building a distributed full-text index for the web

We identify crucial design issues in building a distributed inverted index for a large collection of Web pages. We introduce a novel pipelining technique for structuring the core index-building system that substantially reduces the index construction ...
Dynamic maintenance of web indexes using landmarks
WWW '03: Proceedings of the 12th international conference on World Wide Web

Recent work on incremental crawling has enabled the indexed document collection of a search engine to be more synchronized with the changing World Wide Web. However, this synchronized collection is not immediately searchable, because the keyword index ...
Efficient Textual Web Retrieval using Wavelet Tree

Searching on the web is one of the most progressive and expanding field nowadays. A large amount of information is available on the World Wide Web, motivating the need of efficient text indexing method that support fast text retrieval. In the past, two ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

WWW '01: Proceedings of the 10th international conference on World Wide Web

May 2001

770 pages

ISBN:1581133480

DOI:10.1145/371920

Chairmen:
Vincent Y. Shen
Hong Kong Univ. of Science and Technology
,
Nobuo Saito
Keio Univ., Japan
,
Michael R. Lyu
Chinese Univ. of Hong Kong, HK
,
Mary Ellen Zurko
Iris Associates,USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 April 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

WWW01

Sponsor:

IW3C2

WWW01: Hypermedia Track of the 10th International World Wide Web Conference

May 1 - 5, 2001

Hong Kong, Hong Kong

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

52
Total Citations
View Citations
872
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wu CZhu QZhang YXie XQin HZhou YZhang PYang W(2018)Movement-Oriented Objectified Organization and Retrieval Approach for Heterogeneous GeoVideo DataISPRS International Journal of Geo-Information10.3390/ijgi70702557:7(255)Online publication date: 28-Jun-2018
https://doi.org/10.3390/ijgi7070255
Goodwin BHopcroft MLuu DClemmer ACurmei MElnikety SHe YKando NSakai TJoho HLi Hde Vries AWhite R(2017)BitFunnelProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3077136.3080789(605-614)Online publication date: 7-Aug-2017
https://dl.acm.org/doi/10.1145/3077136.3080789
Jiang KYang Y(2016)Efficient dynamic pruning on largest scores first (LSF) retrievalFrontiers of Information Technology & Electronic Engineering10.1631/FITEE.150019017:1(1-14)Online publication date: 9-Jan-2016
https://doi.org/10.1631/FITEE.1500190
Abiri FKahani MZarinkalam F(2014)An entity based RDF indexing schema using Hadoop and HBase2014 4th International Conference on Computer and Knowledge Engineering (ICCKE)10.1109/ICCKE.2014.6993400(68-73)Online publication date: Oct-2014
https://doi.org/10.1109/ICCKE.2014.6993400
Sharma DJain S(2014)Content sharing in information storage and retrieval system using tree representation of documents2014 Conference on IT in Business, Industry and Government (CSIBIG)10.1109/CSIBIG.2014.7056941(1-4)Online publication date: Mar-2014
https://doi.org/10.1109/CSIBIG.2014.7056941
Al-Bahadili HAl-Saab S(2013)Development of a Novel Compressed Index-Query Web Search Engine ModelNetwork and Communication Technology Innovations for Web and IT Advancement10.4018/978-1-4666-2157-2.ch019(275-293)Online publication date: 2013
https://doi.org/10.4018/978-1-4666-2157-2.ch019
Li BZhang JChen MZhang JWang KMeng D(2013)DIFTSASProceedings of the 2013 IEEE 16th International Conference on Computational Science and Engineering10.1109/CSE.2013.193(1303-1309)Online publication date: 3-Dec-2013
https://dl.acm.org/doi/10.1109/CSE.2013.193
Simha DLu MChiueh T(2012)An update-aware storage system for low-locality update-intensive workloadsACM SIGPLAN Notices10.1145/2248487.215101647:4(375-386)Online publication date: 3-Mar-2012
https://dl.acm.org/doi/10.1145/2248487.2151016
Simha DLu MChiueh T(2012)An update-aware storage system for low-locality update-intensive workloadsACM SIGARCH Computer Architecture News10.1145/2189750.215101640:1(375-386)Online publication date: 3-Mar-2012
https://dl.acm.org/doi/10.1145/2189750.2151016
Simha DLu MChiueh THarris TScott M(2012)An update-aware storage system for low-locality update-intensive workloadsProceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems10.1145/2150976.2151016(375-386)Online publication date: 3-Mar-2012
https://dl.acm.org/doi/10.1145/2150976.2151016
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

References

Cited By

Index Terms

Recommendations