Improving Throughput of a Pipeline Model Indexer
Article No.: 2, Pages 1 - 4
Abstract
There are many competing models for the indexing process of an information retrieval system, one of which is a pipeline based model. Information retrieval is also an inherently parallel process, indexing one document is independent of another document. A pipeline model allows for easy experimentation on the parallelism within an indexer. In this paper we investigate areas within a pipeline where indexing throughput can be increased, as well as exploiting the inherent parallelism of indexing.
References
[1]
J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, 51(1):107--113.
[2]
T. Elsayed, F. Ture, and J. Lin. Brute-Force Approaches to Batch Retrieval: Scalable Indexing with MapReduce, or Why Bother? Technical report, HCIL-2010-23, University of Maryland.
[3]
S. Heinz and J. Zobel. Efficient Single-Pass Index Construction for Text Databases. JASIST, 54(8):713--729.
[4]
D. Hiemstra. Personal communication, 2013.
[5]
D. Hiemstra and C. Hauff. MapReduce for Information Retrieval Evaluation: "Let's Quickly Test This on 12 TB of Data". In Multilingual and Multimodal Information Access Evaluation, pages 64--69. 2010.
[6]
D. Hiemstra and C. Hauff. MIREX: MapReduce Information Retrieval Experiments, 2010.
[7]
R. McCreadie, C. Macdonald, and I. Ounis. MapReduce Indexing Strategies: Studying Scalability and Efficiency. IP&M, 48(5):873--888.
[8]
R. M. C. McCreadie, C. Macdonald, and I. Ounis. On Single-Pass Indexing with MapReduce. SIGIR '09, pages 742--743.
[9]
S. Melink, S. Raghavan, B. Yang, and H. Garcia-Molina. Building a Distributed Full-text Index for the Web. ACM Trans. Inf. Syst., 19(3):217--241.
[10]
B. Ribeiro-Neto, E. S. Moura, M. S. Neubert, and N. Ziviani. Efficient Distributed Algorithms to Build Inverted Files. SIGIR '99, pages 105--112.
[11]
A. Trotman, X.-F. Jia, and M. Crane. Towards an Efficient and Effective Search Engine. In SIGIR 2012 OSIR Workshop, pages 40--47.
[12]
Z. Wei and J. Jaja. An Optimized High-Throughput Strategy for Constructing Inverted Files. IEEE Transactions on Parallel and Distributed Systems, 23(11):2033--2044, Nov 2012.
Index Terms
- Improving Throughput of a Pipeline Model Indexer
Recommendations
Transformations techniques for extracting parallelism in non-uniform nested loops
Executing a program in parallel machines needs not only to find sufficient parallelism in a program, but it is also important that we minimize the synchronization and communication overheads in the parallelized program. This yields to improve the ...
Improving the parallelism of iterative methods by aggressive loop fusion
Traditionally, loop nests are fused only when the data dependences in the loop nests are not violated. This paper presents a new loop fusion algorithm that is capable of fusing loop nests in the presence of fusion-preventing anti-dependences. All the ...
Affine and unimodular transformations for non-uniform nested loops
ICCOMP'08: Proceedings of the 12th WSEAS international conference on ComputersPerformance improvement in the modern parallel machines needs not only to find sufficient parallelism in a program, but it is also important that we minimize the synchronization and communication overheads in the parallelized program. Parallelizing and ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2015
72 pages
Copyright © 2015 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
In-Cooperation
- ACM: Association for Computing Machinery
- Univ. of Western Sydney: University of Western Sydney
- SIGIR: ACM Special Interest Group on Information Retrieval
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 December 2015
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Conference
ADCS '15
ADCS '15: The 20th Australasian Document Computing Symposium
December 8 - 9, 2015
NSW, Parramatta, Australia
Acceptance Rates
ADCS '15 Paper Acceptance Rate 5 of 14 submissions, 36%;
Overall Acceptance Rate 30 of 57 submissions, 53%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 51Total Downloads
- Downloads (Last 12 months)3
- Downloads (Last 6 weeks)1
Reflects downloads up to 16 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in