Article

Relevance score normalization for metasearch

Authors:

Javed A. AslamAuthors Info & Claims

CIKM '01: Proceedings of the tenth international conference on Information and knowledge management

Pages 427 - 433

https://doi.org/10.1145/502585.502657

Published: 05 October 2001 Publication History

Abstract

Given the ranked lists of documents returned by multiple search engines in response to a given query, the problem of metasearch is to combine these lists in a way which optimizes the performance of the combination. This problem can be naturally decomposed into three subproblems: (1) normalizing the relevance scores given by the input systems, (2) estimating relevance scores for unretrieved documents, and (3) combining the newly-acquired scores for each document into one, improved score.Research on the problem of metasearch has historically concentrated on algorithms for combining (normalized) scores. In this paper, we show that the techniques used for normalizing relevance scores and estimating the relevance scores of unretrieved documents can have a significant effect on the overall performance of metasearch. We propose two new normalization/estimation techniques and demonstrate empirically that the performance of well known metasearch algorithms can be significantly improved through their use.

References

[1]

TREC 2, Gaithersburg, MD, USA, Mar. 1994. U.S. Government Printing Office, Washington D.C.

[2]

TREC 5, Gaithersburg, MD, USA, 1997. U.S. Government Printing Office, Washington D.C.

[3]

ACM SZGZR 2001, New Orleans, Louisiana, USA, 2001. ACM Press, New York.

[4]

J. Aslam and M. Montague. Models for metasearch. In ACM SIGIR 2001 {3}.

Digital Library

[5]

B. T. Bartell. Optimizing Ranking Functions: A Connectionist Approach to Adaptive Information Retrieval. PhD thesis, University of California, San Diego, 1994.

Digital Library

[6]

N. Belkin, P. Kantor, C. Cool, and R. Quatrain. Combining evidence for information retrieval. In TREC 2 {I}, pages 35-43.

[7]

W. B. Croft. Combining approaches to information retrieval. In W. B. Croft, editor, Advances in Information Retrieval: Recent Research jrvm the Center for Intelligent Information Retrieval, chapter 1. Kluwer, 2000.

[8]

E. A. Fox, M. P. Koushik, J. Shaw, R. Modlin, and D. Rao. Combining evidence from multiple searches. In TREC 1, pages 319-328, Gaithersburg, MD, USA, Mar. 1993. U.S. Government Printing Office, Washington D.C.

[9]

E. A. Fox and J. A. Shaw. Combination of multiple searches. In TREC 2 {l}, pages 243-249.

[10]

K. L. Fox, 0. Frieder, M. Knepper, and E. Snowberg. SENTINEL: A multiple engine information retrieval and visualization system. Journal of the ASIS, 50(7), May 1999.

Digital Library

[11]

D. A. Hull, J. 0. Pedersen, and H. Schiitze. Method combination for document filtering. In ACM SIGIR '96, pages 279-287, Zurich, Switzerland, 1996. ACM Press, New York.

Digital Library

[12]

J. H. Lee. Analyses of multiple evidence combination. In ACM SIGIR '97, pages 267-275, Philadelphia, Pennsylvania, USA, July 1997. ACM Press, New York.

Digital Library

[13]

R. Manmatha, T. Rath, and F. Feng. Modeling score distributions for combining the outputs of search engines. In ACM SIGIR 2001 {3}.

Digital Library

[14]

M. Montague and J. Aslam. Metasearch consistency. In ACM SIGIR 2001 {3}.

Digital Library

[15]

K. B. Ng. An Investigation of the Conditions for Effective Data Fusion in Information Retrieval. PhD thesis, School of Communication, Information, and Library Studies, Rutgers University, 1998.

[16]

K. B. Ng and P. B. Kantor. An investigation of the preconditions for effective data fusion in ir: A pilot study. In Proceedings of the 61th Annual Meeting of the American Society for Information Science, 1998.

[17]

K. B. Ng, D. Loewenstern, C. Basu, H. Hirsh, and P. B. Kantor. Data fusion of machine-learning methods for the TREC5 routing task (and other work). In TREC 5 {2}, pages 477-487.

[18]

Content-Based Multimedia Information Access (RIAO), Paris, France, Apr. 2000.

[19]

E. W. Selberg. Towards Comprehensive Web Search. PhD thesis, University of Washington, 1999.

Digital Library

[20]

J. A. Shaw and E. A. Fox. Combination of multiple searches. In TREC 3, pages 105-108, Gaithersburg, MD, USA, Apr. 1995. U.S. Government Printing Office, Washington D.C.

[21]

P. Thompson. A combination of expert opinion approach to probabilistic information retrieval, part 1: the conceptual model. Information Processing and Management, 26(3):371-382, 1990.

Digital Library

[22]

C. C. Vogt. Adaptive Combination of Evidence for Information Retrieval. PhD thesis, University of California, San Diego, 1999.

Digital Library

[23]

C. C. Vogt. How much more is better? Characterizing the effects of adding more IR systems to a combination. In RIAO {18}, pages 457-475.

[24]

C. C. Vogt and G. W. Cottrell. Fusion via a linear combination of scores. Information Retrieval, 1(3):151-173, Oct. 1999.

Digital Library

[25]

C. C. Vogt, G. W. Cottrell, R. K.Belew, and B. T. Bartell. Using relevance to train a linear mixture of experts. In TREC 5 {2}, pages 503-515.

Cited By

Cachel KRundensteiner ESerra ESpezzano F(2024)Wise Fusion: Group Fairness Enhanced Rank FusionProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679649(163-174)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679649
Wang SGeng MLin BSun ZWen MLiu YLi LBissyandé TMao X(2024)Fusing Code SearchersIEEE Transactions on Software Engineering10.1109/TSE.2024.340304250:7(1852-1866)Online publication date: Jul-2024
https://doi.org/10.1109/TSE.2024.3403042
Bassani ERomelli LAl Hasan MXiong L(2022)ranx.fuse: A Python Library for MetasearchProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557207(4808-4812)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557207
Show More Cited By

Index Terms

Relevance score normalization for metasearch
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Database query processing
  2. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment
    2. Retrieval tasks and goals
      1. Document filtering
      2. Information extraction
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Database theory
      1. Database query processing and optimization (theory)

Recommendations

Web metasearch: rank vs. score based rank aggregation methods
SAC '03: Proceedings of the 2003 ACM symposium on Applied computing

Given a set of rankings, the task of ranking fusion is the problem of combining these lists in such a way to optimize the performance of the combination. The ranking fusion problem is encountered in many situations and, e.g., metasearch is a prominent ...
Building efficient and effective metasearch engines

Frequently a user's information needs are stored in the databases of multiple search engines. It is inconvenient and inefficient for an ordinary user to invoke multiple search engines and identify useful documents from the returned results. To support ...
Using Online Relevance Feedback to Build Effective Personalized Metasearch Engine
WISE '01: Proceedings of the Second International Conference on Web Information Systems Engineering (WISE'01) Volume 1 - Volume 1

Metasearch Engine is popular for facilitating users' queries over multiple search engines and increasing the coverage of the WWW. How to rank the merged results becomes crucial for the success of metasearch engines. Many current metasearch engines have ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '01: Proceedings of the tenth international conference on Information and knowledge management

October 2001

616 pages

ISBN:1581134363

DOI:10.1145/502585

Editors:
Henrique Paques
Georgia Institute of Technology
,
Ling Liu
Georgia Institute of Technology
,
David Grossman
Illinois Institute of Technology
,
General Chair:
Calton Pu
Georgia Institute of Technology

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

CIKM01

Sponsor:

CIKM01: International Conference on Information and Knowledge Management

October 5 - 10, 2001

Georgia, Atlanta, USA

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

112
Total Citations
View Citations
1,212
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)4

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cachel KRundensteiner ESerra ESpezzano F(2024)Wise Fusion: Group Fairness Enhanced Rank FusionProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679649(163-174)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679649
Wang SGeng MLin BSun ZWen MLiu YLi LBissyandé TMao X(2024)Fusing Code SearchersIEEE Transactions on Software Engineering10.1109/TSE.2024.340304250:7(1852-1866)Online publication date: Jul-2024
https://doi.org/10.1109/TSE.2024.3403042
Bassani ERomelli LAl Hasan MXiong L(2022)ranx.fuse: A Python Library for MetasearchProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557207(4808-4812)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557207
Kumar RGupta AArora HRaman B(2021)CBSN: Comparative measures of normalization techniques for brain tumor segmentation using SRCNetMultimedia Tools and Applications10.1007/s11042-021-10565-081:10(13203-13235)Online publication date: 16-Mar-2021
https://doi.org/10.1007/s11042-021-10565-0
Kato SShimizu TFujita SSakai T(2020)Unsupervised Answer Retrieval with Data Fusion for Community Question AnsweringInformation Retrieval Technology10.1007/978-3-030-42835-8_2(10-21)Online publication date: 27-Feb-2020
https://doi.org/10.1007/978-3-030-42835-8_2
Yang HGonçalves T(2020)Aggregation on Learning to Rank for Consumer Health Information RetrievalModelling and Development of Intelligent Systems10.1007/978-3-030-39237-6_6(81-93)Online publication date: 17-Jan-2020
https://doi.org/10.1007/978-3-030-39237-6_6
Lipani ALosada DZuccon GLupu M(2019)Fixed-Cost Pooling StrategiesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2019.2947049(1-1)Online publication date: 2019
https://doi.org/10.1109/TKDE.2019.2947049
He JPing QLou WChen C(2019)PaperPolesJournal of the Association for Information Science and Technology10.1002/asi.2417170:8(843-857)Online publication date: 2-Jul-2019
https://dl.acm.org/doi/10.1002/asi.24171
Xu DShaikh N(2018)A Heuristic Approach for Ranking Items Based on Inputs from Multiple ExpertsInternational Journal of Information Systems and Social Change10.4018/IJISSC.20180701019:3(1-22)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.4018/IJISSC.2018070101
(2018)Activity-based linkage and ranking methods for personal dataspaceInternational Journal of Mobile Communications10.1504/IJMC.2018.09138116:3(266-285)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1504/IJMC.2018.091381
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents