research-article

An Optimization Framework for Merging Multiple Result Lists

Authors:

W. Bruce Croft,

Daniel SheldonAuthors Info & Claims

CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

Pages 303 - 312

https://doi.org/10.1145/2806416.2806489

Published: 17 October 2015 Publication History

Abstract

Developing effective methods for fusing multiple ranked lists of documents is crucial to many applications. Federated web search, for instance, has become a common practice where a query is issued to different verticals and a single ranked list of blended results is created. While federated search is regarded as collection fusion, data fusion techniques aim at improving search coverage and precision by combining multiple search runs on a single document collection. In this paper, we study in depth and extend a neural network-based approach, LambdaMerge, for merging results of ranked lists drawn from one (i.e., data fusion) or more (i.e., collection fusion) verticals. The proposed model considers the impact of the quality of documents, ranked lists and verticals for producing the final merged result in an optimization framework. We further investigate the potential of incorporating deep structures into the model with an aim of determining better combinations of different evidence. In the experiments on collection fusion and data fusion, the proposed approach significantly outperforms several standard baselines and state-of-the-art learning-based approaches.

References

[1]

J. A. Aslam and M. Montague. Bayes optimal metasearch: A probabilistic model for combining the results of multiple retrieval systems (poster session). In Proceedings of SIGIR, SIGIR '00, pages 379--381, 2000.

Digital Library

[2]

J. A. Aslam and M. Montague. Models for metasearch. In Proceedings of SIGIR, SIGIR '01, pages 276--284, 2001.

Digital Library

[3]

S. M. Beitzel, E. C. Jensen, A. Chowdhury, D. Grossman, O. Frieder, and N. Goharian. On fusion of effective retrieval strategies in the same information retrieval system. JASIST, 55:859--868, 2004.

Digital Library

[4]

Y. Bengio. Learning deep architectures for ai. Found. Trends Mach. Learn., 2(1):1--127, Jan. 2009.

Digital Library

[5]

Y. Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, pages 437--478. Springer, 2012.

[6]

C. Burges, R. Ragno, and Q. Le. Learning to rank with non-smooth cost functions. In Advances in Neural Information Processing Systems 19, January 2007.

[7]

C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proceedings of ICML, ICML '05, pages 89--96, 2005.

Digital Library

[8]

C. J. Burges. From ranknet to lambdarank to lambdamart: An overview. Technical report, June 2010.

[9]

J. Callan. Distributed information retrieval. In In: Advances in Information Retrieval, pages 127--150. Kluwer Academic Publishers, 2000.

[10]

J. P. Callan, Z. Lu, and W. B. Croft. Searching distributed collections with inference networks. In Proceedings of SIGIR, SIGIR '95, pages 21--28, 1995.

Digital Library

[11]

G. V. Cormack, C. L. A. Clarke, and S. Buettcher. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of SIGIR, SIGIR '09, pages 758--759, 2009.

Digital Library

[12]

W. B. Croft. Combining approaches to information retrieval. Advances in Information Retrieval, Chapter 1, The Information Retrieval Series, 7:1--36, 2000.

[13]

T. Demeester, D. Trieschnigg, D. Nguyen, and D. Hiemstra. Overview of the trec 2013 federated web search track. In Proceedings of TREC, 2013.

[14]

T. Demeester, D. Trieschnigg, D. Nguyen, Z. K., and D. Hiemstra. Overview of the trec 2014 federated web search track. In Proceedings of TREC, 2014.

[15]

L. Deng, X. He, and J. Gao. Deep stacking networks for information retrieval. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 3153--3157, May 2013.

[16]

L. Deng and D. Yu. Deep learning: Methods and applications. Technical Report MSR-TR-2014-21, January 2014.

[17]

E. A. Fox and J. A. Shaw. Combination of multiple searches. In The Second Text REtrieval Conference (TREC-2), pages 243--252, 1994.

[18]

F. Guan, S. Zhang, C. Liu, X. Yu, Y. Liu, and X. Cheng. Ictnet at federated web search track 2014. In Proceedings of TREC, 2014.

[19]

D. Hawking and P. Thomas. Server selection methods in hybrid portal search. In Proceedings of SIGIR, SIGIR '05, pages 75--82, 2005.

Digital Library

[20]

P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of CIKM, CIKM '13, pages 2333--2338, 2013.

Digital Library

[21]

S. Jin and M. Lan. Simple may be best - a simple and effective method for federated web search via search engine impact factor estimation. In Proceedings of TREC, 2014.

[22]

A. Khudyak Kozorovitsky and O. Kurland. Cluster-based fusion of retrieved lists. In Proceedings of SIGIR, SIGIR '11, pages 893--902, 2011.

Digital Library

[23]

O. Kurland, A. Shtok, D. Carmel, and S. Hummel. A unified framework for post-retrieval query-performance prediction. In Proceedings of ICTIR, ICTIR'11, pages 15--26, 2011.

Digital Library

[24]

J. H. Lee. Analyses of multiple evidence combination. In Proceedings of SIGIR, SIGIR '97, pages 267--276, 1997.

Digital Library

[25]

D. Lillis, F. Toolan, R. Collier, and J. Dunnion. Probabilistic data fusion on a large document collection. Artif. Intell. Rev., 26(1--2):23--34, Oct. 2006.

Digital Library

[26]

D. Lillis, F. Toolan, R. Collier, and J. Dunnion. Extending probabilistic data fusion using sliding windows. In Proceedings of ECIR, ECIR'08, pages 358--369, 2008.

Digital Library

[27]

G. Mesnil, X. He, L. Deng, and Y. Bengio. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In INTERSPEECH, pages 3771--3775, 2013.

[28]

D. Metzler and W. B. Croft. A markov random field model for term dependencies. In Proceedings of SIGIR, SIGIR '05, pages 472--479, 2005.

Digital Library

[29]

T. Mikolov, W.-t. Yih, and G. Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746--751. Citeseer, 2013.

[30]

M. Montague and J. A. Aslam. Condorcet fusion for improved retrieval. In Proceedings of CIKM, CIKM '02, pages 538--548, 2002.

Digital Library

[31]

A. Mourao, F. Martins, and J. Magalhaes. Novasearch at trec 2013 federated web search track: Experiments with rank fusion. In The 22nd Text Retrieval Conference, TREC 13, 2013.

[32]

D. Sheldon, M. Shokouhi, M. Szummer, and N. Craswell. Lambdamerge: Merging the results of query reformulations. In Proceedings of WSDM, WSDM '11, pages 795--804, 2011.

Digital Library

[33]

Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil. A latent semantic model with convolutional-pooling structure for information retrieval. In Proceedings of CIKM, CIKM '14, pages 101--110, 2014.

Digital Library

[34]

M. Shokouhi. Segmentation of search engine results for effective data-fusion. In Proceedings of ECIR, ECIR'07, pages 185--197, 2007.

Digital Library

[35]

M. Shokouhi and L. Si. Federated search. Found. Trends Inf. Retr., 5(1):1--102, Jan. 2011.

Digital Library

[36]

M. Shokouhi and J. Zobel. Robust result merging using sample-based score estimates. ACM Trans. Inf. Syst., 27(3):14:1--14:29, May 2009.

Digital Library

[37]

A. Shtok, O. Kurland, D. Carmel, F. Raiber, and G. Markovits. Predicting query performance by query-drift estimation. ACM Trans. Inf. Syst., 30(2):11:1--11:35, May 2012.

Digital Library

[38]

L. Si and J. Callan. A semisupervised learning method to merge search engine results. ACM Trans. Inf. Syst., 21(4):457--491, Oct. 2003.

Digital Library

[39]

C. C. Vogt and G. W. Cottrell. Fusion via a linear combination of scores. Inf. Retr., 1(3):151--173, Oct. 1999.

Digital Library

[40]

E. M. Voorhees, N. K. Gupta, and B. Johnson-laird. The collection fusion problem. In Proceedings of the TREC, pages 95--104, 1995.

[41]

Q. Wu, C. J. Burges, K. M. Svore, and J. Gao. Adapting boosting for information retrieval measures. Inf. Retr., 13(3):254--270, June 2010.

Digital Library

[42]

S. Wu and S. McClean. Performance prediction of data fusion for information retrieval. Inf. Process. Manage., 42(4):899--915, July 2006.

Digital Library

Cited By

Garba AKhalid SAleryni AUllah ITairan NShah HMumin D(2024)Utilizing Ant Colony Optimization for Result Merging in Federated SearchEngineering, Technology & Applied Science Research10.48084/etasr.730214:4(14832-14839)Online publication date: 2-Aug-2024
https://doi.org/10.48084/etasr.7302
Tyomkin LKurland OHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Analyzing Fusion Methods Using the Condorcet RuleProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657912(2281-2285)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657912
Stamatis VSalampasis M(2020)Results Merging in the Patent DomainProceedings of the 24th Pan-Hellenic Conference on Informatics10.1145/3437120.3437313(229-232)Online publication date: 20-Nov-2020
https://dl.acm.org/doi/10.1145/3437120.3437313
Show More Cited By

Index Terms

An Optimization Framework for Merging Multiple Result Lists
1. Information systems
  1. Information retrieval

Recommendations

The early fusion strategy for search result diversification
ACM TURC '17: Proceedings of the ACM Turing 50th Celebration Conference - China

A typical strategy for search result diversification is a two-stage process: first we use a traditional search engine to obtain a ranked list of documents, in which relevance is the only concern; then the results are re-ranked so as to promote ...
A Fuzzy Search Engine Weighted Approach to Result Merging for Metasearch
RSFDGrC '07: Proceedings of the 11th International Conference on Rough Sets, Fuzzy Sets, Data Mining and Granular Computing

Each search engine queried by a metasearch engine returns results in the form of a result list of documents. The key issue is to combine these lists to achieve the best performance. The salient contribution of this paper is a result merging model that ...
Expert agreement and content based reranking in a meta search environment using Mearf
WWW '02: Proceedings of the 11th international conference on World Wide Web

Recent increase in the number of search engines on the Web and the availability of meta search engines that can query multiple search engines makes it important to find effective methods for combining results coming from different sources. In this paper ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

October 2015

1998 pages

ISBN:9781450337946

DOI:10.1145/2806416

General Chairs:
James Bailey
The University of Melbourne
,
Alistair Moffat
The University of Melbourne
,
Program Chairs:
Charu C. Aggarwal
IBM
,
Maarten de Rijke
University of Amsterdam
,
Ravi Kumar
Google
,
Vanessa Murdock
Microsoft
,
Timos Sellis
RMIT University
,
Jeffrey Xu Yu
Chinese University of Hong Kong

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM'15

Sponsor:

CIKM'15: 24th ACM International Conference on Information and Knowledge Management

October 18 - 23, 2015

Melbourne, Australia

Acceptance Rates

CIKM '15 Paper Acceptance Rate 165 of 646 submissions, 26%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
384
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)5

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Garba AKhalid SAleryni AUllah ITairan NShah HMumin D(2024)Utilizing Ant Colony Optimization for Result Merging in Federated SearchEngineering, Technology & Applied Science Research10.48084/etasr.730214:4(14832-14839)Online publication date: 2-Aug-2024
https://doi.org/10.48084/etasr.7302
Tyomkin LKurland OHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Analyzing Fusion Methods Using the Condorcet RuleProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657912(2281-2285)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657912
Stamatis VSalampasis M(2020)Results Merging in the Patent DomainProceedings of the 24th Pan-Hellenic Conference on Informatics10.1145/3437120.3437313(229-232)Online publication date: 20-Nov-2020
https://dl.acm.org/doi/10.1145/3437120.3437313
Huang XQi JSun YZhang RZheng H(2019)CARL: Aggregated Search with Context-Aware Module Embedding Learning2019 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN.2019.8851716(1-8)Online publication date: Jul-2019
https://doi.org/10.1109/IJCNN.2019.8851716
Xu DShaikh N(2018)A Heuristic Approach for Ranking Items Based on Inputs from Multiple ExpertsInternational Journal of Information Systems and Social Change10.4018/IJISSC.20180701019:3(1-22)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.4018/IJISSC.2018070101
Kurland OCulpepper JCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Fusion in Information RetrievalThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210186(1383-1386)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210186
Onal KZhang YAltingovde IRahman MKaragoz PBraylan ADang BChang HKim HMcnamara QAngert ABanner EKhetan VMcdonnell TNguyen AXu DWallace BRijke MLease M(2018)Neural information retrievalInformation Retrieval10.1007/s10791-017-9321-y21:2-3(111-182)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s10791-017-9321-y
Benham RCulpepper J(2017)Risk-Reward Trade-offs in Rank FusionProceedings of the 22nd Australasian Document Computing Symposium10.1145/3166072.3166084(1-8)Online publication date: 7-Dec-2017
https://dl.acm.org/doi/10.1145/3166072.3166084
Anava YShtok AKurland ORabinovich EMukhopadhyay SZhai CBertino ECrestani FMostafa JTang JSi LZhou XChang YLi YSondhi P(2016)A Probabilistic Fusion FrameworkProceedings of the 25th ACM International on Conference on Information and Knowledge Management10.1145/2983323.2983739(1463-1472)Online publication date: 24-Oct-2016
https://dl.acm.org/doi/10.1145/2983323.2983739
Yan RSong YWu HPerego RSebastiani FAslam JRuthven IZobel J(2016)Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation SystemProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval10.1145/2911451.2911542(55-64)Online publication date: 7-Jul-2016
https://dl.acm.org/doi/10.1145/2911451.2911542

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents