research-article

Relevance Estimation with Multiple Information Sources on Search Engine Result Pages

Authors:

Qi TianAuthors Info & Claims

CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

Pages 627 - 636

https://doi.org/10.1145/3269206.3271673

Published: 17 October 2018 Publication History

Abstract

Relevance estimation is among the most important tasks in the ranking of search results because most search engines follow the Probability Ranking Principle. Current relevance estimation methodologies mainly concentrate on text matching between the query and Web documents, link analysis and user behavior models. However, users judge the relevance of search results directly from Search Engine Result Pages (SERPs), which provide valuable signals for reranking. Morden search engines aggregate heterogeneous information items (such as images, news, and hyperlinks) to a single ranking list on SERPs. The aggregated search results have different visual patterns, textual semantics and presentation structures, and a better strategy should rely on all these information sources to improve ranking performance. In this paper, we propose a novel framework named Joint Relevance Estimation model (JRE), which learns the visual patterns from screenshots of search results, explores the presentation structures from HTML source codes and also adopts the semantic information of textual contents. To evaluate the performance of the proposed model, we construct a large scale practical Search Result Relevance (SRR) dataset which consists of multiple information sources and 4-grade relevance scores of over 60,000 search results. Experimental results show that the proposed JRE model achieves better performance than state-of-the-art ranking solutions as well as the original ranking of commercial search engines.

References

[1]

Javad Azimi, Ruofei Zhang, Zhou Yang, Vidhya Navalpakkam, Jianchang Mao, and Xiaoli Fern. 2012. The impact of visual appearance on user response in online display advertising. 457--458.

Digital Library

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. Computer Science (2014).

[3]

Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th international conference on World wide web. ACM, 1--10.

Digital Library

[4]

Danqi Chen, Weizhu Chen, Haixun Wang, Zheng Chen, and Qiang Yang. 2012. Beyond ten blue links: enabling user click modeling in federated web search. In Proceedings of the fifth ACM international conference on Web search and data mining. ACM, 463--472.

Digital Library

[5]

Kan Chen, Trung Bui, Chen Fang, Zhaowen Wang, and Ram Nevatia. 2017. AMC: Attention guided multi-modal correlation learning for image search. arXiv preprint arXiv:1704.00763 (2017).

[6]

Haibin Cheng, Roelof Van Zwol, Javad Azimi, Eren Manavoglu, Ruofei Zhang, Yang Zhou, and Vidhya Navalpakkam. 2012. Multimedia features for click prediction of new ads in display advertising. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 777--785.

Digital Library

[7]

Dan C Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, and Jürgen Schmidhuber. 2011. Flexible, high performance convolutional neural networks for image classification. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence, Vol. 22. Barcelona, Spain, 1237.

Digital Library

[8]

Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, Vol. 70, 4 (1968), 213.

[9]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 248--255.

[10]

Georges E. Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 331--338.

Digital Library

[11]

Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. 2015. The pascal visual object classes challenge: A retrospective. International journal of computer vision, Vol. 111, 1 (2015), 98--136.

Digital Library

[12]

Yixing Fan, Jiafeng Guo, Yanyan Lan, Jun Xu, Liang Pang, and Xueqi Cheng. 2017. Learning Visual Features from Snapshots for Web Search. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 247--256.

Digital Library

[13]

Yoav Freund, Raj D. Iyer, Robert E. Schapire, and Yoram Singer. 1998. An Efficient Boosting Algorithm for Combining Preferences. In Fifteenth International Conference on Machine Learning. 170--178.

Digital Library

[14]

Jerome H. Friedman. 2001. Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics, Vol. 29, 5 (2001), 1189--1232.

[15]

Fan Guo, Chao Liu, and Yi Min Wang. 2009. Efficient multiple-click models in web search. In Proceedings of the Second ACM International Conference on Web Search and Data Mining. ACM, 124--131.

Digital Library

[16]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[17]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997), 1735--1780.

Digital Library

[18]

Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In International Conference on Neural Information Processing Systems. 2042--2050.

Digital Library

[19]

Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In ACM Conference on Knowledge Discovery and Data Mining. 133--142.

Digital Library

[20]

D. Kinga and J. Ba Adam. 2015. A method for stochastic optimization. In International Conference on Learning Representations (ICLR) .

[21]

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, Vol. 123, 1 (2017), 32--73.

Digital Library

[22]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.

Digital Library

[23]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740--755.

[24]

Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2016. Knowing when to look: Adaptive attention via A visual sentinel for image captioning. arXiv preprint arXiv:1612.01887 (2016).

[25]

Cheng Luo, Yiqun Liu, Tetsuya Sakai, Fan Zhang, Min Zhang, and Shaoping Ma. 2017. Evaluating Mobile Search with Height-Biased Gain. (2017).

[26]

Minh Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. Computer Science (2015).

[27]

Thomas Mandl. 2006. Implementation and evaluation of a quality-based search engine. In Proceedings of the seventeenth conference on Hypertext and hypermedia. ACM, 73--84.

Digital Library

[28]

Schütze Manning Raghavan. 2008. Introduction to Information Retrieval. Journal of the American Society for Information Science & Technology, Vol. 43, 3 (2008), 824--825.

[29]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems, Vol. 26 (2013), 3111--3119.

Digital Library

[30]

L Page. 1999. The PageRank Citation Ranking: Bringing Order to the Web. Stanford Digital Libraries Working Paper, Vol. 9, 1 (1999), 1--14.

[31]

Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text Matching as Image Recognition. In AAAI. 2793--2799.

Digital Library

[32]

Tao Qin, Tie Yan Liu, Jun Xu, and Hang Li. 2010. LETOR: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval, Vol. 13, 4 (2010), 346--374.

Digital Library

[33]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2017. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, Vol. 39, 6 (2017), 1137--1149.

Digital Library

[34]

Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations & Trends® in Information Retrieval, Vol. 3, 4 (2009), 333--389.

Digital Library

[35]

Kalervo Rvelin, Kek, and Jaana Inen. 2002. Cumulated gain-based evaluation of IR techniques. Acm Transactions on Information Systems, Vol. 20, 4 (2002), 422--446.

Digital Library

[36]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[37]

Ruihua Song, Haifeng Liu, Ji-Rong Wen, and Wei-Ying Ma. 2004. Learning block importance models for web pages. In Proceedings of the 13th international conference on World Wide Web. ACM, 203--211.

Digital Library

[38]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1--9.

[39]

Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2015. A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations. (2015), 2835--2841.

Digital Library

[40]

Chao Wang, Yiqun Liu, Meng Wang, Ke Zhou, Jian-yun Nie, and Shaoping Ma. 2015. Incorporating non-sequential behavior into click models. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 283--292.

Digital Library

[41]

Chao Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013. Incorporating vertical results into search click models. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. ACM, 503--512.

Digital Library

[42]

Qiang Wu, Christopher J. C. Burges, Krysta M. Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval, Vol. 13, 3 (2010), 254--270.

Digital Library

[43]

Jun Xu and Hang Li. 2007. AdaRank:a boosting algorithm for information retrieval. 391--398.

Digital Library

[44]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Computer Science (2015), 2048--2057.

Digital Library

[45]

Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, and Chikashi Nobata. 2016. Ranking Relevance in Yahoo Search. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 323--332.

Digital Library

[46]

Masrour Zoghi, Tomáš Tunys, Lihong Li, Damien Jose, Junyan Chen, Chun Ming Chin, and Maarten de Rijke. 2016. Click-based hot fixes for underperforming torso queries. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 195--204.

Digital Library

Cited By

Ye ZXie XAi QLiu YWang ZSu WZhang M(2023)Relevance Feedback with Brain SignalsACM Transactions on Information Systems10.1145/363787442:4(1-37)Online publication date: 18-Dec-2023
https://dl.acm.org/doi/10.1145/3637874
Breuer TFuhr NSchaer P(2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
https://dl.acm.org/doi/10.1145/3623640
Yuan HDou ZZhou YGuo YWen JFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)VILE: Block-Aware Visual Enhanced Document RetrievalProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615107(3104-3113)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615107
Show More Cited By

Index Terms

Relevance Estimation with Multiple Information Sources on Search Engine Result Pages
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

Search Result Reranking with Visual and Structure Information Sources

Relevance estimation is among the most important tasks in the ranking of search results. Current methodologies mainly concentrate on text matching, link analysis, and user behavior models. However, users judge the relevance of search results directly ...
Search Engine Optimization by Re-Ranking the Product Search Result Based on User Click Data
AISS '21: Proceedings of the 3rd International Conference on Advanced Information Science and System

Blibli.com provides a search engine for its customers. It used Solr search engine with only plain BM25 similarity function which is based on probability. In order to improve search engine performance, this research tried to implement an algorithm that ...
Intent-based diversification of web search results: metrics and algorithms

We study the problem of web search result diversification in the case where intent based relevance scores are available. A diversified search result will hopefully satisfy the information need of user-L.s who may have different intents. In this context, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

October 2018

2362 pages

ISBN:9781450360142

DOI:10.1145/3269206

General Chair:
Alfredo Cuzzocrea
University of Trieste, Italy
,
Program Chairs:
James Allan
University of Massachusetts, USA
,
Norman Paton
University of Manchester, United Kingdom
,
Divesh Srivastava
AT&T Labs Research, USA
,
Rakesh Agrawal
Data Insights Lab, USA
,
Andrei Broder
Google Research, USA
,
Mohammed Zaki
Rensselaer Polytechnic Institute, USA
,
Selcuk Candan
Arizona State University, USA
,
Alexandros Labrinidis
University of Pittsburgh, USA
,
Assaf Schuster
Technion, Israel
,
Haixun Wang
Google Research, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Key Basic Research Program
Natural Science Foundation of China

Conference

CIKM '18

Sponsor:

CIKM '18: The 27th ACM International Conference on Information and Knowledge Management

October 22 - 26, 2018

Torino, Italy

Acceptance Rates

CIKM '18 Paper Acceptance Rate 147 of 826 submissions, 18%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
761
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)8

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ye ZXie XAi QLiu YWang ZSu WZhang M(2023)Relevance Feedback with Brain SignalsACM Transactions on Information Systems10.1145/363787442:4(1-37)Online publication date: 18-Dec-2023
https://dl.acm.org/doi/10.1145/3637874
Breuer TFuhr NSchaer P(2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
https://dl.acm.org/doi/10.1145/3623640
Yuan HDou ZZhou YGuo YWen JFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)VILE: Block-Aware Visual Enhanced Document RetrievalProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615107(3104-3113)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615107
Milton APera M(2023)Into the Unknown: Exploration of Search Engines’ Responses to Users with Depression and AnxietyACM Transactions on the Web10.1145/358028317:4(1-29)Online publication date: 18-Jan-2023
https://dl.acm.org/doi/10.1145/3580283
Xie XDong QWang BLv FYao TGan WWu ZLi XLi HLiu YMa JChen HDuh WHuang HKato MMothe JPoblete B(2023)T2Ranking: A Large-scale Chinese Benchmark for Passage RankingProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591874(2681-2690)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591874
Gorodishchev AGorodishcheva ABaigozhina DKovalev G(2023)Web 3.0 Credibility: Principles for Ranking Media Sources2023 Communication Strategies in Digital Society Seminar (ComSDS)10.1109/ComSDS58064.2023.10130407(184-188)Online publication date: 12-Apr-2023
https://doi.org/10.1109/ComSDS58064.2023.10130407
Ye ZXie XLiu YWang ZLi XLi JChen XZhang MMa SAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Why Don't You ClickProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3532082(633-645)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3532082
Chen YLiu SLiu ZSun WBaltrunas LSchroeder B(2022)WANDS: Dataset for Product Search Relevance AssessmentAdvances in Information Retrieval10.1007/978-3-030-99736-6_9(128-141)Online publication date: 5-Apr-2022
https://doi.org/10.1007/978-3-030-99736-6_9
Chen JMao JLiu YZhang MMa SCaverlee JHu XLalmas MWang W(2020)A Context-Aware Click Model for Web SearchProceedings of the 13th International Conference on Web Search and Data Mining10.1145/3336191.3371819(88-96)Online publication date: 20-Jan-2020
https://dl.acm.org/doi/10.1145/3336191.3371819
Zhang JMao JLiu YZhang RZhang MMa SXu JTian QZhu WTao DCheng XCui PRundensteiner ECarmel DHe QXu Yu J(2019)Context-Aware Ranking by Constructing a Virtual Environment for Reinforcement LearningProceedings of the 28th ACM International Conference on Information and Knowledge Management10.1145/3357384.3357945(1603-1612)Online publication date: 3-Nov-2019
https://dl.acm.org/doi/10.1145/3357384.3357945
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents