short-paper

Open access

On Optimizing Top-K Metrics for Neural Ranking Models

Authors:

Michael Bendersky,

Marc NajorkAuthors Info & Claims

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 2303 - 2307

https://doi.org/10.1145/3477495.3531849

Published: 07 July 2022 Publication History

Abstract

Top-K metrics such as NDCG@K are frequently used to evaluate ranking performance. The traditional tree-based models such as LambdaMART, which are based on Gradient Boosted Decision Trees (GBDT), are designed to optimize NDCG@K using the LambdaRank losses. Recently, there is a good amount of research interest on neural ranking models for learning-to-rank tasks. These models are fundamentally different from the decision tree models and behave differently with respect to different loss functions. For example, the most popular ranking losses used in neural models are the Softmax loss and the GumbelApproxNDCG loss. These losses do not connect to top-K metrics such as NDCG@K naturally. It remains a question on how to effectively optimize NDCG@K for neural ranking models. In this paper, we follow the LambdaLoss framework and design novel and theoretically sound losses for NDCG@K metrics, while the original LambdaLoss paper can only do so using an unsound heuristic. We study the new losses on the LETOR benchmark datasets and show that the new losses work better than other losses for neural ranking models.

References

[1]

Sebastian Bruch, Shuguang Han, Michael Bendersky, and Marc Najork. 2020. A Stochastic Treatment of Learning to Rank Scoring Functions. In Proceedings of the 13th International Conference on Web Search and Data Mining. 61--69.

Digital Library

[2]

Sebastian Bruch, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2019. An Analysis of the Softmax Cross Entropy Loss for Learning-to-Rank with Binary Relevance. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR '19). 75--78.

Digital Library

[3]

Sebastian Bruch, Masrour Zoghi, Michael Bendersky, and Marc Najork. 2019. Revisiting approximate metric optimization in the age of deep neural networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1241--1244.

Digital Library

[4]

Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning. 89--96.

Digital Library

[5]

Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11, 23--581 (2010), 81.

[6]

Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. 2006. Learning to Rank with Nonsmooth Cost Functions. In Proceedings of the 19th International Conference on Neural Information Processing Systems (NIPS'06). 193--200.

Digital Library

[7]

Olivier Chapelle and Mingrui Wu. 2010. Gradient descent optimization of smoothed information retrieval metrics. Information retrieval 13, 3 (2010), 216-- 235.

[8]

Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Transactions on Information Systems (TOIS) 35, 2 (2016), 1--31.

Digital Library

[9]

Rolf Jagerman, Harrie Oosterhuis, and Maarten de Rijke. 2019. To model or to intervene: A comparison of counterfactual and online learning to rank from user interactions. In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval. 15--24.

Digital Library

[10]

Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20, 4 (2002), 422--446.

Digital Library

[11]

Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. 781--789.

Digital Library

[12]

Hyunsung Lee, Sangwoo Cho, Yeongjae Jang, Jaekwang Kim, and Honguk Woo. 2021. Differentiable ranking metric using relaxed sorting for top-k recommendation. IEEE Access 9 (2021), 114649--114658.

[13]

Pan Li, Zhen Qin, Xuanhui Wang, and Donald Metzler. 2019. Combining decision trees and neural networks for learning-to-rank in personal search. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2032--2040.

Digital Library

[14]

Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval. Found. Trends Inf. Retr. 3, 3 (mar 2009), 225--331.

Digital Library

[15]

Harrie Oosterhuis and Maarten de Rijke. 2020. Policy-aware unbiased learning to rank for top-k rankings. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 489--498.

Digital Library

[16]

Rama Kumar Pasumarthi, Sebastian Bruch, Xuanhui Wang, Cheng Li, Michael Bendersky, Marc Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, and Stephan Wolf. 2019. Tf-ranking: Scalable tensorflow library for learning-to-rank. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2970--2978.

Digital Library

[17]

Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. CoRR abs/1306.2597 (2013). http://arxiv.org/abs/1306.2597

[18]

Tao Qin, Tie-Yan Liu, and Hang Li. 2010. A general approximation framework for direct optimization of information retrieval measures. Information retrieval 13, 4 (2010), 375--397.

[19]

Zhen Qin, Suming J. Chen, Donald Metzler, Yongwoo Noh, Jingzheng Qin, and Xuanhui Wang. 2020. Attribute-Based Propensity for Unbiased Learning in Recommender Systems: Algorithm and Case Studies. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2359--2367.

Digital Library

[20]

Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2021. Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?. In International Conference on Learning Representations.

[21]

Pradeep Ravikumar, Ambuj Tewari, and Eunho Yang. 2011. On NDCG consistency of listwise ranking methods. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 618--626.

[22]

Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. 2008. Softrank: optimizing non-smooth rank metrics. In Proceedings of the 2008 International Conference on Web Search and Data Mining. 77--86.

Digital Library

[23]

Hamed Valizadegan, Rong Jin, Ruofei Zhang, and Jianchang Mao. 2009. Learning to rank by optimizing ndcg measure. Advances in neural information processing systems 22 (2009).

[24]

Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to Rank with Selection Bias in Personal Search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '16). 115--124.

Digital Library

[25]

Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 610--618.

Digital Library

[26]

Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2018. The lambdaloss framework for ranking metric optimization. In Proceedings of the 27th ACM international conference on information and knowledge management. 1313--1322.

Digital Library

[27]

Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning. 1192--1199.

Digital Library

[28]

Le Yan, Zhen Qin, Rama Kumar Pasumarthi, Xuanhui Wang, and Mike Bendersky. 2021. Diversification-Aware Learning to Rank using Distributed Representation. In Proceedings of the Web Conference 2021 (WWW '21). 127--136.

Digital Library

Cited By

Kang Jde Rijke MOosterhuis HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted TreesProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657918(2390-2394)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657918
Srinivasan SSheng SDeshmukh RLuo CDattatreya YSanyal SS V N VChua TNgo CKumar RLauw HKa-Wei Lee R(2024)Bi-CAT: Improving Robustness of LLM-based Text Rankers to Conditional Distribution ShiftsCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3651947(1626-1633)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3651947
Wang YWang ZYang JWen SKong DLi HGai KChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking SystemsProceedings of the ACM on Web Conference 202410.1145/3589334.3645605(3798-3809)Online publication date: 13-May-2024
https://doi.org/10.1145/3589334.3645605
Show More Cited By

Index Terms

On Optimizing Top-K Metrics for Neural Ranking Models
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Learning to rank

Recommendations

The LambdaLoss Framework for Ranking Metric Optimization
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

How to optimize ranking metrics such as Normalized Discounted Cumulative Gain (NDCG) is an important but challenging problem, because ranking metrics are either flat or discontinuous everywhere, which makes them hard to be optimized directly. Among ...
Quality-biased ranking for queries with commercial intent
WWW '13 Companion: Proceedings of the 22nd International Conference on World Wide Web

Modern search engines are good enough to answer popular commercial queries with mainly highly relevant documents. However, our experiments show that users behavior on such relevant commercial sites may differ from one to another web-site with the same ...
An Alternative Cross Entropy Loss for Learning-to-Rank
WWW '21: Proceedings of the Web Conference 2021

Listwise learning-to-rank methods form a powerful class of ranking algorithms that are widely adopted in applications such as information retrieval. These algorithms learn to rank a set of items by optimizing a loss that is a function of the entire set—...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2022

3569 pages

ISBN:9781450387323

DOI:10.1145/3477495

General Chairs:
Enrique Amigo
UNED
,
Pablo Castells
UAM and Amazon
,
Julio Gonzalo
UNED
,
Program Chairs:
Ben Carterette
Spotify
,
J. Shane Culpepper
RMIT University
,
Gabriella Kazai
Waseda University

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '22

Sponsor:

SIGIR

SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2022

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
987
Total Downloads

Downloads (Last 12 months)501
Downloads (Last 6 weeks)81

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kang Jde Rijke MOosterhuis HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted TreesProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657918(2390-2394)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657918
Srinivasan SSheng SDeshmukh RLuo CDattatreya YSanyal SS V N VChua TNgo CKumar RLauw HKa-Wei Lee R(2024)Bi-CAT: Improving Robustness of LLM-based Text Rankers to Conditional Distribution ShiftsCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3651947(1626-1633)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3651947
Wang YWang ZYang JWen SKong DLi HGai KChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking SystemsProceedings of the ACM on Web Conference 202410.1145/3589334.3645605(3798-3809)Online publication date: 13-May-2024
https://doi.org/10.1145/3589334.3645605
Landoni MHuibers TMurgia EPera M(2024)Good for Children, Good for All?Advances in Information Retrieval10.1007/978-3-031-56066-8_24(302-313)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56066-8_24
Qin ZJagerman RPasumarthi RZhuang HZhang HBai AHui KYan LWang XOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)RD-SuiteProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3667673(35748-35760)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3667673
Lyzhin IUstimenko AGulin AProkhorenkova LKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Which tricks are important for learning to rank?Proceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619376(23264-23278)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619376
Bai AJagerman RQin ZYan LKar PLin BWang XBendersky MNajork MFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Regression Compatible Listwise Objectives for Calibrated Ranking with Binary RelevanceProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614712(4502-4508)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614712
Zhuang HQin ZJagerman RHui KMa JLu JNi JWang XBendersky MChen HDuh WHuang HKato MMothe JPoblete B(2023)RankT5: Fine-Tuning T5 for Text Ranking with Ranking LossesProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3592047(2308-2313)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3592047
Yan LQin ZWang XBendersky MNajork MZhang ARangwala H(2022)Scale Calibration of Deep Ranking ModelsProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3539072(4300-4309)Online publication date: 14-Aug-2022
https://dl.acm.org/doi/10.1145/3534678.3539072

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents