short-paper

Open access

Relevance under the Iceberg: Reasonable Prediction for Extreme Multi-label Classification

Authors:

Wei-Cheng Chang,

Hsiang-Fu YuAuthors Info & Claims

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1870 - 1874

https://doi.org/10.1145/3477495.3531767

Published: 07 July 2022 Publication History

Abstract

In the era of big data, eXtreme Multi-label Classification (XMC) has already become one of the most essential research tasks to deal with enormous label spaces in machine learning applications. Instead of assessing every individual label, most XMC methods rely on label trees or filters to derive short ranked label lists as prediction, thereby reducing computational overhead. Specifically, existing studies obtain ranked label lists with a fixed length for prediction and evaluation. However, these predictions are unreasonable since data points have varied numbers of relevant labels. The greatly small and large list lengths in evaluation, such as Precision@5 and Recall@100, can also lead to the ignorance of other relevant labels or the tolerance of many irrelevant labels. In this paper, we aim to provide reasonable prediction for extreme multi-label classification with dynamic numbers of predicted labels. In particular, we propose a novel framework, Model-Agnostic List Truncation with Ordinal Regression (MALTOR), to leverage the ranking properties and truncate long ranked label lists for better accuracy. Extensive experiments conducted on six large-scale real-world benchmark datasets demonstrate that MALTOR significantly outperforms statistical baseline methods and conventional ranked list truncation methods in ad-hoc retrieval with both linear and deep XMC models. The results of an ablation study also shows the effectiveness of each individual component in our proposed MALTOR.

References

[1]

Hosein Azarbonyad, Mostafa Dehghani, Maarten Marx, and Jaap Kamps. 2021. Learning to rank for multi-label text classification: combining different sources of information. Natural Language Engineering 27, 1 (2021), 89--111.

[2]

Rohit Babbar and Bernhard Schölkopf. 2017. DiSMEC: Distributed sparse machines for extreme multi-label classification. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. 721--729.

Digital Library

[3]

Rohit Babbar and Bernhard Schölkopf. 2019. Data scarcity, robustness and extreme multi-label classification. Machine Learning 108, 8 (2019), 1329--1351.

Digital Library

[4]

Dara Bahri, Yi Tay, Che Zheng, Donald Metzler, and Andrew Tomkins. 2020. Choppy: Cut transformer for ranked list truncation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1513--1516.

Digital Library

[5]

Tal Baumel, Jumana Nassour-Kassis, Raphael Cohen, Michael Elhadad, and Noémie Elhadad. 2018. Multi-label classification of patient notes: case study on ICD code assignment. In Workshops at the thirty-second AAAI conference on artificial intelligence.

[6]

Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateek Jain. 2015. Sparse Local Embeddings for Extreme Multi-label Classification. In NIPS, Vol. 29. 730--738.

Digital Library

[7]

Wei-Cheng Chang, Daniel Jiang, Hsiang-Fu Yu, Choon Hui Teo, Jiong Zhang, Kai Zhong, Kedarnath Kolluri, Qie Hu, Nikhil Shandilya, Vyacheslav Ievgrafov, et al. 2021. Extreme multi-label learning for semantic matching in product search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2643--2651.

Digital Library

[8]

Wei Chu and S Sathiya Keerthi. 2005. New approaches to support vector ordinal regression. In Proceedings of the 22nd international conference on Machine learning. 145--152.

Digital Library

[9]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, 4171--4186.

[10]

Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. the Journal of machine Learning research 9 (2008), 1871--1874.

[11]

Eibe Frank and Mark Hall. 2001. A simple approach to ordinal classification. In European conference on machine learning. Springer, 145--156.

Digital Library

[12]

Jyun-Yu Jiang, Patrick H Chen, Cho-Jui Hsieh, and Wei Wang. 2020. Clustering and constructing user coresets to accelerate large-scale top-k recommender systems. In Proceedings of The Web Conference 2020. 2177--2187.

Digital Library

[13]

Sujay Khandagale, Han Xiao, and Rohit Babbar. 2020. Bonsai: diverse and shallow trees for extreme multi-label classification. Machine Learning 109, 11 (2020), 2099--2119.

Digital Library

[14]

Ling Li and Hsuan-Tien Lin. 2006. Ordinal regression by extended binary classification. Advances in neural information processing systems 19 (2006).

[15]

Yen-Chieh Lien, Daniel Cohen, and W Bruce Croft. 2019. An assumption-free approach to the dynamic truncation of ranked lists. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. 79--82.

Digital Library

[16]

Weiwei Liu, Donna Xu, IvorWTsang, andWenjie Zhang. 2018. Metric learning for multi-output tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 2 (2018), 408--422.

Digital Library

[17]

Peter McCullagh. 1980. Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological) 42, 2 (1980), 109--127.

[18]

Alexandru Niculescu-Mizil and Ehsan Abbasnejad. 2017. Label filters for large scale multilabel classification. In Artificial intelligence and statistics. PMLR, 1448--1457.

[19]

Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In Proceedings of the 2018 World Wide Web Conference. 993--1002.

Digital Library

[20]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.

[21]

Christopher Winship and Robert D Mare. 1984. Regression models with ordinal variables. American sociological review (1984), 512--525.

[22]

ChenWu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, and Xueqi Cheng. 2021. Learning to Truncate Ranked Lists for Information Retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4453--4461.

[23]

Shunyao Wu, Yuzhu Chen, Zhiruo Li, Jian Li, Fengyang Zhao, and Xiaoquan Su. 2021. Towards multi-label classification: Next step of machine learning for microbiome research. Computational and Structural Biotechnology Journal (2021).

[24]

Yiming Yang and Siddharth Gopal. 2012. Multilabel classification with meta-level features in a learning-to-rank framework. Machine Learning 88, 1 (2012), 47--68.

Digital Library

[25]

Ian EH Yen, Xiangru Huang, Wei Dai, Pradeep Ravikumar, Inderjit Dhillon, and Eric Xing. 2017. Ppdsparse: A parallel primal-dual sparse method for extreme classification. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 545--553.

Digital Library

[26]

Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. 2019. AttentionXML: Label Tree-based Attention-Aware ee Model for High-Performance Extreme Multi-Label Text Classification. Advances in Neural Information Processing Systems 32 (2019), 5820--5830.

[27]

Hsiang-Fu Yu, Kai Zhong, Jiong Zhang,Wei-Cheng Chang, and Inderjit S Dhillon. 2022. PECOS: Prediction for enormous and correlated output spaces. the Journal of machine Learning research (2022).

Digital Library

[28]

Jiong Zhang, Wei-Cheng Chang, Hsiang-Fu Yu, and Inderjit S Dhillon. 2021. Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification. In Advances in Neural Information Processing Systems.

[29]

Wenbin Zheng, Xiaping Fu, and Yibin Ying. 2014. Spectroscopy-based food classification with extreme learning machine. Chemometrics and Intelligent Laboratory Systems 139 (2014), 42--47.

Cited By

Ye HSunderraman RJi S(2024)MatchXML: An Efficient Text-Label Matching Framework for Extreme Multi-Label Text ClassificationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.337475036:9(4781-4793)Online publication date: Sep-2024
https://doi.org/10.1109/TKDE.2024.3374750
Zhang JWang YChang WLi WJiang JHsieh CYu HFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Build Faster with Less: A Journey to Accelerate Sparse Model Building for Semantic Matching in Product SearchProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614661(4960-4966)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614661
Zhang DTaneva-Popova BYoshioka MKiseleva JAliannejadi M(2023)A Theoretical Analysis of Out-of-Distribution Detection in Multi-Label ClassificationProceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3578337.3605116(275-282)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.1145/3578337.3605116

Index Terms

Relevance under the Iceberg: Reasonable Prediction for Extreme Multi-label Classification
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information retrieval

Recommendations

Generalized Zero-Shot Extreme Multi-label Learning
KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

Extreme Multi-label Learning (XML) involves assigning the subset of most relevant labels to a data point from millions of label choices. A hitherto unaddressed challenge in XML is that of predicting unseen labels with no training points. These form a ...
Collaborative learning of supervision and correlation for generalized zero-shot extreme multi-label learning
Abstract
Generalized zero-shot extreme multi-label learning (GZXML) aims to predict relevant labels for unknown instances from a set of seen and unseen labels and is widely used in engineering applications. Since the supervisory information of the ...
Cost-sensitive classification with inadequate labeled data

It is an actual and challenging issue to learn cost-sensitive models from those datasets that are with few labeled data and plentiful unlabeled data, because some time labeled data are very difficult, time consuming and/or expensive to obtain. To solve ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2022

3569 pages

ISBN:9781450387323

DOI:10.1145/3477495

General Chairs:
Enrique Amigo
UNED
,
Pablo Castells
UAM and Amazon
,
Julio Gonzalo
UNED
,
Program Chairs:
Ben Carterette
Spotify
,
J. Shane Culpepper
RMIT University
,
Gabriella Kazai
Waseda University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '22

Sponsor:

SIGIR

SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2022

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
590
Total Downloads

Downloads (Last 12 months)266
Downloads (Last 6 weeks)15

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ye HSunderraman RJi S(2024)MatchXML: An Efficient Text-Label Matching Framework for Extreme Multi-Label Text ClassificationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.337475036:9(4781-4793)Online publication date: Sep-2024
https://doi.org/10.1109/TKDE.2024.3374750
Zhang JWang YChang WLi WJiang JHsieh CYu HFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Build Faster with Less: A Journey to Accelerate Sparse Model Building for Semantic Matching in Product SearchProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614661(4960-4966)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614661
Zhang DTaneva-Popova BYoshioka MKiseleva JAliannejadi M(2023)A Theoretical Analysis of Out-of-Distribution Detection in Multi-Label ClassificationProceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3578337.3605116(275-282)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.1145/3578337.3605116

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents