Ranking-Incentivized Document Manipulations for Multiple Queries
Pages 61 - 70
Abstract
In competitive retrieval settings, document publishers (authors) modify their documents in response to induced rankings so as to potentially improve their future rankings. Previous work has focused on analyzing ranking-incentivized document modifications for a single query. We present a novel theoretical and empirical study of document modification strategies applied for improved ranking for multiple queries; e.g., those representing the same information need. Using game theoretic analysis, we show that in contrast to the single-query setting, an equilibrium does not necessarily exist. We empirically study document modification strategies in the multiple-queries setting by organizing ranking competitions. In contrast to previous ranking competitions devised for the single-query setting, we also used a neural ranker and allowed in some competitions the use of generative AI tools to modify documents. We found that publishers tend to mimic content from documents highly ranked in the past, as in the single-query setting, although this was a somewhat less emphasized trend when generative AI tools were allowed. We also demonstrate the merits of using information induced from multiple queries to predict which document might be the highest ranked in the next ranking for a given query.
References
[1]
Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2016. UQV100: A Test Collection with Query Variability. In Proceedings of SIGIR. 465--474.
[2]
Ran Ben-Basat, Moshe Tennenholtz, and Oren Kurland. 2015. The Probability Ranking Principle is Not Optimal in Adversarial Retrieval Settings. In Proceedings of ICTIR. 51--60.
[3]
Ran Ben-Basat, Moshe Tennenholtz, and Oren Kurland. 2017. A Game Theoretic Analysis of the Adversarial Retrieval Setting. J. Artif. Intell. Res., Vol. 60 (2017), 1127--1164.
[4]
Michael Bendersky, W. Bruce Croft, and Yanlei Diao. 2011. Quality-biased ranking of web documents. In Proceedings of WSDM. 95--104.
[5]
Michael Bendersky, Donald Metzler, and W. Bruce Croft. 2011. Parameterized concept weighting in verbose queries. In Proceedings of SIGIR. 605--614.
[6]
Carlos Castillo and Brian D. Davison. 2010. Adversarial Web Search. Foundations and Trends in Information Retrieval, Vol. 4, 5 (2010), 377--486.
[7]
Xuanang Chen, Ben He, Le Sun, and Yingfei Sun. 2023. Defense of Adversarial Ranking Attack in Text Retrieval: Benchmark and Baseline via Detection. CoRR, Vol. abs/2307.16816 (2023).
[8]
Xuanang Chen, Ben He, Zheng Ye, Le Sun, and Yingfei Sun. 2023. Towards Imperceptible Document Manipulations against Neural Ranking Models. In Findings of the Association for Computational Linguistics: ACL. 6648--6664.
[9]
Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of SIGIR. 758--759.
[10]
Shlomo Geva, Jaap Kamps, and Ralf Schenkel (Eds.). 2012. Focused Retrieval of Content and Structure, 10th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2011, Saarbrücken, Germany, December 12-14, 2011, Revised Selected Papers.
[11]
Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2018. Ranking Robustness Under Adversarial Document Manipulations. In Proceedings of SIGIR. 395--404.
[12]
Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2020. Ranking-Incentivized Quality Preserving Content Modification. In Proceedings of SIGIR. 259--268.
[13]
Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2021. Driving the Herd: Search Engines as Content Influencers. In Proceedings of SIGIR. 586--595.
[14]
Zoltán Gyöngyi and Hector Garcia-Molina. 2005. Web Spam Taxonomy. In Proceedings of AIRWeb 2005, First International Workshop on Adversarial Information Retrieval on the Web. 39--47.
[15]
Rachel Hartman, Aaron Moss, Shalom Jaffe, Cheskie Rosenzweig, Leib Litman, and Jonathan Robinson. 2023. Introducing Connect by CloudResearch: Advancing Online Participant Recruitment in the Digital Age.
[16]
Sandy Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. 2017. Adversarial Attacks on Neural Network Policies. In Proceedings of ICLR.
[17]
Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. In Proc. of EMNLP. 2021--2031.
[18]
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019. Certified Robustness to Adversarial Word Substitutions. In Proceedings of EMNLP-IJCNLP. 4127--4140.
[19]
Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of SIGIR. 154--161.
[20]
Felix Kreuk, Yossi Adi, Moustapha Cissé, and Joseph Keshet. 2018. Fooling End-To-End Speaker Verification With Adversarial Examples. In Proceedings of ICASSP. 1962--1966.
[21]
Oren Kurland and Moshe Tennenholtz. 2022. Competitive Search. In Proceedings of SIGIR. 2838--2849.
[22]
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020. Pretrained Transformers for Text Ranking: BERT and Beyond. CoRR, Vol. abs/2010.06467 (2020).
[23]
Jiawei Liu, Yangyang Kang, Di Tang, Kaisong Song, Changlong Sun, Xiaofeng Wang, Wei Lu, and Xiaozhong Liu. 2022. Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models. In Proceedings of SIGSAC. 2025--2039.
[24]
Tie-Yan Liu. 2011. Learning to Rank for Information Retrieval. Springer. I-XVII, 1--285 pages.
[25]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023. Topic-oriented Adversarial Attacks against Black-box Neural Ranking Models. In Proceedings of SIGIR. 1700--1709.
[26]
John F. Nash. 1950. Equilibrium Points in N-Person Games. Proc. of the National Academy of Sciences of the United States of America, Vol. 36.1 (1950), 48--49.
[27]
Noam Nisan, Tim Roughgarden, Éva Tardos, and Vijay V. Vazirani (Eds.). 2007. Algorithmic Game Theory. Cambridge University Press. Section 19.3.
[28]
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085 (2019).
[29]
OpenAI. 2023. GPT-4 Technical Report. CoRR, Vol. abs/2303.08774 (2023).
[30]
Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks against Machine Learning. In Proceedings of AsiaCCS. 506--519.
[31]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vander Plas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, Vol. 12 (2011), 2825--2830.
[32]
Nimrod Raifer, Fiana Raiber, Moshe Tennenholtz, and Oren Kurland. 2017. Information Retrieval Meets Game Theory: The Ranking Competition Between Documents' Authors. In Proceedings of SIGIR. 465--474.
[33]
Stephen E. Robertson. 1977. The Probability Ranking Principle in IR. Journal of Documentation (1977), 294--304. Reprinted in K. Sparck Jones and P. Willett (eds), Readings in Information Retrieval, pp. 281--286, 1997.
[34]
Junshuai Song, Jiangshan Zhang, Jifeng Zhu, Mengyun Tang, and Yong Yang. 2022. TRAttack: Text Rewriting Attack Against Text Retrieval. In Proceedings of RepL4NLP@ACL. 191--203.
[35]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proceedings of ICLR.
[36]
Ziv Vasilisky, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2023. Content-Based Relevance Estimation in Retrieval Settings with Ranking-Incentivized Document Manipulations. In Proceedings of SIGIR. 205--2014.
[37]
Yumeng Wang, Lijun Lyu, and Avishek Anand. 2022. BERT Rankers are Brittle: A Study using Adversarial Document Perturbations. In Proceedings of ICTIR. 115--120.
[38]
William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Trans. Inf. Syst., Vol. 28, 4 (2010), 20:1--20:38.
[39]
Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2022. PRADA: Practical Black-Box Adversarial Attacks against Neural Ranking Models. arxiv: 2204.01321
[40]
Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. 2023. Are Neural Ranking Models Robust? ACM Trans. Inf. Syst., Vol. 41, 2 (2023), 29:1--29:36.
[41]
Qiang Wu, Christopher J. C. Burges, Krysta Marie Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval, Vol. 13, 3 (2010), 254--270.
[42]
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan L. Yuille. 2017. Adversarial Examples for Semantic Segmentation and Object Detection. In Proceedings of ICCV 2017. 1378--1387.
[43]
Cheng Xiang Zhai and John D. Lafferty. 2006. A risk minimization framework for information retrieval. Inf. Process. Manag., Vol. 42, 1 (2006), 31--55.
[44]
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. 2019. Theoretically Principled Trade-off between Robustness and Accuracy. In Proceedings of ICML. 7472--7482.
Index Terms
- Ranking-Incentivized Document Manipulations for Multiple Queries
Recommendations
Clustering queries for better document ranking
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge managementDifferent queries require different ranking methods. It is however challenging to determine what queries are similar, and how to rank documents for them. In this paper, we propose a new method to cluster queries according to the similarity determined ...
Ranking Robustness Under Adversarial Document Manipulations
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information RetrievalFor many queries in the Web retrieval setting there is an on-going ranking competition: authors manipulate their documents so as to promote them in rankings. Such competitions can have unwarranted effects not only in terms of retrieval effectiveness, ...
Context-sensitive document ranking
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge managementRanking is a main research issue in IR-styled keyword search over a set of documents. In this paper, we study a new keyword search problem, called context-sensitive document ranking, which is to rank documents with an additional context that provides ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
August 2024
267 pages
ISBN:9798400706813
DOI:10.1145/3664190
- General Chair:
- Harrie Oosterhuis,
- Program Chairs:
- Hannah Bast,
- Chenyan Xiong
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 August 2024
Check for updates
Badges
Best Paper
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
ICTIR '24
Sponsor:
ICTIR '24: The 2024 ACM SIGIR International Conference on the Theory of Information Retrieval
July 13, 2024
Washington DC, USA
Acceptance Rates
ICTIR '24 Paper Acceptance Rate 26 of 45 submissions, 58%;
Overall Acceptance Rate 235 of 527 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 95Total Downloads
- Downloads (Last 12 months)95
- Downloads (Last 6 weeks)75
Reflects downloads up to 26 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in