research-article

Open access

Representation Online Matters: Practical End-to-End Diversification in Search and Recommender Systems

Authors:

Ashudeep Singh,

Nadia FawazAuthors Info & Claims

FAccT '23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency

Pages 1735 - 1746

https://doi.org/10.1145/3593013.3594112

Published: 12 June 2023 Publication History

All formats PDF

Abstract

As the use of online platforms continues to grow across all demographics, users often express a desire to feel represented in the content. To improve representation in search results and recommendations, we introduce end-to-end diversification, ensuring that diverse content flows throughout the various stages of these systems, from retrieval to ranking. We develop, experiment, and deploy scalable diversification mechanisms in multiple production surfaces on the Pinterest platform, including Search, Related Products, and New User Homefeed, to improve the representation of different skin tones in beauty and fashion content. Diversification in production systems includes three components: identifying requests that will trigger diversification, ensuring diverse content is retrieved from the large content corpus during the retrieval stage, and finally, balancing the diversity-utility trade-off in a self-adjusting manner in the ranking stage. Our approaches, which evolved from using Strong-OR logical operator to bucketized retrieval at the retrieval stage and from greedy re-rankers to multi-objective optimization using determinantal point processes for the ranking stage, balances diversity and utility while enabling fast iterations and scalable expansion to diversification over multiple dimensions. Our experiments indicate that these approaches significantly improve diversity metrics, with a neutral to a positive impact on utility metrics and improved user satisfaction, both qualitatively and quantitatively, in production.

References

[1]

Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Ieong. 2009. Diversifying Search Results. In Proceedings of the Second ACM International Conference on Web Search and Data Mining (Barcelona, Spain) (WSDM ’09). Association for Computing Machinery, New York, NY, USA, 5–14. https://doi.org/10.1145/1498759.1498766

Digital Library

[2]

Ashton Anderson, Lucas Maystre, Ian Anderson, Rishabh Mehrotra, and Mounia Lalmas. 2020. Algorithmic Effects on the Diversity of Consumption on Spotify. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Machinery, New York, NY, USA, 2155–2165. https://doi.org/10.1145/3366423.3380281

Digital Library

[3]

Jon Louis Bentley. 1975. Multidimensional Binary Search Trees Used for Associative Searching. Commun. ACM 18, 9 (sep 1975), 509–517. https://doi.org/10.1145/361002.361007

Digital Library

[4]

Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems (Barcelona, Spain) (NIPS’16). Curran Associates Inc., Red Hook, NY, USA, 4356–4364. https://proceedings.neurips.cc/paper_files/paper/2016/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf

Digital Library

[5]

Jaime Carbonell and Jade Goldstein. 1998. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Melbourne, Australia) (SIGIR ’98). Association for Computing Machinery, New York, NY, USA, 335–336. https://doi.org/10.1145/290941.291025

Digital Library

[6]

Ben Carterette. 2009. An Analysis of NP-Completeness in Novelty and Diversity Ranking. In Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory (Cambridge, UK) (ICTIR ’09). Springer-Verlag, Berlin, Heidelberg, 200–211. https://doi.org/10.1007/978-3-642-04417-5_18

Digital Library

[7]

Laming Chen, Guoxin Zhang, and Hanning Zhou. 2018. Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (Montréal, Canada) (NIPS’18). Curran Associates Inc., Red Hook, NY, USA, 5627–5638. https://proceedings.neurips.cc/paper_files/paper/2018/file/dbbf603ff0e99629dda5d75b6f75f966-Paper.pdf

Digital Library

[8]

Konstantina Christakopoulou, Alex Beutel, Rui Li, Sagar Jain, and Ed H. Chi. 2018. Q&R: A Two-Stage Approach toward Interactive Recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 139–148. https://doi.org/10.1145/3219819.3219894

Digital Library

[9]

Charles L.A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, and Ian MacKinnon. 2008. Novelty and Diversity in Information Retrieval Evaluation. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Singapore, Singapore) (SIGIR ’08). Association for Computing Machinery, New York, NY, USA, 659–666. https://doi.org/10.1145/1390334.1390446

Digital Library

[10]

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An Experimental Comparison of Click Position-Bias Models. In Proceedings of the 2008 International Conference on Web Search and Data Mining (Palo Alto, California, USA) (WSDM ’08). Association for Computing Machinery, New York, NY, USA, 87–94. https://doi.org/10.1145/1341531.1341545

Digital Library

[11]

Michael Curtiss, Iain Becker, Tudor Bosman, Sergey Doroshenko, Lucian Grijincu, Tom Jackson, Sandhya Kunnatur, Soren Lassen, Philip Pronin, Sriram Sankar, Guanghao Shen, Gintaras Woss, Chao Yang, and Ning Zhang. 2013. Unicorn: A System for Searching the Social Graph. Proc. VLDB Endow. 6, 11 (aug 2013), 1150–1161. https://doi.org/10.14778/2536222.2536239

Digital Library

[12]

Nadia Fawaz, Bhawna Juneja, and David Xue. 2020. Powering inclusive search & recommendations with our new visual skin tone model. In Pinterest Engineering Blog. https://medium.com/pinterest-engineering/powering-inclusive-search-recommendations-with-our-new-visual-skin-tone-model-1d3ba6eeffc7

[13]

Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi. 2019. Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD ’19). Association for Computing Machinery, New York, NY, USA, 2221–2231. https://doi.org/10.1145/3292500.3330691

Digital Library

[14]

Jennifer Gillenwater, Alex Kulesza, and Ben Taskar. 2012. Near-Optimal MAP Inference for Determinantal Point Processes. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2 (Lake Tahoe, Nevada) (NIPS’12). Curran Associates Inc., Red Hook, NY, USA, 2735–2743. https://proceedings.neurips.cc/paper_files/paper/2012/file/6c8dba7d0df1c4a79dd07646be9a26c8-Paper.pdf

[15]

Pinterest Inc.2023. Pinterest Announces First Quarter 2023 Results. https://investor.pinterestinc.com/press-releases/press-releases-details/2023/Pinterest-Announces-First-Quarter-2023-Results/default.aspx. Accessed: 2023-05-02.

[16]

Piotr Indyk, Rajeev Motwani, Prabhakar Raghavan, and Santosh Vempala. 1997. Locality-Preserving Hashing in Multidimensional Spaces. In Proceedings of the Twenty-Ninth Annual ACM Symposium on Theory of Computing (El Paso, Texas, USA) (STOC ’97). Association for Computing Machinery, New York, NY, USA, 618–625. https://doi.org/10.1145/258533.258656

Digital Library

[17]

Rishabh Iyer and Jeff Bilmes. 2013. Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2 (Lake Tahoe, Nevada) (NIPS’13). Curran Associates Inc., Red Hook, NY, USA, 2436–2444. https://proceedings.neurips.cc/paper_files/paper/2013/file/a1d50185e7426cbb0acad1e6ca74b9aa-Paper.pdf

[18]

Matthew Kay, Cynthia Matuszek, and Sean A. Munson. 2015. Unequal Representation and Gender Stereotypes in Image Search Results for Occupations. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3819–3828. https://doi.org/10.1145/2702123.2702520

Digital Library

[19]

Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (aug 2009), 30–37. https://doi.org/10.1109/MC.2009.263

Digital Library

[20]

Alex Kulesza and Ben Taskar. 2012. Determinantal Point Processes for Machine Learning. Now Publishers Inc., Hanover, MA, USA. https://doi.org/10.1561/2200000044

[21]

Lei Li, Dingding Wang, Tao Li, Daniel Knox, and Balaji Padmanabhan. 2011. SCENE: A Scalable Two-Stage Personalized News Recommendation System. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (Beijing, China) (SIGIR ’11). Association for Computing Machinery, New York, NY, USA, 125–134. https://doi.org/10.1145/2009916.2009937

Digital Library

[22]

Odile Macchi. 1975. The coincidence approach to stochastic point processes. Advances in Applied Probability 7, 1 (1975), 83–122. https://doi.org/10.2307/1425855

[23]

Yu A. Malkov and D. A. Yashunin. 2020. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Trans. Pattern Anal. Mach. Intell. 42, 4 (apr 2020), 824–836. https://doi.org/10.1109/TPAMI.2018.2889473

Digital Library

[24]

Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, and Jure Leskovec. 2020. PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Association for Computing Machinery, New York, NY, USA, 2311–2320. https://doi.org/10.1145/3394486.3403280

Digital Library

[25]

Jing Qin. 2021. A Survey of Long-Tail Item Recommendation Methods. Wireless Communications and Mobile Computing 2021 (2021), 1–14. https://doi.org/10.1155/2021/7536316

Digital Library

[26]

Filip Radlinski, Paul N. Bennett, Ben Carterette, and Thorsten Joachims. 2009. Redundancy, Diversity and Interdependent Document Relevance. SIGIR Forum 43, 2 (dec 2009), 46–52. https://doi.org/10.1145/1670564.1670572

Digital Library

[27]

Filip Radlinski, Robert Kleinberg, and Thorsten Joachims. 2008. Learning Diverse Rankings with Multi-Armed Bandits. In Proceedings of the 25th International Conference on Machine Learning (Helsinki, Finland) (ICML ’08). Association for Computing Machinery, New York, NY, USA, 784–791. https://doi.org/10.1145/1390156.1390255

Digital Library

[28]

Nielsen Media Research. 2023. Seen on screen: The impact of diverse talent in media. https://www.nielsen.com/insights/2023/seen-on-screen-the-impact-of-diverse-talent-in-media/. Accessed: 2023-01-31.

[29]

Pew Research. 2018. Gender and Jobs in Online Image Searches. https://www.pewresearch.org/social-trends/2018/12/17/gender-and-jobs-in-online-image-searches/. Accessed: 2023-01-31.

[30]

Pew Research. 2021. Social Media Fact Sheet. https://www.pewresearch.org/internet/fact-sheet/social-media/. Accessed: 2023-01-31.

[31]

Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal 27, 3 (1948), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

[32]

Mark Wilhelm, Ajith Ramanathan, Alexander Bonomo, Sagar Jain, Ed H. Chi, and Jennifer Gillenwater. 2018. Practical Diversified Recommendations on YouTube with Determinantal Point Processes. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy) (CIKM ’18). Association for Computing Machinery, New York, NY, USA, 2165–2173. https://doi.org/10.1145/3269206.3272018

Digital Library

[33]

Peter N. Yianilos. 1993. Data Structures and Algorithms for Nearest Neighbor Search in General Metric Spaces. In Proceedings of the Fourth Annual ACM-SIAM Symposium on Discrete Algorithms (Austin, Texas, USA) (SODA ’93). Society for Industrial and Applied Mathematics, USA, 311–321. https://dl.acm.org/doi/10.5555/313559.313789

Digital Library

[34]

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 974–983. https://doi.org/10.1145/3219819.3219890

Digital Library

[35]

Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Megahed, and Ricardo Baeza-Yates. 2017. FA*IR: A Fair Top-k Ranking Algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (Singapore, Singapore) (CIKM ’17). Association for Computing Machinery, New York, NY, USA, 1569–1578. https://doi.org/10.1145/3132847.3132938

Digital Library

[36]

Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning Fair Representations. In Proceedings of the 30th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 28). PMLR, Atlanta, Georgia, USA, 325–333. https://proceedings.mlr.press/v28/zemel13.html

[37]

ChengXiang Zhai, William W. Cohen, and John Lafferty. 2015. Beyond Independent Relevance: Methods and Evaluation Metrics for Subtopic Retrieval. SIGIR Forum 49, 1 (jun 2015), 2–9. https://doi.org/10.1145/2795403.2795405

Digital Library

[38]

Xiaojin Zhu, Andrew Goldberg, Jurgen Van Gael, and David Andrzejewski. 2007. Improving Diversity in Ranking using Absorbing Random Walks. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference. Association for Computational Linguistics, Rochester, New York, 97–104. https://aclanthology.org/N07-1013

Cited By

Srinivasan HSchumann CSinha AMadras DOlanubi GBeutel ARicco SChen J(2024)Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People ImagesProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658940(797-821)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658940
Gleason JGhosh ARobertson RWilson CChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Perceptions in Pixels: Analyzing Perceived Gender and Skin Tone in Real-world Image Search ResultsProceedings of the ACM Web Conference 202410.1145/3589334.3645666(1249-1259)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645666

Index Terms

Representation Online Matters: Practical End-to-End Diversification in Search and Recommender Systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Intent-aware search result diversification
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

Search result diversification has gained momentum as a way to tackle ambiguous queries. An effective approach to this problem is to explicitly model the possible aspects underlying a query, in order to maximise the estimated relevance of the retrieved ...
Multidimensional search result diversification: diverse search results for diverse users
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

Hundreds of millions of people today rely on Web based Search Engines to satisfy their information needs. In order to meet the expectations of this vast and diverse user population, the search engine should present a list of results such that the ...
Ranking in folksonomy systems: can context help?
CIKM '08: Proceedings of the 17th ACM conference on Information and knowledge management

Folksonomy systems have shown to contribute to the quality of Web search ranking strategies. In this paper, we analyze and compare different graph-based ranking algorithms, namely FolkRank, SocialPageRank, and SocialSimRank. We enhance these algorithms ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

FAccT '23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency

June 2023

1929 pages

ISBN:9798400701924

DOI:10.1145/3593013

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

FAccT '23

FAccT '23: the 2023 ACM Conference on Fairness, Accountability, and Transparency

June 12 - 15, 2023

IL, Chicago, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
1,510
Total Downloads

Downloads (Last 12 months)1,129
Downloads (Last 6 weeks)133

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Srinivasan HSchumann CSinha AMadras DOlanubi GBeutel ARicco SChen J(2024)Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People ImagesProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658940(797-821)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658940
Gleason JGhosh ARobertson RWilson CChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Perceptions in Pixels: Analyzing Perceived Gender and Skin Tone in Real-world Image Search ResultsProceedings of the ACM Web Conference 202410.1145/3589334.3645666(1249-1259)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645666

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents