Targeted Training for Multi-organization Recommendation
Authors: 
Kiran Tomlinson, Mengting Wan, Cao Lu, Brent Hecht, 
Jaime Teevan, Longqi YangAuthors Info & Claims
Article No.: 12, Pages 1 - 18
Abstract
Making recommendations for users in diverse organizations (orgs) is a challenging task for workplace social platforms such as Microsoft Teams and Slack. The current industry-standard model training approaches either use data from all organizations to maximize information or train organization-specific models to minimize noise. Our real-world experiments show that both approaches are poorly suited for the multi-org recommendation setting where different organizations’ interaction patterns vary in their generalizability. We introduce targeted training, which improves on standard practices by automatically selecting a subset of orgs for model development whose data are cleanest and best represent global trends. We demonstrate how and when targeted training improves over global training through theoretical analysis and simulation. Our experiments on large-scale datasets from Microsoft Teams, SharePoint, Stack Exchange, DBLP, and Reddit show that in many cases targeted training can improve mean average precision (MAP) across orgs by 10–15% over global training, is more robust to orgs with lower data quality, and generalizes better to unseen orgs. Our training framework is applicable to a wide range of inductive recommendation models, from simple regression models to graph neural networks (GNNs).
References
[1]
Lada A. Adamic and Eytan Adar. 2003. Friends and neighbors on the web. Social Networks 25, 3 (2003), 211–230.
[2]
Rebecca R. Andridge and Roderick J. A. Little. 2010. A review of hot deck imputation for survey non-response. International Statistical Review 78, 1 (2010), 40–64.
[3]
Sylvain Arlot and Alain Celisse. 2010. A survey of cross-validation procedures for model selection. Statistics Surveys 4 (2010), 40–79.
[4]
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The Pushshift Reddit dataset. In Proceedings of the ICWSM. 830–839.
[5]
Robert M. Bell and Yehuda Koren. 2007. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proceedings of the ICDM. IEEE, 43–52.
[6]
James Bennett and Stan Lanning. 2007. The netflix prize. In Proceedings of the KDD Cup and Workshop. Vol. 2007, New York, NY, 35.
[7]
Alex Beutel, Ed H. Chi, Zhiyuan Cheng, Hubert Pham, and John Anderson. 2017. Beyond globally optimal: Focused learning for improved recommendations. In Proceedings of the WWW. 203–212.
[8]
Jesús Bobadilla, Fernando Ortega, Antonio Hernando, and Abraham Gutiérrez. 2013. Recommender systems survey. Knowledge-based Systems 46 (2013), 109–132.
[9]
Dirk Bollen, Bart P. Knijnenburg, Martijn C. Willemsen, and Mark Graus. 2010. Understanding choice overload in recommender systems. In Proceedings of the RecSys. 63–70.
[10]
Renaud Bourassa. 2018. Building recommender systems with strict privacy boundaries. In Proceedings of the RecSys. 486–486.
[11]
Sergey Brin and Lawrence Page. 1998. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems 30, 1-7 (1998), 107–117.
[12]
Wei Chen, Wynne Hsu, and Mong Li Lee. 2013. Making recommendations from multiple domains. In Proceedings of the KDD. 892–900.
[13]
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide and deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. 7–10.
[14]
Evangelia Christakopoulou and George Karypis. 2016. Local item-item models for top-n recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems. 67–74.
[15]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the RecSys. 191–198.
[16]
Inderjit S. Dhillon. 2001. Co-clustering documents and words using bipartite spectral graph partitioning. In Proceedings of the KDD. 269–274.
[17]
Ignacio Fernández-Tobías, Iván Cantador, Marius Kaminskas, and Francesco Ricci. 2012. Cross-domain recommender systems: A survey of the state-of-the-art. In Proceedings of the Spanish Conference on Information Retrieval. 1–12.
[18]
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the WWW. 173–182.
[19]
Larry V. Hedges. 1982. Estimation of effect size from a series of independent experiments. Psychological Bulletin 92, 2 (1982), 490.
[20]
Liang Hu, Jian Cao, Guandong Xu, Longbing Cao, Zhiping Gu, and Can Zhu. 2013. Personalized recommendation via cross-domain triadic factorization. In Proceedings of the WWW. 595–606.
[21]
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the ICLR.
[22]
Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer 42, 8 (2009), 30–37.
[23]
Adit Krishnan, Mahashweta Das, Mangesh Bendre, Hao Yang, and Hari Sundaram. 2020. Transfer learning via contextual invariants for one-to-many cross-domain recommendation. In Proceedings of the SIGIR. 1081–1090.
[24]
Joonseok Lee, Samy Bengio, Seungyeon Kim, Guy Lebanon, and Yoram Singer. 2014. Local collaborative ranking. In Proceedings of the 23rd International Conference on World Wide Web. 85–96.
[25]
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine 37, 3 (2020), 50–60.
[26]
David Liben-Nowell and Jon Kleinberg. 2007. The link-prediction problem for social networks. Journal of the Association for Information Science and Technology 58, 7 (2007), 1019–1031.
[27]
Pasquale Lops, Dietmar Jannach, Cataldo Musto, Toine Bogers, and Marijn Koolen. 2019. Trends in content-based recommendation. User Modeling and User-Adapted Interaction 29, 2 (2019), 239–249.
[28]
Mark E. J. Newman. 2002. Assortative mixing in networks. Physical Review Letters 89, 20 (2002), 208701.
[29]
Mark E. J. Newman and Michelle Girvan. 2004. Finding and evaluating community structure in networks. Physical Review E 69, 2 (2004), 026113.
[30]
Jan Overgoor, George Pakapol Supaniratisai, and Johan Ugander. 2020. Scaling choice models of relational social data. In Proceedings of the KDD. 1990–1998.
[31]
Martin Riedmiller and Heinrich Braun. 1992. Rprop-a fast adaptive learning algorithm. In Proceedings of the ISCIS VII), Universitat. Citeseer.
[32]
Aditya Sakhuja. 2021. Building a Multi-tenant Content-based Recommender with Automated Training. Retrieved from https://pycon.blogspot.com/2021/05/building-multi-tenant-content-based.html, accessed 3/1/2023.
[33]
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. 2015. Autorec: Autoencoders meet collaborative filtering. In Proceedings of the WWW. 111–112.
[34]
Anu Sivunen and Kaisa Laitinen. 2019. Digital communication environments in the workplace. In Workplace Communication, Leena Mikkola and Maarit Valo (Eds.). Routledge, New York, NY, 41–53.
[35]
Stack Exchange, Inc.2021. Stack Exchange Data Dump. Retrieved from https://archive.org/details/stackexchange, accessed September 7, 2021.
[36]
Mervyn Stone. 1974. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B 36, 2 (1974), 111–133.
[37]
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. ArnetMiner: Extraction and mining of academic social networks. In Proceedings of the KDD’08. 990–998.
[38]
Vincent A. Traag, Ludo Waltman, and Nees Jan Van Eck. 2019. From Louvain to Leiden: Guaranteeing well-connected communities. Scientific Reports 9, 1 (2019), 1–12.
[39]
Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and Yi Zhou. 2019. A hybrid approach to privacy-preserving federated learning. In Proceedings of the AISec. 1–11.
[40]
Xinxi Wang and Ye Wang. 2014. Improving content-based and hybrid music recommendation using deep learning. In Proceedings of the MM. 627–636.
[41]
Duncan J. Watts and Steven H. Strogatz. 1998. Collective dynamics of ‘small-world’ networks. Nature 393, 6684 (1998), 440–442.
[42]
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. 2022. Graph neural networks in recommender systems: a survey. Comput. Surveys 55, 5 (2022), 1–37.
[43]
Jie Xu, Benjamin S. Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. 2021. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research 5, 1 (2021), 1–19.
[44]
Huan Yan, Xiangning Chen, Chen Gao, Yong Li, and Depeng Jin. 2019. Deepapf: Deep attentive probabilistic factorization for multi-site video recommendation. TC 2, 130 (2019), 17–883.
[45]
Longqi Yang, Tobias Schnabel, Paul N. Bennett, and Susan Dumais. 2021. Local factor models for large-scale inductive recommendation. In Proceedings of the 15th ACM Conference on Recommender Systems. 252–262.
[46]
Yisong Yue, Thomas Finley, Filip Radlinski, and Thorsten Joachims. 2007. A support vector method for optimizing average precision. In Proceedings of the SIGIR. 271–278.
[47]
Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. 2016. Collaborative knowledge base embedding for recommender systems. In Proceedings of the KDD. 353–362.
[48]
Muhan Zhang and Yixin Chen. 2018. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems, Vol. 31.
[49]
Muhan Zhang and Yixin Chen. 2020. Inductive matrix completion based on graph neural networks. In Proceedings of the ICLR.
[50]
Ping Zhang. 1993. Model selection via multifold cross validation. The Annals of Statistics 21, 1 (1993), 299–313.
[51]
Qian Zhang, Dianshuang Wu, Jie Lu, Feng Liu, and Guangquan Zhang. 2017. A cross-domain recommender system with consistent information transfer. Decision Support Systems 104 (2017), 49–63.
[52]
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys 52, 1 (2019), 1–38.
[53]
Shuai Zhang, Lina Yao, and Xiwei Xu. 2017. AutoSVD++: An efficient hybrid collaborative filtering model via contractive auto-encoders. In Proceedings of the SIGIR. 957–960.
[54]
Yu Zhang, Bin Cao, and Dit-Yan Yeung. 2010. Multi-domain collaborative filtering. In Proceedings of the UAI. 725–732.
[55]
Cheng Zhao, Chenliang Li, and Cong Fu. 2019. Cross-domain recommendation via preference propagation graphnet. In Proceedings of the CIKM. 2165–2168.
[56]
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018. DRN: A deep reinforcement learning framework for news recommendation. In Proceedings of the WWW. 167–176.
[57]
Lei Zheng, Vahid Noroozi, and Philip S. Yu. 2017. Joint deep modeling of users and items using reviews for recommendation. In Proceedings of the WSDM. 425–434.
[58]
Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. 2021. Cross-domain recommendation: Challenges, progress, and prospects. In Proceedings of the IJCAI.
Index Terms
- Targeted Training for Multi-organization Recommendation
Recommendations
Multi-label co-training
IJCAI'18: Proceedings of the 27th International Joint Conference on Artificial IntelligenceMulti-label learning aims at assigning a set of appropriate labels to multi-label samples. Although it has been successfully applied in various domains in recent years, most multi-label learning methods require sufficient labeled training samples, ...
Inductive Semi-supervised Multi-Label Learning with Co-Training
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningIn multi-label learning, each training example is associated with multiple class labels and the task is to learn a mapping from the feature space to the power set of label space. It is generally demanding and time-consuming to obtain labels for training ...
Activity Recommendation with Partners
Recommending social activities, such as watching movies or having dinner, is a common function found in social networks or e-commerce sites. Besides certain websites which manage activity-related locations (e.g., foursquare.com), many items on product ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
September 2023
118 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 July 2023
Online AM: 03 June 2023
Accepted: 21 May 2023
Revised: 02 March 2023
Received: 28 November 2022
Published in TORS Volume 1, Issue 3
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 234Total Downloads
- Downloads (Last 12 months)72
- Downloads (Last 6 weeks)8
Reflects downloads up to 14 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in