research-article

On the Diversity and Explainability of Recommender Systems: A Practical Framework for Enterprise App Recommendation

Authors:

Latrice Barnett,

Markus Anderle,

Harshavardhan Utharavalli,

Steven HOIAuthors Info & Claims

CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Pages 4302 - 4311

https://doi.org/10.1145/3459637.3481940

Published: 30 October 2021 Publication History

Abstract

This paper introduces an enterprise app recommendation problem with a new "to-business'' use case, which aims to assist a sales team acting as the bridge connecting the applications and developers with the customers who apply these apps to solve their business problems. Our recommender system is an assistant to the sales team, helping recommend relevant apps to the customers for their businesses and increasing the likelihood of improving sales revenue. Besides recommendation accuracy, recommendation diversity and explainability are even more crucial since they provide more exposure opportunities for app developers and improve the transparency and trustworthiness of the recommender system. To allow the sales team to explore unpopular but relevant apps and understand why such apps are recommended, we propose a novel framework for improving aggregate recommendation diversity and generating recommendation explanations, which supports a wide variety of models for improving recommendation accuracy. The model in our framework is simple yet effective, which can be trained in an end-to-end manner and deployed as a recommendation service easily. Furthermore, our framework can also apply to other generic recommender systems for improving diversity and generating explanations. Experiments on public and private datasets demonstrate the effectiveness of our framework and solution.

References

[1]

Gediminas Adomavicius and YoungOk Kwon. 2012. Improving Aggregate Recommendation Diversity Using Ranking-Based Techniques. IEEE Trans. on Knowl. and Data Eng., Vol. 24, 5 (2012), 896--911.

Digital Library

[2]

Erik Brynjolfsson, Yu (Jeffrey) Hu, and Duncan Simester. 2011. Goodbye Pareto Principle, Hello Long Tail: The Effect of Search Costs on the Concentration of Product Sales. Management Science, Vol. 57, 8 (2011), 1373--1386.

Digital Library

[3]

Erik Brynjolfsson, Yu (Jeffrey) Hu, and Michael D. Smith. 2003. Consumer Surplus in the Digital Economy: Estimating the Value of Increased Product Variety at Online Booksellers. Management Science, Vol. 49, 11 (2003), 1580--1596.

Digital Library

[4]

Shuo Chang, F. Maxwell Harper, and Loren Gilbert Terveen. 2016. Crowd-Based Personalized Natural Language Explanations for Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16). Association for Computing Machinery, 175--182.

Digital Library

[5]

Jingwu Chen, Fuzhen Zhuang, Xin Hong, Xiang Ao, Xing Xie, and Qing He. 2018b. Attention-Driven Factor Model for Explainable Personalized Recommendation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '18). 909--912.

Digital Library

[6]

Laming Chen, Guoxin Zhang, and Hanning Zhou. 2018a. Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS'18). 5627--5638.

Digital Library

[7]

Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2019. Personalized Fashion Recommendation with Visual Explanations Based on Multimodal Attention Network: Towards Visually Explainable Recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'19). 765--774.

Digital Library

[8]

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & Deep Learning for Recommender Systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems (DLRS 2016). 7--10.

Digital Library

[9]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16). 191--198.

Digital Library

[10]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 a. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171--4186.

[11]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 b. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4171--4186.

[12]

Tim Donkers, Benedikt Loepp, and Jürgen Ziegler. 2017. Sequential User-Based Recurrent Neural Network Recommendations. In Proceedings of the Eleventh ACM Conference on Recommender Systems (RecSys '17). 152--160.

Digital Library

[13]

Azin Ghazimatin, Oana Balalau, Rishiraj Saha Roy, and Gerhard Weikum. 2020. PRINCE: Provider-Side Interpretability with Counterfactual Explanations in Recommender Systems. In Proceedings of the 13th International Conference on Web Search and Data Mining (WSDM '20). Association for Computing Machinery, 196--204.

Digital Library

[14]

Carlos A. Gomez-Uribe and Neil Hunt. 2016. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Transactions on Management Information Systems, Vol. 6, 4 (2016).

Digital Library

[15]

Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI'17). AAAI Press, 1725--1731.

Digital Library

[16]

F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst., Vol. 5, 4 (2015).

Digital Library

[17]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) (ICCV '15). 1026--1034.

Digital Library

[18]

Yu-Chieh Ho, Yi-Ting Chiang, and Jane Yung-Jen Hsu. 2014. Who Likes It More? Mining Worth-Recommending Items from Long Tails by Modeling Relative Preference. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining (WSDM '14). 253--262.

Digital Library

[19]

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining (ICDM '08). 263--272.

Digital Library

[20]

Yejin Kim, Kwangseob Kim, Chanyoung Park, and Hwanjo Yu. 2019. Sequential and Diverse Recommendation with Long Tail. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019. 2740--2746.

Digital Library

[21]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015.

[22]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (NIPS'12). 1097--1105.

Digital Library

[23]

Greg Linden, Brent Smith, and Jeremy York. 2003. Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Computing, Vol. 7 (2003), 76--80.

Digital Library

[24]

Julian McAuley and Jure Leskovec. 2013. Hidden Factors and Hidden Topics: Understanding Rating Dimensions with Review Text. In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys '13). 165--172.

Digital Library

[25]

Georgina Peake and Jun Wang. 2018. Explanation Mining: Post Hoc Interpretability of Latent Factor Models for Recommendation Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19--23, 2018. 2060--2069.

Digital Library

[26]

Steffen Rendle. 2010. Factorization Machines. In Proceedings of the 2010 IEEE International Conference on Data Mining (ICDM '10). 995--1000.

Digital Library

[27]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). 1135--1144.

Digital Library

[28]

Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. 2007. Restricted Boltzmann Machines for Collaborative Filtering. In Proceedings of the 24th International Conference on Machine Learning (ICML '07). 791--798.

Digital Library

[29]

Ying Shan, T. Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. 2016. Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). 255--262.

Digital Library

[30]

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML'17). 3319--3328.

Digital Library

[31]

Nava Tintarev and Judith Masthoff. 2012. Evaluating the effectiveness of explanations for recommender systems: Methodological issues and empirical studies on the impact of personalization. User Modelling and User-Adapted Interaction, Vol. 22, 4-5 (Oct. 2012), 399--439.

Digital Library

[32]

Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & Cross Network for Ad Click Predictions. In Proceedings of the ADKDD'17 (ADKDD'17).

Digital Library

[33]

Yichao Wang, Xiangyu Zhang, Zhirong Liu, Zhenhua Dong, Xinhua Feng, Ruiming Tang, and Xiuqiang He. 2020. Personalized Re-ranking for Improving Diversity in Live Recommender Systems. arxiv: cs.IR/2004.06390

[34]

Mark Wilhelm, Ajith Ramanathan, Alexander Bonomo, Sagar Jain, Ed H. Chi, and Jennifer Gillenwater. 2018. Practical Diversified Recommendations on YouTube with Determinantal Point Processes. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18). 2165--2173.

Digital Library

[35]

Mi Zhang and Neil Hurley. 2008. Avoiding Monotony: Improving the Diversity of Recommendation Lists. In Proceedings of the 2008 ACM Conference on Recommender Systems (RecSys '08). 123--130.

Digital Library

[36]

Yongfeng Zhang, Guokun Lai, Min Zhang, Yi Zhang, Yiqun Liu, and Shaoping Ma. 2014. Explicit Factor Models for Explainable Recommendation Based on Phrase-Level Sentiment Analysis. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14). 83--92.

Digital Library

[37]

Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '18). 1059--1068.

Digital Library

[38]

Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005. Improving Recommendation Lists through Topic Diversification. In Proceedings of the 14th International Conference on World Wide Web (WWW '05). 22--32.

Digital Library

Cited By

Yu DWang XXiong YShen XWu RWang DZou ZXu G(2023)MHANER: A Multi-source Heterogeneous Graph Attention Network for Explainable Recommendation in Online GamesACM Transactions on Intelligent Systems and Technology10.1145/362624315:4(1-23)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3626243
Krook JBlockx J(2023)Recommender Systems, Autonomy and User EngagementProceedings of the First International Symposium on Trustworthy Autonomous Systems10.1145/3597512.3599712(1-9)Online publication date: 11-Jul-2023
https://dl.acm.org/doi/10.1145/3597512.3599712
Liang GWen JZhou W(2022)Individual Diversity Preference Aware Neural Collaborative FilteringKnowledge-Based Systems10.1016/j.knosys.2022.109730258:COnline publication date: 22-Dec-2022
https://dl.acm.org/doi/10.1016/j.knosys.2022.109730

Index Terms

On the Diversity and Explainability of Recommender Systems: A Practical Framework for Enterprise App Recommendation
1. Computing methodologies
  1. Machine learning
    1. Learning settings
      1. Learning from implicit feedback
2. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Information retrieval diversity
    2. Retrieval tasks and goals
      1. Recommender systems

Recommendations

User Personality and User Satisfaction with Recommender Systems

In this study, we show that individual users' preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. We demonstrate that we can extract strong signals about individual ...
Research on Recommendation List Diversity of Recommender Systems
ICMECG '08: Proceedings of the 2008 International Conference on Management of e-Commerce and e-Government

Recommender systems have emerged in the past several years as an effective way to help people cope with the problem of information overload. Most research up to this point has focused on improving the accuracy of recommender systems. However, considering ...
Using Multidimensional Clustering Based Collaborative Filtering Approach Improving Recommendation Diversity
WI-IAT '12: Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 03

In this paper, we present a hybrid recommendation approach for discovering potential preferences of individual users. The proposed approach provides a flexible solution that incorporates multidimensional clustering into a collaborative filtering ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

October 2021

4966 pages

ISBN:9781450384469

DOI:10.1145/3459637

General Chairs:
Gianluca Demartini
The University of Queensland, Australia
,
Guido Zuccon
The University of Queensland, Australia
,
Program Chairs:
J. Shane Culpepper
RMIT University, Australia
,
Zi Huang
The University of Queensland, Australia
,
Hanghang Tong
University of Illinois at Urbana-Champaign, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '21

Sponsor:

CIKM '21: The 30th ACM International Conference on Information and Knowledge Management

November 1 - 5, 2021

Queensland, Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
491
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)3

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu DWang XXiong YShen XWu RWang DZou ZXu G(2023)MHANER: A Multi-source Heterogeneous Graph Attention Network for Explainable Recommendation in Online GamesACM Transactions on Intelligent Systems and Technology10.1145/362624315:4(1-23)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3626243
Krook JBlockx J(2023)Recommender Systems, Autonomy and User EngagementProceedings of the First International Symposium on Trustworthy Autonomous Systems10.1145/3597512.3599712(1-9)Online publication date: 11-Jul-2023
https://dl.acm.org/doi/10.1145/3597512.3599712
Liang GWen JZhou W(2022)Individual Diversity Preference Aware Neural Collaborative FilteringKnowledge-Based Systems10.1016/j.knosys.2022.109730258:COnline publication date: 22-Dec-2022
https://dl.acm.org/doi/10.1016/j.knosys.2022.109730

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents