research-article

Evaluating recommender behavior for new users

Authors:

Joseph A. KonstanAuthors Info & Claims

RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems

Pages 121 - 128

https://doi.org/10.1145/2645710.2645742

Published: 06 October 2014 Publication History

Abstract

The new user experience is one of the important problems in recommender systems. Past work on recommending for new users has focused on the process of gathering information from the user. Our work focuses on how different algorithms behave for new users. We describe a methodology that we use to compare representatives of three common families of algorithms along eleven different metrics. We find that for the first few ratings a baseline algorithm performs better than three common collaborative filtering algorithms. Once we have a few ratings, we find that Funk's SVD algorithm has the best overall performance. We also find that ItemItem, a very commonly deployed algorithm, performs very poorly for new users. Our results can inform the design of interfaces and algorithms for new users.

Supplementary Material

JPG File (p121-sidebyside.jpg)

Download
16.13 KB

MP4 File (p121-sidebyside.mp4)

Download
55.36 MB

References

[1]

A. Bellogín. Performance prediction and evaluation in Recommender Systems: an Information Retrieval perspective. PhD thesis, Universidad Autónoma de Madrid, Madrid, Spain, Oct. 2012.

[2]

P. Cremonesi, F. Garzottto, and R. Turrin. User effort vs. accuracy in rating-based elicitation. In RecSys '12. ACM, 2012.

Digital Library

[3]

P. Cremonesi, Y. Koren, and R. Turrin. Performance of recommender algorithms on top-n recommendation tasks. In RecSys '10. ACM, 2010.

Digital Library

[4]

S. Drenner, S. Sen, and L. Terveen. Crafting the initial user experience to achieve community goals. In RecSys '08. ACM, 2008.

Digital Library

[5]

M. Ekstrand and J. Riedl. When recommenders fail: predicting recommender failure for algorithm selection and combination. In RecSys '12. ACM, 2012.

Digital Library

[6]

M. D. Ekstrand, M. Ludwig, J. A. Konstan, and J. T. Riedl. Rethinking the recommender research ecosystem: Reproducibility, openness, and lenskit. In RecSys '11. ACM, 2011.

Digital Library

[7]

S. Funk. Netflix update: Try this at home. http://sifter.org/~simon/journal/20061211.html, Dec. 2006.

[8]

N. Golbandi, Y. Koren, and R. Lempel. Adaptive bootstrapping of recommender systems using decision trees. In WSDM '11. ACM, 2011.

Digital Library

[9]

N. Golbandi, Y. Koren, and R. Lempel. Adaptive bootstrapping of recommender systems using decision trees. In WSDM '11. ACM, 2011.

Digital Library

[10]

J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst., 22(1):5--53, Jan. 2004.

Digital Library

[11]

N. Lathia, S. Hailes, L. Capra, and X. Amatriain. Temporal diversity in recommender systems. In SIGIR '10. ACM, 2010.

Digital Library

[12]

C. D. Manning, P. Raghavan, and H. Schütze. Introduction to information retrieval, volume 1. Cambridge university press Cambridge, 2008.

Digital Library

[13]

S. M. McNee, J. Riedl, and J. A. Konstan. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In CHI EA '06. ACM, 2006.

Digital Library

[14]

P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl. Grouplens: an open architecture for collaborative filtering of netnews. In CSCW '94. ACM, 1994.

Digital Library

[15]

B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-based collaborative filtering recommendation algorithms. In WWW '01. ACM, 2001.

Digital Library

[16]

G. Shani and A. Gunawardana. Evaluating recommendation systems. In F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor, editors, Recommender Systems Handbook, pages 257--297. Springer US, 2011.

[17]

M. C. Willemsen, B. P. Knijnenburg, M. P. Graus, L. C. Velter-Bremmers, and K. Fu. Using latent features diversification to reduce choice difficulty in recommendation lists. RecSys '11, 2011.

[18]

C.-N. Ziegler, S. M. McNee, J. A. Konstan, and G. Lausen. Improving recommendation lists through topic diversification. In WWW '05. ACM, 2005.

Digital Library

Cited By

Fernandez MBellogín ACantador I(2024)Analysing the Effect of Recommendation Algorithms on the Spread of MisinformationProceedings of the 16th ACM Web Science Conference10.1145/3614419.3644003(159-169)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3614419.3644003
Yadav SSrivastav S(2024)Making Use of Coverage to Assess the Research Paper Recommender System2024 1st International Conference on Advances in Computing, Communication and Networking (ICAC2N)10.1109/ICAC2N63387.2024.10895344(374-379)Online publication date: 16-Dec-2024
https://doi.org/10.1109/ICAC2N63387.2024.10895344
Klimashevskaia AJannach DElahi MTrattner C(2024)A survey on popularity bias in recommender systemsUser Modeling and User-Adapted Interaction10.1007/s11257-024-09406-0Online publication date: 1-Jul-2024
https://doi.org/10.1007/s11257-024-09406-0
Show More Cited By

Index Terms

Evaluating recommender behavior for new users
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Document filtering
      2. Information extraction

Recommendations

A new approach to evaluating novel recommendations
RecSys '08: Proceedings of the 2008 ACM conference on Recommender systems

This paper presents two methods, named Item- and User-centric, to evaluate the quality of novel recommendations. The former method focuses on analyzing the item-based recommendation network. The aim is to detect whether the network topology has any ...
A Scalable, Accurate Hybrid Recommender System
WKDD '10: Proceedings of the 2010 Third International Conference on Knowledge Discovery and Data Mining

Recommender systems apply machine learning techniques for filtering unseen information and can predict whether a user would like a given resource. There are three main types of recommender systems: collaborative filtering, content-based filtering, and ...
On Unexpectedness in Recommender Systems: Or How to Better Expect the Unexpected
Special Sections on Diversity and Discovery in Recommender Systems, Online Advertising and Regular Papers

Although the broad social and business success of recommender systems has been achieved across several domains, there is still a long way to go in terms of user satisfaction. One of the key dimensions for significant improvement is the concept of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems

October 2014

458 pages

ISBN:9781450326681

DOI:10.1145/2645710

General Chairs:
Alfred Kobsa
University of California, Irvine
,
Michelle Zhou
IBM
,
Program Chairs:
Martin Ester
Simon Fraser University
,
Yehuda Koren
Google

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 October 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Division of Information and Intelligent Systems

Conference

RecSys'14

Sponsor:

SIGCHI

RecSys'14: Eighth ACM Conference on Recommender Systems

October 6 - 10, 2014

California, Foster City, Silicon Valley, USA

Acceptance Rates

RecSys '14 Paper Acceptance Rate 35 of 234 submissions, 15%;

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
1,464
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)8

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fernandez MBellogín ACantador I(2024)Analysing the Effect of Recommendation Algorithms on the Spread of MisinformationProceedings of the 16th ACM Web Science Conference10.1145/3614419.3644003(159-169)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3614419.3644003
Yadav SSrivastav S(2024)Making Use of Coverage to Assess the Research Paper Recommender System2024 1st International Conference on Advances in Computing, Communication and Networking (ICAC2N)10.1109/ICAC2N63387.2024.10895344(374-379)Online publication date: 16-Dec-2024
https://doi.org/10.1109/ICAC2N63387.2024.10895344
Klimashevskaia AJannach DElahi MTrattner C(2024)A survey on popularity bias in recommender systemsUser Modeling and User-Adapted Interaction10.1007/s11257-024-09406-0Online publication date: 1-Jul-2024
https://doi.org/10.1007/s11257-024-09406-0
Su HLi JDu ZZhu LLu KShen H(2023)Cross-domain Recommendation via Dual Adversarial AdaptationACM Transactions on Information Systems10.1145/363252442:3(1-26)Online publication date: 11-Nov-2023
https://dl.acm.org/doi/10.1145/3632524
Su HZhang YYang XHua HWang SLi JAl Hasan MXiong L(2022)Cross-domain Recommendation via Adversarial AdaptationProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557277(1808-1817)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557277
Natarajan SVairavasundaram SKotecha KIndragandhi VPalani SSaini JRavi L(2022)CD-SemMF: Cross-Domain Semantic Relatedness Based Matrix Factorization Model Enabled With Linked Open Data for User Cold Start IssueIEEE Access10.1109/ACCESS.2022.317556610(52955-52970)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3175566
Zhang ZDong MOta KZhang YRen Y(2021)LBCF: A Link-Based Collaborative Filtering for Overfitting Problem in Recommender SystemIEEE Transactions on Computational Social Systems10.1109/TCSS.2021.30814248:6(1450-1464)Online publication date: Dec-2021
https://doi.org/10.1109/TCSS.2021.3081424
Bellogín ASaid A(2021)Improving accountability in recommender systems research through reproducibilityUser Modeling and User-Adapted Interaction10.1007/s11257-021-09302-x31:5(941-977)Online publication date: 21-Oct-2021
https://doi.org/10.1007/s11257-021-09302-x
AlRossais NKudenko DYuan T(2021)Improving cold-start recommendations using item-based stereotypesUser Modeling and User-Adapted Interaction10.1007/s11257-021-09293-931:5(867-905)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1007/s11257-021-09293-9
He FSun CLiu Y(2021)What social characteristics enhance recommender systems? The effects of network embeddedness and preference heterogeneityElectronic Commerce Research10.1007/s10660-021-09517-523:3(1807-1827)Online publication date: 28-Oct-2021
https://doi.org/10.1007/s10660-021-09517-5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten