research-article

Public Access

The Impact of More Transparent Interfaces on Behavior in Personalized Recommendation

Authors:

Tobias Schnabel,

Saleema Amershi,

Paul N. Bennett,

Thorsten JoachimsAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 991 - 1000

https://doi.org/10.1145/3397271.3401117

Published: 25 July 2020 Publication History

Abstract

Many interactive online systems, such as social media platforms or news sites, provide personalized experiences through recommendations or news feed customization based on people's feedback and engagement on individual items (e.g., liking items). In this paper, we investigate how we can support a greater degree of user control in such systems by changing the way the system allows people to gauge the consequences of their feedback actions. To this end, we consider two important aspects of how the system responds to feedback actions: (i) immediacy, i.e., how quickly the system responds with an update, and (ii) visibility, i.e., whether or not changes will get highlighted. We used both an in-lab qualitative study and a large-scale crowd-sourced study to examine the impact of these factors on people's reported preferences and observed behavioral metrics. We demonstrate that UX design which enables people to preview the impact of their actions and highlights changes results in a higher reported transparency, an overall preference for this design, and a greater selectivity in which items are liked.

References

[1]

Xavier Amatriain and Justin Basilico. 2015. Recommender systems in industry: A Netflix case study. In Recommender Systems Handbook. Springer, 385--419.

[2]

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et almbox. 2019. Guidelines for human-ai interaction. In CHI. 1--13.

[3]

Fedor Bakalov, Marie-Jean Meurs, Birgitta König-Ries, Bahar Sateli, René Witte, Greg Butler, and Adrian Tsang. 2013. An approach to controlling user models and personalization effects in recommender systems. In IUI. 49--56.

[4]

JesúS Bobadilla, Fernando Ortega, Antonio Hernando, and JesúS Bernal. 2012. A collaborative filtering approach to mitigate the new user cold start problem. Knowledge-based systems, Vol. 26 (2012), 225--238.

[5]

Svetlin Bostandjiev, John O'Donovan, and Tobias Höllerer. 2012. TasteWeights: a visual interactive hybrid recommender system. In RecSys. 35--42.

[6]

Eli T. Brown, Remco Chang, and Alex Endert. 2016. Human-Machine-Learner Interaction: The Best of Both Worlds. In CHI Workshop on Human Centered Machine Learning.

[7]

Dan Cosley, Shyong K Lam, Istvan Albert, Joseph A Konstan, and John Riedl. 2003. Is seeing believing?: how recommender system interfaces affect users' opinions. In CHI. 585--592.

[8]

Anita Crescenzi, Diane Kelly, and Leif Azzopardi. 2016. Impacts of time constraints and system delays on user experience. In SIGIR. 141--150.

[9]

Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, Vol. 41, 6 (1990), 391--407.

[10]

Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In CHI. 278--288.

[11]

Motahhare Eslami, Karrie Karahalios, Christian Sandvig, Kristen Vaccaro, Aimee Rickman, Kevin Hamilton, and Alex Kirlik. 2016. First I like it, then I hide it: Folk theories of social feeds. In CHI. 2371--2382.

[12]

James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: interactive concept learning in image search. In CHI. 29--38.

[13]

Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline and online evaluation of news recommender systems at swissinfo. ch. In RecSys. 169--176.

[14]

Donald A Hantula, Diane DiClemente Brockman, and Carter L Smith. 2008. Online shopping as foraging: The effects of increasing delays on purchasing and patch residence. IEEE Transactions on Professional Communication, Vol. 51, 2 (2008), 147--154.

[15]

Chen He, Denis Parra, and Katrien Verbert. 2016. Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications, Vol. 56 (2016), 9--27.

Digital Library

[16]

Dietmar Jannach, Sidra Naveed, and Michael Jugovac. 2016. User control in recommender systems: Overview and interaction challenges. In International Conference on Electronic Commerce and Web Technologies. 21--33.

[17]

Yucheng Jin, Bruno Cardoso, and Katrien Verbert. 2017. How do different levels of user control affect cognitive load and acceptance of recommendations?. In RecSys Workshop on Interfaces and Human Decision Making for Recommender Systems. 35--42.

[18]

Michael Jugovac and Dietmar Jannach. 2017. Interacting with recommenders-overview and research directions. ACM Transactions on Interactive Intelligent Systems (TiiS), Vol. 7, 3 (2017), 10.

[19]

Yvonne Kammerer, Rowan Nairn, Peter Pirolli, and Ed H Chi. 2009. Signpost from the masses: learning effects in an exploratory social tag search browser. In CHI. 625--634.

[20]

Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In CHI. 453--456.

[21]

Bart P Knijnenburg, Svetlin Bostandjiev, John O'Donovan, and Alfred Kobsa. 2012. Inspectability and control in social recommenders. In RecSys. 43--50.

[22]

Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of explanatory debugging to personalize interactive machine learning. In IUI. 126--137.

[23]

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In WWW. 661--670.

[24]

Pasquale Lops, Marco De Gemmis, and Giovanni Semeraro. 2011. Content-based recommender systems: State of the art and trends. In Recommender Systems Handbook. Springer, 73--105.

[25]

Sean M McNee, Shyong K Lam, Joseph A Konstan, and John Riedl. 2003. Interfaces for eliciting new user preferences in recommender systems. In UMAP. 178--187.

[26]

Jakob Nielsen. 1994. Usability engineering. Elsevier.

Digital Library

[27]

Jakob Nielsen. 1995. 10 usability heuristics for user interface design. Nielsen Norman Group, Vol. 1, 1 (1995).

[28]

Jakob Nielsen. 1999. User interface directions for the Web. Commun. ACM, Vol. 42, 1 (1999), 65--72.

Digital Library

[29]

Donald A Norman. 1988. The psychology of everyday things. Basic books.

[30]

John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. Peer Chooser: visual interactive recommendation. In CHI. 1085--1088.

[31]

Denis Parra and Peter Brusilovsky. 2015. User-controllable personalization: A case study with SetFusion. International Journal of Human-Computer Studies, Vol. 78 (2015), 43--67.

Digital Library

[32]

Pearl Pu and Li Chen. 2006. Trust building with explanation interfaces. In IUI. 93--100.

[33]

Pearl Pu, Li Chen, and Rong Hu. 2011. A user-centric evaluation framework for recommender systems. In RecSys. 157--164.

[34]

Pernilla Qvarfordt, Gene Golovchinsky, Tony Dunnigan, and Elena Agapie. 2013. Looking ahead: query preview in exploratory search. In SIGIR. 243--252.

[35]

Joseph John Rocchio. 1971. Relevance feedback in information retrieval. The SMART retrieval system: experiments in automatic document processing (1971), 313--323.

[36]

J Ben Schafer, Joseph A Konstan, and John Riedl. 2002. Meta-recommendation systems: user-controlled integration of diverse recommendations. In CIKM. 43--51.

[37]

James Schaffer, Tobias Hollerer, and John O'Donovan. 2015. Hypothetical recommendation: A study of interactive profile manipulation behavior for recommender systems. In FLAIRS.

[38]

Donald A Schön. 2017. The reflective practitioner: How professionals think in action. Routledge.

[39]

Ben Shneiderman. 1984. Response time and display rate in human performance with computers. Comput. Surveys, Vol. 16, 3 (1984), 265--285.

Digital Library

[40]

Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. Interactions, Vol. 4, 6 (1997), 42--61.

Digital Library

[41]

Jacob Solomon. 2014. Customization bias in decision support systems. In CHI. 3065--3074.

[42]

E Isaac Sparling and Shilad Sen. 2011. Rating: how difficult is it?. In RecSys. 149--156.

[43]

Michael Terry and Elizabeth D Mynatt. 2002. Side views: persistent, on-demand previews for open-ended tasks. In UIST. 71--80.

[44]

Nava Tintarev and Judith Masthoff. 2007. A survey of explanations in recommender systems. In International Conference on Data Engineering. 801--810.

Digital Library

[45]

Kristen Vaccaro, Dylan Huang, Motahhare Eslami, Christian Sandvig, Kevin Hamilton, and Karrie Karahalios. 2018. The Illusion of Control: Placebo Effects of Control Settings. In CHI. 16.

[46]

Flavian Vasile, Elena Smirnova, and Alexis Conneau. 2016. Meta-Prod2Vec: Product embeddings using side-information for recommendation. In RecSys. 225--232.

[47]

Katrien Verbert, Denis Parra, Peter Brusilovsky, and Erik Duval. 2013. Visualizing recommendations to support exploration, transparency and controllability. In IUI. 351--362.

[48]

Wesley Waldner and Julita Vassileva. 2014. Emphasize, don't filter!: displaying recommendations in Twitter timelines. In RecSys. 313--316.

[49]

Feng Wang and Xiangshi Ren. 2009. Empirical evaluation for finger input properties in multi-touch interaction. In CHI. 1063--1072.

[50]

Chao-Yuan Wu, Christopher V Alvino, Alexander J Smola, and Justin Basilico. 2016. Using navigation to improve recommendations in real-time. In RecSys. 341--348.

Cited By

Chen CEickhoff CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Evaluating Search System Explainability with Psychometrics and CrowdsourcingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657796(1051-1061)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657796
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Conclusions and Open ChallengesTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_6(143-146)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_6
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Privacy and SecurityTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_5(103-141)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_5
Show More Cited By

Index Terms

The Impact of More Transparent Interfaces on Behavior in Personalized Recommendation
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems
    2. Users and interactive retrieval
      1. Personalization

Recommendations

Personalized news recommendation based on click behavior
IUI '10: Proceedings of the 15th international conference on Intelligent user interfaces

Online news reading has become very popular as the web provides access to news articles from millions of sources around the world. A key challenge of news websites is to help users find the articles that are interesting to read. In this paper, we ...
Effects of Personalized and Aggregate Top-N Recommendation Lists on User Preference Ratings

Prior research has shown a robust effect of personalized product recommendations on user preference judgments for items. Specifically, the display of system-predicted preference ratings as item recommendations has been shown in multiple studies to bias ...
Generalized Collaborative Personalized Ranking for Recommendation
Web and Big Data
Abstract
Data sparsity is a common problem in collaborative ranking for personalized recommendation with implicit feedback. Several previous work tried to ‘borrow’ feedback information from users’ neighborhood as their prior preferences to alleviate this ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
892
Total Downloads

Downloads (Last 12 months)333
Downloads (Last 6 weeks)37

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen CEickhoff CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Evaluating Search System Explainability with Psychometrics and CrowdsourcingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657796(1051-1061)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657796
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Conclusions and Open ChallengesTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_6(143-146)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_6
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Privacy and SecurityTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_5(103-141)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_5
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)TransparencyTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_4(69-102)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_4
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Biases, Fairness, and Non-discriminationTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_3(29-67)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_3
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Regulatory InitiativesTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_2(11-27)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_2
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)IntroductionTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_1(1-10)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_1
Stray JHalevy AAssar PHadfield-Menell DBoutilier CAshar ABakalar CBeattie LEkstrand MLeibowicz CMoon Sehat CJohansen SKerlin LVickrey DSingh SVrijenhoek SZhang AAndrus MHelberger NProutskova PMitra TVasan N(2023)Building Human Values into Recommender Systems: An Interdisciplinary SynthesisACM Transactions on Recommender Systems10.1145/36322972:3(1-57)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3632297
Barker MKallina EAshok DCollins KCasovan AWeller ATalwalkar AChen VBhatt U(2023)FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning PipelinesProceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3617694.3623239(1-15)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3617694.3623239
Luria M(2023)Co-Design Perspectives on Algorithm Transparency Reporting: Guidelines and PrototypesProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594064(1076-1087)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594064
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents