CARE: An Infrastructure for Evaluation of Carousel-Based Recommender Interfaces
Pages 41 - 44
Abstract
Carousel-based interfaces are integral in enhancing user experience in online recommender systems like streaming services or e-commerce platforms, yet their usability evaluation often lacks standardization. Existing work on evaluating recommender systems, from toolkits to infrastructure, mainly assesses recommendation algorithms rather than user experience. This focus leads to a limited understanding of recommender systems’ effectiveness, as it overlooks the role of user interface design, especially carousel-based interfaces, in user experience. In response, this paper introduces a web-based infrastructure for the usability assessment of carousel-based interfaces. Our infrastructure is adaptable for various domains and setups, and its modular design allows for potential expansion.
Supplemental Material
ZIP File
Demonstration video
- Download
- 8.21 MB
References
[1]
Vito Walter Anelli, Alejandro Bellogin, Antonio Ferrara, Daniele Malitesta, Felice Antonio Merra, Claudio Pomo, Francesco Maria Donini, and Tommaso Di Noia. 2021. Elliot: A Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. Association for Computing Machinery, New York, NY, USA, 2405–2414. https://doi.org/10.1145/3404835.3463245
[2]
Vito Walter Anelli, Alejandro Bellogín, Antonio Ferrara, Daniele Malitesta, Felice Antonio Merra, Claudio Pomo, Francesco Maria Donini, and Tommaso Di Noia. 2021. V-Elliot: Design, Evaluate and Tune Visual Recommender Systems. In Proceedings of the 15th ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 768–771. https://doi.org/10.1145/3460231.3478881
[3]
Federico Bianchi, Patrick John Chia, Jacopo Tagliabue, Ciro Greco, Gabriel S P Moreira, Davide Eynard, Fahd Husain, and Claudio Pomo. 2023. EvalRS 2023: Well-Rounded Recommender Systems for Real-World Deployments. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Long Beach, CA) (KDD ’23). Association for Computing Machinery, New York, NY, USA, 5851–5852. https://doi.org/10.1145/3580305.3599222
[4]
Robin Burke, Michael Ekstrand, Daniel Kluver, Bart Knijnenburg, Joseph Konstan, and Edward Malthouse. 2024. POPROX: Platform for OPen Recommendation and Online eXperimentation. https://poprox.ai/
[5]
Nicolas Burny. 2020. Towards Supporting Reproducibility of Experimental Studies in GUI Visual Design. In Companion Proceedings of the 12th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Sophia Antipolis, France) (EICS ’20 Companion). Association for Computing Machinery, New York, NY, USA, Article 13, 4 pages. https://doi.org/10.1145/3393672.3398644
[6]
Patrik Dokoupil and Ladislav Peska. 2023. EasyStudy: Framework for Easy Deployment of User Studies on Recommender Systems. In Proceedings of the 17th ACM Conference on Recommender Systems (Singapore, Singapore) (RecSys ’23). Association for Computing Machinery, New York, NY, USA, 1196–1199. https://doi.org/10.1145/3604915.3610640
[7]
Nicolò Felicioni, Maurizio Ferrari Dacrema, and Paolo Cremonesi. 2021. Measuring the User Satisfaction in a Recommendation Interface with Multiple Carousels. In Proceedings of the 2021 ACM International Conference on Interactive Media Experiences (Virtual Event, USA) (IMX ’21). Association for Computing Machinery, New York, NY, USA, 212–217. https://doi.org/10.1145/3452918.3465493
[8]
Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proceedings of the Fourth ACM Conference on Recommender Systems (Barcelona, Spain) (RecSys ’10). Association for Computing Machinery, New York, NY, USA, 257–260. https://doi.org/10.1145/1864708.1864761
[9]
Dietmar Jannach, Michael Jugovac, and Lucas Lerche. 2012. Recommender systems beyond accuracy: A multi-criteria evaluation approach. In Recommender Systems Handbook. Springer Berlin Heidelberg, Berlin, Germany, 297–331.
[10]
Joseph A Konstan and John Riedl. 2012. Recommender systems: from algorithms to user experience. User modeling and user-adapted interaction 22 (2012), 101–123. https://doi.org/10.1007/s11257-011-9112-x
[11]
Benedikt Loepp and Jürgen Ziegler. 2023. How Users Ride the Carousel: Exploring the Design of Multi-List Recommender Interfaces From a User Perspective. In Proceedings of the 17th ACM Conference on Recommender Systems (Singapore, Singapore) (RecSys ’23). Association for Computing Machinery, New York, NY, USA, 1090–1095. https://doi.org/10.1145/3604915.3610638
[12]
Alexandra Papoutsaki, Patsorn Sangkloy, James Laskey, Nediyana Daskalova, Jeff Huang, and James Hays. 2016. WebGazer: Scalable Webcam Eye Tracking Using User Interactions. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI). AAAI, ACM, New York, NY, 3839–3845. https://doi.org/10.5555/3061053.3061156
[13]
Behnam Rahdari, Peter Brusilovsky, Daqing He, Khushboo Thaker, Zhimeng Luo, and Young Ji Lee. 2022. Helper: an interactive recommender system for ovarian cancer patients and caregivers. In 16th ACM Conference on Recommender Systems. ACM, New York, NY, 644–647. https://doi.org/10.1145/3523227.3551471
[14]
Behnam Rahdari, Peter Brusilovsky, and Branislav Kveton. 2024. Towards Simulation-Based Evaluation of Recommender Systems with Carousel Interfaces. ACM Trans. Recomm. Syst. 1, 1 (jan 2024), 1. https://doi.org/10.1145/3643709
[15]
Behnam Rahdari, Peter Brusilovsky, and Alireza Javadian Sabet. 2021. Controlling Personalized Recommendations in Two Dimensions with a Carousel-Based Interface. In Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS’21) at 2021 ACM Conference on Recommender Systems (RecSys’21)(CEUR Workshop Proceeding, Vol. 2948). CEUR, Online, 112–122.
[16]
Behnam Rahdari, Branislav Kveton, and Peter Brusilovsky. 2022. From Ranked Lists to Carousels: A Carousel Click Model. arXiv preprint arXiv:2209.13426 1 (2022), 1.
[17]
Behnam Rahdari, Branislav Kveton, and Peter Brusilovsky. 2022. The Magic of Carousels: Single vs. Multi-List Recommender Systems. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media. ACM, New York, YN, 166–174. https://doi.org/10.1145/3511095.3531278
[18]
Aghiles Salah, Quoc-Tuan Truong, and Hady W. Lauw. 2020. Cornac: A Comparative Framework for Multimodal Recommender Systems. J. Mach. Learn. Res. 21, 1, Article 95 (jan 2020), 5 pages.
[19]
Tobias Schnabel, Paul N. Bennett, Susan T. Dumais, and Thorsten Joachims. 2018. Short-Term Satisfaction and Long-Term Coverage: Understanding How Users Tolerate Algorithmic Exploration. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey, CA, USA) (WSDM ’18). Association for Computing Machinery, New York, NY, USA, 513–521. https://doi.org/10.1145/3159652.3159700
Recommendations
How Users Ride the Carousel: Exploring the Design of Multi-List Recommender Interfaces From a User Perspective
RecSys '23: Proceedings of the 17th ACM Conference on Recommender SystemsMulti-list interfaces are widely used in recommender systems, especially in industry, showing collections of recommendations, one below the other, with items that have certain commonalities. The composition and order of these “carousels” are usually ...
Towards Simulation-Based Evaluation of Recommender Systems with Carousel Interfaces
Offline data-driven evaluation is considered a low-cost and more accessible alternative to the online empirical method of assessing the quality of recommender systems. Despite their popularity and effectiveness, most data-driven approaches are unsuitable ...
Trust-inspiring explanation interfaces for recommender systems
A recommender system's ability to establish trust with users and convince them of its recommendations, such as which camera or PC to purchase, is a crucial design factor especially for e-commerce environments. This observation led us to build a trust ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2024
182 pages
ISBN:9798400705090
DOI:10.1145/3640544
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 April 2024
Check for updates
Qualifiers
- Demonstration
- Research
- Refereed limited
Data Availability
Demonstration video https://dl.acm.org/doi/10.1145/3640544.3645223#iuicompanion24-9_video.zip
Conference
IUI '24: 29th International Conference on Intelligent User Interfaces
March 18 - 21, 2024
SC, Greenville, USA
Acceptance Rates
Overall Acceptance Rate 746 of 2,811 submissions, 27%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 76Total Downloads
- Downloads (Last 12 months)76
- Downloads (Last 6 weeks)12
Reflects downloads up to 01 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format