Abstract
Recommender systems (RSs) have become essential tools in e-commerce applications, helping users in the decision-making process. Evaluation on these tools is, however, a major divergence point nowadays, since there is no consensus regarding which metrics are necessary to consolidate new RSs. For this reason, distinct frameworks have been developed to ease the deployment of RSs in research and/or production environments. In the present work, we perform an extensive study of the most popular evaluation metrics, organizing them into three groups: Effectiveness-based, Complementary Dimensions of Quality and Domain Profiling. Further, we consolidate a framework named FAiR to help researchers in evaluating their RSs using these metrics, besides identifying the characteristics of data collections that may intrinsically affect RSs performance. FAiR is compatible with the output format of the main existing RSs libraries (i.e., MyMediaLite and LensKit).
This work was partially supported by CNPq, CAPES, Fapemig, INWEB, MAsWeb.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Available: https://github.com/recommenders/rival.
- 2.
Available at: https://www.librec.net/ .
- 3.
Available at: https://github.com/RankSys.
- 4.
LensKit follows all the formats used by FAiR. For MyMediaLite, there is an option to change the format of the recommendation lists to the output format of that library.
References
Anderson, C.: Long tail, the, revised and updated edition: why the future of business is selling less of more (2008)
Barjasteh, I., Forsati, R., Masrour, F., Esfahanian, A.H., Radha, H.: Cold-start item and user recommendation with decoupled completion and transduction. In: Proceedings of the 9th ACM RecSys, pp. 91–98. ACM (2015)
Beel, J., Gipp, B., Langer, S., Breitinger, C.: Research-paper recommender systems: a literature survey. Int. J. Digit. Libr. 17(4), 305–338 (2016)
Bobadilla, J., Ortega, F., Hernando, A., Gutiérrez, A.: Recommender systems survey. Knowl.-Based Syst. 46, 109–132 (2013)
Chiancone, A., Franzoni, V., Li, Y., Markov, K., Milani, A.: Leveraging zero tail in neighbourhood for link prediction. In: 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), vol. 3, pp. 135–139. IEEE (2015)
Chiancone, A., Franzoni, V., Niyogi, R., Milani, A.: Improving link ranking quality by quasi-common neighbourhood. In: 2015 15th International Conference on Computational Science and Its Applications (ICCSA), pp. 21–26. IEEE (2015)
Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the Fourth ACM Conference on Recommender Systems, pp. 39–46. ACM (2010)
Ekstrand, M.D., Ludwig, M., Kolb, J., Riedl, J.T.: Lenskit: a modular recommender framework. In: Proceedings of the Fifth ACM Conference on Recommender Systems, pp. 349–350. ACM (2011)
Gantner, Z., Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: MyMediaLite: a free recommender system library. In: Proceedings of 5th ACM RecSys (2011)
Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. (TOIS) 22(1), 5–53 (2004)
Jannach, D., Zanker, M., Felfernig, A., Friedrich, G.: Recommender Systems: An Introduction. Cambridge University Press, Cambridge (2010)
Kawai, M., Nogami, S.: A hybrid recommender system of collaborative and content based filtering. Int. Inf. Inst. (Tokyo) Inf. 19(6B), 2177 (2016)
Konstan, J.A., Riedl, J.: Research resources for recommender systems. In: CHI 1999 Workshop Interacting with Recommender Systems (1999)
Kouki, P., Fakhraei, S., Foulds, J., Eirinaki, M., Getoor, L.: Hyper: a flexible and extensible probabilistic framework for hybrid recommender systems. In: Proceedings of the 9th ACM RecSys, pp. 99–106 (2015)
Lu, J., Wu, D., Mao, M., Wang, W., Zhang, G.: Recommender system application developments: a survey. Decis. Support Syst. 74, 12–32 (2015)
Martin, R., Walid, M., Robert, W., Thomas, Z.: Recommendation Systems in Software Engineering. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-45135-5
McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI 2006 Extended Abstracts on Human Factors in Computing Systems, pp. 1097–1101. ACM (2006)
Modarresi, K.: Recommendation system based on complete personalization. In: ICCS 2016, California, USA, pp. 2190–2204 (2016)
Ozok, A.A., Fan, Q., Norcio, A.F.: Design guidelines for effective recommender system interfaces based on a usability criteria conceptual model: results from a college student population. Behav. Inf. Technol. 29(1), 57–83 (2010)
Puthiya Parambath, S.A., Usunier, N., Grandvalet, Y.: A coverage-based approach to recommendation diversity on similarity graph. In: Proceedings of the 10th ACM RecSys, pp. 15–22 (2016)
Schwartz, B.: The Paradox of Choice: Why More Is Less, Revised edn. HarperCollins, New York City (2009)
Son, L.H.: Dealing with the new user cold-start problem in recommender systems: a comparative review. Inf. Syst. 58, 87–104 (2016)
Vargas, S., Castells, P.: Rank and relevance in novelty and diversity metrics for recommender systems. In: Proceedings of the Fifth ACM RecSys, pp. 109–116. ACM (2011)
Zhang, Y.C., Séaghdha, D.Ó., Quercia, D., Jambor, T.: Auralist: introducing serendipity into music recommendation. In: Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, pp. 13–22. ACM (2012)
Acknowledgments
This work was partially supported by CNPq, CAPES, Fapesp, and Finep.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Carvalho, D. et al. (2018). FAiR: A Framework for Analyses and Evaluations on Recommender Systems. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2018. ICCSA 2018. Lecture Notes in Computer Science(), vol 10962. Springer, Cham. https://doi.org/10.1007/978-3-319-95168-3_26
Download citation
DOI: https://doi.org/10.1007/978-3-319-95168-3_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-95167-6
Online ISBN: 978-3-319-95168-3
eBook Packages: Computer ScienceComputer Science (R0)