Abstract
The past decade has seen a fast rise in popularity of recommendation systems provided by many entertainment and social media services. However, despite the recognised advances in different recommendation approaches and technologies, there remain many challenges, particularly in TV content recommendation systems. More precisely, machine learning based TV content recommendation systems suffer from a class imbalance problem; hence, it is difficult to evaluate the system using traditional metrics. Moreover, specific challenges arise during the development phase, when the system operates in ‘offline’ mode. This means the recommendations are not actually presented to users - making it even more difficult to measure the quality of those recommendations. This paper presents a proof-of-concept demonstrator of a television recommendation system, based on Content-based Filtering, as a contribution towards building a full-scale intelligent recommendation system. New evaluation metrics are proposed for ‘offline’ testing mode, while also tackling the class imbalance problem. The experimental results, based on real usage data, are promising and help in defining the future path as presented along with the conclusion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abel, F., Deldjoo, Y., Elahi, M., Kohlsdorf, D.: RecSys challenge 2017: offline and online evaluation. In: RecSys 2017 - Proceedings of the 11th ACM Conference on Recommender Systems (2017). https://doi.org/10.1145/3109859.3109954
Beel, J., Langer, S.: A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In: Kapidakis, S., Mazurek, C., Werla, M. (eds.) TPDL 2015. LNCS, vol. 9316, pp. 153–168. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24592-8_12
Cañamares, R., Castells, P., Moffat, A.: Offline evaluation options for recommender systems. Inf. Retriev. J. 23(4), 387–410 (2020). https://doi.org/10.1007/s10791-020-09371-3
Çano, E., Morisio, M.: Hybrid recommender systems: a systematic literature review. Intell. Data Anal. 21(6), 1487–1524 (2017)
Ferraro, A., Bogdanov, D., Choi, K., Serra, X.: Using offline metrics and user behavior analysis to combine multiple systems for music recommendation (2019)
Ferro, N., et al.: From evaluating to forecasting performance: how to turn information retrieval, natural language processing and recommender systems into predictive sciences: manifesto from dagstuhl perspectives workshop 17442, Dagstuhl Manifestos, vol. 7, no. 1, pp. 96–139 (2018)
Gama, J., Sebastião, R., Rodrigues, P.P.: On evaluating stream learning algorithms. Mach. Learn. 90(3), 317–346 (2012). https://doi.org/10.1007/s10994-012-5320-9
Ge, M., Delgado-Battenfeld, C., Jannach, D.: Beyond accuracy: evaluating recommender systems by coverage and serendipity. In: 4th ACM Conference on Recommender Systems, pp. 257–260 (2010)
Hu, Y., Volinsky, C., Koren, Y.: Collaborative filtering for implicit feedback datasets. In: IEEE International Conference on Data Mining, ICDM (2008)
Jeunen, O.: Revisiting offline evaluation for implicit-feedback recommender systems. In: Proceedings of the 13th ACM Conference on Recommender Systems, pp. 596–600. ACM, NY, USA (2019)
Krauth, K., et al.: Do offline metrics predict online performance in recommender systems? (2020)
Li, S., Abbasi-Yadkori, Y., Kveton, B., Muthukrishnan, S., Vinay, V., Wen, Z.: Offline evaluation of ranking policies with click models. In: 24th International Conference on Knowledge Discovery & Data Mining, NY, USA, pp. 1685–1694 (2018)
Myttenaere, A.D., Grand, B.L., Golden, B., Rossi, F.: Reducing offline evaluation bias in recommendation systems (2014)
Peska, L., Vojtas, P.: Off-line vs. on-line evaluation of recommender systems in small e-commerce. In: Proceedings of the 31st ACM Conference on Hypertext and Social Media, pp. 291–300. ACM, NY, USA (2020)
Rao, S., et al.: Learning to be Relevant. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2625–2633. ACM, New York (2019)
Shani, G., Gunawardana, A.: Evaluating recommendation systems. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 257–297. Springer, Boston, MA (2011). https://doi.org/10.1007/978-0-387-85820-3_8
Silveira, T., Zhang, M., Lin, X., Liu, Y., Ma, S.: How good your recommender system is? A survey on evaluations in recommendation. Int. J. Mach. Learn. Cybern. 10(5), 813–831 (2017). https://doi.org/10.1007/s13042-017-0762-9
Acknowledgment
This work has been supported by NORTE-06-3559-FSE-000018, integrated in the invitation NORTE-59-2018-41, aimed at Hiring of Highly Qualified Human Resources, co-financed by the Regional Operational Programme of the North 2020, thematic area of Competitiveness and Employment, through the European Social Fund (ESF).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Simões, L., Shah, V., Silva, J., Rodrigues, N., Leite, N., Lopes, N. (2021). New Performance Metrics for Offline Content-Based TV Recommender System. In: Boratto, L., Faralli, S., Marras, M., Stilo, G. (eds) Advances in Bias and Fairness in Information Retrieval. BIAS 2021. Communications in Computer and Information Science, vol 1418. Springer, Cham. https://doi.org/10.1007/978-3-030-78818-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-78818-6_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78817-9
Online ISBN: 978-3-030-78818-6
eBook Packages: Computer ScienceComputer Science (R0)