Abstract
Cross-validation is commonly used to select the recommendation algorithms that will generalize best on yet unknown data. Yet, in many situations the available dataset used for cross-validation is scarce and the selected algorithm might not be the best suited for the unknown data. In contrast, established companies have a large amount of data available to select and tune their recommender algorithms, which therefore should generalize better. These companies often make their recommender systems available as black-boxes, i.e., users query the recommender through an API or a browser. This paper proposes RecRank, a technique that exploits a black-box recommender system, in addition to classic cross-validation. RecRank employs graph similarity measures to compute a distance between the output recommendations of the black-box and of the considered algorithms. We empirically show that RecRank provides a substantial improvement (33%) for the selection of algorithms for the MovieLens dataset, in comparison with standalone cross-validation.
G. Damaskinos—Work done during an internship at Technicolor - Rennes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
If the recommender only outputs a top-N list, the output for each item is the rank (e.g., value \(\in [1,5]\) for top-5 outputs).
- 2.
- 3.
- 4.
References
Agarwal, A., Wang, X., Li, C., Bendersky, M., Najork, M.: Offline comparison of ranking functions using randomized data. In: REVEAL (2018)
Beel, J., Langer, S.: A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In: Kapidakis, S., Mazurek, C., Werla, M. (eds.) TPDL 2015. LNCS, vol. 9316, pp. 153–168. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24592-8_12
Damaskinos, G.: RecRank source-code. https://github.com/gdamaskinos/RecRank
Damaskinos, G., Guerraoui, R., Patra, R.: Capturing the moment: lightweight similarity computations. In: ICDE, pp. 747–758. IEEE (2017). https://doi.org/10.1109/ICDE.2017.126
Dodge, Y.: The Concise Encyclopedia of Statistics. Springer Science & Business Media, New York (2008). https://doi.org/10.1007/978-0-387-32833-1
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Hou, L., Liu, K., Liu, J.: Navigated random walks on Amazon book recommendation network. In: Cherifi, C., Cherifi, H., Karsai, M., Musolesi, M. (eds.) COMPLEX NETWORKS 2017 2017. SCI, vol. 689, pp. 935–945. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-72150-7_75
Hussein, A., Gaber, M.M., Elyan, E., Jayne, C.: Imitation learning: a survey of learning methods. CSUR 50(2), 21 (2017)
Jenson, G.: Recommenders list (2019). https://github.com/grahamjenson/list_of_recommender_systems
Kille, B., et al.: Overview of CLEF newsreel 2015: news recommendation evaluation lab. In: International Conference of the CLEF Initiative (2015)
Košir, A., Odić, A., Tkalčič, M.: How to improve the statistical power of the 10-fold cross validation scheme in recommender systems. In: RepSys, pp. 3–6. ACM (2013)
Kowald, D., Kopeinik, S., Lex, E.: The TagRec framework as a toolkit for the development of tag-based recommender systems. In: UMAP, pp. 23–28. ACM, New York (2017). https://doi.org/10.1145/3099023.3099069
Le Merrer, E., Trédan, G.: The topological face of recommendation. In: Cherifi, C., Cherifi, H., Karsai, M., Musolesi, M. (eds.) COMPLEX NETWORKS 2017 2017. SCI, vol. 689, pp. 897–908. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-72150-7_72
Lécuyer, M., et al.: XRay: enhancing the web’s transparency with differential correlation. In: USENIX Security Symposium, pp. 49–64 (2014)
Pan, W., Xiang, E.W., Liu, N.N., Yang, Q.: Transfer learning in collaborative filtering for sparsity reduction. In: AAAI (2010)
Polino, A., Pascanu, R., Alistarh, D.: Model compression via distillation and quantization. In: ICLR (2018)
IMDB via RapidAPI query limit. https://rapidapi.com/blog/how-to-use-imdb-api/
Said, A., Bellogín, A.: Rival: a toolkit to foster reproducibility in recommender system evaluation. In: RecSys, pp. 371–372. ACM (2014)
Vartak, M., Thiagarajan, A., Miranda, C., Bratman, J., Larochelle, H.: A meta-learning perspective on cold-start recommendations for items. In: NIPS, pp. 6904–6914 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Damaskinos, G., Guerraoui, R., Le Merrer, E., Neumann, C. (2021). The Imitation Game: Algorithm Selection by Exploiting Black-Box Recommenders. In: Georgiou, C., Majumdar, R. (eds) Networked Systems. NETYS 2020. Lecture Notes in Computer Science(), vol 12129. Springer, Cham. https://doi.org/10.1007/978-3-030-67087-0_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-67087-0_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67086-3
Online ISBN: 978-3-030-67087-0
eBook Packages: Computer ScienceComputer Science (R0)