short-paper

Should we embed? A study on the online performance of utilizing embeddings for real-time job recommendations

Authors:

Markus Reiter-Haas,

Tomislav Duricic,

Valentin Slawicek,

Elisabeth LexAuthors Info & Claims

RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems

Pages 496 - 500

https://doi.org/10.1145/3298689.3346989

Published: 10 September 2019 Publication History

Abstract

In this work, we present the findings of an online study, where we explore the impact of utilizing embeddings to recommend job postings under real-time constraints. On the Austrian job platform Studo Jobs, we evaluate two popular recommendation scenarios: (i) providing similar jobs and, (ii) personalizing the job postings that are shown on the homepage. Our results show that for recommending similar jobs, we achieve the best online performance in terms of Click-Through Rate when we employ embeddings based on the most recent interaction. To personalize the job postings shown on a user's homepage, however, combining embeddings based on the frequency and recency with which a user interacts with job postings results in the best online performance.

References

[1]

F. Abel, A. Benczúr, D. Kohlsdorf, M. Larson, and R. Pálovics. Recsys challenge 2016: Job recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, pages 425--426. ACM, 2016.

Digital Library

[2]

F. Abel, Y. Deldjoo, M. Elahi, and D. Kohlsdorf. Recsys challenge 2017: Offline and online evaluation. In Proceedings of the Eleventh ACM Conference on Recommender Systems, pages 372--373. ACM, 2017.

Digital Library

[3]

S. T. Al-Otaibi and M. Ykhlef. A survey of job recommender systems. International Journal of Physical Sciences, 7(29):5127--5142, 2012.

[4]

J. R. Anderson, D. Bothell, M. D. Byrne, S. Douglass, C. Lebiere, and Y. Qin. An integrated theory of the mind. Psychological review, 111(4):1036, 2004.

[5]

O. Barkan and N. Koenigstein. Item2vec: neural item embedding for collaborative filtering. In 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1--6. IEEE, 2016.

[6]

J. Beel, M. Genzmehr, S. Langer, A. Nürnberger, and B. Gipp. A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. In Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation, pages 7--14. ACM, 2013.

Digital Library

[7]

P. G. Campos, F. Díez, and I. Cantador. Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Modeling and User-Adapted Interaction, 24(1-2):67--119, 2014.

Digital Library

[8]

B. Chandramouli, J. J. Levandoski, A. Eldawy, and M. F. Mokbel. Streamrec: a real-time recommender system. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, pages 1243--1246. ACM, 2011.

Digital Library

[9]

C. Eksombatchai, P. Jindal, J. Z. Liu, Y. Liu, R. Sharma, C. Sugnet, M. Ulrich, and J. Leskovec. Pixie: A system for recommending 3+ billion items to 200+ million users in real-time. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 1775--1784. International World Wide Web Conferences Steering Committee, 2018.

Digital Library

[10]

T. George and S. Merugu. A scalable collaborative filtering framework based on co-clustering. In Fifth IEEE International Conference on Data Mining (ICDM'05), pages 4--pp. IEEE, 2005.

Digital Library

[11]

Y. Huang, B. Cui, W. Zhang, J. Jiang, and Y. Xu. Tencentrec: Real-time stream recommendation in practice. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 227--238. ACM, 2015.

Digital Library

[12]

K. Kenthapadi, B. Le, and G. Venkataraman. Personalized job recommendation system at linkedin: Practical challenges and lessons learned. In Proceedings of the Eleventh ACM Conference on Recommender Systems, pages 346--347. ACM, 2017.

Digital Library

[13]

D. Kowald, S. C. Pujari, and E. Lex. Temporal effects on hashtag reuse in twitter: A cognitive-inspired hashtag recommendation approach. In Proceedings of the 26th International Conference on World Wide Web, pages 346--347, 2017.

Digital Library

[14]

E. Lacic, D. Kowald, and E. Lex. Neighborhood troubles: On the value of user pre-filtering to speed up and enhance recommendations. In Proceedings of the International Workshop on Entity Retrieval (EYRE'2018) co-located with CIKM'18, 2018.

[15]

E. Lacic, D. Kowald, D. Parra, M. Kahr, and C. Trattner. Towards a scalable social recommender engine for online marketplaces: The case of apache solr. In Proceedings of the 23rd International Conference on World Wide Web, pages 817--822. ACM, 2014.

Digital Library

[16]

E. Lacic, D. Kowald, M. Reiter-Haas, V. Slawicek, and E. Lex. Beyond accuracy optimization: On the value of item embeddings for student job recommendations. In Proceedings of the Workshop on Multi-dimensional Information Fusion for User Modeling and Personalization (IFUP'2018) co-located with WSDM'18, 2018.

[17]

Q. Le and T. Mikolov. Distributed representations of sentences and documents. In International conference on machine learning (ICML'14), pages 1188--1196, 2014.

Digital Library

[18]

G. Linden, B. Smith, and J. York. Amazon. com recommendations: Item-to-item collaborative filtering. IEEE Internet computing, (1):76--80, 2003.

Digital Library

[19]

T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111--3119, 2013.

Digital Library

[20]

C. Musto, G. Semeraro, M. de Gemmis, and P. Lops. Learning word embeddings from wikipedia for content-based recommender systems. In European Conference on Information Retrieval, pages 729--734. Springer, 2016.

[21]

M. J. Pazzani and D. Billsus. Content-based recommendation systems. In The adaptive web, pages 325--341. Springer, 2007.

[22]

V.-T. Phi, L. Chen, and Y. Hirate. Distributed representation based recommender systems in e-commerce. In DEIM Forum, 2016.

[23]

C. Qin, H. Zhu, T. Xu, C. Zhu, L. Jiang, E. Chen, and H. Xiong. Enhancing person-job fit for talent recruitment: An ability-aware neural network approach. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 25--34. ACM, 2018.

Digital Library

[24]

R. Rafter, K. Bradley, and B. Smyth. Personalised retrieval for online recruitment services. In The BCS/IRSG 22nd Annual Colloquium on Information Retrieval (IRSG 2000), Cambridge, UK, 5--7 April, 2000, 2000.

[25]

A. Said, J. Lin, A. Bellogín, and A. de Vries. A month in the life of a production news recommender system. In Proceedings of the 2013 workshop on Living labs for information retrieval evaluation, pages 7--10. ACM, 2013.

Digital Library

[26]

J. Yuan, W. Shalaby, M. Korayem, D. Lin, K. AlJadda, and J. Luo. Solving cold-start problem in large-scale recommendation engines: A deep learning approach. In 2016 IEEE International Conference on Big Data (Big Data), pages 1901--1910. IEEE, 2016.

[27]

C. Zhang and X. Cheng. An ensemble method for job recommender systems. In Proceedings of the Recommender Systems Challenge, RecSys Challenge '16, pages 2:1--2:4. ACM, 2016.

Digital Library

[28]

C. Zhu, H. Zhu, H. Xiong, C. Ma, F. Xie, P. Ding, and P. Li. Person-job fit: Adapting the right talent for the right job with joint representation learning. ACM Transactions on Management Information Systems (TMIS), 9(3):12, 2018.

Digital Library

Cited By

Mashayekhi YLi NKang BLijffijt JDe Bie T(2024)A Challenge-based Survey of E-recruitment Recommendation SystemsACM Computing Surveys10.1145/365994256:10(1-33)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3659942
Tran VSalha-Galvan GSguerra BHennequin R(2024)Transformers Meet ACT-R: Repeat-Aware and Sequential Listening Session RecommendationProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688139(486-496)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688139
Bogers TGraus DKaya MJohnson CDecorte JDe Bie T(2024)Fourth Workshop on Recommender Systems for Human Resources (RecSys in HR 2024)Proceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3687109(1222-1226)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3687109
Show More Cited By

Index Terms

Should we embed? A study on the online performance of utilizing embeddings for real-time job recommendations
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

Content-Based approaches for Cold-Start Job Recommendations
RecSys Challenge '17: Proceedings of the Recommender Systems Challenge 2017

This paper provides an overview of the approach we adopted as team Lunatic Goats for the ACM RecSys Challenge 2017 [7]. The competition, organized by XING.com, focuses on a cold start job recommendation scenario. The goal was to design and tune a ...
Regularizing Matrix Factorization with User and Item Embeddings for Recommendation
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

Following recent successes in exploiting both latent factor and word embedding models in recommendation, we propose a novel Regularized Multi-Embedding (RME) based recommendation model that simultaneously encapsulates the following ideas via ...
Practical Lessons for Job Recommendations in the Cold-Start Scenario
RecSys Challenge '17: Proceedings of the Recommender Systems Challenge 2017

The 2017 ACM RecSys Challenge focuses on the problem of job recommendations on XING in a cold-start scenario. In this paper we describe our solution as well as some practical lessons learned from the competition. We model this task as a binary ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems

September 2019

635 pages

ISBN:9781450362436

DOI:10.1145/3298689

General Chairs:
Toine Bogers
Aalborg University Copenhagen, Denmark
,
Alan Said
University of Gothenburg, Sweden
,
Program Chairs:
Peter Brusilovsky
University of Pittsburgh
,
Domonkos Tikk
Gravity R&D, Hungary

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

RecSys '19

RecSys '19: Thirteenth ACM Conference on Recommender Systems

September 16 - 20, 2019

Copenhagen, Denmark

Acceptance Rates

RecSys '19 Paper Acceptance Rate 36 of 189 submissions, 19%;

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
355
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)3

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mashayekhi YLi NKang BLijffijt JDe Bie T(2024)A Challenge-based Survey of E-recruitment Recommendation SystemsACM Computing Surveys10.1145/365994256:10(1-33)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3659942
Tran VSalha-Galvan GSguerra BHennequin R(2024)Transformers Meet ACT-R: Repeat-Aware and Sequential Listening Session RecommendationProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688139(486-496)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688139
Bogers TGraus DKaya MJohnson CDecorte JDe Bie T(2024)Fourth Workshop on Recommender Systems for Human Resources (RecSys in HR 2024)Proceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3687109(1222-1226)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3687109
Kowald DReiter-Haas MKopeinik SSchedl MLex E(2024)Transparent Music Preference Modeling and Recommendation with a Model of Human Memory TheoryA Human-Centered Perspective of Intelligent Personalized Environments and Systems10.1007/978-3-031-55109-3_4(113-136)Online publication date: 1-May-2024
https://doi.org/10.1007/978-3-031-55109-3_4
Kaya MBogers T(2023)An Exploration of Sentence-Pair Classification for Algorithmic RecruitingProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3610657(1175-1179)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3610657
Moscati MWallmann CReiter-Haas MKowald DLex ESchedl M(2023)Integrating the ACT-R Framework with Collaborative Filtering for Explainable Sequential Music RecommendationProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3608838(840-847)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3608838
Bogers TGraus DKaya MJohnson CDecorte J(2023)Third Workshop on Recommender Systems for Human Resources (RecSys in HR 2023)Proceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3608755(1244-1247)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3608755
Kwieciński RMelniczak GGórecki T(2023)Comparison of Real-Time and Batch Job RecommendationsIEEE Access10.1109/ACCESS.2023.324935611(20553-20559)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3249356
Graus DBogers TKaya MGutiérrez FVerbert K(2022)Report on the 1st workshop on recommender systems for human resources (RecSys in HR 2021) at RecSys 2021ACM SIGIR Forum10.1145/3527546.352756755:2(1-14)Online publication date: 17-Mar-2022
https://dl.acm.org/doi/10.1145/3527546.3527567
Bogers TGraus DKaya MGutiérrez FMesbah SJohnson C(2022)Second Workshop on Recommender Systems for Human Resources (RecSys in HR 2022)Proceedings of the 16th ACM Conference on Recommender Systems10.1145/3523227.3547414(671-674)Online publication date: 12-Sep-2022
https://dl.acm.org/doi/10.1145/3523227.3547414
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents